# Actual temporal hp_d versus the F1 paper bootstrap — 2026-10-03

> **2026-10-03 follow-up:** [HPD_SHADOW_CALIBRATION.md](HPD_SHADOW_CALIBRATION.md) now traces the defective column into uncalibrated virtual-row counts and verifies selected actual Hotpants regenerations. The full temporal/light-curve goal remains open.

**Conclusion:** the temporal investigation and the newer neighbour audit use
different hp_d generations. Their underlying RAW pixels are identical at the
common FFI 602, but their fixed-aperture measurements differ. Background
subtraction dominates several large differences. This comparison identifies
where to investigate the requested hp_d repair; it does not identify which
background is physically correct or supply a temporal correction. The newer
removal ledger cannot be treated as provenance for the historical template.

The full instrument-free/photon-limited goal remains unachieved. Current RAW
photometry is not replaced. All code stays local.

## Scope and input identity

Run: `/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/`.
Owned worktree: `/home/kshukawa/syndiff_pipeline/dev/.worktrees/temporal-raw-epsf`,
branch `codex/temporal-raw-epsf`, scripts under `temporal_raw_epsf/`.

Common exposure: `tess2020120182919-s0024-2-2`, frame index 602, the first
second-orbit exposure. Inputs and SHA256 hashes are recorded in
`results/hpd_generation_bridge/summary.json`.

| Input generation | Authoritative source |
|---|---|
| Historical hp_d cube | `temporal_epsf_20260929/extract_hpd.py` extracts production `data/s0024/c2/k2/diff_linear/hp_d/` |
| Historical template | Frozen lane `diff_config.yaml` points to `templates_linear/oversampling_1`; `linear_mode_assembly.json` records 18 linear groups |
| Newer paper hp_d | `paper_dataset_20261001/F1/bootstrap/diff/hp_d/`, oversampling-4 bootstrap |
| Newer removal ledger | `gaia_neighbour_joint_fit_20261002/ledger/F1/`, linked to recipe `e17a198a4942aa2d` |

The neighbouring remap directory also contains a field-mode manifest written
later than the historical linear template. Do not infer the template's
producing geometry simply from that directory name or newest manifest. The
actual template assembly record and lane input paths govern this comparison.

Both input RAW FITS hashes and pixel arrays match exactly. Historical RAW
and hp_d cube stamps match the directly read files exactly at all 431
auxiliary-source positions. The paper RAW/background/hp_c/hp_d identity closes
within 0.962 float32 epsilon-scaled pixel units, below the declared bound of
8. No generation is selected because it looks flatter.

The peer corrected C4 neighbour/ePSF training pilot is complete, but did not
establish magnitude-shape closure: see
`/astro/armin/koji/syndiff/dev_runs/neighbour_joint_training_20261002/README.md`.
That field is S23 C4K2, not this S24 C2K2 time series. Its initial flux-prior
runs were explicitly superseded. No peer files or jobs are altered here.

## Matched measurement and decomposition

At this single exposure,

$$D_j=R-B_j-H_j,\qquad
\Delta f=\sum_p w_p(D_{\rm paper,p}-D_{\rm legacy,p})
=\sum_p w_p[(B_{\rm legacy,p}-B_{\rm paper,p})+(H_{\rm legacy,p}-H_{\rm paper,p})].$$

Here $D$ is hp_d, $R$ is the unsubtracted calibrated RAW image, $B$ is the
saved background image, and $H$ is the convolved-template term including
its fitted background convention, all in electrons/second per pixel. The
generation index $j$ distinguishes legacy and paper. The pixel index $p$
runs over a fixed 15×15 stamp. The dimensionless $w$ are the existing r3
aperture/sky weights, identical for both generations; $\Delta f$ is the
aperture-rate change in electrons/second. Fractional changes divide by the
same measured RAW r3 rate. One ppt is a fractional change of 0.001.

The legacy hp_c was not persisted; its effective image is obtained from the
other three images by this identity. That is algebraic bookkeeping, not an
independent replay of its saved kernel or proof of template correctness.
The paper hp_c is independently saved. The component sums close to the
fixed-aperture difference within the declared numerical tolerance.

No flux, PSF, WCS, or mask is fitted in this diagnostic. It uses r3 for the
comparison, including science targets whose adopted final extraction uses
a larger aperture/profile. One of 165 eligible science stars has a fatal
hp_d pixel on this r3 support in both generations; it is explicitly flagged
and excluded only from paired hp_d summaries. All 235 reference apertures
are complete. The original RAW science cohort is unchanged.

| Role | Complete / total | Median / p95 absolute generation difference [ppt] |
|---|---:|---:|
| Science | 164 / 165 | 0.272 / 1.766 |
| Reference | 235 / 235 | 0.632 / 5.540 |

These are differences between reductions of the same photons, not temporal
RMS, calibration accuracy, or missing-neighbour flux estimates.

![All-source generation differences and their additive components](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/hpd_generation_bridge/generation_impact.png)

![Underlying absolute and difference stamp stacks](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/hpd_generation_bridge/generation_stamps.png)

The absolute panels share display scales and RAW aperture normalization.
The difference panels also share a scale. No image interpolation is applied.
Display-only local sky planes are removed from the absolute panels; the
generation difference is preserved directly. The stacks show broad structure,
so treating their aperture projection as a central missing-source component
would not be justified.

## Background differences require a separate repair test

The two largest absolute science differences are +24.03 and −5.05 ppt.
Their background components are +24.04 and −5.21 ppt, respectively. The two
largest reference differences are −146.46 and +38.15 ppt; corresponding
background components are −147.58 and +40.80 ppt. This is deliberately an
outlier inspection, not a population estimate.

![Actual background/template components for the largest differences](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/hpd_generation_bridge/assessment/background_component_examples.png)

The images show substantial curved background structure near some apertures.
Display sky-plane removal leaves the fixed-aperture measurement unchanged,
verified numerically. A background difference identifies a processing
difference; neither of the two maps is established as astrophysical truth.
The present RAW extraction does not subtract either saved background image,
so these large values are not automatically errors in its current light curves.

## Omitted-source geometry on the actual target set

Exact Gaia IDs link the newer paper ledger to the temporal targets. Within
three native pixels, 81/165 science targets and 83/235 references have at
least one recorded neighbour absent from the newer paper scene. There are
125 and 110 such target-neighbour pairs, respectively. This includes
catalogue positions on source cores as shown below.

![Actual hp_d examples with paper omitted-source IDs marked](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/hpd_generation_bridge/omission_examples.png)

![Central distribution and omitted-neighbour catalogue magnitude contrast](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/hpd_generation_bridge/assessment/generation_central_population.png)

These are catalogue-epoch relative positions through the common FFI WCS,
not a new proper-motion-corrected scene. Missing proper motion is flagged
for 7 science pairs and 6 reference pairs. Their catalogue magnitudes are
mostly much fainter than the primary. No tight flux prior is inferred from
those values. A removed-source record still does not certify complete photon
removal; all full-positive permissions remain false in the source ledger.
Most importantly, this ledger describes the paper template, not the historical
production template. It must not be used to add full positive ePSFs blindly.

## Reproducibility and next action

- `hpd_generation_bridge.py`: cluster 1686 on plscience3, exit 0, 23 seconds,
  two requested CPUs/4000 MB. Frozen commands/code: `condor/hpd_generation_bridge/`.
- `hpd_generation_assessment.py`: cluster 1695 on plscience3, exit 0, 16 seconds,
  two requested CPUs/2000 MB. Frozen command/code: `condor/hpd_generation_assessment/`.
- Source hashes, complete-cohort flags, exact-ID neighbour pairs, raw stamp
  arrays and measurements are in `results/hpd_generation_bridge/`.
- All five figure products were visually inspected. The initial read-only
  preflight attempted bitwise operations on the float-valued legacy mask;
  it was corrected to explicit finite checking and integer conversion before
  either science job ran. No result depends on that failed inspection.
- [Existing canvas](http://localhost:18888/temporal-raw-epsf/canvas/temporal_raw_epsf.html).

The next discriminating hp_d repair test must isolate stellar-light transfer
through the actual background/removal operator using paired source injections,
or a justified independent scene, rather than taking a newer reduction as
truth. Reconstruct the legacy removal lineage before assigning its missing
source response. For the main RAW photometry, a neighbour correction needs
S24 faint-source profiles and signal-preserving temporal validation; the
failed population-PSF neighbour tests and inconclusive C4 pilot are not enough.

**Conclusion:** this bridge establishes a real generation mismatch and
locates significant background-dependent differences on our actual targets.
It prevents an unsupported transplant of the paper ledger into the time
series. It is progress toward the requested hp_d repair, not the repair or
proof of photon-limited light curves.