# Column calibration trace and actual hp_d shadow regeneration — 2026-10-03

> **Illumination/centroid follow-up, 2026-10-03:** [ILLUMINATION_CENTROID.md](ILLUMINATION_CENTROID.md) adds nonlinear-response tests, fresh matched centroid transfer, latest-generation covariance and an independent two-minute coverage diagnostic. Absolute WCS and the full photon-limit goal remain open.

> **Repaired-surface follow-up, 2026-10-03:** [REPAIRED_BACKGROUND_RESPONSE.md](REPAIRED_BACKGROUND_RESPONSE.md) completes the 36-epoch repaired-surface validation and full-cadence refit. Selected stars improve further, persistent groups remain.

**Conclusion:** a time-dependent defect in full-FFI column 1067 is present in
the uncalibrated collateral pixels and is strongly propagated by column-smear
subtraction. Our background fit then used the defective calibrated column as
sky. The previously tested background repair now survives actual Hotpants
regeneration at four epochs. This is a verified local processing improvement,
not a complete detector calibration, corrected full hp_d time series, or
photon-noise acceptance. The broader goal remains active.

Run: `/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/`.
Code: `/home/kshukawa/syndiff_pipeline/dev/.worktrees/temporal-raw-epsf/`,
branch `codex/temporal-raw-epsf`, scripts in `temporal_raw_epsf/`.
All code stays local. Production products and the frozen v1 photometry are unchanged.

## Calibration is spatial, not one scalar per CCD

The actual inputs are SPOC-calibrated `_ffic` images in electrons/second.
The corresponding uncalibrated `_ffir` images contain digital counts. Each
CCD has four readout channels, separate gain/nonlinearity coefficients,
a 2-D bias model, exposure-dependent collateral corrections including column
smear, and a pixel flat. See the [NASA product description](https://heasarc.gsfc.nasa.gov/docs/tess/data-products.html)
and the earlier `CALIBRATION_AUDIT.md` for the exact files and header audit.
A CCD-wide light-curve trend alone cannot distinguish calibration, sky,
PSF/motion response, or shared-reference errors.

The frozen legacy `mask_settings.yaml` has no explicit bad-column layer.
The extraction's column list is also present in the earlier S24 C2K2
`forward_epsf_wcs/template_kernel/common.py`: crop columns
222, 424, 1023, 1024, 1032, 1033, 1376, 1386, 1505, 1967, 1968.
`SINGLE_FFI_TEST_PLAN_20260924.md` explicitly describes unmasked bad columns
found on September 23, although its broader exclusion intervals differ.
This is an empirical, field-specific provenance; it is not an official
universal detector mask. The new tests do not transplant it to other CCDs.

## Direct science/collateral comparison

`badcolumn_calibration_trace.py` reads the existing 36 uncalibrated/calibrated
FFI pairs, public 2-D bias and flat files. SHA256 hashes are saved in
`results/badcolumn_calibration_trace/summary.json`. Compare full-FFI column
1067 (crop 1023) with columns 1064–1066, all in output B. Use 144 fixed rows
selected by catalogue masks and detector boundaries, without clipping on
measured counts. Virtual rows are full-FFI 2060 through 2067 inclusive.

The first-order count diagnostic is

$$A_t=\frac{g_B}{\tau}\left[d^{\rm sci}_t-d^{\rm virt}_t\right].$$

Here $t$ is exposure; $g_B$ is header output-B gain in electrons/ADU;
$\tau$ is header integration time per readout in seconds;
$d^{\rm sci}_t$ is the median, over the fixed 144 science rows, of the
target-column count minus the mean of its three comparison columns;
and $d^{\rm virt}_t$ is the same contrast averaged over eight virtual rows.
Both contrasts are in ADU/readout after dividing raw coadds by header
`NREADOUT` and subtracting the public 2-D bias. $A_t$ is an approximate
pre-flat rate contrast in electrons/second/pixel. It omits nonlinear response,
row-bias differences and exact exposure/deadtime conventions; it is not a
replacement CAL model. Compare it with the archived calibrated image
multiplied by the public flat, retaining reference-file ambiguity.

| FFI | Science contrast before virtual subtraction [e−/s/pixel] | Science minus virtual contrast [e−/s/pixel] | Archived CAL × public flat contrast [e−/s/pixel] |
|---|---:|---:|---:|
| 1 | +49.558 | −67.528 | −67.729 |
| 601 | +48.721 | −67.299 | −67.447 |
| 698 | −35.918 | −504.762 | −505.241 |
| 1225 | +48.349 | −67.133 | −67.315 |

![Measured 36-epoch science and virtual-row contrasts](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/badcolumn_calibration_trace/column_history.png)

The virtual-row excess increases from approximately 44 to 176 ADU/readout
between the first and high-background examples. The first-order subtraction
closely follows the observed calibrated deficit. This identifies the
processing route, not the correct physical response of damaged science versus
collateral pixels. The science column itself also changes before subtraction.
Do not simply add back the virtual term and declare the pixels repaired.
These fixed-row contrasts differ from the whole-column medians in
`BACKGROUND_DETECTOR_MASK.md`; their denominators and reference pixels differ.

![Underlying observed science and virtual-row pixels](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/badcolumn_calibration_trace/science_and_virtual_pixels.png)

The panels remove only the same-row mean of the three comparison columns
for display. Every row of panels shares a colour scale; different panel
rows have different units. The high-background column is dark in science
counts and bright in the virtual rows before CAL; it becomes much darker
after calibration. The electronics origin is unresolved.

## Independently reproduce the historical subtraction

`hpd_shadow_replay.py` freezes the actual Hotpants Python package and compiled
extension, with hashes. It uses the saved per-FFI kernel coefficients and
actual oversampling-1 templates at FFIs 1, 601, 698, 1225. Kernel files omit
fit shape, so native 2048² and original padded 2064² geometry are both tested.
Only the eight-pixel-padded geometry reproduces the archive: maximum interior
closure is below 0.929 float32 epsilon-scaled units over 4,064,256 pixels
per exposure. The native-geometry control fails by orders of magnitude.
This is arithmetic identity, not detector accuracy.

`hpd_shadow_refit.py` then repeats complete C-backend fits using the frozen
lane settings and actual reference catalogue. All four fresh baseline images
are bit-identical to the archived images, including the high-background
frame's saved kernel coefficients. Catalogue/padding conventions are preserved.

For each exposure compare:

1. Original background and mask, full refit: exact baseline replay.
2. Corrected background, original saved kernel: independent template convolution.
3. Corrected background, original Hotpants mask, full kernel refit.
4. Corrected background and explicit bad-column Hotpants exclusions, full refit.

The corrected background is the previously tested component-anomaly,
positive-RMS, harmonic-fill candidate with detector-column exclusions.
The total change from historical biharmonic processing combines those changes;
it must not all be attributed to the column-mask change alone. The preceding
controlled harmonic-mask comparison isolates that contribution separately.

The matched source-rate change is

$$\Delta_{s,t}=10^6\frac{\sum_p o_{s,t,p}
   (D^{\rm new}_{t,p}-D^{\rm old}_{t,p})}{F_{s,t}}.$$

Here $s$ is star; $p$ indexes its fixed measured stamp pixels;
$o_{s,t,p}$ is the original dimensionless RAW profile-plus-plane source-rate
operator, including its numerical ridge; $D$ is hp_d in electrons/second/pixel;
$F_{s,t}$ is the same original physically corrected RAW source rate in
electrons/second; and $\Delta$ is a reduction change in ppm, not an error
relative to known stellar truth. No profile or WCS is refitted here.

| FFI | Previously selected offset source: full change [ppm] | Same source: fixed old kernel [ppm] | Median / p95 absolute full change, 164 complete eligible sources [ppm] |
|---|---:|---:|---:|
| 1 | +17880.17 | +17896.69 | 163.98 / 785.89 |
| 601 | +16267.09 | +16326.13 | 141.66 / 832.89 |
| 698 | +94528.69 | +94491.54 | 204.77 / 1335.41 |
| 1225 | +15854.10 | +15874.68 | 163.76 / 781.81 |

![Actual regenerated subtraction images and fixed-kernel comparison](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/hpd_shadow_assessment/four_frames_readable/subtraction_images.png)

![Source measurements on unchanged support](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/hpd_shadow_assessment/four_frames_readable/source_changes.png)

All 172 science sources are retained in the tables. The original variable
selection yields 165 eligible; 164 have common nonfatal Hotpants support.
No stars are selected because the correction improves their curves. Actual
underlying difference stacks by magnitude, colour and CCD location follow;
these are changes between reductions, not residuals against known truth.

![Magnitude and colour difference stacks](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/hpd_shadow_assessment/four_frames_readable/property_stacks.png)

![CCD-cell difference stacks](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/hpd_shadow_assessment/four_frames_readable/spatial_stacks.png)

## Signal and numerical checks

A +1-ppt extended-source perturbation at FFI 698 repeats the candidate
background and Hotpants fit. The r3 aperture recovery error is −0.08045 ppm
of the original RAW aperture rate, or −0.008045% of the injected amplitude.
The largest induced error in another auxiliary source is 0.0000332 ppm.
The injected frame's kernel is identical to the candidate baseline kernel.
This is one conditional empirical core/wing source family; WCS, template,
catalogue and uncertainties remain fixed. It is not universal signal preservation.

![Actual injected pixels and recovered-minus-injected image](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/hpd_shadow_assessment/four_frames_readable/injection_images.png)

HOTPANTS' aggregate `x2norm` is nonfinite in these runs and is not used as an
acceptance statistic. Its conditional output noise is not an updated physical
noise/covariance budget. Float32 arithmetic is checked exactly at the actual
subtraction interface, separately from double-precision closure bounds.

Successful jobs on plscience3: saved-kernel replay 2192; high-background
refit/injection 2197; other epochs 2200/2201/2202; assessment 2203; additional
stacks/readable labels 2209; calibration trace 2208. Exact frozen commands
and logs are under `condor/`. Initial replay 2189 lacked a finite C-extension
threshold setting; refit 2193 used a colliding pandas `.pad` accessor;
2194 incorrectly demanded double-precision equality from float32 subtraction.
All failed before accepted products. Corrections preserve the failures and
check the correct contracts; they do not relax a scientific residual gate.
The two assessment source tables and saved measured image arrays agree exactly
after presentation changes. All scientific figures were inspected.

## Remaining work

Recompute the temporal sky-response model with the corrected background
surfaces; the current full-cadence profile result remains conditional on the
older surfaces. Validate fresh hp_d/RAW astrometry and source-scene accounting
before using regenerated hp_d WCS. Expand the repaired background sample and
test independent held-out sky, source shapes and amplitudes. A full hp_d time
series, broader/fainter/crowded populations, remaining multiscale trends,
reference-star covariance and a revised photon-noise budget remain required.

**Conclusion:** calibration is not one number per CCD. A specific column's
science/collateral mismatch is now traced through the real counts, and the
downstream background/subtraction repair is tested beyond an algebraic map
change. The origin of every systematic and photon-limited photometry remain
unproven; no CCD-wide empirical flattening is justified by this result.