# Background response and full-cadence profile re-learning — 2026-10-03

> **Repaired-surface follow-up, 2026-10-03:** [REPAIRED_BACKGROUND_RESPONSE.md](REPAIRED_BACKGROUND_RESPONSE.md) completes the 36-epoch repaired-surface validation and full-cadence refit. Selected stars improve further, persistent groups remain.

**Conclusion:** independently inferred sky response, followed by profile
re-learning and reference recalibration, substantially reduces large orbit-
shaped excursions in three diagnostic stars. It gives a modest overall
improvement on the unchanged 165-star eligible cohort. The persistent
neighbourhood trends remain, two conditional association flags remain, and
the updated complete noise budget is not established. This is a working
diagnostic result, not instrument-free/photon-limited acceptance. The original
goal remains active; no production or frozen accepted product is replaced.

Run: `/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/`.
Code: `/home/kshukawa/syndiff_pipeline/dev/.worktrees/temporal-raw-epsf/`,
local branch `codex/temporal-raw-epsf`. Nothing pushed or merged.
This extends the run's `STRAP_SKY_VALIDATION.md` and `STRAP_BACKGROUND.md`.

## 1. Separate background response from static residual light

The existing 36-epoch sample supplies 312 usable strap columns. The same
110,806 pad-6 sky pixels are evaluated at every epoch. Training uses either
the original source exclusions or an extra six-pixel expansion. Existing bad
columns remain excluded, and input-array/surface hashes are recorded.

$$R_{t,y,x}=B_{0,t,y,x}q_x(y)+c_{y,x}+\epsilon_{t,y,x}.$$

Here $t$ is exposure; $y,x$ are zero-based crop-local detector coordinates;
$R$ is measured calibrated RAW minus the frozen convolved template, in
electrons/second/pixel; $B_0$ is the historical pre-strap surface in the same
units; $q_x(y)$ is a dimensionless quadratic response along each column;
$c$ is a static additive residual in electrons/second/pixel; and $\epsilon$
is the remaining pixel error. $c$ can contain template or stellar-wing light.
It is fitted only on available sky pixels and is never filled under stars.
Template generations may differ between epochs, limiting physical attribution.

For the response-plus-static model, subtract the training-epoch mean at each
pixel, solve the resulting three-coefficient ordinary least-squares problem,
then recover $c$ from those training means. Compare against surface-only,
static-additive-only and response-only models on identical pixels. Withhold
whole two-day blocks with 0.25-day guards; separately train on each orbit and
predict the other. No stellar light curve trains these fits.

| Model, pad-6 training | Two-day held-out MSE | Opposite-orbit MSE |
|---|---:|---:|
| Surface only | 24.2157 | 24.2157 |
| Static additive only | 11.2982 | 10.0381 |
| Response only | 4.1847 | 4.0723 |
| Response plus static | 0.5646 | 0.5482 |

MSE is in $(\mathrm{electrons/second})^2$ per evaluated pixel, averaged equally
over the selected exposures. It is a conditional prediction error, not photon
variance. The minimum training illumination fraction is 0.1063 and the maximum
scaled normal-matrix condition number is 38.49; no evaluated polynomial has a
nonpositive response row. Those numerical checks do not establish detector-
truth calibration or uncertainty on a stellar correction.

![Held-out temporal prediction](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/temporal_strap_validation/temporal_scores.png)

![Underlying withheld pixel residuals](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/temporal_strap_validation/temporal_images.png)

![Response consistency between independently trained orbits](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/temporal_strap_validation/orbit_response.png)

Known quadratic response and arbitrary static-light recovery, physical-unit
rescaling, held-out-count perturbation invariance and zero-illumination
degeneracy controls pass. The real-data worker also perturbs withheld counts
and requires exactly unchanged fitted parameters.

## 2. Changing the background alone is insufficient

For each stellar stamp, use only the response trained on the opposite orbit
and fit a local response-weighted sky plane:

$$D_{s,t,p}=f_{s,t}P_{s,t,p}
+q_{o(t),p}\sum_{a=0}^{2}\beta_{s,t,a}b_a(p)+e_{s,t,p}.$$

Here $s$ is source and $p$ a native stamp pixel. $D$ is its actual calibrated
RAW rate in electrons/second/pixel. $f$ is the fitted source rate in the
declared aperture convention, in electrons/second; $P$ is its normalized
recorded temporal pixel profile; $q_{o(t),p}$ is the response inferred from
the orbit opposite to the evaluated orbit $o(t)$; $b_a$ are dimensionless
constant/x/y plane basis values; $\beta_a$ are free sky-plane coefficients in
electrons/second; and $e$ is the pixel residual in electrons/second. WCS remains
in the existing exported convention. No static $c$ map is subtracted beneath
a star.

The 36-epoch fixed-profile test preserves source/reference membership, count
restoration and the old reference transport. It replays old reference
apertures exactly and archived source fluxes within 3.68e-8 fraction. It
reduces excursions in three selected stars but leaves the two preselected
groups nearly unchanged. More importantly, changing only the sky basis while
retaining old profiles introduces visible column residuals in those stars.

![Actual fixed-profile conditional source histories](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/response_plane_sample/individual_histories.png)

![Measured pixels exposing the fixed-profile inconsistency](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/response_plane_sample/pixel_evidence.png)

This is consistent with background structure having entered the previously
learned recorded profiles. A better sky predictor alone does not complete
the photometric change; it changes how the same pixels are divided between
source and background.

## 3. Relearn all temporal profiles with consistent aperture conventions

For all 172 science sources and 1,224 quality-zero FFIs, fit the response-aware
outer-ring sky and normalize the training image by its matching r5 aperture:

$$Y_{s,t,p}=\frac{D_{s,t,p}-\widehat B_{s,t,p}}{A_{5,s,t}},
\qquad P_{s,t,p}=\sum_u H_{s,t,u}Y_{s,u,p}.$$

Here $\widehat B$ is the fitted response-weighted outer-ring sky in
electrons/second/pixel, $A_5$ is the matching five-pixel aperture/sky-operator
rate in electrons/second, and $Y$ is a normalized recorded pixel image. $H$ is
the dimensionless existing temporal profile predictor based on measured WCS,
shape state, cubic pointing terms and local quadratic time/state interpolation.
It uses the other interleaved exposures within the same orbit, has a 0.5-day
local scale and never uses an exposure's own pixels. This is exposure holdout,
not a claim that profile shapes are predicted from another orbit. The **sky
response**, separately, is trained on the opposite orbit.

The old extraction is reconstructed first. All 172 sources replay to maximum
fractional flux difference 3.80e-13. Every predictor has zero self-weight and
no cross-orbit weight/fallback. New profiles retain their matching r5
normalization within 1e-8. Off-strap software controls provide an unchanged-
response comparison. No per-star temporal flux trend is fitted.

The stored physical onboard count-loss image is projected into the new
aperture operator rather than silently retaining the old aperture correction:

$$F_{s,t}=f_{s,t}-\sum_p w_{5,s,t,p}\Delta I_{s,t,p}.$$

Here $F$ is the restored rate in electrons/second, $w_5$ is the new
dimensionless r5 aperture/sky operator, and $\Delta I$ is the existing modeled
retained-minus-ordinary pixel rate in electrons/second/pixel. The analogous
r3 projection is applied to auxiliary references. This keeps the measurement
convention consistent, but **does not resimulate photon expectations** with
the new sky/profile model; that physical uncertainty remains open.

![Actual old/new recorded profiles and differences, in a common display convention](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/response_profile_full/evidence/profiles.png)

![Matching measured pixel residuals and their differences](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/response_profile_full/evidence/residuals.png)

The absolute profile panels share a signed-log scale so both core and wings
are visible. Difference panels have their own common linear scale. For
display only, both profiles are expressed in the old r5 convention. Residuals
use the same old source-rate denominator. The off-strap example is unchanged;
the other profiles lose column-like components. Their total pixel residuals
can remain similar despite a material change in inferred source flux. Low
pixel residual alone is therefore not sufficient photometric evidence.

## 4. Rebuild reference transport and check signals

All 431 auxiliary apertures are remeasured; the same 235 disjoint references
and 165 eligible science stars are used for comparison. The seven massive
variables are excluded by the original selection, not a new favorable cut.
The r5/r3 transport is refit from simultaneous aperture ratios using the
existing source-fold and two-day exclusions. Explanatory features remain
fixed. Reference precision is both held at its original values as a control
and refitted from reference histories. No science light-curve amplitude
trains the reference ensemble.

Independent analytic primary profiles with 1-ppt, 5/10/20/30-day signals are
inserted into the actual pixels. Normalization, profile prediction and flux
fitting are repeated, followed by a matched transport/precision refit.
The maximum amplitude error is 0.1748% before reference recalibration and
0.4998% afterward. Final median/p95 paired signal-error RMS is 1.686/3.497
ppm. These paired errors are not photon noise. Global response, WCS and
physical photon expectations stay fixed; injections before nonlinear onboard
stacking and global response refitting are still required for full closure.

![Actual known/recovered signals and population errors](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/response_profile_full/evidence/signal_validation.png)

Reference-variable injections at aperture level give median/p95 imported
variation 87.82/120.15 ppm for the chosen 1-ppt reference signals. Independent
reference-half predictions differ by median/p95 91.14/109.59 ppm RMS. These
are finite-ensemble sensitivities, not a measured pure detector-noise floor,
and must not be silently relabeled astrophysical signal in science targets.

![Actual reference histories and calibration sensitivities](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/response_profile_full/reference_evidence/reference_sensitivity.png)

## 5. What changed in the actual light curves

Six-hour bins below are diagnostics only; no binning or trend subtraction
changes the saved exposure-level light curves. All numbers include real
stellar variability and shared reference uncertainty.

| Quantity [ppt RMS] | Current | New, fixed precision | New, refitted precision |
|---|---:|---:|---:|
| Median eligible source | 0.53490 | 0.49143 | 0.49476 |
| Full-cohort median history | 0.07661 | 0.06898 | 0.07282 |
| Preselected 10-star group | 0.13663 | 0.11304 | 0.12264 |
| Preselected 16-star group | 0.19484 | 0.18647 | 0.18328 |

![Matched full-cadence and all-group comparison](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/response_profile_full/assessment/lightcurve_assessment.png)

![Underlying constituent histories; missing time bins remain blank](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/response_profile_full/assessment/constituent_histories.png)

The same four diagnostic sources are retained:

| Gaia ID | Current six-hour RMS [ppt] | New, refitted precision [ppt] |
|---|---:|---:|
| 1629810213318424704 | 0.4534 | 0.4580 |
| 1625600183294357504 | 4.0266 | 0.4058 |
| 1627184373392387072 | 5.4977 | 0.8958 |
| 1603178323705979648 | 3.4923 | 0.6945 |

![Actual histories of those four sources](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/response_profile_full/assessment/diagnostic_stars.png)

Large orbit-shaped excursions shrink substantially in three stars. The
comparatively quiet off-strap example avoids a new large background trend.
Remaining variation is not classified as astrophysical by this result alone.
These selected examples are not a population-level acceptance claim.

All 354 predeclared magnitude, colour, spatial and neighbourhood groups are
retained. A 4,096-draw whole-history sign association test flags one group in
the current curves and two in each new arm. The new refitted-precision flags
are knn16_93 and knn16_104, with conditional family probabilities 0.00366 and
0.02587. Shared calibration covariance and non-sign-symmetric astrophysics
limit that test; neither flags nor non-detections identify instrumental
causation. The known-invalid slow-star phase-null argument is not used.

## 6. Standard residual grids and coverage limits

The standard colour × flux-derived Tmag × 3×3 CCD grids use actual residuals
and full fitted-model snapshots at FFIs 1, 601, 698 and 1225. Both arms use
the same 165 eligible input sources, pixel masks, WCS positions, old source
rate denominators and old median fluxes for bin membership. The standard
bilinear rasterizer is used, not an invented optical oversampling. The
canonical magnitude range contains 159 of those input sources. Numerous
faint/colour/spatial bins are sparse or empty and do not establish absence
of residuals. Width annotations are descriptive derivatives of these
rasters, not an independent optical jitter measurement.
Canonical plots retain their standard per-plot percentile colour limits;
the native-pixel comparisons above use explicitly shared before/after limits.

![Matched baseline absolute grid](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/response_profile_full/residual_grids/baseline/resgrid_raw_response_20261003_baseline.png)

![Relearned absolute grid](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/response_profile_full/residual_grids/relearned/resgrid_raw_response_20261003_relearned.png)

![Relearned relative grid; cell-wide residual removed](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/response_profile_full/residual_grids/relearned/resgrid_raw_response_20261003_relearned_rel.png)

All six generated absolute/relative grids, including the software pilot,
were inspected. Its seven eligible sources leave every bin below MIN_N=5;
those panels are deliberately blank. The plotting adapter changes only
empty-list handling, not the threshold or bin definitions. Both baseline and
new residual patterns remain visible and broadly similar; this is not a
clean-residual claim.

The three unique labels are published in the existing private
[residual-grid gallery](http://localhost:18888/residual-grids/resgrid.html).
All 115 prior entries were preserved and copied image hashes verified. The
legacy external Claude Artifact mirror was not republished because no Artifact
tool is available. The investigation's existing private results canvas is
updated separately; no new server or tunnel is used.

## 7. Provenance and remaining work

Frozen jobs on plscience3: temporal preparation 2071, temporal validation
2076, fixed-profile comparison 2085, response-profile preparation 2094,
software pilot 2099, full profile run 2104, reference calibration 2109,
standard grids 2114, multiscale assessment 2119, time-display correction
2124, profile/signal evidence 2129 and reference evidence 2142. Required outputs and terminal results
are checked. The time-display correction makes missing six-hour bins explicit
without changing numerical values. Source snapshots/hashes and exact commands
are in their `condor/` directories.

Outputs: `results/temporal_strap_validation/`, `response_plane_sample/`,
`response_profile_pilot/`, and `response_profile_full/`. The latter contains
one source file per auxiliary star; `reference_calibration/photometry.npz`
holds the complete 1,224-epoch science curves, injection/control products and
source/frame identities. This is a development product, not a replacement
for `products/temporal_epsf_wcs_candidate_v1/`.

Next requirements remain substantial:

- Recompute noise propagation for the new profiles, sky operators, count-loss
  projections and reference covariance. The old noise denominator is not an
  acceptance test for the new estimator, and archive uncertainty is not pure
  Poisson variance.
- Test whether epoch-dependent sky response explains residual orbit-edge
  features after profiling static residual light. Use withheld pixels and
  preserve the existing science support; do not choose amplitudes by stellar
  flatness.
- Quantify reference-variability import with the new inputs. The earlier
  reference-engineering experiment must not simply be repeated unchanged;
  any revisit must explicitly use the corrected apertures/profiles and be
  compared with the current reference ensemble on identical cohorts.
- Preserve global-response/WCS and photon-mean uncertainty in full-chain
  signals; inspect fresh absolute astrometry where new profile conventions
  matter. Broader/fainter/crowded populations, hp_d missing-light/offset repair
  and independent-sector validation remain required.

A separate mask audit has now found that the historical background fit admits
some columns already excluded from stellar extraction, including strongly
negative crop-column 1023 values near the earlier hp_d-offset outlier. A shadow
extra-exclusion test now completes at four epochs; see
BACKGROUND_DETECTOR_MASK_20261003.md (run copy BACKGROUND_DETECTOR_MASK.md).
Production masks and images remain untouched. This is a distinct background-
generation issue and must be evaluated before treating the conditional
response map as a detector calibration. Its results are not included in the
photometry above.

**Conclusion:** a physically motivated, independently tested background
component has now been carried through full-cadence temporal profile learning
and photometry. It repairs a substantial part of several large excursions,
but it has not removed all systematics or proved photon-noise dominance.