# Actual RAW background-curvature comparison — 2026-10-03

**Conclusion:** the completed 36-epoch test does not support a general RAW
background correction. The two predefined group histories remain essentially
unchanged. Some strongly affected individual stars improve, while another
comparatively flat star acquires variation. Background curvature is relevant
to particular stars but this estimator is not uniformly trustworthy. Smaller
median intervention is not proof of sky accuracy or photon-limited curves.
No accepted photometry or production file is replaced.

Run: `/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/`.
Code: `/home/kshukawa/syndiff_pipeline/dev/.worktrees/temporal-raw-epsf/`,
local branch `codex/temporal-raw-epsf`. No push, PR or merge.

## Fixed measurement and conditional intervention

The current extraction operates on unsubtracted SPOC-calibrated RAW pixels.
It fits a temporal source profile and local sky plane. A global background
can affect the flux through curvature that the local plane cannot absorb.
We subtract three background alternatives from the actual RAW image: the
historical map; a component-detector-only replacement; and the combined
component detector, harmonic fill and positive RMS interpolation candidate.
See `BACKGROUND_CANDIDATES.md` for their provenance and rejected alternatives.

Frames 1, 601 and 602 sample the beginning/end of orbit 1 and start of orbit 2.
All 172 science measurements and 235 disjoint references remain; summaries
use the same 165 eligible science stars and 354 predefined groups. No new
variable rejection, trend fit or flux-based source selection is introduced.
The background, source profile and fixed weights all use the same detector
pixels. Direct FFI stamps, not an interpolated light curve, are remeasured.

For fixed source and background design columns, the profile estimator is a
linear pixel operator:

$$\Delta F_{s,t}=-\sum_p o_{s,t,p} B_{t,p},\qquad
\Delta A_{r,t}=-\sum_p w_{r,p} B_{t,p}.$$

Here $s$ is a science source, $r$ a disjoint reference source, $t$ an exposure
and $p$ a detector pixel. $B$ is the tested global background in electrons
per second per pixel. $o$ is the dimensionless source-flux operator derived
from the archived profile-plus-plane weighted fit, including its numerical
ridge; $w$ is the existing dimensionless fixed-r3 aperture/sky weight.
$\Delta F$ and $\Delta A$ are the induced science/reference rate changes in
electrons per second. Profile, WCS, pixel mask and fit weights are held fixed.

The consistently propagated relative change is

$$\delta\ell_{s,t}=\log\!\left(1+\frac{\Delta F_{s,t}}{F_{s,t}}\right)
-\sum_r W_{r,s}\log\!\left(1+\frac{\Delta A_{r,t}}{A_{r,t}}\right),
\qquad F'_{c,s,t}=F_{c,s,t}\exp(\delta\ell_{s,t}).$$

Here $F$ and $A$ are the existing physically count-corrected science and
reference rates in electrons per second, $W$ is the fixed dimensionless
reference-weight matrix (each science column sums to one), $F_c$ is the
current reference-calibrated science rate, and $F'_c$ is the conditional
alternative rate. $\delta\ell$ is a dimensionless log-flux change. Learned
reference aperture transport, original reference normalizations and count
restoration remain fixed. This deliberately isolates background subtraction;
it is not a complete recalibration with altered input means.

## Three-epoch result

The following numbers describe the RMS of **the change**, after subtracting
its three-epoch mean per source. They are not the scatter of corrected stars.

| Background subtracted | Median change RMS [ppm] | 95th percentile [ppm] | Maximum [ppm] |
|---|---:|---:|---:|
| Historical | 85.96 | 399.59 | 1953.55 |
| Component detector only | 86.25 | 400.53 | 1952.24 |
| Combined positive-RMS candidate | 11.31 | 63.77 | 322.86 |

![Actual conditional changes by magnitude, colour and preselected group](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/background_curvature_photometry/curvature_photometric_effect.png)

The combined candidate's background curvature has broad structure across the
science stamps. The faint-bin normalized stack has the largest amplitude.
This plot removes a plane only for display and uses a shared colour scale;
the flux measurements retain the original background including ridge leakage.
These are background images, not data-minus-stellar-model residuals.

![Measured background curvature behind the changes](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/background_curvature_photometry/curvature_stacks.png)

Both previously selected groups retain their large early-to-late changes.
Each source/method has one constant three-sample normalization in this figure;
no temporal trend is removed. The plotted frame indices are sampling labels,
not uniform time intervals. These three points cannot establish full-sector
performance or separate remaining astrophysical and instrumental variation.

![Actual group samples before and after subtraction](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/background_curvature_photometry/actual_group_samples.png)

## Validation and implementation failure

The archived RAW profile flux replays within 2.84e-8 fraction. Direct
remeasurement of RAW minus each background agrees with the linear flux
operator to 8.74e-11 electrons/second maximum. No tested science/reference
rate becomes nonpositive. Saved float32 candidate-minus-historical maps give
a conservative first-order rounding bound of 0.0214 ppm maximum; this is not
background/model uncertainty.

Job 1955 failed a display-plane equivalence assertion before creating any
science output. The archived solver uses a 1e-10 scaled numerical ridge and
therefore does not reject a plane exactly. The revised calculation retains
the original solver, explicitly accounts for its plane response, and passes
the original tolerance. Maximum plane leakage is 4.64e-5 ppm. No scientific
threshold was loosened and no source was removed. Job 1962 then completed
with exit 0 in 18 seconds (two CPUs/4,000 MB requested, 101 MB reported peak).
All three figures were inspected before this record was written.

Script: `temporal_raw_epsf/background_curvature_photometry.py`.
Frozen commands/code/hashes: `condor/background_curvature_photometry/` and
`condor/background_curvature_photometry_ridge/`.
Arrays/tables/figures: `results/background_curvature_photometry/`.
The saved conditional fluxes retain all three epochs and all source IDs;
`group_changes.csv` includes all 354 groups rather than only the two plots.

## Completed extension and acceptance gates

`background_sample_worker.py` extends to the already chosen 36 FFIs. It
freezes actual epoch inputs, qualifies historical float32 replay,
then runs the combined candidate and a paired +1-ppt source perturbation.
Each stage uses a fresh process to isolate experimental SEP overrides.
Incomplete outputs stop for inspection; existing completed stages are reused.
The last-epoch worker 1965 completed successfully. Eight four-epoch workers
1966–1973 cover the remaining 32 sample epochs. Worker 1973 stopped at FFI
288 because two pixels, at crop coordinates (945,1635) and (1783,1870),
differed by one adjacent float32 value (1.526e-5 electrons/second). Neither
overlaps any of the 431 science/reference stamps. The explicit
`background_replay_guard.py` qualification now permits only adjacent-float
changes outside every complete stamp and requires exact equality on all
measurement stamps. It records full-image inequality rather than changing
the original replay report. New worker 1978 resumes this batch, reusing its
completed inputs and replay. Same-process candidate pairing remains required.
The single source's core varies with its archived RAW profile; the empirical
outer-wing family and catalogue/kernel masks remain fixed. This is temporal
coverage, not independent validation of arbitrary source morphology.

`background_curvature_photometry.py --sample` compares original/combined
backgrounds on all completed sample epochs. It changes no unsampled epoch.
Further sources and shapes, partial-neighbour accounting, mean/normalization
refits, covariance and complete noise accounting are still required. A
smaller group RMS alone is not permission to subtract astrophysical signals.

All 36 frames completed. A second replay stop at frame 600 was one adjacent
float32 difference at crop (1076,1024), also outside every measurement stamp.
The same unchanged guard qualified it; worker 1985 resumed the two remaining
epochs after original worker 1967 exited 1. Thus 34 full images replay
bit-exactly and all 36 replay exactly on every science/reference stamp. All
paired harmonic solves converge below 1.01e-10 relative residual. These
selected-source controls are deterministic responses, not noise floors.

The generalized evaluator was separately checked by job 1984: every saved
array equals the original three-epoch result exactly. Identity checks cover
both science/reference source IDs, frame order and physical rate conventions.
Pilot group assessment 1986 was rerendered by 1987 to fix a clipped row label;
its full numerical group table is unchanged. Empty masked pixels remain
missing rather than receiving zeros.

### All 36 sampled epochs

Job 1992 applies both backgrounds to actual RAW science/reference pixels;
1993 assesses all 354 groups. All rates remain positive. Maximum direct
remeasurement/operator disagreement is 1.17e-10 electrons/second; archived
RAW profile replay differs by at most 3.68e-8 fraction. The saved background
difference rounding bound is 0.0702 ppm and ridge plane leakage is 0.000231
ppm maximum. Neither is a physical error estimate.

| Background subtracted | Median change RMS [ppm] | 95th percentile [ppm] | Maximum [ppm] |
|---|---:|---:|---:|
| Historical | 213.91 | 1021.18 | 27333.80 |
| Combined positive-RMS candidate | 43.78 | 597.08 | 3573.15 |

These changes are larger than in the three-epoch pilot, showing why the
middle-of-orbit observations matter. They do not measure corrected scatter.

![36-epoch intervention by magnitude, colour and predefined group](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/background_curvature_sample/curvature_photometric_effect.png)

![Matching endpoint background-curvature stamps](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/background_curvature_sample/curvature_stacks.png)

The actual group RMS values below use a single sample-mean normalization per
source/method, no trend fit and the same membership. Nonuniform epoch sampling,
stellar variability and shared references remain; these are descriptive
values, not multiple-testing significance or independent instrumental power.

| Predefined group | Current RAW RMS [ppm] | Historical subtraction | Combined subtraction |
|---|---:|---:|---:|
| grid4_shift0_1_0, 10 stars | 228.56 | 233.55 | 233.55 |
| knn16_104, 16 stars | 252.03 | 298.77 | 253.12 |

![Actual 36-epoch group histories](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/background_curvature_sample/actual_group_samples.png)

![All predefined groups and source positions](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/background_curvature_sample/assessment/multiscale_comparison.png)

![Historical, combined and difference background images in every 3x3 cell](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/background_curvature_sample/assessment/spatial_background_images.png)

### Largest individual interventions

The four largest combined-response RMS sources are inspected explicitly,
not rejected. Their selection is post hoc and diagnostic. In three cases
the large orbit-shaped excursions shrink, but in Gaia 1629810213318424704
the candidate introduces variation into a comparatively flat current curve.
That star is also the target chosen earlier, independently of these outcomes,
for the background perturbation tests. Small differential signal absorption
does not prove that the absolute background beneath it is accurate.

![Largest conditional individual interventions](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/background_curvature_sample/outliers_fitmask/individual_interventions.png)

The image diagnostic shows the actual RAW-minus-template region, the two
background curvatures and the current RAW-minus-fitted-scene residuals at
the endpoints and largest-response epochs. Background row scales are shared;
the stellar residual row has its own much smaller scale. The historical
background has a pronounced broad central feature in the high-background
epochs; the combined version reduces it but leaves structure. This is not
proof that the combined background is correct or that every improvement is
instrumental. The exact fit exclusions include all configured masks, not
only catalogue bits. The first rendering's clipped labels are corrected in
the fit-mask figure; numerical photometry is unchanged.

![Actual images and fitting exclusions at the strongest intervention](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/background_curvature_sample/outliers_fitmask/outlier_images.png)

**Conclusion:** this background family does not resolve the two persistent
group histories, and must not be applied indiscriminately to RAW curves.
It reveals a mixture of local effects that the three endpoints missed.
The FFI-698 component split and 36-epoch strap-only follow-up are now recorded
in STRAP_BACKGROUND_20261003.md (run copy: STRAP_BACKGROUND.md). They separate
a surface-curvature problem from column-shaped response in three other stars.
Independent background prediction and strap-only signal tests remain needed.
Gain/flat accuracy, neighbour light, complete noise accounting and the original
goal remain open.