# Static PSF refinement from retained pixels: held-out failure

> Follow-up: [PARTIAL_POPULATION_PRIOR.md](PARTIAL_POPULATION_PRIOR.md) now learns a source-excluded Gaussian shape prior from fully observed other stars. Neither prior-only nor posterior improves the required photometry; neither is adopted.

**2026-10-03 conclusion:** five static shape modes fitted only to retained
pixels do not provide a reliable missing-light correction. For the heavier
mask, retained-pixel fit quality can improve while hidden-core errors grow.
Held-out photometric accuracy worsens, and the worst primary-injection
amplitude error exceeds the existing 0.5% criterion. This model is not adopted.
The result limits this basis/prior/transport recipe; it does not rule out
better physically constrained or population-informed PSF models. Full goal
remains active and unmet.

Run: `/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/`.
Worktree: `/home/kshukawa/syndiff_pipeline/dev/.worktrees/temporal-raw-epsf`;
branch `codex/temporal-raw-epsf`, local only. Prior goal turn: progress, with
the partial-support inventory and irregular-mask validation completed.

## Model and genuine exclusions

For each clean control star, refine the existing opposite-source-fold
population reference with five static latent-coefficient modes:

$$P_{s,t}=\mathcal L_{s,t}\!\left(C_s+\sum_{k=1}^{5}\theta_{s,k}M_{s,k}\right),
\qquad D_{s,t,p}=F_{s,t}P_{s,t,p}+\sum_{j=1}^{3}b_{s,t,j}B_{s,t,p,j}+\epsilon_{s,t,p}.$$

Here $s$ labels a source, $t$ an exposure and $p$ a detector pixel. $C$ is
the fixed population latent spline; $M$ consists of two first and three
second finite-difference modes of its coefficients, with zero coefficient
sum. These are declared shape modes, not exact astrometric derivatives.
$\theta$ is constant over a fit's training orbit; its first two components
have pixel units and its other three have pixel-squared units. The fixed
linear renderer $\mathcal L$ applies measured WCS positioning, exact spline
pixel integration and inherited first-order blur/focus transport to every
mode. $P$ is dimensionless pixel response per source rate; $D$ is the
observed calibrated RAW rate in electrons/second/pixel; $F$ is free source
rate in electrons/second for each exposure; $B$ is the dimensionless existing
response-weighted sky-plane basis; $b$ gives free per-exposure sky coefficients
in electrons/second/pixel; and $\epsilon$ is the remaining pixel residual.

The objective is retained-pixel weighted least squares plus a Gaussian prior
on scaled parameters:

$$\chi^2=\sum_{t\in\mathcal T_s}\sum_{p\in\mathcal R_s}
 \frac{(D_{s,t,p}-F_{s,t}P_{s,t,p}-\sum_j b_{s,t,j}B_{s,t,p,j})^2}{U_{s,t,p}^2}
 +\sum_k u_{s,k}^2,\qquad u_{s,k}=\theta_{s,k}/\sigma_k.$$

Here $\mathcal T_s$ is the opposite orbit from evaluation; $\mathcal R_s$
is the intersection of the original clean-source mask and the artificial
pattern over the entire stamp, including the sky ring. $U$ is archived pixel
uncertainty in electrons/second/pixel. The fixed scales $\sigma$ are 0.05
pixel for the first two modes and 0.05 pixel squared for the other three;
$u$ is dimensionless and bounded between -4 and +4. Linear nuisance flux/sky
parameters are solved at every optimizer step. The analytic derivative
includes their response to the static parameters.

Hidden pixels and held-out epochs are indexed out before fitting. Full
aperture rates are evaluation measurements, never training normalizations.
The two fixed mask cases are the preceding benchmark's mask 2 and mask 4;
all 172 clean controls and the same 36 FFIs are retained. The 165-source
eligible population is unchanged. Global WCS, sky response and temporal
fields are inherited, so this is conditional validation of the new fit,
not an independent complete-pipeline holdout.

## Two photometric estimators and the support failure

The aperture estimator divides the measured partial-r3 rate by the model's
partial/complete-r3 response. It requires at least ten retained outer-ring
pixels for its sky estimate. One eligible source fails that support gate
for each tested mask, leaving 164/165 evaluable aperture histories. Those
failures remain explicit; no threshold is relaxed.

The separate joint estimator fits flux and sky over all retained pixels,
then projects the fitted source into the complete-r3 observable. It does
not use a ring-only data estimate and retains all 165 eligible sources.
The comparison is to the actually measured complete-r3 rate of these clean
controls. This is known support for the artificial-mask benchmark, **not
independent astrophysical or detector-calibration truth**. No observed
orbit-by-orbit normalization is applied.

## Numerical checks and failed attempts

Three unit checks pass: analytic versus numerical variable-projection
derivatives, known static-shape recovery with freely varying source flux,
and exact invariance to hidden-pixel/evaluation-epoch perturbations.
All 1,376 real fits converge; eight reach a parameter bound. Four
catalogue-spaced sources supply independent starts and sixteen real-data
exclusion checks. Maximum independent-start objective difference is
6.67e-16 fraction and scaled parameter difference 2.61e-8. Training and
evaluation source coefficients remain positive. Numerical convergence does
not establish scientific adequacy.

The initial job stopped at the sky-ring support gate. A second run completed
all numerical products but failed to serialize a NumPy integer in its fit
record. Both remain on disk. The authoritative `partial_static_psf_checked_v2`
run fixes record serialization, preserves unsupported aperture flags, and
completes both estimators. Every numerical array replays exactly for all 94
saved sources from the first attempt and all 172 from the second. The
assessment records these checks; failed attempts are not labeled successful.

## Held-out outcome

| Mask / estimator | Evaluable eligible stars | Baseline median/p95 temporal error [ppt] | Static-fit median/p95 temporal error [ppt] |
|---|---:|---:|---:|
| Mask 2, aperture | 164 | 0.4851 / 1.5230 | 0.7689 / 4.4528 |
| Mask 2, joint | 165 | 0.6343 / 2.5751 | 0.8482 / 5.6636 |
| Mask 4, aperture | 164 | 0.1230 / 0.3253 | 0.1286 / 0.3273 |
| Mask 4, joint | 165 | 0.2609 / 0.8764 | 0.2732 / 1.0325 |

These are 36-epoch errors relative to the same measured complete-r3
observable, not the science candidate's full-cadence light-curve RMS. The
stricter retained sky-ring support differs from the preceding aperture-only
benchmark, so its earlier numbers are not the baseline here.

![Actual held-out photometry and primary transfer](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/partial_static_psf_checked_v2/assessment/photometry_and_signal.png)

![Actual source histories on the opposite orbit](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/partial_static_psf_checked_v2/assessment/heldout_histories.png)

The known 1-ppt source injection uses the measured full-support empirical
profile solely to construct the added signal. It does not enter fitted
templates or hidden-pixel predictions. Static shape parameters are refitted
from retained training pixels. For all 172 sources, the joint estimator's
worst amplitude error is 0.5323% for mask 2 and 0.0718% for mask 4. The heavier
mask therefore fails the existing 0.5% amplitude criterion. This injection
remains conditional on fixed global response/WCS and photon expectations;
it is not a fresh pre-readout physical signal simulation.

![Actual known and recovered primary signals](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/partial_static_psf_checked_v2/assessment/primary_signal_histories.png)

## Images explain why retained-pixel fit quality is insufficient

The first displayed source develops a large hidden-core residual after the
static correction. Other sources change less. The generalization plot shows
many improvements on retained pixels but a much wider failure tail on
hidden pixels for the heavier mask. The image predictions, missing-pixel
errors and photometric outcomes must be evaluated together.

![Actual RAW, model and residual pixels on shared scales](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/partial_static_psf_checked_v2/assessment/heldout_pixels.png)

![Retained versus hidden-pixel generalization](/astro/armin/koji/syndiff/dev_runs/temporal_raw_epsf_20261001/results/partial_static_psf_checked_v2/assessment/pixel_generalization.png)

## Required residual grids

Four model grids—baseline and fit for each mask—each with absolute and
cell-relative versions were generated and inspected. They evaluate the
same full original clean-source pixels, including hidden pixels, on FFIs
1, 601, 698 and 1225. All images share the current unchanged per-epoch r5
rate as their normalization; standard flux-derived Tmag uses its full-time
median, and colour is Gaia BP-RP. There are 165 inputs, with 159 inside the
standard magnitude/position range. Sparse bins remain blank; they do not
establish good faint-star performance.

The canonical grid renderer selects its display limit separately for each
figure. Use the common-scale image comparison above for amplitude comparisons;
do not interpret a whiter auto-scaled grid as improvement. Relative grids
remove each cell's all-star stack and are not absolute residuals.

All four labels are in the [existing shared residual gallery](http://localhost:18888/residual-grids/resgrid.html):
`raw_partial_static_m2_population_20261003`,
`raw_partial_static_m2_static_fit_20261003`,
`raw_partial_static_m4_population_20261003`, and
`raw_partial_static_m4_static_fit_20261003`.
All 119 prior entries were preserved, giving 123. Copied image hashes match.
The external Claude Artifact mirror was not republished because its tool is
unavailable; the shared portal is updated. The main investigation canvas
embeds all eight inspected grid images alongside the outcome.

## Reproducibility and next model

Sources: `partial_static_model.py`, `partial_static_psf.py`,
`partial_static_assessment.py`, and `test_partial_static_model.py` under the
worktree's `temporal_raw_epsf/`. Frozen execution directories are
`condor/partial_static_psf`, `partial_static_psf_checked`,
`partial_static_psf_checked_v2`, and `partial_static_assessment`. The first
two failed for the recorded reasons; the last two exit 0. Fits use four
workers with a 6,000-MB local resource hint, assessment two CPUs/4,000 MB.
The resource hints are not enforced local memory limits.

The authoritative arrays, fit parameters, bound/convergence records and
source scores are under `results/partial_static_psf_checked_v2/`; figures,
standard-grid inputs, replay checks and gallery publication record are under
its `assessment/` directory. No WCS headers, production pixels, reference
membership, exported candidate or uncertainty floor is changed.

Next test a better constrained missing-profile prediction, for example a
shape prior learned from fully observed *other* stars rather than allowing
weakly constrained individual modes to determine hidden cores. Such a prior
must use the same opposite-target-fold population reference and relevant
state convention throughout; naively reusing the other stars' reverse-fold
models could leak the target's hidden pixels into its prior. Source/time
exclusions and known astrophysical signals must remain explicit. More
flexibility or lower retained-pixel loss alone is not a success criterion.
The paired signal histories also expose small orbit-dependent offsets after
refitting. A separate flux-normalized shape objective is a useful control
for modulation-dependent weighting of a misspecified static profile; it must
recompute its own observable normalization and repeat signal tests rather
than rescale the observed light curve separately on each orbit.

**Conclusion:** this static five-mode correction does not close the missing-
light problem and is rejected as a replacement. The validated solver and
hidden-pixel benchmark make the failure measurable; the broader-reference
and instrument-free photometry goals remain open.