# PS1 removal inventory audit: paper C4

2026-10-02. Completed diagnostic milestone. Production code and science products unchanged.

## Headline / Conclusion

The existing CSV is **not a complete inventory of removed sources**. Its unmatched rows mostly describe zero-flux pixel components, but the RP<18 accounting catalogue demonstrably excludes faint Gaia sources whose centre pixels are removed. There are also off-cell association failures and a confirmed radius-cap bookkeeping inconsistency.

The audited field is paper **C4: sector 23, camera 4, CCD 2**, frame `tess2020087042920`. The frozen paper code uses `footprint_v1`, not the older segment removal in the live checkout.

Full results canvas: [shared portal](http://localhost:18888/ps1-removal-audit-c4/), [saved HTML](/astro/armin/koji/syndiff/dev_runs/ps1_removal_audit_20261002/index.html).

## Verified findings

| Check | Result | Evidence |
|---|---|---|
| Original ledgers | 975 cells; 43,352 rows; 33,571 distinct known Gaia IDs. Original hashes, recipe IDs and fingerprints checked. | `tables/inventory.json`, `figs/inventory.png` |
| Unmatched components | 702; 700 report zero flux; 582 centroids within 10 PS1 pixels of a cell edge. | `tables/unmatched_with_audit_classification.csv`, `figs/inventory.png` |
| Zero-flux pixel check | Four sampled zero-flux components in two replayed cells had only zero-valued pixels before the saturation pass. This is not a pixel audit of all 700. | Three `*_audit.json`; `figs/skycell.2528.005_stamps.png` and `figs/skycell.2332.095_stamps.png` |
| Nonzero unmatched cases | Both have nearby bright Gaia stars centred outside the skycell. The ten-pixel inward lookup fails; saturation removal subsequently ignores off-cell stars when assigning identities. | `figs/skycell.2390.080_stamps.png`, `figs/skycell.2332.095_stamps.png` |
| Deeper Gaia check in 2528.005 | G<21, no RP requirement: 21 Gaia sources absent from the original catalogue and ledger have finite nonzero centre pixels explicitly zeroed. Twenty have RP≥18; one has no RP. | `tables/additional_gaia_centres.csv`, `figs/skycell.2528.005_deep_stamps.png`, complete gallery `figs/all_missing_1.png`–`all_missing_3.png` |
| Frozen replay | Exactly matches published images, including NaN locations, for all 118,950,692 pixels in cells 2528.005, 2390.080 and 2332.095. Record counts agree: 60, 78, 37. | Three `*_audit.json` |
| Radius-cap regression | A neighbour 800 pixels from the bright trigger, in the same connected component, is recorded as removed while its pixels survive the 480-pixel cap. Controlled test, not a measured C4 occurrence rate. | `tables/synthetic_cap.json`, `figs/synthetic_cap.png` |

G and RP are Gaia broad-band and red-photometer magnitudes. T is the pipeline's Gaia-derived TESS-equivalent magnitude. The original removal trigger is T<13; the accounting catalogue is RP<18. These are different cuts.

![Full ledger inventory](/astro/armin/koji/syndiff/dev_runs/ps1_removal_audit_20261002/figs/inventory.png)

![Measured images behind the faint-source finding](/astro/armin/koji/syndiff/dev_runs/ps1_removal_audit_20261002/figs/skycell.2528.005_deep_stamps.png)

![Nonzero unmatched footprint at the cell edge](/astro/armin/koji/syndiff/dev_runs/ps1_removal_audit_20261002/figs/skycell.2390.080_stamps.png)

![Radius-cap bookkeeping counterexample](/astro/armin/koji/syndiff/dev_runs/ps1_removal_audit_20261002/figs/synthetic_cap.png)

## Mechanism and boundaries

The pipeline first zeros pixels outside SEP source segments and its bright-core mask. The paper code forms connected components of SEP segments, bright cores and saturated/star-core mask pixels. Bright-trigger removal zeros a selected component within a 480-pixel disc. A secondary saturation pass zeros remaining components. Catalogue recording uses rounded source-centre positions, not full stellar profiles or a radial cross-match.

For unmatched component 2390.080:16237, nearby Gaia source `2150034897230546432` has T=9.30 and x=-2.89 PS1 pixels. For 2332.095:20016, source `1415769418609732096` has T=7.29 and y=6320.95, beyond the upper cell edge. Both are in the input projection catalogue but the primary lookup returns no valid pixel within ten inward steps. The secondary pass excludes off-cell centres. The saved candidate associations retain the original `source_id=-1`; the audit does not silently replace uncertain identities.

The centre-pixel test establishes an accounting failure, not each star's removed flux. It does not separate blends, certify all Gaia sources as astrophysical stars, account for partly deleted wings with surviving centres, or measure a paper-fit bias. Catalogue depth G<21 is itself finite. Four selected interior examples appear above; all 21 candidate positions appear in the canvas gallery.

Within the original RP<18 catalogue there were no omitted centres explicitly zeroed by the star-removal pass in the three replayed cells, and no recorded centres retained nonzero flux. Two original-catalogue centres were zeroed at the earlier background-suppression stage, one in 2528.005 and one in 2390.080; this is a separate category, not proof that each star's entire flux was removed.

The live checkout's legacy event exporter drops unknown IDs entirely. That path is not the authoritative paper per-cell ledger. Shared caches, frozen recipe and cell provenance must be resolved before comparing CSV populations.

## Required implementation

1. Persist exact pixel masks and component geometry, keeping background suppression and explicit source removal distinct. Record input/output flux, finite/changed pixel counts, WCS, recipe and catalogue provenance.
2. Separate bright triggering from inclusive source accounting. Retain the scientific T<13 policy while accounting with deeper Gaia without mandatory RP, plus PS1 detections and unmatched components with independent IDs and sky positions.
3. Associate source footprints with actual deleted support, recording partial overlaps and uncertain associations. Centre-label membership is insufficient for complete flux accounting.
4. Apply the removal radius cap to bookkeeping too; retain every trigger-to-component association; diagnose off-cell lookup failures and preserve unknowns through downstream exports.
5. Backfill immutable paper products through deterministic replay into versioned companion ledgers. Any change to removal physics needs a new recipe and validation.

## Records and reproduction

- Run directory: `/astro/armin/koji/syndiff/dev_runs/ps1_removal_audit_20261002/`.
- Paper bootstrap: `/astro/armin/koji/syndiff/dev_runs/paper_dataset_20261001/C4/bootstrap/`.
- Verified input inventory: `/astro/armin/koji/syndiff/dev_runs/epsf_stage_trace_20261002/fields/C4/tables/cell_inventory.json`.
- Frozen source: `/home/kshukawa/syndiff_pipeline/.claude/worktrees/paper-pin`, commit `648c0bcf4b559691177f7b4764f721ac54aebdc8`.
- Frozen combined recipe: `e17a198a4942aa2d`; band weights r/i/z/y = 0.254/0.4368/0.1654/0.1438; projection catalogue `gaia_dr3_projection_rp18_v1`.
- Original-band cache for 2528.005: `/astro/armin/koji/syndiff/dev_runs/epsf_atlas_20261001/dip/ps1check/skycell.2528.005_I0U0.npz`. Its prior independent refetch verification is documented by the stage-trace investigation. The two nonzero unmatched cells were freshly downloaded and replayed in this audit.
- Diagnostic source: `/home/kshukawa/syndiff_pipeline/doc/ps1_removal_audit_canvas/audit.py` and `publish.py`, copied into the run's `code/` directory. All raster/data outputs reside on `/astro`; local canvas links are symlinks.

Commands, from any directory, with absolute paths:

```bash
eval "$(mamba shell hook --shell bash)"
mamba activate syndiff
export OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 PYTHONDONTWRITEBYTECODE=1
python /home/kshukawa/syndiff_pipeline/doc/ps1_removal_audit_canvas/audit.py --out /astro/armin/koji/syndiff/dev_runs/ps1_removal_audit_20261002 --inventory --synthetic
python /home/kshukawa/syndiff_pipeline/doc/ps1_removal_audit_canvas/audit.py --out /astro/armin/koji/syndiff/dev_runs/ps1_removal_audit_20261002 --cell skycell.2528.005 --deep
python /home/kshukawa/syndiff_pipeline/doc/ps1_removal_audit_canvas/audit.py --out /astro/armin/koji/syndiff/dev_runs/ps1_removal_audit_20261002 --cell skycell.2390.080 --fetch
python /home/kshukawa/syndiff_pipeline/doc/ps1_removal_audit_canvas/audit.py --out /astro/armin/koji/syndiff/dev_runs/ps1_removal_audit_20261002 --cell skycell.2332.095 --fetch
python /home/kshukawa/syndiff_pipeline/doc/ps1_removal_audit_canvas/publish.py
```

The rerun uses this audit's cached public PS1 downloads and deeper Gaia response. Pixel maps identify finite nonzero changes; they do not imply that all raw background flux is stellar.

**Conclusion:** the unmatched population is primarily a record-type/edge-association issue, while genuinely incomplete faint-source accounting is directly demonstrated. This audit is sufficient to reject the current ledger as a complete removal catalogue; implementing and validating a replacement remains the next step.


## Ledger repair plan — awaiting approval

2026-10-02. User assigned this chat ownership of fixing the ledger, including seams and PS1 catalogue matching, and requested approval before implementation. The plan is [plan.html](plan.html), with source [ledger_fix_plan.md](ledger_fix_plan.md) and `/home/kshukawa/syndiff_pipeline/doc/ps1_ledger_fix_plan_20261002.md`. No implementation, catalogue campaign, jobs or science products changed for this planning step.

### Saved SEP caches clarified (2026-10-02)

User correctly recalled saved segmentation products. The standard combined-cell payload has only `combined_image` and `combined_mask` in its NPZ, but diagnostic caches are richer. Eight C4 caches under `epsf_atlas_20261001/dip/stagetrace/cells/` include boolean `segb` (SEP-positive union bright-core pixels), original PS1 flags `M0`, and pre/post-removal images. A read-only check on 2528.005 reconstructs the audited connected-component labels exactly from `segb` union saturated/star-core flags with 8-connectivity, without rerunning SEP. Paper-figure `scene_components*.npz` additionally hold labelled `segmap` and deletion masks, with different preprocessing/provenance requiring validation before reuse. The approval plan now explicitly inventories and reuses these caches first; SEP reruns are conditional on missing or incompatible inputs. No implementation has started.