# Removal-ledger repair: approval plan

**2026-10-02 — approved by the user ("alright, please get going"). Isolated implementation and stages A–E are underway. Production merge, template replacement and removal-physics changes remain outside this approval.**

Owner: this Codex chat. Scope: correct and validate removed-source accounting, including faint neighbours, PS1 catalogue associations, and all seam contributions. Start with paper C4; finish validation on all five paper FFIs. Keep the existing [C4 audit](http://localhost:18888/ps1-removal-audit-c4/) as the evidence baseline.

## 1. The contract we will implement

The ledger must answer two different questions without confusing them:

1. **What pixels did this exact processing recipe change, and why?** Every deletion operation has reproducible geometry and provenance, including operations that change no finite flux.
2. **Which astronomical sources are affected in this particular final template?** Sources have Gaia and/or PS1 identities where defensible, explicit ambiguous/unmatched states otherwise, and links to their affected pixels after ownership, padding, convolution and mapping.

Pixel accounting will be exhaustive. Source accounting will cover the catalogues and detected sources used, with declared completeness limits. We cannot honestly promise identities or individual fluxes for arbitrarily faint, unresolved, or fully masked stars. Such light must remain represented as unresolved/unattributed flux, never silently disappear from accounting.

**Default scientific policy:** retain the existing bright-star trigger and removal algorithm. Deeper catalogues improve accounting, not the set of removal triggers. Correcting the ledger must leave the frozen template pixels unchanged. Any needed change to seam pixels or removal physics is isolated, demonstrated, and brought back for separate approval.

**Cache-first clarification (user correction, 2026-10-02):** saved SEP products already exist in investigation/figure caches. Inventory and validate these before scheduling any SEP replay. The standard paper combined-cell payload stores the processed image, PS1 quality mask and removal rows, but that is not the only available cache. The C4 stage-trace cache contains `segb` (SEP-positive pixels union bright-core mask), `M0`, and before/after images for eight cells. For 2528.005, `segb` plus the saturation flags reconstructs the audited connected-component labels exactly, without SEP. The paper-figure `scene_components*.npz` files also contain labelled `segmap` and deletion/background masks, though their recipe and preprocessing must be checked before reuse. Our audit has already saved exact changed-pixel masks and footprint labels for three cells. Prefer these validated artifacts; reconstruct missing footprint information cheaply where possible; rerun SEP only for missing or incompatible inputs. Catalogue matching and seam transport do not themselves require a new SEP pass.

## 2. Reuse the previous seam work, with explicit version control

I located Claude session `857ff00a-2939-49f6-bcd4-773cf15a5c4f`, titled **“seam light double-count production temp…”**, and read its handoff alongside source and experiment records.

| Existing work | Required reuse |
|---|---|
| `doc/seam_neighbour_fix_plan_20260930.md`; `seam-fix` / `25f8d97` | Clean same-projection snapshots, correct last-row padding, canonical-cell construction, neighbour fingerprints, projection-wide Gaia coverage, and off-cell removal handling. Use implemented code rather than treating the old design document as current truth. |
| `vrow-onewriter` / `1edfce9` | Single pixel owner for vertical overlaps, complementing the horizontal owner rule. Reuse `canonical_cell` placement and ownership functions. |
| `d128192`; `paper_dataset_20261001/template_v3/README.md` | Later per-band routing through the canonical implementation. Its alternative star-model removal was not enabled for that rebuild and will not be enabled by this task. |
| `padding_correction.py` and its full-halo/linearity tests | Ordered cross-projection replacement and convolution of the signed correction over the full padded domain. Keep exterior light that blurs into the recipient. |
| `vrow_validate_20261001/README.md` | Same-projection validation is useful evidence, but is not certification of cross-projection seams. Retain the known F2 cross-projection reproducer. |

**First deliverable:** a per-field product manifest identifying actual combined/convolved versions, removal convention, mappings, ownership rules, ordered padding inputs, raw-input/catalogue checksums and code SHA, together with a saved-SEP/mask inventory declaring what can be reused and what genuinely requires recomputation. Historical v2 products and v3 products get separate ledger views; never apply v3 ownership retrospectively to v2 images.

Implement on an isolated `codex/ps1-removal-ledger` worktree based on the verified relevant seam lineage. Do not edit `paper-pin`, `vrow-onewriter`, other agents' worktrees, or the live pipeline. Audit the recorded raw-skycell fingerprint issue as part of provenance; use actual content checksums for ledger replay rather than trusting a constant `missing` token.

## 3. Data model: preserve the facts before matching catalogues

Authoritative outputs will be versioned Parquet tables plus compressed pixel geometry; CSVs are convenient exports, not the only record.

| Artifact | Essential content |
|---|---|
| `manifest.json` | Schema, code/algorithm versions, input checksums, catalogue query/coverage metadata, WCS conventions, stage recipe and output-template fingerprint; status `complete` or explicit reason for incompleteness. |
| `regions.parquet` + pixel masks | Stable region/operation ID, skycell/projection/band, triggering source(s), original segmentation labels, **actual capped support**, stage/reason, pixel area, finite/changed/NaN counts, signed and absolute image-flux change. Store intended zeroing support separately from pixels whose value actually changed. |
| `sources.parquet` | Stable ledger source key; exact Gaia IDs and PS1 IDs; image-only source IDs; original/propagated coordinates, epoch/uncertainty, photometry and flags, source type, identity status. Missing identity stays nullable, never a shared deduplication key of `-1`. |
| `associations.parquet` | Many-to-many source↔region and catalogue↔detection candidates; matching evidence, separation, ambiguity/blend status, support definition, centre and footprint intersection, partial-removal status. Preserve all trigger↔component links. |
| `template_contributions.parquet` | Which region contributes to a specified final template, through which owning cell, donor patch, kernel, mapping/group and trim; distinguishing removed-in-a-cell from actually contributing to that template. |

Keep background suppression, bright-trigger deletion, saturation deletion, input invalid pixels and later template-mask exclusion distinct. Sequential masks may overlap; count each measured pixel change once at the stage where it occurred. A zero-flux component is retained as an operation/geometry record, not promoted to a fictitious star.

Statuses include `unaffected`, `centre_removed`, `partial_on_defined_support`, `fully_removed_on_defined_support`, `no_finite_flux`, and `uncertain`. “Fully removed” requires an explicitly defined finite source support; it never claims that an infinite stellar PSF vanished.

Region flux is measured directly from image differences. For blends, do not assign the entire region flux to every linked source or sum those associations as independent flux. Any individual-star removed-flux estimate requires a documented PSF/deblending model and uncertainty; it remains an estimate, separate from exact pixel bookkeeping.

## 4. Gaia + PS1 + image matching

**Gaia accounting:** query all available DR3 sources over the required sky regions without mandatory RP/BP or a bright magnitude cut. Tile/page the queries to completion. Catalogue size can change batching and storage, not silently introduce a magnitude truncation. Keep the existing trigger catalogue and threshold separate, so additional rows do not change template pixels.

**PS1 accounting:** use the DR2 stack catalogue for faint sources in the stacked images, retaining per-detection identities and available projection/skycell/filter identifiers. Supplement with mean/forced-mean information where useful; single-epoch detections are a diagnostic fallback, not the default bulk download. Discover and freeze actual API metadata before selecting columns. Store available r/i/z/y photometry, positions/errors/epochs, morphology, saturation/quality flags, primary-detection flags and detection counts. Do not require a source to exist in every band or Gaia.

MAST provides stack, mean, detection and forced-mean interfaces, metadata, query counts and paging. We will test count/page agreement, overlap deduplication and coverage before treating a catalogue tile as complete. [MAST API documentation](https://catalogs.mast.stsci.edu/docs/panstarrs.html)

The PS1 documentation warns that `PrimaryDetection` is not a guaranteed unique key, some objects lack a primary detection, and `BestDetection` is unreliable. Therefore retain the original detection records and deduplicate catalogue **objects** separately from image **measurements**, without discarding non-primary-only sources. It also documents catalogue/image astrometric differences: matching must validate local offsets. [MAST PS1 FAQ](https://outerspace.stsci.edu/spaces/PANSTARRS/pages/298812205/PS1%2BFAQ%2B-%2BFrequently%2Basked%2Bquestions)

**Image evidence:** retain the pipeline's native SEP segmentation and also detect/account for sources on the pre-removal r/i/z/y images and combined image. This second detection pass is diagnostic only; it must not feed back into deletion. Preserve unmatched detections, extended objects and plausible artifacts with distinct labels. Catalogue-only sources within deleted support are retained even when the image detector cannot deblend them from a bright halo.

**Association procedure:**

1. Construct coverage from all primary and padding-donor cells, including removal reach and convolution/reprojection support. A faint star outside the nominal target cell can still matter.
2. Preserve original catalogue coordinates. Propagate Gaia positions where proper motion and a usable image/catalogue epoch exist; account for the stack's epoch spread and flag missing motion/epoch information.
3. Measure a local catalogue-to-image offset using clean isolated stars. Freeze matching tolerances from this calibration and positional uncertainties before evaluating removed-source completeness; do not alter the science-image WCS.
4. Generate candidates using calibrated angular uncertainties, then use local PS1 image/detection evidence and band consistency. Keep close pairs and one-to-many blends ambiguous when needed; never assign the nearest star merely to eliminate an unmatched row.
5. Intersect source support with actual deleted support, including wings when the centre survives. A conservative candidate window is not itself proof of removed flux; uncertainty or unavailable support is recorded explicitly.
6. Deduplicate the same physical source across Gaia, PS1, skycells and bands while preserving all underlying measurements. Keep 64-bit identifiers lossless in CSV/JSON and browser display.

## 5. Seam correctness: carry the ledger through the real template operator

**Same-projection overlaps:** reuse the template's exact horizontal/vertical writer rules, row alignment and missing-neighbour fallback. Track donor identity through copied strips. Do not independently OR neighbouring deletion masks, sum duplicate source rows, or choose ownership by the catalogue's primary-detection flag.

**Cross-projection edges/corners:** reuse the exact ordered replacement masks, donor fingerprints, WCS transforms, finite/NaN handling, Gaussian support and recipient cropping. Model the signed seam correction once. A donor outside a cell or science crop must remain eligible if its light can enter after convolution. Transport region labels as provenance; transport image flux through the actual interpolation/convolution weights. A convolved Boolean mask is not a source identity map.

**Downsample and final template:** record the actual regmap, drift/group offsets, native-pixel ownership, exact seam corrections, mask exclusions, oversampling and crop. Distinguish contributions rejected by ownership from those that survive into the final template. A source can be removed in a discarded overlap copy without contributing any missing light to the assembled template.

**Independent closure test:** run the frozen template operator twice with all decisions fixed—once on the background-suppressed image before explicit removal and once after explicit removal—and compare with the transported removal image:

$$D_{\mathrm{template}}=\mathcal{T}(I_{\mathrm{before}})-\mathcal{T}(I_{\mathrm{after}}).$$

- $D_{\mathrm{template}}$: template flux difference, in the assembled template's flux units per output pixel; computed from the two controlled renders.
- $\mathcal{T}$: the selected product's fixed ownership, padding, convolution, reprojection, masking and downsampling operator, taken from its manifests and code version; it includes the full neighbour domain.
- $I_{\mathrm{before}}$: the collection of background-suppressed PS1 cell images immediately before explicit source deletion, in native PS1 combination flux units per input pixel; measured during replay.
- $I_{\mathrm{after}}$: the corresponding cell images after explicit deletion, in the same input units; required to reproduce the frozen products.

The two renders share validity masks, decisions and kernels, including NaN conventions. We will test linearity rather than assume it across value-dependent operations. Background suppression and downstream masking get separate controlled differences. This is an operator/accounting check, not an estimate of individual blended stellar flux or a refit of Hotpants/background parameters.

Existing cross-projection image defects are not hidden by this test: the ledger must first reproduce the selected product honestly; ideal-image seam consistency is an additional gate. Any remaining image defect is a named blocker to claiming end-to-end seam correctness, with its own proposed science change for approval.

## 6. Implementation stages and acceptance gates

| Stage | Work and deliverable | Exit gate |
|---|---|---|
| A — Lock the target | Product-version matrix; isolated worktree; inspect current active runs read-only; select immutable C4/F2 seam fixtures and known audit failures. | Every target resolves to explicit inputs and ownership version. Missing provenance blocks only affected products; no silent latest-pointer substitution. |
| B — Exact cell ledger | Instrument/replay applied masks, cap-aware region records, zero-flux classification, full trigger links and fail-closed publication. Add source- and operation-ID schema plus backward-compatible reader. | All selected replay images bit-identical, including NaNs. Every operation/pixel change represented; the 480-pixel cap counterexample no longer yields a false removed-neighbour claim. |
| C — Inclusive matching | Gaia/PS1 cached catalogues, pre-removal detections, calibrated associations, partial-support states, preserved unknowns. | Recover all 21 known omitted C4 Gaia sources. Resolve or explicitly mark the two off-cell halo associations. Query truncation/error cannot look like an empty sky; ambiguous and no-RP test cases preserved. |
| D — Seam transport | Shared ownership helpers and donor lineage through same-projection and cross-projection paths; operator-difference renderer and contribution tables. | Independent small-mosaic oracle agrees; no double counting or dropped exterior-halo contribution. Tests cover edges, corners and priority changes. |
| E — Real-data validation | Full C4 ledger and all its contributing seams; adversarial F2 cross-projection case; then all five paper FFIs. Per-field completeness/ambiguity tables and before/after maps. | Exact source-ID accounting, all synthetic truth cases correct, every template contribution traceable, numerical closure within preregistered precision limits, no unexplained seam residual or silent incomplete catalogue. |
| F — Review and release | Migration/backfill tool, runtime integration, docs, stable CSV views, results canvas and change review. | User reviews the completed implementation and evidence before production merge, template replacement, or changing the paper's selected dataset. |

**Regression matrix:** horizontal/vertical overlaps; first/last/ragged/missing rows; four edges and corners; cross-projection donor priorities; stars outside the cell/crop; masked saturated cores; faint neighbours whose centres survive but whose support overlaps; 480-pixel caps; crowded blends; extended sources; duplicate PS1 detections; proper motion; missing catalogue photometry; empty/failed/truncated queries; source-ID precision; cache-hit/skip/retry/partial publication; sparse/row and stream/Zarr paths; combined and per-band routes; os1 and os4 with common geometry.

**Numerical gates:** exact equality for copied ownership labels, integer IDs, membership, masks and frozen replay pixels; no nonzero finite changed pixel without a record. For weighted transforms, preregister absolute/relative floating-point bounds from an independent float64 impulse/mosaic oracle, input scale and operation count—not from fitting the observed seam residuals. Enforce both pixel residual and integrated-flux checks; publish residual stamps and worst cases. Do not demand os1/os4 equality where their actual geometries differ; test equivalence on identical geometry and diagnose the real products separately.

**Source-quality gates:** truth-labelled synthetic matches and ambiguous blends must produce the expected decisions. Real match-quality reports are stratified by magnitude, crowding, distance to bright stars and seam type. A high match fraction alone is not a pass: inspect false associations and report unresolved cases. No claims of catalogue completeness beyond recorded survey/detection coverage.

## 7. Publication, compatibility and resource limits

- Add ledger artifacts keyed by existing immutable image fingerprints **and separate ledger/catalogue/schema hashes**. Improving accounting must not force unnecessary image-cache invalidation or change old image identities.
- Backfill by exact historical replay; refuse to certify a ledger whose replay differs from its target image. Old CSV readers get explicit compatibility views; the new complete export preserves unknown/PS1-only/partial entries and all region associations.
- Write staged artifacts and atomically publish a manifest only after required tables/masks validate. Missing catalogues, failed coordinate transforms or absent neighbour provenance produce explicit incomplete/failure states, not apparently complete empty catalogues.
- Pilot the three audited C4 cells and a minimal seam mosaic first. Measure memory, compressed-ledger size, catalogue volume and runtime before the full-field pass. Use sparse masks/region runs and spatial indexes; do not generate a dense image per catalogue source.
- Reuse cached PS1 inputs and fixed mappings. Do not rebuild all convolved stores just to add records. Use the HTCondor skill if resource measurements require the pool; no paid Colab work is part of this plan.
- Preserve all published images and other agents' work. No cleanup/deletion, job cancellation, scheduling changes, or production merge is included in initial approval.
- Store new implementation-validation outputs under `/astro/armin/koji/syndiff/dev_runs/ps1_ledger_fix_20261002/` after approval, with README, exact commands, manifests and INDEX entry. Publish progress and validation on the existing audit portal; include original/removed/retained images, source overlays, seam-difference maps and unresolved examples. Keep all records current and checkpoint only owned code paths.

## 8. What approval means

Approval authorizes stages A–E: isolated implementation, catalogue retrieval/cache, tests, pilot and staged validation/backfill over the five paper fields. I will return a reviewable change and evidence. It does **not** authorize altering the bright-star scientific policy, enabling `starmodel_v1`, replacing frozen templates, adopting new paper products, or merging into the live pipeline. If a genuine image/seam defect prevents validation, I will isolate it and propose the minimal extra change before proceeding with that dependency.

**Recommended decision:** approve this plan with exact pixel accounting plus Gaia/PS1/image-source associations, separate cell and final-template ledgers, and seam closure as mandatory acceptance gates. This gives a testable definition of correctness while retaining honest uncertainty for unresolved stellar light.