PS1 source-removal ledger audit

Approved implementation: Gaia + PS1 ledger plan · Progress and validation.

Paper C4: sector 23, camera 4, CCD 2 · 2026-10-02 · frozen footprint_v1 recipe e17a198a4942aa2d

Conclusion: the existing ledger is not a complete inventory of removed sources. Most unmatched rows are zero-flux component records, not missing Gaia identities. But the RP<18 accounting catalogue demonstrably omits faint Gaia sources whose centre pixels are removed. Two nonzero unmatched footprints also expose an off-cell association failure.

No production code, catalogue, template, or fit was changed. This report and its supplementary CSVs describe the existing products; they do not replace their authoritative ledger.

What the paper pipeline actually removes

The code first detects PS1 sources with SEP, masking bright cores, then zeros pixels outside detected source segments and the bright-core mask. It forms connected components from source segments, bright-core pixels, and pixels carrying both the PS1 saturation and star-core bits.

The bright-star pass uses catalogue TESS-equivalent magnitude T<13. It zeros the selected component within a 480-PS1-pixel disc around the triggering star. The saturation-flag pass can zero remaining whole components. Thus the deleted object is a pixel footprint, potentially containing several sources or fragments of a halo. It is not a fitted, individually subtracted star.

The accounting catalogue is Gaia DR3 RP<18. Here RP is Gaia red-photometer magnitude; G is Gaia broad-band magnitude; T is the pipeline's Gaia-derived TESS-equivalent magnitude. The removal trigger threshold and catalogue completeness limit are different quantities. There is no ordinary radial Gaia cross-match in the footprint ledger: assignment tests the rounded catalogue centre's component label.

Verified full C4 inventory

Re-read all 975 authoritative cell ledgers, checked their saved SHA-256 hashes, recipe IDs and fingerprints. These are per-cell records; skycell overlaps mean they are not unique-star counts.

Record categoryRowsMeaning
catalog_bright_star27,687Bright catalogue trigger
catalog_neighbor14,897Catalogue centre sharing a selected component
quality_flag_star66Catalogue centre inside a flag-selected component
quality_flag_no_star702No in-cell catalogue centre assigned; this is a component record

There are 43,352 rows and 33,571 distinct known Gaia IDs. Of the 702 unmatched components, 700 have zero recorded flux; 582 centroids are within 10 PS1 pixels of a cell edge. A zero sum alone is not a full pixel-level proof for all 700. Four zero-flux components in the replayed cells were checked directly: all their pixels were already zero before the saturation pass.

Full-ledger counts and unmatched-component positions. Red crosses identify the only two unmatched records with nonzero recorded flux. The source cells extend beyond the active TESS science area.
Full-ledger counts and unmatched-component positions. Red crosses identify the only two unmatched records with nonzero recorded flux. The source cells extend beyond the active TESS science area.
Actual PS1 images for three zero-flux unmatched fragments and an ordinary recorded faint neighbour. The fourth column isolates only the labelled component: the bright halo surrounding a tiny zero-flux fragment is a different component.
Actual PS1 images for three zero-flux unmatched fragments and an ordinary recorded faint neighbour. The fourth column isolates only the labelled component: the bright halo surrounding a tiny zero-flux fragment is a different component.

The two nonzero unmatched footprints

Both have a nearby bright Gaia source already present in the projection catalogue, but centred outside the skycell. The primary off-cell lookup returns no usable pixel within its ten-pixel inward search. The later saturation pass removes the halo; its association loop considers only in-cell catalogue centres, so the bright source is not eligible and the record receives source_id=-1.

Cell / componentNearby Gaia source (candidate association)Geometry and validation
2390.080 / 162372150034897230546432; T=9.30Star x=-2.89 PS1 px, just outside left edge; 30.69 px from flux centroid. Whole-cell replay exactly equals published image.
2332.095 / 200161415769418609732096; T=7.29Star y=6320.95 PS1 px, beyond upper edge; 83.11 px from flux centroid. Whole-cell replay exactly equals published image.

The images support bright-star halo associations. These are labelled candidates in the supplementary table, not blindly substituted IDs. A centroid distance is not a calibrated cross-match probability, and a halo can contain additional faint sources.

Nonzero unmatched halo at the left skycell edge. Green markers include the off-cell Gaia star; the white dashed line marks the cell edge. The faint visible source in the halo is not individually identified by this component record.
Nonzero unmatched halo at the left skycell edge. Green markers include the off-cell Gaia star; the white dashed line marks the cell edge. The faint visible source in the halo is not individually identified by this component record.
Second nonzero unmatched halo at the upper skycell edge, plus a zero-flux component and a recorded neighbour. The off-cell bright source is outside the recorded footprint despite its halo entering the image.
Second nonzero unmatched halo at the upper skycell edge, plus a zero-flux component and a recorded neighbour. The off-cell bright source is outside the recorded footprint despite its halo entering the image.

Confirmed omitted faint Gaia sources

For cell 2528.005, queried an independent deeper Gaia DR3 catalogue with G<21 and no RP requirement, then projected it with the same WCS. Of 1,437 in-cell sources, 764 are absent from the original input catalogue. 21 of those absent sources have finite, nonzero centre pixels after background suppression that become exactly zero in the published source-removal output. Twenty have RP≥18 and one lacks RP. None of the 21 is in the original removal ledger.

This is a direct lower-bound demonstration of incomplete accounting. It does not measure each blended star's removed flux, prove Gaia completeness below G=21, or account for partly deleted wings whose source centre survives. All 21 positions are shown below on the same flux scale. Grey marks nonfinite or unavailable pixels.

Four interior examples of omitted Gaia sources. Raw, background-suppressed, and published images are shown at identical positions; cyan crosses mark the omitted sources.
Four interior examples of omitted Gaia sources. Raw, background-suppressed, and published images are shown at identical positions; cyan crosses mark the omitted sources.
Complete gallery of the 21 newly identified omitted Gaia centres, page 1 of 3. Source IDs and Gaia magnitudes accompany every row.
Complete gallery of the 21 newly identified omitted Gaia centres, page 1 of 3. Source IDs and Gaia magnitudes accompany every row.
Complete gallery of the 21 newly identified omitted Gaia centres, page 2 of 3. Source IDs and Gaia magnitudes accompany every row.
Complete gallery of the 21 newly identified omitted Gaia centres, page 2 of 3. Source IDs and Gaia magnitudes accompany every row.
Complete gallery of the 21 newly identified omitted Gaia centres, page 3 of 3. Source IDs and Gaia magnitudes accompany every row.
Complete gallery of the 21 newly identified omitted Gaia centres, page 3 of 3. Source IDs and Gaia magnitudes accompany every row.

Separate confirmed bookkeeping defect: radius cap

A controlled test using the actual pinned removal function and a known connected segmentation shows a distant faint neighbour recorded as removed even though its pixels remain unchanged. Removal applies the 480-pixel disc; neighbour recording tests only whether the component label was selected. This is a reproducible code defect, not a measured frequency across C4. No recorded in-cell source centre retained nonzero flux in the three real-cell replays.

Controlled regression demonstration. Red: triggering T=10 star. Cyan: T=16 neighbour 800 pixels away in the same component. The neighbour is listed as removed while its input pixels survive.
Controlled regression demonstration. Red: triggering T=10 star. Cyan: T=16 neighbour 800 pixels away in the same component. The neighbour is listed as removed while its input pixels survive.

Validation and limits

Frozen-code replay matches every published pixel, including NaN locations, in cells 2528.005, 2390.080 and 2332.095: 118,950,692 pixels in total. The first uses a previously independently verified original-band cache; the other two were downloaded anew from PS1. Replay row counts also agree: 60, 78 and 37. Within the original RP<18 catalogues, these three cells have no unrecorded source centres explicitly zeroed by the star-removal pass. This limited centre check does not certify source wings or fainter populations.

Background suppression is a separate loss stage. In the first two cells one original-catalogue centre apiece was zeroed there without a source-removal record; this does not by itself demonstrate the loss of an identifiable star's full flux. Exact changed-pixel masks and catalogue-centre tables are saved for all three replayed cells.

The live checkout uses older segment-based removal, whereas these paper templates use footprint_v1. The legacy event export also drops source_id=-1 rows. Those observations concern different paths and must not be confused with the verified paper ledger. No complete PS1-source inventory or deleted-pixel mask was persisted by the original paper removal.

Required changes before calling the inventory complete

  1. Save the exact pixel decisions. Persist actual removal support, changed-pixel masks, per-component area/flux, WCS, recipe and catalogue provenance. Separate background suppression, bright-trigger removal, saturation removal, and downstream masking.
  2. Separate bright triggers from source accounting. Keep the intended T<13 trigger policy, but use a deeper accounting catalogue without requiring RP, supplemented by PS1 detections. Preserve unmatched detections and non-stellar components with their own IDs and sky coordinates.
  3. Account for the actual deleted footprint. Associate all affected source footprints with the actual mask, including partial overlaps and off-cell halos. Record ambiguity and fractional removal estimates separately; a centre-label test cannot certify complete stellar flux removal.
  4. Repair the proven inconsistencies. Use the same capped support for neighbour recording and deletion; retain all trigger-to-component associations; diagnose off-cell lookup failures explicitly; never discard unknown records during downstream export.
  5. Keep frozen templates immutable during audit. Backfill evidence from deterministic replay into versioned companion ledgers. A change to removal physics or trigger catalogue needs a new recipe and fresh template validation.

Download the evidence

Code pin: 648c0bcf4b559691177f7b4764f721ac54aebdc8. Data and figures: /astro/armin/koji/syndiff/dev_runs/ps1_removal_audit_20261002/. Source inputs: /astro/armin/koji/syndiff/dev_runs/paper_dataset_20261001/C4/bootstrap/; authoritative cell inventory: /astro/armin/koji/syndiff/dev_runs/epsf_stage_trace_20261002/fields/C4/tables/cell_inventory.json.

Conclusion: The unmatched-row count is dominated by zero-flux footprint bookkeeping, but complete removed-source accounting is demonstrably missing. The evidence warrants fixing and validating the ledger before interpreting it as a complete scene catalogue. The next implementation should preserve exact pixel masks and use deeper, inclusive source accounting; this audit has not measured the resulting paper-fit bias.