cpu · continue_F1_0: absolute and relative residual grids
2026-10-08 · Implemented in the pipeline checkout · isolated branch commit 3091aab0.
Conclusion: The existing Colab controller now runs scene-fit campaigns. A T4 completed continue → independent frozen → dependent polish for F1 fold0, synchronized verified artifacts, restored local evaluation paths, and shut down automatically. Steady training steps are 9.5–15.2× faster than an 8-core CPU job; the short 160-step fits are only 1.6–1.7× faster including compilation. Peak GPU memory was 1,569 MB. No sessions remain, usage is 0.00/hr; account balance decreased by 0.17 CU across the failed startup and successful pilot.
Three colour.dag arms, 160 stage-3 steps each, identical F1/S24 C2K2 scenes, checkpoint lineage, colour catalogue, recipes, adapter and scientific source (colour-radial 226b4e45). JAX 0.9.2, NumPy 2.4.6, Optax 0.2.8 on both backends. CPU job 7472 requested eight cores and 10 GB. GPU run colour-scene-t4-pilot3-20261008 used Tesla T4. CPU and GPU package hashes differ only for the campaign launcher, whose CUDA-selection and temporary-file handling were corrected. Scientific source and all input hashes match.
Each iteration synchronizes through conversion of the loss to a Python float. Medians use steps 11–159 (149 samples per arm). Whole-fit timings include initial solve, compilation, optimization and final output; they exclude VM provisioning, input transfer, artifact sync and held-out evaluation. The 8,000-step panel adds measured median step costs to the measured setup overhead; it is a projection with no convergence or stop-rule guarantee. One GPU executes fits serially; colour.dag currently permits 12 CPU fits simultaneously, so per-fit acceleration is not the campaign's overall speedup.
| Arm | CPU s/step | T4 s/step | Step speedup | CPU fit seconds | T4 fit seconds |
|---|---|---|---|---|---|
| continue_F1_0 | 0.824 | 0.054 | 15.2× | 184.4 | 106.1 |
| frozen_F1_0 | 0.840 | 0.079 | 10.7× | 188.8 | 111.3 |
| polish_F1_0 | 0.830 | 0.088 | 9.5× | 182.1 | 114.6 |
Both backends evaluated on the untouched eval_T17 scene, using the same stored noise and fold0 evaluation contract. All 2,062 held-out source IDs and held masks match per arm. Flux differences remain small but are not bitwise: median shifts −0.0075% to +0.0165%; 95th absolute differences 0.058–0.145%. Median relative core chi-square changes are below 0.03%; individual stars can differ more (95th absolute relative chi-square differences 2.3–5.4%). These are unconverged 160-step optimizer trajectories, not a convergence comparison or evidence of improved photometry.
| Arm | Held stars | Median flux shift | 95th absolute flux difference | Median relative core χ² change |
|---|---|---|---|---|
| continue_F1_0 | 2062 | -0.0075% | 0.0577% | -0.0137% |
| frozen_F1_0 | 2062 | 0.0073% | 0.0803% | -0.0273% |
| polish_F1_0 | 2062 | 0.0165% | 0.1453% | 0.0014% |
The red-star stacks use T10–13 and BP−RP > 1.25; GPU model/residual rasters are converted to the CPU flux denominator before differencing. Data light is shown next to the measured residuals. CPU, GPU and their differences share a colour scale; differences are small compared with the existing core/shoulder structures. The complete colour × magnitude × CCD grids below retain counts; empty bright bins have too few stars to interpret.
Source snapshots and hashed inputs, native recipes plus argv, warm-start metadata and flux state, optional adapters, renderer drop-slot masks, dependency order, fresh process per fit, wall-time budget and STOP, nested verified sync, local metadata restoration and automatic shutdown. The campaign tests pass (18); existing scene/background/runtime tests pass (33, one skipped). W3 launch-contract import was checked; W3 full GPU fits were not run. Full 8,000-step convergence, other fields, fresh multi-stage schedules and simultaneous fits on one GPU are not validated by this pilot.
Fixed during the pilot: generic JAX_PLATFORMS=gpu also probes unavailable ROCm → explicit cuda; Colab's preinstalled incompatible CUDA13 plugin → remove before pinned CUDA12 install; cached verified files incorrectly counted as download failures → skip them; scene outputs nested under job names → verify required products and restore local metadata; expired Drive OAuth → direct upload available by default; NFS cleanup race → do not reject a validated package for transient cleanup failure.
python -m syndiff_pipeline.forward_model.colab_scene colour-spec --help python -m syndiff_pipeline.forward_model.colab_scene contract-spec --help python scripts/colab_job.py publish-scene --spec /astro/your/spec.json --archive /astro/your/campaign.zip python scripts/colab_job.py submit-scene --gpu T4 --run-id YOUR_UNIQUE_RUN --archive /astro/your/campaign.zip --no-discord
Guide: /home/kshukawa/syndiff_pipeline/docs/markdown/colab_scene_campaigns.md. Record and reproduction scripts: /astro/armin/koji/syndiff/dev_runs/colab_scene_campaigns_20261008. Raw verified GPU products and locally usable result views: /astro/armin/koji/syndiff/output/colab/jobs/colour-scene-t4-pilot3-20261008. Scientific model source: /home/kshukawa/syndiff_pipeline/.claude/worktrees/colour-radial.
Conclusion: T4 is adequate for these F1 scene fits and substantially accelerates long training loops. The reusable runner is operationally tested. Use it for selected experiments and bounded campaigns; matching a 12-job Condor campaign's throughput and accepting final science fits still require field-specific, converged comparisons.