bug: release pipeline force-commits simulated workspace datasets (git add -f)

[Jammy2211]

bug: release pipeline force-commits simulated workspace datasets (git add -f)

Autonomy: supervised
Difficulty: large (executed as two scoped tasks — see Scope split)
Found: 2026-07-09 (verify_install check F). Diagnosis corrected 2026-07-13 — the
original "15×15 smoke artifact" framing below the fold was disproven; the real defect is
the -f force-commit, not simulator env.

Supersedes the old framing (issued/release_ships_smoke_datasets.md, retired
2026-07-13). The datasets aren't 15×15 because a smoke run shrank them at release —
generate.py never runs simulators. They're committed at all because pre_build.sh
force-stages dataset/ against .gitignore; some are incidentally 15×15 because a
prior local smoke run (PYAUTO_SMALL_DATASETS=1) left them on disk before pre_build.

Corrected diagnosis

Design intent (encoded in each workspace .gitignore): dataset/** is gitignored
except a small allowlist of real observational data un-ignored via ! lines. All
simulated datasets are meant to be generated at runtime, never committed. The
self-provision mechanism already exists — al.util.dataset.should_simulate(path) +
a simulator subprocess (canonical example scripts/imaging/modeling.py:96-102).

Two coupled bugs:

1. PyAutoBuild/pre_build.sh runs git add -f dataset/ — the -f overrides .gitignore,
   committing simulated datasets that should be ignored.
2. A minority of example scripts load simulated data without the should_simulate
   guard, so they only work because the -f leak committed the data for them.

Phase-2 cross-repo audit (2026-07-13, read-only)

Repo	ignore rule	tracked non-allowlisted	size	simulators	orphan dirs	loader
autolens_workspace	dataset/** + 14 ! allowlist	242	8.2 MB	63	7	from_fits
autogalaxy_workspace	dataset/** + 2 ! allowlist	134	1.4 MB	32	5	from_fits
autofit_workspace	bare dataset/	409	2.6 MB	3	—	from_json
HowToFit	bare dataset/	408	2.6 MB	2	—	from_json
HowToGalaxy	bare dataset/	21	2.8 MB	4	—	from_fits
HowToLens	bare dataset/	27	0.9 MB	7	—	from_fits

Notebook/Colab CWD gate → PASS. The guard's relative subprocess path resolves in both
notebook paths: Colab via setup_colab.setup() → os.chdir(workspace_dir)
(setup_colab.py:250); CI nbconvert via run_smoke.py:118 (cwd=str(WORKSPACE)). Purge is
CWD-safe provided each simulated loader is guarded and its simulator exists.

Orphans (no simulator reference — do not blind-purge):

• autolens (7): cluster/csv_api_example, imaging/simulated_lens,
  interferometer/{many_visibilities, simpleold, simulated_lens},
  point_source/{simulated_lens, start_here_example}
• autogalaxy (5): database/simple__{0,1,2}, {imaging,interferometer}/simulated_galaxy

Scope split

• This issue (bug: release pipeline force-commits simulated workspace datasets (git add -f) #126) = Group A: autolens_workspace + autogalaxy_workspace — allowlist
  regime, from_fits + should_simulate idiom already present, simulators exist. The 4-leg
  fix applies as written. Plus universal leg 1 (the -f drop).
• Group B = the 4 bare-dataset/ repos (autofit, HowToFit, HowToGalaxy, HowToLens) —
  small toy/tutorial data, possibly committed-by-design; needs a policy decision (allowlist
  vs purge+guard, and a from_json guard idiom for the fit repos). Filed separately as
  bug/pyautobuild/release_datasets_group_b_policy.md.

Plan (high-level)

1. Leg 1 — stop the leak (universal): drop -f from git add dataset/ in
   pre_build.sh so staging honors .gitignore.
2. Leg 2 — guard the problem set (Group A): add the should_simulate + simulator
   subprocess to the truly-unguarded simulated loaders (small set; not the sibling scripts
   that rely on a folder's guarded start_here.py).
3. Leg 3 — purge (Group A): git rm the non-allowlisted committed simulated datasets,
   excluding the classified orphans. Users/CI/Colab regenerate via the simulators.
4. Leg 4 — guard leg: an allowlist-based assertion (git ls-files dataset/ ⊆ .gitignore
   allowlist — NOT git check-ignore, which mis-flags tracked + negated paths), wired as a
   PyAutoHeart leg or pre_build assertion, landed after the purge so it goes green.

Plan (detailed)

• Leg 1 — PyAutoBuild/pre_build.sh: git add -f dataset/ → git add dataset/.
  Already-tracked real (allowlisted) data stays tracked; only allowlisted paths can ever be
  staged henceforth.
• Leg 2 — per Group A repo, for each from_fits script that loads a non-allowlisted
  (simulated) dataset and lacks should_simulate: insert the guard block mirroring
  autolens_workspace/scripts/imaging/modeling.py:96-102, targeting that folder's simulator.
  Exclude scripts whose data is provided by a guarded start_here.py in the same folder.
• Leg 3 — git rm -r --cached (then remove from tree) each non-allowlisted, non-orphan
  dataset dir in autolens + autogalaxy; verify a simulator produces it first. Orphans parked
  for a per-dir decision (dead → purge; consumer-but-no-simulator → needs a simulator).
• Leg 4 — allowlist guard: parse each workspace .gitignore !dataset/... lines, assert
  every git ls-files dataset/ entry is under an allowlisted prefix. Fail loud otherwise.

Autonomy

--auto, effective level supervised (bug cap). Plan written here; implementation
proceeds; run parks at ship sign-off with a batched question (no auto-PR, no merge).
Orphan-dir dispositions and the precise leg-2 set are supervised judgment forks → batched
questions on this issue.

Impact

Every pip/Colab/tagged-zip user ships ~10 MB of simulated datasets that should be
runtime-generated; some are degenerate 15×15 from a pre-release smoke run. Open =
release-blocker.

[Jammy2211]

Evidence correction + sharper picture

The check-F clone that surfaced this was at tag 2026.5.29.4 (matching its pip-installed autolens — PyPI's stack latest; the tag-matched clone worked exactly as designed), not 2026.7.6.649 as originally written. Re-verified directly against git:

State	imaging/simple	imaging/dark_matter_subhalo	overall
tag 2026.5.29.4 (= what every pip/Colab user gets today)	15×15	15×15	30/49 ≤20px
tag 2026.7.6.649 (current)	100×100 (fixed 05-29)	15×15	28/49 ≤20px
origin/main today	100×100	15×15	~28/49

So there is no release-lineage re-smallification — strike that leg of the original report. The mechanism is simpler: the 2026-05-21 "pre build" commits wrote smoke-regenerated (PYAUTO_SMALL_DATASETS) datasets straight to main; only imaging/simple was subsequently restored. The fix legs stand, re-prioritised:

1. Repair the data on main (regenerate the ~28 datasets full-size) — ideally before the upcoming release, otherwise the new tag inherits them and the freshly Colab-enabled notebooks will demo on degenerate data.
2. Guard: assert example data.fits shapes exceed the smoke footprint (Heart check or pre-commit in the pipeline) so a future smoke-env run can't be committed again.
3. Audit autogalaxy_workspace / autofit_workspace / HowTo repos for the same class (the HowTo checkouts currently carry large uncommitted dataset churn locally — possibly full-size regenerations; verify before reusing).

Leg 1 is the release-blocking one; the "stop the leak" pipeline leg may already be moot if pre-build no longer runs simulators under smoke env — verify while fixing.

[Jammy2211]

Diagnosis corrected + scoped (2026-07-13, --auto supervised).

The original "15×15 smoke artifact" framing is disproven — generate.py never runs simulators, so the release isn't shrinking data. The real defect is pre_build.sh's git add -f dataset/, which force-commits simulated datasets against each workspace's .gitignore allowlist.

A read-only Phase-2 cross-repo audit (see the rewritten issue body) found two .gitignore regimes, so this splits:

• bug: release pipeline force-commits simulated workspace datasets (git add -f) #126 = Group A (autolens + autogalaxy): allowlist regime, should_simulate+from_fits idiom present, simulators exist → the 4-leg fix applies as written.
• Group B (autofit / HowToFit / HowToGalaxy / HowToLens): bare dataset/, tiny toy/tutorial data possibly committed-by-design → a policy decision, filed separately.

Notebook/Colab CWD gate verified PASS. Orphan dirs classified (autolens 7, autogalaxy 5) as do-not-blind-purge. Proceeding through implementation; will park at ship sign-off with a batched question.

[Jammy2211]

⏸️ Parked at ship sign-off + a design fork (supervised --auto)

What's done and ready: Leg 1 (the -f drop) is implemented and committed on
feature/release-ships-simulated-datasets (4600e6f, PyAutoBuild). It's the clean,
universal, safe part — staging now honors each workspace's .gitignore so no new
simulated data can be force-committed. Ship sign-off for it parks here (no push/PR
until you approve — supervised contract).

The fork: during execution I built the precise path-based provisioning map, and it
diverges from the plan. The prompt scoped leg 2 as "~7 unguarded scripts"; the real
graph across the two Group-A repos is:

Category	autolens	autogalaxy	meaning
A. Purge-safe — a should_simulate-guarded script already regenerates it	9	4	purge now, no code change
B. Needs-guard — a simulator exists but no guarded caller invokes it	9	7	add a guard to the topic entry script, then purge
C. Orphan / ambiguous — no simulator found, or a multi//nested edge case	14	10	needs per-dir judgment

Full lists

autolens A (9): group/{operated,simple,simple__no_lens_light,sky_background}, imaging/{extra_galaxies,simple,simple__no_lens_light}, interferometer/simple, weak/simple
autolens B (9): {group,imaging,interferometer}/dark_matter_subhalo, imaging/{lens_light_asymmetric,light_operated,sky_background}, interferometer/extra_galaxies, point_source/{deblending,simple}
autolens C (14): cluster/csv_api_example, imaging/{dark_matter_subhalo_no_lens_light,misc,samples,simulated_lens}, interferometer/{datacube,many_visibilities,simpleold,simulated_lens}, multi/{imaging,interferometer}, point_source/{samples,simulated_lens,start_here_example}

autogalaxy A (4): imaging/{extra_galaxies,sersic_x2,simple}, interferometer/simple
autogalaxy B (7): imaging/{asymmetric,clumpy,operated,simple__sersic,sky_background}, interferometer/{clumpy,extra_galaxies}
autogalaxy C (10): database/simple__{0,1,2}, imaging/{ellipse,misc,samples,simulated_galaxy}, interferometer/simulated_galaxy, multi/{imaging,interferometer}

Two more wrinkles the plan didn't anticipate:

• Guard ownership — for the 16 B-dirs, which script owns the new guard? Several
  candidates are guide/tutorial scripts where injecting a subprocess simulate-block is a
  prose change (Opus, careful), not a mechanical edit.
• Category C isn't uniform. multi/* and interferometer/datacube are almost certainly
  mis-flagged (their simulators are nested deeper than my topic/name matcher looked) → really
  A/B. database/simple__{0,1,2} (autogalaxy) look committed-by-design aggregator data —
  same class as the Group-B fit repos. The rest (simulated_lens, simpleold,
  many_visibilities, csv_api_example, start_here_example, simulated_galaxy) match the
  prompt's "dead orphan" prediction.

My recommendation

1. Approve leg 1 → I push it and open its PyAutoBuild PR now (it's independent and
   valuable — stops the bleed immediately).
2. Greenlight the surgery approach for legs 2–3: execute A (purge, 13 dirs) + B
   (guard the topic entry → purge, 16 dirs) with run_smoke.py validation after each repo;
   reclassify C as I go — multi/+datacube fold into A/B, the 6 dead orphans get purged
   as dead (recoverable via git history), and database/simple__* moves to the Group-B
   prompt. Then leg 4 (allowlist guard) lands with the purge so it goes green.
3. Leg 4 wiring: pre_build assertion (blocks the release commit at the exact leak point),
   allowlist-presence-gated so bare-dataset/ Group-B repos are skipped-with-notice until
   they adopt allowlists.

What each answer does:

• "Approve leg 1 + greenlight the surgery" → I resume, ship leg 1's PR, then do the A+B
  guard/purge across both repos behind the four-leg gate, parking again only if smoke or
  review goes red or a C-dir needs a real call.
• "Approve leg 1 only" → I open leg 1's PR and leave 2–4 parked for a dedicated session.
• "Review the per-dir C dispositions first" → I post the itemized C table (consumer +
  simulator evidence per dir) before any purge.

No merge/close will happen autonomously either way.

[Jammy2211]

Resumed (go + route purges through PyAutoGut). Leg 1 PR open: #150 (pending-release). Now executing legs 2-4 on the worktree: add should_simulate guards to the B-set topic entries, purge A+B+dead-orphans via the Gut lifecycle (condemned.md type:file entries, sweep-after clock, archive-ref=pre-delete SHA; irreplaceable orphans get a durable archive/condemned/* ref first — no bare git rm), smoke-validate, then leg 4. Ship parks at PR-open per supervised.

[Jammy2211]

✅ Legs 1 & 3 (safe core) shipped — smoke-validated — + a checkpoint on the remainder

PRs open (pending-release, merge stays human):

• PyAutoBuild#150 — leg 1, drop -f from pre_build staging.
• autolens_workspace#272 — purge 19 dirs / 145 files (category A + multi/).
• autogalaxy_workspace#129 — purge 13 dirs / 93 files.

Validation: smoke re-run after each purge — autolens 9/9, autogalaxy 8/8. The
example entries self-provision their data via simulator subprocess post-purge (the
notebook entries too — the CWD gate holds in practice). All purges recorded via
PyAutoGut (PyAutoMind/condemned.md, two type: file entries, sweep-after
2026-08-13, archive-ref = pre-purge SHA — bytes recoverable from history).

The classification correction that made this safe

The prompt's "~7 unguarded scripts" (and my earlier "16 needs-guard") were both wrong:
the workspaces already self-provision almost everything via the older
if not dataset_path.exists(): subprocess.run([...simulator...]) idiom — my audits had
keyed on the string should_simulate, missing 130+ .exists()-guarded scripts. Corrected
signal (invokes a simulator subprocess, any condition): A purge-safe 17+11, B needs-guard
just 1, C 14+10. So leg 2 is effectively moot — no guard-injection surgery was needed.
I did not mass-convert .exists()→should_simulate() (130+ scripts, out of scope; the
prompt's shrink-under-smoke benefit is real but a separate cleanup).

A second finding forced the purge to cover more than A: a plain git add dataset/ (post--f
drop) still re-stages already-tracked dirs that a smoke run regenerates at 15×15 — so
leg 1 alone doesn't stop the leak for tracked dirs; they must be purged. That's why multi/
is included and why the remaining C must be resolved before leg 4 can land.

Checkpoint — the remaining ~12 non-allowlisted dirs (deferred)

I shipped only what smoke can prove. The rest splits three ways — one decision needed:

1. Regenerable, not smoke-covered (simulator verified, but no smoke entry exercises them):
   imaging/simulated_lens, interferometer/simulated_lens, point_source/simulated_lens,
   imaging/simulated_galaxy, interferometer/simulated_galaxy,
   imaging/dark_matter_subhalo_no_lens_light, point_source/start_here_example,
   imaging/ellipse (autogalaxy). → purge (low risk; recoverable via Gut) — I'll spot-run
   one consumer each rather than rely on smoke.
2. Genuinely dead (no simulator, no consumer found): cluster/csv_api_example,
   imaging/misc, imaging/samples, point_source/samples, interferometer/many_visibilities,
   interferometer/simpleold. → archive bytes to a durable archive/condemned/* ref, then
   purge (irreplaceable-by-simulator, so a real Gut archive not just a SHA).
3. Needs a human call: interferometer/datacube (nested-real — the datacube epic's data,
   consumed via dataset_name="sim_simple"; I won't touch it without your ok) and
   database/simple__{0,1,2} (autogalaxy aggregator data — the Group-B committed-by-design
   class; belongs with that prompt, not here).

Leg 4 (allowlist guard as a pre_build assertion) lands once (1)+(2) clear, so it goes green.

Ask: OK to proceed with (1) purge + (2) archive-then-purge in a follow-up commit on these
same branches, leave datacube alone, and route database/* to Group B? Or would you rather
eyeball the dead-set (2) before anything is removed?

[Jammy2211]

Remainder resolved (your "drift" read was right). Re-investigated the deferred set — most were not dead, just regenerable/producer-output my static scan mis-flagged (cross-topic dataset_names, and a samples name-collision with search samples.csv). Expanded the purge accordingly (PRs #272/#129 updated, still smoke-green 9/9 + 8/8):

• Purged additionally: producer-output dirs (simulated_lens/simulated_galaxy, misc, start_here_example, csv_api_example = script scratch-output, dark_matter_subhalo_no_lens_light) + guarded-consumer dirs (datacube guarded×4, ellipse guarded×3). autolens now 28 dirs, autogalaxy 17.
• Killed interferometer/simpleold (stale/old, per your call).
• Kept interferometer/many_visibilities (active, per your call).
• Still deferred (small): the samples population (real drift — needs the auto-simulate guard added at its provisioning point, which name-collides with search samples.csv so it needs a careful read not a blind purge); interferometer/dark_matter_subhalo (verify); database/simple__{0,1,2} → Group B.

Leg 4 (pre_build allowlist assertion) lands once samples+those clear. Everything recorded via PyAutoGut (condemned.md). Merge stays human.

[Jammy2211]

All four legs complete for Group A (autolens + autogalaxy). ✅

• Leg 1 — -f drop (PyAutoBuild#150).
• Leg 2 — moot: the workspaces already self-provision via the .exists() idiom.
• Leg 3 — purged every non-allowlisted regenerable dataset (autolens + los_halos simulated outputs; autogalaxy). samples turned out guarded-provisioned (advanced dataset_label=samples guides) → purged; interferometer/dark_matter_subhalo (no consumer) → purged; simpleold killed. Kept + allowlisted: interferometer/many_visibilities (active) and database/simple__{0,1,2} (committed-by-design aggregator data — the Group-B b2 resolution for these). All via PyAutoGut.
• Leg 4 — check_dataset_allowlist.py wired into pre_build (PyAutoBuild#150). Both workspaces pass (autolens 96 tracked files, autogalaxy 16 — all allowlisted); Group-B repos skip cleanly. It caught a real leak the manual audit missed — a mixed los_halos dir (real .npy + simulated .fits).

Final smoke: autolens 9/9, autogalaxy 8/8. The three PRs (#150, autolens#272, autogalaxy#129) are ready for review/merge (human). Remaining: Group B (autofit/HowToFit/HowToGalaxy/HowToLens) is its own prompt (release_datasets_group_b_policy.md) — the leg-4 guard already skips them until they adopt the allowlist regime. This issue can close once the three PRs merge.