fix: test-mode-gate singular/non-PD inversion crash (release tail)

[Jammy2211]

Overview

Release re-validation (run 29341418859, wheels 2026.7.14.1.dev65901) reported two chained-pipeline FAILs — interferometer/.../multi_gaussian_expansion/slam.py (numpy.linalg.LinAlgError: Singular matrix) and imaging/features/pixelization/cpu_fast_modeling.py (autoarray.exc.InversionException). Investigation + reproduction (dev venv, jax 0.10.2, matches release) shows this is a flaky test-mode tail, not a deterministic library bug: real inference is already protected by the resample guard in log_likelihood_function; the crash fires only in PYAUTO_TEST_MODE chained runs where an under-converged/fabricated model reaches an unguarded downstream inversion, and only tips into a raised exception under specific numeric conditions. Fix is therefore test-mode-gated in PyAutoArray and must not perturb real-mode numerics.

Plan

• In normal mode, inversion behaviour is byte-for-byte unchanged (still raises → still resamples in the guarded likelihood).
• Only when is_test_mode() (from autoconf), the inversion sites return a benign dummy instead of raising — neutralising the singular case wherever the inversion is invoked (likelihood, preload, result, chaining), which is why per-path guards in PyAutoLens are unnecessary.
• No conditioning floor / jitter: that would shift every real evidence/reconstruction and conflicts with the "no silent guards / fix the producer" rule.
• Validate with a numpy-only unit test; do not use the flaky end-to-end scripts as a gate.

Detailed implementation plan

Affected Repositories

• PyAutoArray (primary)

Branch Survey

Repository	Current Branch	Dirty?
./PyAutoArray	feature/ticks-minus-in-math	clean (unrelated; new worktree off origin/main)

Suggested branch: feature/inversion-testmode-singular-guard

Implementation Steps

1. autoarray/inversion/inversion/inversion_util.py reconstruction_positive_negative_from (~:224) — wrap xp.linalg.solve in try/except np.linalg.LinAlgError; in test mode return xp.ones_like(data_vector), else raise exc.InversionException.
2. inversion_util.py reconstruction_positive_only_from (~:344-353) — the except already raises InversionException; in test mode return xp.ones_like(data_vector) instead.
3. autoarray/inversion/inversion/abstract.py log_det_curvature_reg_matrix_term (~:715-721) — wrap the Cholesky; in test mode return 0.0 on LinAlgError (else the crash moves here once the reconstruction stops raising).
4. abstract.py log_det_regularization_matrix_term (~:740-746) — same.
5. Small shared helper to keep the gate DRY; import is_test_mode from autoconf.

Key Files

• autoarray/inversion/inversion/inversion_util.py — the two reconstruction solvers that raise on singular.
• autoarray/inversion/inversion/abstract.py — the two log-det Cholesky evidence terms that also raise on singular.
• test_autoarray/inversion/inversion/ — numpy-only unit test for the guard.

Testing

• Numpy-only unit test: a deliberately singular curvature_reg_matrix (a) raises normally, (b) under monkeypatched PYAUTO_TEST_MODE returns finite dummies.
• No JAX in unit tests; JAX path covered by existing *_workspace_test parity scripts.

Notes

Follow-up to PyAutoHeart#72 and the G fix PyAutoLens#607. Diagnosis reversed the original prompt's scope (dropped the conditioning floor + PyAutoLens guards). Autonomy: supervised — filed and implemented under --auto; ship sign-off parks for a human.

[Jammy2211]

⏸️ Parked for ship sign-off (--auto, supervised)

Implementation is complete and committed locally on feature/inversion-testmode-singular-guard (commit 48fae1ac), not pushed — the supervised autonomy contract makes ship sign-off (push + PR-open) a human checkpoint.

What was done

The four inversion raise-sites now return a benign dummy only when autoconf.is_test_mode() is true, else re-raise exactly as before:

Site	Normal mode (unchanged)	Test mode
inversion_util.reconstruction_positive_negative_from	raise LinAlgError	return ones_like(data_vector)
inversion_util.reconstruction_positive_only_from	raise InversionException	return ones_like(data_vector)
abstract.log_det_curvature_reg_matrix_term	raise LinAlgError	return 0.0
abstract.log_det_regularization_matrix_term	raise LinAlgError	return 0.0

numpy-only by construction (the JAX linalg path returns NaN rather than raising). No conditioning floor — real-mode numerics are byte-for-byte unchanged.

Diff: abstract.py +40/-… · inversion_util.py +20 · test_inversion_util.py +53 (3 files, +101/-12).

Test evidence

• 2 new numpy-only guard tests (singular matrix → raises normally; under monkeypatched PYAUTO_TEST_MODE → finite dummy).
• test_autoarray/inversion/inversion/test_inversion_util.py + test_abstract.py: 41 passed.
• Full test_autoarray/inversion/: 233 passed.

The decision for you

Sign off to ship? On approval, /ship_library runs the four-leg gate (full PyAutoArray suite, review, Heart) and pushes + opens the pending-release PR (merge stays human).

• Approve → resume /ship_library on this task.
• Change dummy semantics / sites → tell me what to adjust; the commit amends in place.
• Prefer validation-tiering instead (exclude these scripts from the test-mode set rather than a library change) → say so and I'll revert the branch and re-scope to a PyAutoHeart harness change.

Reproduction note: the reported LinAlgError/InversionException did not reproduce deterministically on clean main (TEST_MODE=1 and =2 both pass; the fragility shows as mapper_util divide-by-zero on near-singular samples). This fix removes the flaky crash regardless of which site tips over, without touching real inference.

[Jammy2211]

🔴 Ship blocked — Heart RED (parked, nothing pushed)

Human ship sign-off received, four-leg gate resumed. Tests green (test_autoarray/inversion/ 233 passed), but the Heart readiness gate is RED, which blocks commit/push/PR at every autonomy level. Verbatim RED reasons from pyauto-heart readiness:

1. PyAutoArray: on branch feature/ticks-minus-in-math (not main)
2. release validation FAILED (stage integrate)

The corrective-PR exception for Heart RED is a contemporaneous human act, never reachable under --auto — this is an --auto run, so it does not apply. It would also be a poor fit: reason #1 is unrelated branch state, and reason #2 is the structural perf-flake integrate tail, which this fix does not causally repair (it removes flaky test-mode singular crashes; the integrate RED is cleared by the advisory-tier work + PyAutoArray returning to main).

Parked. The fix is complete, tested, and committed locally (48fae1ac on feature/inversion-testmode-singular-guard) — not pushed. It ships via /ship_library once Heart clears (PyAutoArray back on main + integrate validation passing). No code was modified to make any leg pass.

[Jammy2211]

✅ Corrective-PR exception authorized — PR opened

Human authorized the corrective-PR exception in-session (2026-07-14) for the named RED reason:

release validation FAILED (stage integrate)

PR #389 opened (pending-release) — the test-mode singular-inversion guard (commit 48fae1ac), reason-scoped, no mixed scope.

Causal map: integrate FAILs slam.py + cpu_fast_modeling.py (run 29341418859) → unguarded test-mode singular inversion → 4-site is_test_mode() dummy guard → PR #389. Follow-up to PyAutoLens#607.

Sibling RED remains: PyAutoArray: on branch feature/ticks-minus-in-math (not main) — Heart stays RED after any merge until PyAutoArray's main checkout returns to main and integrate validation re-runs on fresh wheels.

Caveat (recorded): these FAILs did not reproduce deterministically on clean main; the fix is test-mode-gated. Causal confirmation comes from the post-merge re-validation, not before.

Merge is a separate human act — not performed by this action. After a human merge: build fresh wheels → re-run release integration on main → obtain a new Heart verdict before any release decision.

[Jammy2211]

✅ Merged (corrective PR #389) — awaiting corrective validation

PR #389 squash-merged to main by explicit human direction (commit 1ff90287fc7654f864fb4c51cba7ea69686a7b66, 2026-07-14). Issue kept open until corrective validation reruns.

Recovery sequence (required before any release decision):

1. Build fresh post-merge wheels.
2. Re-run release integration on main.
3. Obtain a new Heart verdict over that fresh evidence.

Heart stays RED until the sibling reason is also cleared: PyAutoArray: on branch feature/ticks-minus-in-math (not main) (the main checkout must return to main), plus the perf-flake tail (no_run SLOW PRs alwt#169 / agwt#73). Release remains blocked throughout.

Caveat carried forward: these FAILs did not reproduce deterministically pre-merge; whether the integrate stage actually flips is confirmed only by the re-validation above.

[Jammy2211]

Corrective validation GREEN — closing

The fresh mode=release validation run on the current main HEADs passed end-to-end:

• Stage 2 TestPyPI rehearsal v2026.7.14.1.dev66001 — all 5 wheels built + uploaded.
• Stage 3 wheel-based real-search integration — 542 passed, 0 failed, 0 timeout, 87 skipped.
• Ingested → release_ready: true; Heart is now YELLOW (no RED reasons remain).

This fix is on the validated main HEADs (commit_shas matched), so its corrective validation is satisfied. Closing.