ngmix: config-selectable UberSeg neighbour masking (BLEND_HANDLING)

[cailmdaley]

Closes #776.

What

A config-selectable UberSeg neighbour treatment for the ngmix module, wired end-to-end and off by default:

• BLEND_HANDLING = {noisefill, uberseg} (default noisefill — the historical behaviour, byte-for-byte unchanged, asserted in tests). uberseg hard-masks (weight → 0) every stamp pixel closer to a neighbour's segmentation footprint than to the central object's, leaving the image untouched — Erin Sheldon's MEDS/ngmix treatment.
• The segmentation input, per @aguinot's answer on UberSeg neighbour masking (Erin Sheldon) — config-selectable, validated against noise-fill #776: the tile SExtractor run emits SEGMENTATION (one config line); a CLASSIC vignetmaker run cuts integer, non-interpolated seg stamps in the coadd frame; that single stamp is overlaid unchanged on every per-epoch stamp — no reprojection (UNIONS coadds carry no field rotation vs single exposures, so residual offsets are a few px at most).
• DILATE_NEIGHBOUR (int, default 1): binary dilation of the neighbour footprints to absorb those few-px offsets. Geometrically self-targeting: for well-separated objects the Voronoi bisector already claims the whole neighbour side and dilation adds nothing; it only bites for close blends — exactly the offset-sensitive regime.
• NGMIX_NEIGHBOUR_FLAG propagated to the final catalogue (does the seg stamp contain any non-central object?), enabling downstream systematics splits.

Design notes

• uberseg_weight reimplements meds._uberseg.uberseg_tree (nearest-segment Voronoi partition) with a scipy cKDTree, validated bit-for-bit against Sheldon's reference get_uberseg (only measure-zero boundary ties differ). Reimplemented rather than depended on: esheldon/meds drags the fitsio/esutil stack for a single small algorithm.
• The central object is identified by its SExtractor NUMBER (seg labels are NUMBERs from the same run) — robust to registration offsets, which could otherwise let a neighbour's label sit on the centre pixel and invert the mask. The centre-pixel label is kept as a per-object diagnostic (warns on mismatch; the object is dropped if its footprint is absent).
• Seg catalogue row-alignment with the tile catalogue is asserted at load (fail fast on length or NUMBER mismatch).

Why (fitting, not shearing)

Neighbour handling matters for the model fit: masking strictly along one side of a galaxy makes the weight anisotropic and lets the fit be "pulled" from the unmasked side. UberSeg reduces that pull by construction; 4-fold weight symmetrisation (@aguinot's least-biased endpoint) removes it entirely and is the declared follow-on PR — it composes with, and is orthogonal to, this one. The metacal shearing side (discontinuities, noise-fill of flagged pixels) is untouched here.

Validation & default

Unit tests cover the mask geometry (vs brute force and vs Sheldon), byte-identity of the default, dilation additivity, the fail-fast guards, and the flag reaching the final catalogue (56 ngmix/make_cat tests; full module+unit sweep green). The production default stays noisefill until a real-data A/B (FLAGS=2-inclusive B-modes) says otherwise — that comparison is tracked separately and is not a merge condition for this dormant, default-unchanged capability.

— Claude, on behalf of Cail

[aguinot]

Hello,
I just want to mentioned that I looked again at the DES Y6 config file and they do not symmetrize the uberseg weights. I am not sure I know why.. I am discussion with the people who did the analysis to understand this change.

[aguinot]

Regarding the implementation, I see that you are doing a pure python with KDTree. What is the compute time? Is it trackable?

[cailmdaley]

here's what Claude found, scipy cKDTree uses Cython and the inner loop is C++:

┌───────────────────────────────────┬────────────┬──────────────────┬───────┐
│              n_neigh              │ ours (μs)  │ Sheldon's C (μs) │ ratio │
├───────────────────────────────────┼────────────┼──────────────────┼───────┤
│ 0 (isolated, 45% of real objects) │ 26.6 ± 0.1 │ 39.7 ± 0.1       │ 0.67× │
├───────────────────────────────────┼────────────┼──────────────────┼───────┤
│ 1 (35% of real objects)           │ 990 ± 71   │ 1094 ± 44        │ 0.91× │
├───────────────────────────────────┼────────────┼──────────────────┼───────┤
│ 2 (14% of real objects)           │ 1059 ± 77  │ 1155 ± 69        │ 0.92× │
├───────────────────────────────────┼────────────┼──────────────────┼───────┤
│ 3 (4% of real objects)            │ 1168 ± 88  │ 1206 ± 46        │ 0.97× │
├───────────────────────────────────┼────────────┼──────────────────┼───────┤
│ 4 (rare)                          │ 1214 ± 52  │ 1226 ± 58        │ 0.99× │
└───────────────────────────────────┴────────────┴──────────────────┴───────┘

We compared uberseg_weight()'s cKDTree implementation against your meds._uberseg.uberseg_tree reference at production stamp size (51×51px), averaging timing over 20 independent random neighbour geometries per neighbour count to get a stable estimate, and also summed the cost over every real object (30,435) in one actual UNIONS tile using real catalogue positions. The two are effectively tied — ours runs 5–10% faster in the realistic 0–2-neighbour regime that covers 94% of real objects (isolated objects, the 45% majority, are ~33% faster for us specifically), and the whole real tile's UberSeg pass costs under 20 seconds either way.

[cailmdaley]

thank you for the comment and checking with DES folks, let me know if you have any more comments on this PR!

[aguinot]

Interesting, I was not expecting scipy to be faster, good to know!

[cailmdaley]

Merged into develop with origin/develop merged in first (through #843; all conflict resolutions additive). As validation, a noisefill control run of this exact merged tree on the image simulations reproduces the baseline catalog bit-for-bit — the new code is a verified no-op when BLEND_HANDLING is left at its default. The uberseg A/B characterization on the image sims is in progress under #813. Unit tests (20) and CI pass on the merged head.

— Claude (Fable), on behalf of Cail.