Fix OSE POD pagination truncating to first 2000 wells

[jirhiker]

Problem

nm_pod_age_by_county / nm_pod_age_points computed near-empty (33 counties, total_recent_pods max = 1; points collection had only 3 features), despite ~15,600 PODs in the source with a finish_dat in the last 10 years.

Root cause

The OSE OSE_Points_of_Diversion FeatureServer caps a single page at maxRecordCount = 2000, but NMOSEPODSiteSource.chunk_size was 5000. The pagination loop stops when a page returns fewer rows than chunk_size:

if len(rs) < self.chunk_size:  # 2000 < 5000 -> breaks after page 1
    break

So the first full 2000-row page read as "short" and the loop stopped after one request. Only the first 2000 PODs were ever fetched statewide — and because the server returns them in OBJECTID order (oldest first), every recent well sat past row 2000 and was dropped. The POD-age products saw ~3 recent wells total.

This silently under-counted every product fed by this source, not just POD-age: well correlation and county/basin well-density were all capped at the 2000 oldest PODs.

Fix

Set chunk_size = 2000 (≤ server maxRecordCount) so full pages read as full and the loop pages to the end.

Verification

• Confirmed maxRecordCount = 2000, supportsPagination = true on the layer.
• Confirmed the source has 15,646 PODs with finish_dat >= 2017-01-01.
• Ran the connector live before/after on a bounded bbox: 2,000 records -> 11,523 records, with completion years 2017–2026 now populated (dozens–100+ per year) where they were previously empty.
• Cross-checked field mapping and filters against the OSE WATERS PODs data dictionary (v8): finish_dat = FINISH_DATE "Date well completed" (correct binning field); pod_status='ACT' (Active) and excluded basins SP/SD/LWD (Surface Permit / Surface Declaration / Livestock Watering Declaration — not drilled wells) are semantically correct.

Follow-up (not in this PR)

Re-run the affected assets to backfill: nm_pod_age_by_county, nm_pod_age_points, well correlation, well density.

🤖 Generated with Claude Code

[github-actions]

Your pull request is automatically being deployed to Dagster Cloud.

Location	Status	Link	Updated
die-orchestration		View in Cloud	Jul 09, 2026 at 05:03 AM (UTC)