Add Ocotillo OGC API source (summary-only) alongside AMP#112
Open
jirhiker wants to merge 2 commits into
Open
Conversation
Adds a new `ocotillo` connector backed by the pygeoapi/PostGIS OGC API - Features service at ocotillo-api.newmexicowaterdata.org/ogcapi, intended as the eventual replacement for the NMBGMR AMP API datastream. The connector runs alongside the existing nmbgmr_amp source (not a replacement yet) so outputs can be compared before any cutover. Enable/ disable with --no-ocotillo. Scope: water levels, chemistry analytes, and TDS. Sites are seeded from the water_wells collection; water-level summaries from water_well_summary; chemistry from major/minor_chemistry_results; TDS from latest_tds_wells. Wiring: SOURCE_DICT, ANALYTE_SOURCE_PAIRS, WATERLEVEL_SOURCE_PAIRS, PARAMETER_SOURCE_MAP, use_source_ocotillo flag, CLI --no-ocotillo, and the Dagster orchestration source list. IMPORTANT - NOT yet at feature parity with AMP. The Ocotillo API currently publishes only pre-aggregated "latest"/"summary" collections and exposes no raw observations, so this connector is SUMMARY-ONLY and several summary columns are necessarily blank. Backend (Ocotillo pygeoapi/PostGIS) changes required to reach AMP parity: 1. Raw observation collections. Add per-observation, filterable-by-datetime collections for depth-to-water and for chemistry results (all samples over time, not just latest). Without these, DIG timeseries output (timeseries_unified / timeseries_separated) yields nothing for Ocotillo and summary mean/earliest cannot be computed. This is the single biggest gap. 2. Chemistry history. major_chemistry_results / minor_chemistry_wells expose only the latest value per analyte as static columns. Feature parity needs full analyte time series so DIG can compute count/min/max/mean/earliest, matching AMP's waterchemistry endpoint. 3. Site metadata on water_wells. water_wells lacks elevation, horizontal datum, vertical datum, aquifer/formation, and USGS site id. AMP's locations endpoint provides these. Currently elevation is null and datum defaults to WGS84. Either enrich water_wells or expose a locations-equivalent with full metadata. 4. Nitrate reporting basis. Confirm whether the `nitrate` column is reported as N or as NO3 and expose units accordingly. AMP reports "Nitrate (as N)" and DIG applies a 4.427 conversion; Ocotillo values are currently taken as-is. Parity requires a documented, unit-tagged basis. 5. TDS count/summary. latest_tds_wells is latest-only; avg_tds_wells is an average. Parity needs full TDS history (or explicit count/min/max/earliest fields) rather than a single latest snapshot. Until (1)-(3) land, requesting timeseries output for Ocotillo is a logged no-op and summary output omits mean/earliest (and, for chemistry/TDS, min/max/nrecords) by design. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…pi-source # Conflicts: # orchestration/resources/die_config.py
|
Your pull request is automatically being deployed to Dagster Cloud.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new
ocotilloconnector backed by the pygeoapi/PostGIS OGC API - Features service atocotillo-api.newmexicowaterdata.org/ogcapi, intended as the eventual replacement for the NMBGMR AMP API datastream (nmbgmr_amp).Runs alongside AMP (not a replacement yet) so outputs can be compared before cutover. Toggle with
--no-ocotillo/--no-nmbgmr-amp.Scope: water levels, chemistry analytes, TDS.
water_wellswater_well_summarymajor_chemistry_results/minor_chemistry_wellslatest_tds_wellsNew files
backend/connectors/ocotillo/mappings.py— collection ids + DIG-param → (collection, column) mapbackend/connectors/ocotillo/transformer.py— site + summary transformers (buildSummaryRecorddirectly from pre-aggregated features)backend/connectors/ocotillo/source.py— site source + water-level/analyte summary sourcestests/test_sources/test_ocotillo.py— 4 live tests (health, WL summary, analyte summary, timeseries no-op), all passingWiring
config.py(SOURCE_DICT,ANALYTE_SOURCE_PAIRS,WATERLEVEL_SOURCE_PAIRS,PARAMETER_SOURCE_MAP,use_source_ocotillo),cli.py(--no-ocotilloonweave+sites),orchestration/resources/die_config.py.The Ocotillo API currently publishes only pre-aggregated latest/summary collections and exposes no raw observations, so this connector is summary-only:
timeseries_unified/timeseries_separated) is a logged no-op.mean,earliest_*(all params); plusmin/max/nrecordsfor chemistry + TDS.elevationblank; datum defaults to WGS84.Backend changes required for AMP parity
Not included in this PR — a working doc lives at
OCOTILLO_AMP_PARITY.md(untracked, not committed). Summary:depth_to_water_observations,chemistry_observations), filterable by well + datetime — biggest gap; unblocks timeseries + mean/earliest.water_wells— elevation, horizontal/vertical datum, USGS id.Verification
die weave waterlevels --output-type summary --bbox ...→summary.csvwithNMBGMR-Ocotillorows; water levels get real count/min/max/latest (e.g. NM-28369: 11 records, 133.17–134.51 ft). Chemistry + TDS confirmed live (calcium, arsenic, ph, specific_conductance, tds). 4/4 tests pass.🤖 Generated with Claude Code