feat(cli): chemistry LIMS ingestion with Google Drive sync (BDMS-1034)#775
Open
jirhiker wants to merge 4 commits into
Open
feat(cli): chemistry LIMS ingestion with Google Drive sync (BDMS-1034)#775jirhiker wants to merge 4 commits into
jirhiker wants to merge 4 commits into
Conversation
Add `oco water-chemistry` commands to ingest lab LIMS .xlsx workbooks into the legacy NMA chemistry tables (NMA_MajorChemistry / NMA_MinorTraceChemistry), porting the analyte mapping, non-detect handling, and EPA-200.7 / low-bromide dedup from the AMPAPI chemfile.py script. - bulk-upload: parse a LIMS .xlsx and load a single file - sync-drive: on-demand, engineer-triggered ingest of new/changed files from a Google Drive folder, with an ingested-file manifest stored in GCS - append semantics: each distinct lab sample (WCLab_ID) becomes a new sample point using the next letter incrementor on the base PointID (MG-030A, MG-030B, ... Z, AA); a lab sample already recorded for the well is skipped, so re-runs are idempotent - add a `cli` dependency group (openpyxl, google-api-python-client); CI syncs it explicitly, the production requirements export excludes it - add docs/chemistry-ingestion-runbook.md Refs BDMS-1034 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Materialized-view refresh is handled by the nightly pg_cron job, so the production/staging/testing deploy workflows no longer need to run `python -m cli.cli refresh-materialized-views`. Refs BDMS-1034 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- CLI serializes its own JSON from the result payload; drop the pretty_json flag threaded through the service and the pre-baked `stdout` field on ChemistryUploadResult. - Remove the unused `--output json` option from `water-chemistry bulk-upload` and `sync-drive`; they are engineer-facing and have no scripted consumer (programmatic callers use the service functions directly). - Replace the FMapper/AnalyteField classes with a frozen `AnalyteMapping` dataclass, a flat `_ANALYTE_MAPPINGS` list, and a `lookup_analyte()` helper backed by a case-insensitive dict; drop the dead list/tuple and reverse-lookup branches. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reflect the renamed analyte map (`lookup_analyte` / `_ANALYTE_MAPPINGS`, was `FMapper`) and the removal of the `--output json` option. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ksmuczynski
reviewed
Jul 16, 2026
Comment on lines
+164
to
+167
| - **Duplicate detection is WCLab_ID-only.** A re-ingest is recognized by the lab | ||
| `WCLab_ID` (SampleNumber). A genuinely new lab sample with a reused SampleNumber | ||
| would be treated as a duplicate and skipped; a re-run of the same sample under a | ||
| new SampleNumber would append a spurious extra lettered sample. |
Contributor
There was a problem hiding this comment.
How likely is this to happen? Should dup detection be across multiple fields, e.g. SampleNumber + AnalysisTime + Param? (field names are from an example lab .xlsx from Sianin)
Contributor
There was a problem hiding this comment.
Nevermind, I see this is already suggested as an improvement in the fuller solution down below.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an
oco water-chemistryCLI to ingest lab LIMS.xlsxworkbooks into the legacy NMA chemistry tables (NMA_MajorChemistry/NMA_MinorTraceChemistry), as an interim workaround while the fuller solution is pursued. Analyte mapping, non-detect handling, and EPA-200.7 / low-bromide dedup are ported from AMPAPIchemfile.py.Interim process: Sianin drops LIMS
.xlsxfiles in a shared Google Drive folder; an engineer (Kelsey) runs the ingest on demand; results are recorded in a GCS manifest. Seedocs/chemistry-ingestion-runbook.md.Commands
oco water-chemistry bulk-upload --file batch.xlsx— load a single workbook.oco water-chemistry sync-drive [--dry-run]— on-demand, engineer-triggered ingest of new/changed files fromCHEMISTRY_DRIVE_FOLDER_ID. No polling/scheduling. An ingested-file manifest (keyed by Drive file id + md5) lives in GCS so re-runs skip unchanged files.Append semantics (BDMS-1034 core)
SamplePointIDin the LIMS file is the base well PointID.WCLab_ID/ SampleNumber) becomes a newNMA_Chemistry_SampleInfoatbase + next letter incrementor—A, B, … Z, AA, …(bijective base-26, matching the ~11k-row migration data whereAis the first sample).WCLab_IDalready recorded for the well is skipped (idempotent).Also
clidependency group (openpyxl,google-api-python-client): CI syncs it with--group cli; the production requirements export (uv export --no-dev) excludes it, so these deps stay out of the API runtime.Testing
ENABLE_PG_CRON=0 uv run pytest tests/test_chemistry_lims.py tests/test_chemistry_drive.py— 23 passed (analyte mapping, ND handling, dedup, suffix base-26 roundtrip, append/skip, Drive sync + manifest with fakes).Notes / open
WCLab_ID-only (documented as a known gap).ENABLE_PG_CRON, so they get no view refresh now — parity change deferred, not in this PR.🤖 Generated with Claude Code
Refs BDMS-1034