You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Batch creation currently writes in this order: append the new batch's ID to each dependency's BatchDependent.Dependents reverse index → create the batch's own (empty) BatchDependent row → CAS the request to Batched (serialization point vs cancel) → BatchStore.Create last.
Because the reverse-index appends happen before the batch exists, two paths leave dangling IDs in dependencies' Dependents lists — IDs of batches that will never exist (batch IDs are never reused; a retry mints a fresh ID):
the cancel-race abandon: the request-claim CAS loses to the cancel controller and batch creation is abandoned after the index appends (batch.go documents the entry "becomes a dangling BatchDependent — tolerated per the 'downstream should handle stale entries' contract");
a crash between the CAS and BatchStore.Create.
Every reader of BatchDependent must therefore implement tolerance. In practice the speculate dependent fan-out publishes wakes for dangling IDs; those messages fail the batch load, reject to the DLQ, and are skipped there as not-found — noise and retry churn on the mainline path.
Proposal
Reorder batch creation: create the batch first, then update the dependents' reverse index, and only then publish the batch into the pipeline.
Rationale: the failure residue flips from "dangling ID that every reader must tolerate" to "created batch whose reverse-index update failed" — and that batch is fine to lose. It has no meaning to the rest of the system: it is never published for speculation and never satisfies any speculation state, so it never advances and nothing downstream acts on it. Dangling IDs, by contrast, leak tolerance requirements into every consumer of the reverse index.
An additional win: with the request-claim CAS moved ahead of all writes (CAS → Create → own BatchDependent row → reverse-index appends → publish), the common failure source — losing the CAS race to cancel — abandons with zero residue, instead of leaving danglers behind.
Caveats to address as part of this change
Orphan batches must stay invisible to conflict analysis. A created-but-unindexed/unpublished batch sits in Created forever. If the active-batch scan used by the conflict analyzer can see it, later batches will list it as a dependency and wait on its terminal state indefinitely — a queue wedge. The reorder needs one of: (a) dependency-eligibility gated on the batch having completed creation (e.g. only batches that reached the published/scored stage are conflict candidates), or (b) a reaper/reconciler that drives stuck Created orphans to a terminal state so any accidental dependents unblock. Note a narrower version of this window exists today (crash between Create and the pipeline publish); the reorder widens it, so the mitigation is needed regardless and becomes load-bearing.
Subscribe-before-publish ordering must be preserved. The reverse-index append is the batch's wake-up subscription: dependents' terminal fan-outs only wake batches listed in Dependents. The batch must not be published into the pipeline until all its index appends have completed — otherwise it can start waiting on a dependency whose terminal transition already fanned out without it, and that missed wake is unrecoverable. (Fresh reads only cover transitions that happen before the batch's first pipeline pass.) The safe order is strictly: Create → index appends → publish.
Partial index appends. With N dependencies, a crash mid-append leaves some dependencies knowing the batch and others not. Since publish is last, such a batch is never woken into the pipeline and falls under caveat 1's orphan handling; redelivery creates a fresh batch ID that redoes all appends. Confirm the redelivery path (re-CAS Batched → Batched, new batch ID) stays intact.
The missing-BatchDependent-row invariant flips. Today the batch's own BatchDependent row is created before the batch, so speculate's respeculateDependents treats a missing row on a terminal batch as a storage invariant violation (nack). Under the new order a batch can briefly exist without its row. Terminal batches will still always have one (only fully-created batches get published and can reach terminal), but the invariant's justification changes and must be re-documented; any orphan reaper must not assume the row exists.
Interim behavior until this lands. The speculate dependent fan-out blind-publishes a wake for every listed dependent: dangling IDs become speculate messages that fail the batch load and are skipped in the DLQ (noise, not a stall). Terminal dependents are also re-woken on self-heal redeliveries; the resulting re-fan-out cascade is bounded in practice by the MySQL queue's publish dedup on (topic, partition_key, id) — see messagequeue: entity-ID message IDs + publish dedup silently drop wake-up events; use intent-scoped message IDs #352 (intent-scoped message IDs) for why that dedup is itself a caveat. A hardened fan-out (load each dependent; skip stale and already-terminal ones) was prototyped during review of feat(speculate): write the speculation tree in shadow mode #353 and removed in favor of this reorder: once the reorder lands, stale entries cease to exist as a class, and only the terminal-dependent skip remains as an optional cascade bound.
Problem
Batch creation currently writes in this order: append the new batch's ID to each dependency's
BatchDependent.Dependentsreverse index → create the batch's own (empty)BatchDependentrow → CAS the request toBatched(serialization point vs cancel) →BatchStore.Createlast.Because the reverse-index appends happen before the batch exists, two paths leave dangling IDs in dependencies'
Dependentslists — IDs of batches that will never exist (batch IDs are never reused; a retry mints a fresh ID):batch.godocuments the entry "becomes a dangling BatchDependent — tolerated per the 'downstream should handle stale entries' contract");BatchStore.Create.Every reader of
BatchDependentmust therefore implement tolerance. In practice the speculate dependent fan-out publishes wakes for dangling IDs; those messages fail the batch load, reject to the DLQ, and are skipped there as not-found — noise and retry churn on the mainline path.Proposal
Reorder batch creation: create the batch first, then update the dependents' reverse index, and only then publish the batch into the pipeline.
Rationale: the failure residue flips from "dangling ID that every reader must tolerate" to "created batch whose reverse-index update failed" — and that batch is fine to lose. It has no meaning to the rest of the system: it is never published for speculation and never satisfies any speculation state, so it never advances and nothing downstream acts on it. Dangling IDs, by contrast, leak tolerance requirements into every consumer of the reverse index.
An additional win: with the request-claim CAS moved ahead of all writes (CAS → Create → own BatchDependent row → reverse-index appends → publish), the common failure source — losing the CAS race to cancel — abandons with zero residue, instead of leaving danglers behind.
Caveats to address as part of this change
Orphan batches must stay invisible to conflict analysis. A created-but-unindexed/unpublished batch sits in
Createdforever. If the active-batch scan used by the conflict analyzer can see it, later batches will list it as a dependency and wait on its terminal state indefinitely — a queue wedge. The reorder needs one of: (a) dependency-eligibility gated on the batch having completed creation (e.g. only batches that reached the published/scored stage are conflict candidates), or (b) a reaper/reconciler that drives stuckCreatedorphans to a terminal state so any accidental dependents unblock. Note a narrower version of this window exists today (crash betweenCreateand the pipeline publish); the reorder widens it, so the mitigation is needed regardless and becomes load-bearing.Subscribe-before-publish ordering must be preserved. The reverse-index append is the batch's wake-up subscription: dependents' terminal fan-outs only wake batches listed in
Dependents. The batch must not be published into the pipeline until all its index appends have completed — otherwise it can start waiting on a dependency whose terminal transition already fanned out without it, and that missed wake is unrecoverable. (Fresh reads only cover transitions that happen before the batch's first pipeline pass.) The safe order is strictly: Create → index appends → publish.Partial index appends. With N dependencies, a crash mid-append leaves some dependencies knowing the batch and others not. Since publish is last, such a batch is never woken into the pipeline and falls under caveat 1's orphan handling; redelivery creates a fresh batch ID that redoes all appends. Confirm the redelivery path (re-CAS
Batched→Batched, new batch ID) stays intact.The missing-BatchDependent-row invariant flips. Today the batch's own
BatchDependentrow is created before the batch, so speculate'srespeculateDependentstreats a missing row on a terminal batch as a storage invariant violation (nack). Under the new order a batch can briefly exist without its row. Terminal batches will still always have one (only fully-created batches get published and can reach terminal), but the invariant's justification changes and must be re-documented; any orphan reaper must not assume the row exists.Interim behavior until this lands. The speculate dependent fan-out blind-publishes a wake for every listed dependent: dangling IDs become speculate messages that fail the batch load and are skipped in the DLQ (noise, not a stall). Terminal dependents are also re-woken on self-heal redeliveries; the resulting re-fan-out cascade is bounded in practice by the MySQL queue's publish dedup on (topic, partition_key, id) — see messagequeue: entity-ID message IDs + publish dedup silently drop wake-up events; use intent-scoped message IDs #352 (intent-scoped message IDs) for why that dedup is itself a caveat. A hardened fan-out (load each dependent; skip stale and already-terminal ones) was prototyped during review of feat(speculate): write the speculation tree in shadow mode #353 and removed in favor of this reorder: once the reorder lands, stale entries cease to exist as a class, and only the terminal-dependent skip remains as an optional cascade bound.
Related