messagequeue: entity-ID message IDs + publish dedup silently drop wake-up events; use intent-scoped message IDs

[behinddwalls]

Problem

The MySQL message queue dedups publishes on the unique key (topic, partition_key, id), and every pipeline controller reuses the entity ID (batch ID, build ID, request ID) as the message ID. Because consumed rows are only deleted by GC (which runs on idle poll ticks), a re-publish for the same entity on the same topic is silently swallowed whenever an earlier row — even an already-consumed one — has not been GC'd yet. The publisher gets a success, no new row is inserted, and the consumer never sees the event.

This turns one-shot wake-up publishes into best-effort delivery, gated by GC timing.

Mechanics

• platform/extension/messagequeue/mysql/message_store.go — InsertDelayed uses INSERT ... ON DUPLICATE KEY UPDATE topic = topic: a repeated publish for the same (topic, partition_key, id) is deliberately treated as an idempotent success ("lets callers safely retry publishes"). That is the right behavior for retries of the same event, but it also fires for new events about the same entity.
• Consumption is offset-based; acked rows stay in the table until GC.
• platform/extension/messagequeue/mysql/subscriber.go — GC only runs every Nth idle poll tick. A busy consumer defers GC, which widens the swallow window exactly when traffic is high.

Where it bites

Every controller publishes with entityqueue.NewMessage(<entityID>, ...):

• score → speculate (msg ID = batch ID)
• buildsignal → speculate (msg ID = batch ID) and its own poll self-reschedule (msg ID = build ID)
• mergesignal → conclude + speculate fan-out (msg ID = batch ID)
• cancel → speculate nudge (msg ID = batch ID)
• speculate → dependent wake-up fan-out (msg ID = dependent batch ID), build/merge/conclude publishes (msg ID = batch ID)

Concrete loss scenario: batch B's speculate message from the score stage is consumed; the row awaits GC. Dependency A then merges and the terminal fan-out publishes "speculate B" to wake it — same key (speculate, queue, B) → silent no-op → B is never woken and stalls until some unrelated event pings it. Poll loops (buildsignal) survive because they re-publish every few seconds until terminal, but one-shot nudges (mergesignal fan-out, cancel nudge, dependent wake-up, DLQ re-trigger tooling) have no retry.

We have hit this before operationally: re-trigger publishes had to mint distinct message IDs to land at all.

Proposal: intent-scoped message IDs

The message ID should identify the intent to process an event occurrence, not the entity it is about. The entity reference already travels in the payload; consumers correlate via payload, not message ID, so this is a publisher-side convention change.

Sketch:

• Wake B because A reached a terminal state → batch/B/wake/batch/A
• Advance B after its build finished → batch/B/buildsignal/build/X
• Poll tick N for build X → build/X/poll/N
• Stage handoff (score → speculate) → batch/B/scored

Properties: a retry of the same intent still dedups (idempotent publish is preserved — same cause, same ID), while a new intent about the same entity can never collide with an older row. IDs stay deterministic (derived from cause), so redelivery/crash-retry paths remain idempotent without random UUIDs.

Suggested scope

1. Define the ID convention in platform/base/messagequeue docs (and decide whether NewMessage should take an explicit intent/cause argument to make the convention hard to miss).
2. Audit every NewMessage call site across submitqueue/, runway/, and tooling; migrate to intent-scoped IDs.
3. Check consumers/DLQ/log correlation for any place that assumes message ID == entity ID.
4. Consider a lint or contract test that rejects bare entity IDs as message IDs on pipeline topics.

Notes

• Pre-existing and systemic; not introduced by the speculation-tree work — that review just made it visible (the new dependent wake-up fan-out inherits the same window as mergesignal/cancel).
• Orthogonal mitigation (GC cadence / delete-on-ack) would shrink the window but not close it; the ID convention closes it.

[behinddwalls]

Linear: CODEM-303