fix(ai): actually enable prompt caching on both provider paths#13
Open
ndemianc wants to merge 1 commit into
Open
fix(ai): actually enable prompt caching on both provider paths#13ndemianc wants to merge 1 commit into
ndemianc wants to merge 1 commit into
Conversation
The agent loop re-sends tools + system + the whole growing transcript every turn; without cache breakpoints each turn re-pays FULL input price for the entire prefix. Symptom: input token usage an order of magnitude above output (e.g. 21M input vs 175k output on a run) and a bill that makes no sense given output is the pricey side. Two latent gaps, both fixed: - providers/anthropic.js: `system` was passed as a plain string, so the advertised cache_control never attached and cache_read_input_tokens stayed 0 forever. Now sends system as an ephemeral-cached block (caches tools+system) and rolls a cache_control breakpoint onto the last message (caches the transcript). 2 of the 4 allowed breakpoints. - providers/openaiCompat.js + translate.js (OpenRouter + the LevelCode Cloud gateway): read only prompt_tokens, so caching was invisible, and sent no breakpoints. Now reads prompt_tokens_details.cached_tokens (splitting it out of prompt_tokens so the context meter total stays exact) for ALL providers, and writes cache_control breakpoints gated to Claude-family model ids via a new isAnthropicFamily() — GPT/Gemini/DeepSeek/etc. auto-cache and would ignore or reject the field. Note: for the metered gateway the client now sends the hint, but the gateway proxy must forward cache_control upstream for the cache to land (server side). Direct-Anthropic (BYOK) and OpenRouter benefit immediately. Adds 4 translate.test.js cases covering the breakpoints + the Claude gate. node --test test/translate.test.js → all pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The symptom
An agent run showed ~21M input tokens vs ~175k output — input an order of magnitude over output, on a metered plan. That is the exact signature of an uncached agent loop: every turn re-sends
tools + system + the entire growing transcriptat full input price, while output stays tiny. Cached input bills at ~0.1×; we were paying 1× on the whole prefix, every turn.Root cause — two latent gaps
providers/anthropic.jspassedsystemas a plain string, so thecache_controlthe header comment advertised never attached.cache_read_input_tokensread 0 forever.providers/openaiCompat.js(serves OpenRouter and the LevelCode Cloud gateway — i.e. the Pro-plan path) read onlyprompt_tokens(blind to caching) and sent no breakpoints.The fix
Anthropic —
system→ an ephemeral-cached block (cachestools+system); a rollingcache_controlbreakpoint on the last message (caches the transcript). 2 of 4 breakpoints.OpenAI-shaped path — two parts:
prompt_tokens_details.cached_tokens, splitting it out ofprompt_tokensso the context-meter total stays exact (mirrors Anthropic's disjoint fields). Surfaces caching for OpenAI/OpenRouter/gateway alike.isAnthropicFamily()— GPT/Gemini/DeepSeek/etc. auto-cache server-side and would ignore or rejectcache_control. This is the guard that keeps the change from breaking every non-Claude request.Honest caveat (server-side follow-up)
The client can only ask for caching. BYO-key Anthropic and OpenRouter benefit immediately. For the metered gateway, the client now sends the hint, but the gateway proxy must forward
cache_controlupstream for the cache to land — a one-line change in the gateway service (thin.ly). Instrumentation ships regardless, socache_read_input_tokensstarts reading real values in the meter today.Tests
node --test test/translate.test.js→ all pass, incl. 4 new cases: system + last-message breakpoints under{cache:true}, the breakpoint landing on atool_resultmessage, legacy shape unchanged without the flag, and theisAnthropicFamilygate.Expected impact: on any multi-turn run,
cache_read_input_tokensgoes 0 → ~transcript size on turn 2, cutting billed input ~60–80%.🤖 Generated with Claude Code