Right company, right quarter, wrong question

Partway through the first replay, a question about Cinder's net margin for Q2 2025 came back with Cinder's operating margin for Q2 2025. Right company. Right quarter. Right filing version, the same two income-statement chunks, the same tool payload, the same model. Every check the cache runs had passed, and the check the paper uses to say whether a served answer is safe had passed too. I had written down, two days earlier, that I expected exactly this to happen. I still sat and looked at it for a while.

The cache in question is FinCacheServe, from a July paper by Lingteng Zeng and Yifan Jin. The setting is an assistant that answers questions about company filings by fetching evidence from the documents and then calling a model, and the idea is that most of those calls are repeats. If you can recognise a repeat, you skip the model. The worry with that idea has always been staleness: a filing gets amended and the saved answer is quietly out of date. Zeng and Jin's answer to the worry is a receipt stapled to every cached answer, listing the document versions, evidence hashes and tool output it depended on, and a rule that the answer may be reused only while every item on the receipt still matches. They report 53.31% of model calls skipped on a 32B model, with zero stale outputs.

I rebuilt the receipt and the rule, replayed 12,000 synthetic requests with 600 filing amendments through them, and measured the one thing the paper does not: whether the reused answer matched what a fresh generation would have said. On staleness, the paper is right. Nothing my copy served was out of date, not once. On the other thing, about one request in twenty got a fresh, cited, current answer to a sibling question, and the paper's own safety number scored every one of those as fine.

That gap is what the rest of this is about, along with two results I did not see coming: the baseline I had felt sorry for turned out to be worse than it looked, and fixing my own retriever made the wrong-answer rate go up rather than down. What I am not doing is reproducing the paper's SEC workloads, its Qwen2.5 and vLLM timings, its eviction policy or its energy estimates, and I am saying nothing about deliberate cache poisoning. There is no language model in my prototype. It is a replay of a request trace through the gate, and a count of what came back.

If you only want the decision: keep the paper's gate, because the freshness half of it works; add the specific question being asked to the cache key, because the paper's key groups siblings; and before any analyst sees a reused answer, regenerate a sample of cache hits and count how many differ. In my replay that one extra key field took wrong answers from 4.82% of requests to none, and served more correct answers than the original did, not fewer.

What Zeng and Jin built

Picture a finance team's filings assistant on a weekday morning. Dashboards and analysts ask the same few dozen questions about the same companies: revenue change, margins, liquidity, debt, what changed in the risk factors. Each request retrieves chunks of a 10-Q, the quarterly report, maybe runs a small calculation tool, and calls a model to write the answer. By ten o'clock most of those calls are repeating yesterday's work.

The paper's position is that the repeated work can go, but only under a contract, because four different things can quietly make a saved answer wrong. A filing amendment restates cost of revenue. An extraction job re-chunks a table. A calculation tool changes version. A model upgrade changes phrasing. None of those make the saved answer look wrong. It still reads fluently and still carries a citation.

Two terms carry everything that follows. An answer cache is a store of whole generated answers, keyed so a later request can be served from it without calling the model; a hit skips both reading the prompt and writing the reply. Dependency-stale is the paper's safety measure (Section 3): a served answer is dependency-stale when a source-document version, evidence fingerprint or tool fingerprint recorded with it conflicts with the current metadata.

Here is how one request moves through it.

First it becomes a signature. Seven fields: company, reporting period, query family, document scope, tool requirement, model identity and decoding configuration (Section 4.1). The families the paper names are deliberately coarse, liquidity, revenue change, debt, margins, risk-factor change, cash-flow movement, filing change, ratio analysis, so that any paraphrase of "gross margin for Acme in Q2" lands in one bucket regardless of wording. Coarse is how FinCacheServe finds the reuse that plain similarity search misses.

Every answer gets a receipt. Each cache entry stores the answer text, cited document and chunk identifiers, chunk hashes, document versions, an evidence fingerprint (an order-stable digest of the cited chunks and their versions), a tool fingerprint (a hash of tool name, version, normalized inputs and returned payload), model identity and decoding settings (Section 4.2). A reverse index maps each source document to the answers that depend on it.

Reuse passes four gates, and updates invalidate. A request finds candidates in its signature bucket, then passes signature, evidence, version, and tool-and-model gates in that order (Section 4.3, Equations 2 to 6). Any mismatch sends it to retrieval, tools, generation and promotion into the cache. When a filing changes, the version store updates first and the reverse index invalidates every dependent answer before later reads see the new version (Section 4.4).

Read the gates as questions. The first asks whether this is the same kind of question about the same company and quarter. The other three ask whether the inputs have changed since the answer was written. There is no gate that asks whether the cached answer was written for this question, and the staleness metric on the right of the figure reads the same records the gates just compared, so it cannot ask either.

Here is the gate as it stands in my copy:

def lookup(self, ctx, corpus, index):
    if ctx.sig_key is None:
        return None
    cands = []
    for e in self.buckets.get(self._key(ctx.sig_key, ctx.hint), []):
        if self.fresh and not (
            e.evidence_fp == ctx.evidence_fp
            and _versions_current(corpus, e)
            and e.tool_fp == ctx.tool_fp
            and e.model == MODEL_ID and e.decoding == DECODING
        ):
            continue
        sim = index.cosine(ctx.qvec, e.qvec)
        if self.tau is not None and sim < self.tau:
            continue
        cands.append((sim, e))
    return max(cands, key=lambda x: x[0])[1] if cands else None

Three tests pin it down. test_amendment_makes_entry_dependency_stale_and_gate_rejects_it restates cost of revenue in one filing and checks that the entry turns dependency-stale and the gate refuses it. test_prior_quarter_amendment_invalidates_quarter_over_quarter_answer does the same through the prior quarter's filing. And test_sibling_question_can_be_served_while_dependency_fresh checks the case this article is about: a hit whose answer differs from a fresh generation while the staleness check still reads false.

Two things in that excerpt are mine. The paper allows a semantic match inside a signature bucket (Section 5) without giving a threshold, so my default picks the most similar entry with no floor and tau is an ablation. And intent_slot=True appends the normalized metric name to the signature, an arm I added after the first run and will come back to.

Fifty-three percent skipped, and one sentence in Section 9

The comparison is set up well. Every safe arm runs on the same requests under the same staleness criterion, and the unsafe arms are there as negative controls, not strawmen. I believe the direction of the 14.34-point gap over versioned semantic caching. What I cannot tell from Table 5 is what share of those skips were correct, because correctness is not what it counts.

The ablations are the strongest part of the paper. Take out the signature layer and safe reuse drops from 53.31% to 10.48%; take out freshness invalidation and the skip rate holds while 126 stale outputs appear (Table 5). Table 9 removes one field at a time and shows each removal exposing a specific unsafe case: drop the period key and a wrong-period probe walks through. On a 2,376-probe near-collision suite, versioned semantic caching accepts 87.18% of the unsafe probes and FinCacheServe accepts none, while accepting every benign paraphrase (Table 10). So the gain comes from the combination the paper claims, and the gate blocks every boundary case the authors wrote down.

The paper knows where its own edge is, and I want to give it credit for saying so. Section 9 states that the consistency claim is dependency freshness with respect to recorded versions and fingerprints, and that factual financial correctness "remains a separate application-quality property." One sentence, and it is carrying every zero in Tables 3, 5, 10 and 11. They are zeros on freshness.

Two more things I would want before trusting the headline elsewhere. No code and no data ship with the paper; a reproducibility package may follow with a future venue or archival release. And the human audit of the normalizer (Table 12) found 384 of 487 analyzable rows signature-safe, 78.9%, with period errors on 27 rows, document-scope errors on 42 and tool-requirement errors on 64. The paper says rows the audit marked unsafe become conservative generation routes; I could not tell from the text whether the running normalizer knows which rows those are.

Five caching papers, one I could rebuild without code

I had five serious candidates on the desk and screened them on claim support, baseline strength, causal evidence, production transfer and testability. Anything first posted before 26 March 2026 was out, which removed Krites, the asynchronous verified caching paper from 13 February 2026, and the INFOCOM semantic caching paper by Liu and colleagues from August 2025. I read FinCacheServe in full and GroundedCache in depth. The other three I screened on abstracts and method summaries, which is a thinner reading, and you should weigh the table accordingly.

FinCacheServe won because it is the only one of the five that claims both more reuse and a precise safety contract, and because that contract's own definition made the claim testable from outside. If your threat is an adversary and not an amendment, LaCache is the one to rebuild instead.

What I wrote down before the first run

Those thresholds went into the repository as PREREGISTRATION.md before anything ran, next to five predictions. FinCacheServe would beat versioned semantic on skip rate by something near the paper's 14 points. Dependency-stale would be zero. Sibling questions sharing a table would collide inside a signature bucket, giving wrong answers on 2% to 5% of requests while the staleness count stayed at zero. Per-metric tools would remove the collisions. And the grounded-style check, which wants the cached answer's words to appear in fresh evidence, would block most computed ratios, because a ratio is not printed verbatim in a filing. The decision rule applies to FinCacheServe with a statement-level tool, since the paper does not specify its tool and a statement extractor is the cheaper one to build. Of the five predictions, two held cleanly, two held in part, and one was wrong. Per-metric tools removed every collision except the risk pairs, which have no tool. The grounded-style check reused computed ratios on 10% to 31% of their requests against 49% to 70% for values printed verbatim in a filing, so it held them back without blocking them; my token-overlap version is crude, and a single digit from a ratio often matches some other number in the chunk. The wrong one was the first.

Everything in the replay is synthetic and seeded. Eight fictional companies, five quarterly filings each, six text chunks per filing rendered from structured facts: two income-statement chunks, two balance-sheet chunks, cash flow and risk factors. Sixteen analyst questions across the paper's families, four paraphrases each, written before the normalizer so the normalizer could not be tuned to them. 2,400 requests per seed with Zipf-popular company, quarter and question keys, one filing amendment every 20 requests, five seeds: 12,000 requests and 600 amendments.

The model is an oracle. It reads the question correctly and answers from the current facts, so every wrong served answer is the cache's doing and nothing else's. I also gave the cache a perfect record of every filing each answer depended on, so a hidden dependency cannot be the explanation either. The normalizer gets family, company and quarter right on all 64 paraphrases. Apple M4 Pro, Python 3.14.7, standard library only.

git clone https://github.com/shravan1996/ship-the-paper-answer-cache
cd ship-the-paper-answer-cache
python3 -m unittest discover -s tests
python3 eval.py

The full implementation and reproducibility notes are in the prototype repository.

Zero stale, 578 wrong

The pre-registered rule fired Stop, on two counts. Wrong answers at 4.82% crossed the 2% line, which is the failure I had predicted. The skip gain was negative, which I had not predicted, and the reason turned out to be a mistake of mine, not the paper's. I had assumed raw skip rate was the right thing to count. It counts a wrong hit as a saving, which is the same blind spot I was about to criticise in the paper. Count correct skips instead and FinCacheServe beats versioned semantic by 20.46 points, a bigger margin than the paper reports.

The baseline I had felt sorry for

Look back at the versioned semantic row. Every arm in that table scored zero on dependency-stale, including one that served another company's numbers on 35.53% of requests.

I had cast versioned semantic caching as the weak but safe option, the one that leaves savings on the table, and that row surprised me more than anything in FinCacheServe's. 49.06% of its hits were another company's or another quarter's answer. Once you see the reason it is mundane, though I am not sure I would have seen it without the test failing first. In a question like "How much of each revenue dollar did Acme keep after cost of revenue in Q2 2025?", the company name is one token among many, and swapping it for Dunmore still scores 0.94 cosine. I had assumed the short template would collide too, and wrote my first test that way. It failed: "What was Acme's gross margin in Q2 2025?" against Dunmore scores 0.765, under the threshold. The longer the question, the less the company weighs. Quarters are worse. "Q2" and "Q3" are each one short token, and swapping one scores 0.856 even on the short template.

The paper's staleness metric cannot see this failure any more than it can see mine. The version check passes because Acme's filing did not change. The paper's near-collision suite does catch it, with the 87.18% unsafe-accept rate in Table 10, which is to the paper's credit; it is simply a different measurement from the headline zero.

I fixed the retriever and the number went up

After the first run I found a problem with my own setup. "What was Cinder's net margin in Q2 2025?" was retrieving the cash-flow chunks, from two different quarters, because the word "net" matched "Net cash provided." Recall of the chunks that actually hold each answer's facts was 0.602. A fair reader would say the collisions were junk evidence matching junk evidence, and I thought so too.

So I reran with two better retrievers, outside the pre-registered decision. Filtering to the named quarter's filing and taking three chunks raised recall to 0.839. Perfect retrieval, handing the gate exactly the chunks that hold the facts, is 1.000.

I expected the wrong-served rate to fall. It rose, and I did not expect that at all. Good retrieval sends operating margin and net margin to exactly the same two income-statement chunks, so their evidence fingerprints match by design; poor retrieval had been hiding collisions behind noisy, mismatched evidence. The evidence gate only separates two questions when their answers happen to live in different places. Gross margin, whose numbers sit in a different chunk from operating and net margin, collided 8 times in the first run. Net and operating margin collided 75.

One more thing I did not expect. The question-in-key arm has a narrower bucket than FinCacheServe, so I expected fewer skips from it. It had fewer raw skips and more correct ones: 58.66% against 57.35%. When a sibling's answer is served, the model never runs, so the right answer is never generated and never cached, and the next request for that question finds the sibling again. A wrong hit stays until an amendment flushes it.

Where this bites in a real finance team

The pairs that share a table

The paper assumes the signature separates questions that need different answers. A real question catalog is full of pairs that share a company, quarter, family and source table: current and quick ratio, operating and free cash flow, revenue and revenue change. The symptom is invisible on the paper's metric and on a latency dashboard; a human reviewer sees a plausible, cited number for the wrong line item. The control is to put the specific metric being asked for into the signature, and to send any request whose metric the normalizer cannot name to generation.

Risk factors have no tool to fingerprint

Tool-backed questions got an accidental second line of defense from the tool fingerprint in my per-metric arm. Text questions did not. Risk factors and risk-factor changes collided 118 times even with per-metric tools, because there was no tool payload to differ. In a real assistant those are the narrative questions, what changed in the MD&A, what management said about liquidity. I would keep reuse off for families without a tool until the shadow check below has measured them.

A normalizer that is right four times in five

Mine was perfect on its 64 paraphrases, which is the kindest possible assumption. The paper's own audit found 78.9% of rows signature-safe. A period error in the normalizer can turn a request for Q3 into a hit on Q2's answer, which is the versioned-semantic failure coming back through a different door. Watch for any family whose shadow wrong-served rate climbs after new question templates ship, fall that family back to generation, and re-audit.

Invalidation during filing season

The reverse index is cheap in memory. On the paper's SQLite write-ahead-log backend at 100k entries, invalidation p95 is 63.44 ms at a fan-out of 16 dependent answers per update and about 165 ms at 128 (Figure 11). Filing season probably means bursts of amendments against popular companies with many dependent answers, and Section 9 says multi-replica invalidation and cross-region consistency need more engineering. Serialize version updates per company, and measure time from filing update to invalidation as a service-level objective of its own.

Why evalability sets the verdict, and cost does not

The average lands on Wait, and I would still wait if every other row went to 5, because of the evalability row. A cache whose published safety number stays at zero while it serves the wrong line item cannot be operated; the number you would page on never moves.

Keep the gate, regenerate a sample

The synchronous path is the paper's with one change: the normalizer emits the specific metric as an eighth signature field, and a request whose metric it cannot name never reuses. Everything else about the gate stays as published, because the freshness half did exactly what it said.

The new part sits off the request path. A sampler takes a fixed share of hits, regenerates them, and compares the regenerated answer with what was served. For numeric answers compare the extracted numbers and not the text, because a real model will reword. Out of that comes a wrong-served rate per question family, with an interval, and it is the number the kill switch reads. I expect the sampler, not the gate, to be where most of the engineering goes: comparing a regenerated ratio with a served one is a string match, and comparing two paragraphs about risk factors is its own evaluation problem.

1. 01

   Pick one question catalog and name every metric in it

   Owner

   Applied science + finance analyst

   Artifact

   A list of the questions the assistant actually receives, each tagged with family and metric, including the siblings that share a table

   Signal

   Every family with two or more metrics is flagged before a cache exists

2. 02

   Build the paper's gate with the metric in the key

   Owner

   Platform

   Artifact

   Signature, receipts, version store, reverse index, plus the metric field and a no-reuse rule when the metric is unknown

   Signal

   Replay of last month's logs shows zero dependency-stale outputs and zero cross-metric hits

3. 03

   Audit the normalizer on real traffic

   Owner

   Applied science

   Artifact

   500 labeled requests with company, period, scope, family, and metric, audited the way the paper's Table 12 was

   Signal

   At least 95% of rows signature-safe, and every unsafe row routed to generation at runtime

4. 04

   Run the cache in shadow, serving nothing

   Owner

   Platform

   Artifact

   A log of would-have-hit decisions next to the fresh answers users actually received

   Signal

   Wrong-served rate per family, with a 95% interval, from comparing the would-have-served answer to the fresh one

5. 05

   Serve hits for one family with sampled regeneration

   Owner

   Platform + risk

   Artifact

   Reuse enabled for one tool-backed family, 5% of hits regenerated off the request path

   Signal

   Upper bound of the wrong-served interval stays below 0.5% for two weeks

6. 06

   Widen, hold, or stop

   Owner

   Risk + product

   Artifact

   A decision record per family with skip rate, wrong-served rate, and invalidation latency

   Signal

   Families widen only on their own evidence; a family without a tool needs its own two clean weeks

Before a family goes live, three things have to be true. A replay of recorded traffic shows zero dependency-stale outputs and zero hits across different metrics. The shadow wrong-served rate's upper 95% bound sits below 0.5% for the family being enabled. And the time from a filing update to the invalidation of every dependent answer is under 1 second at the 99th percentile.

Three things send it back. A sampled wrong-served rate above 1% for any family on a rolling day returns that family to generation. Any hit served for a company, period or metric different from the request's normalized signature is a stop. So is a normalizer audit on new traffic dropping below 90% signature-safe.

A finance analyst reviews every sampled mismatch in the first two weeks of each family, and reviews the question catalog whenever a new metric or template ships. Sampled regeneration can narrow that review to mismatches only. It cannot replace the review of a new family.

So should you build it?

Yes, but not as published and not in front of users yet. Wait for user-facing reuse; build the gate with the metric in the key and run it in shadow on one question family.

The paper earned the test. Its reuse contract is specified well enough to rebuild without code, and on 600 amendments its freshness half served zero stale answers. My prototype fired the pre-registered Stop anyway, because the same contract served a fresh answer to a sibling question on 4.82% of requests, and 6.88% with perfect retrieval, while the paper's metric read zero on both. The paper and the prototype do not disagree about staleness. They measure different things, and where they diverge I let mine drive the decision, because a wrong number delivered with a valid receipt is the failure a finance team cannot see.

I would not call the fix validated either. One extra field in the signature removed every wrong answer in every retrieval setting I ran, and kept 58.66% correct skips, more than the original. But my normalizer was perfect by construction, and the paper's was right on 78.9% of rows. The sampler is what tells you which of those two worlds you are in.

I would change this verdict if:

1. The authors' reproducibility package shows per-metric tools and a question-level signature, and reports a wrong-answer rate measured by regeneration on the operator suite.
2. Shadow regeneration on your own traffic keeps the wrong-served upper bound below 0.5% for two weeks in a family, with the normalizer audited at 95% or better.
3. A replay on a real question catalog shows no family where two metrics share a signature, a table, and a tool payload.

What I would carry to the next cache

If you have an answer cache running over documents that change, tell me your wrong-served rate, how you measured it, and which pair of questions collided first. Dense embeddings are the setting I could not test, and my guess is that they make the sibling problem worse, not better; if your cache runs on them, yours is the report I would read first.

Acknowledgements

This article builds on FinCacheServe: Dependency-Consistent Answer Reuse for Cost-Efficient RAG Serving over Mutable Enterprise Documents by Lingteng Zeng and Yifan Jin.

The prototype reimplements the signature, receipt, and reuse gate from Sections 4.1 to 4.4. It does not use official code, which the paper says may be released with a future version. The grounded-style arm follows the four-gate design of GroundedCache by Syed Huma Shah, reimplemented from the paper rather than taken from its public repository.

The corpus, questions, and trace are synthetic and were written for this prototype. No SEC data was used.

References

Primary research

• FinCacheServe: Dependency-Consistent Answer Reuse for Cost-Efficient RAG Serving over Mutable Enterprise Documents. Lingteng Zeng and Yifan Jin, 2026. First public version 14 July 2026.

Supporting and contrary evidence

• Grounded Cache Routing for Retrieval-Augmented Generation: When Is It Safe to Reuse an Answer?. Syed Huma Shah, 2026. Scores reuse by the rate of wrong cached answers against gold, and finds a lexical support check carries most of the safety.
• Closing the Operational Gap in Semantic Caching. Aditeya Baral, Radoslav Ralev, Iliya Sotirov Zhechev, Srijith Rajamohan, and Jen Agarwal, 2026. Argues semantic cache selection is a threshold problem, not a ranking one.
• Risk-Constrained Freshness-Aware Semantic Caching for Open-Web Retrieval-Augmented LLMs. Muhammad Mansoor, Tahir Ahmad, and Yeo-Chan Yoon, 2026. Freshness as a learned staleness probability for web evidence.
• LaCache: Robust Semantic Caching for LLM Serving. Jiacheng Liang, Yuhui Wang, Tanqiu Jiang, and Ting Wang, 2026. Defends against deliberate cache collisions by also matching early decoded tokens.
• Asynchronous Verified Semantic Caching for Tiered LLM Architectures. Asmit Kumar Singh and colleagues, 2026. Opened, then excluded: first public version 13 February 2026, outside the six-month window.

Prototype

• Source, setup, and evaluation
• Exact tested commit

Sources last checked: 2026-09-26