The old FAQ sorted first

The fix I was about to recommend answered my parental leave question with the cafeteria opening hours.

That was 16 August 2026, partway through a trial I had expected to be dull. A colleague had uploaded the HR folder to a Project and asked how many days of annual leave full-time staff get. The chatbot said fifteen. The policy that took effect on 1 July 2026 says twenty. Nobody had lied to anyone. The 2024 FAQ was still in the folder, it used the words "annual leave" more often than the current policy did, and filename order put it first. I had a tidy explanation for that, and a tidy fix ready to write up: prefer the newest file that matches the question. Then I ran the fix, and it handed me a page about the canteen.

What happens to a number between the folder and the prompt, before any model gets a vote, turned out to be the whole question. I took a ten-file HR export with old and new files mixed together and packed it five ways: paste the whole folder in filename order, retrieve by word overlap, retrieve the top three, prefer the newest file in the overlap shortlist, and hand-pick the three files a new hire would actually be given. No model API. An extractive reader pulled whatever numbers were sitting in whichever files made the pack. The files, the prompts and the arithmetic are in the prototype repository. I am not ranking Claude Projects against ChatGPT Projects against NotebookLM, I did not run a frontier model on the pack (so every sentence about what a chatbot would say is labelled as my inference), and I am not going to walk you through building a vector database, which is a real next step for a library far bigger than this one.

If you want the outcome without the trip, the next section has it.

Five packs, one leave question, one right pack

1. Pasting the whole folder in filename order put both 15 and 20 in the prompt. The 2024 FAQ came first, with three mentions to the 2026 policy's two.
2. Overlap retrieval at k=1 ranked the FAQ above the policy and answered 15. At k=3 the conflict simply came back.
3. Preferring the newest file in the shortlist recovered 20 on leave, then returned no number at all for parental leave or the meal stipend, because it chose newer files that only mention those topics.
4. The curated three-file pack was uniquely 20, 16 and $15. It was the only pack that got all three.
5. Parental leave and the stipend had no stale twin in the folder, and they survived paste-all without trouble. The failure I measured is a mixed-age copy of the same fact, and I put one there on purpose. Embeddings, live Claude or ChatGPT retrieval, and a folder whose files all agree sit outside what I ran.

My call is split. If you can name the current files, paste those and stop. If you cannot, retrieve, then open the file the answer came from before you repeat it. Keep fine-tuning out of it for facts.

Why the folder sorts the old file first

You already have a chatbot open and a folder somewhere: policies, FAQs, an onboarding checklist, a slide dump from last year. Every product invites you to upload the lot. Claude Projects, ChatGPT knowledge files, Gemini notebooks, the paperclip in the chat box. A 200K or 1M token window looks like it will hold a whole handbook, so the implied move is to dump it in today.

The window will hold the bytes. What you need is narrower than that: when someone asks a current question, which number comes back, and can you point at the file that produced it?

Start with the filename. A Drive export often prefixes files with a year, so 2024-leave-faq.md sorts ahead of 2026-annual-leave-policy.md, and paste-all in filename order put the superseded FAQ first in my pack. I had assumed that having the current file somewhere in the pack was enough. Being in there second, with the old number in front of it, is exactly how a reader that takes the first claim it meets answers 15.

The vendor docs describe this honestly once you read them as descriptions of a subset. OpenAI's GPT knowledge article tells you to put reference material in files and rules in instructions. Anthropic's project help says Claude loads what it can, then switches to a project-knowledge search when the library approaches the context window, and that this can expand capacity by up to 10x. Both are telling you the model sees part of the folder. Neither promises that the part is the current file.

Fifteen and twenty in the same prompt

Fifteen and twenty cannot both be the annual leave answer. If the pack contains both, you have an argument, not a knowledge base. Paste-all on my leave question produced [15, 15, 15, 20, 20]: three hits from the FAQ, two from the 2026 policy. Uniqueness failed. Parental leave and the meal stipend, which had no stale twin, survived paste-all uniquely. So the failure I am describing is narrower than "folders are hard": two copies of the same fact, from different years, in one prompt.

There is a third thing to check, and I used to skip it. Cafeteria hours mention parental leave, in a sentence about the stipend not applying on those days. The onboarding checklist tells you to read the cafeteria page. Both files are current. Both can win a keyword shortlist. Neither contains the number. The trial turned that check from a nicety into the whole default, which is the part I get to later.

What the long-context papers measured, and when

None of the evidence below was collected this month, and I want you to read the dates before the numbers.

That paper is older than this week's products. My read is that the shape is still the reason a paste-all pack is not "more complete". A paste-all pack is a longer middle.

I am not going to pretend a 2025 study of GPT-4.1 is a 2026 study of Opus 5. What I will say is that the upload box still invites you to do the one thing both papers make expensive: put more of the folder in, in the order the folder already had.

The capacity badge answers a different question

Hover over a Project and you get a capacity number. 2% used. 35% of 200K. 10x with RAG. Every one of those cells can be right and still point you at the wrong default, because capacity says nothing about whether the pack holds one number for the fact, and holding one number says nothing about whether the file answers the question.

I am not accusing anyone of hiding the retrieval switch. Anthropic documents it. OpenAI documents hybrid search over uploaded files. Google's NotebookLM is the rare product that makes source-grounding the whole point. The misreading is ours: we see "uploaded" and hear "in the prompt, current, alone".

Three questions with the answers written down first

I wanted a test you would recognise as a folder, not as a benchmark.

Three questions, gold numbers fixed in data/questions.json before I packed anything:

1. How many days of annual leave do full-time employees get? 20.
2. How many weeks of parental leave do employees get? 16.
3. What is the daily meal stipend in dollars? 15.

Five packs: paste-all in filename order, overlap k=1, overlap k=3, newest file in the lexical shortlist, and a curated three-file pack (2026 leave policy, parental leave, cafeteria). The reader is extractive, and I kept Claude out of that seat deliberately: if two current numbers disagree in the prompt, a better model does not get a vote. The files already voted.

What I expected, written down at the time: overlap would pick the official 2026 policy, because its title matches the question almost word for word. And if overlap failed, "prefer the newest matching file" would be the working default I sent people away with. I was wrong about both, and the second one is the more useful miss.

The FAQ outscored the policy 13.90 to 8.40

The stop threshold fired twice, on the same question. Dumping did not uniquely determine 20, and keywords preferred the stale FAQ.

The FAQ result is the one that changed how I use the upload box. I had assumed an official policy would beat an FAQ on overlap because it is the "right" document, the one with the entitlement in it. I expected the FAQ to compete. What surprised me was the margin: 13.90 to 8.40, against a policy whose title matches the question almost word for word. The FAQ won because it said "annual leave" in every heading. Keyword density and authority are different things, and the retriever only sees one of them.

The second surprise was smaller and more irritating. I had also assumed that last-claim would quietly rescue a paste-all user, the way recency sometimes rescues a long-context model. Last-claim is 20, yes. First-claim is 15. Liu's U-shape says both ends of the prompt are strong. Filename order in a year-prefixed dump puts 2024 at the strong end people actually read first, and my impression, which I have not measured, is that most real dumps look like this one.

Then the newest file was the cafeteria page

That cafeteria page is the result from the first line, and it killed my working default. I had walked in believing "newest matching file" was the grown-up version of dump-the-folder. What it turned out to be was a fix for the one conflict I had designed, and a fresh failure on the two questions I had not thought were the point. A neighbouring current file that merely mentions the topic is enough to win, and once it wins there is no number to return. The part that surprised me was the shape of the miss. I had braced for a wrong number. I got no number, twice, from files I had written myself, and I should have seen it coming, because the cafeteria sentence about parental leave was mine.

Three ways an upload still hands you the wrong number

The file you uploaded may never have entered the prompt

Anthropic will load a small library into context and search a large one. OpenAI will stuff what fits and hybrid-search the rest. A February user report said the Claude switch can follow file count rather than tokens; I did not verify that. You can verify something cheaper. After you upload, ask the chatbot which file it used, then go and open that file. If it cannot name one, you do not have a source. You have a vibe.

Two current-looking numbers are a tie the model should never break

When 15 and 20 are both in the prompt, "the smarter model will notice the 2026 date" is a hope, and Liu's middle is where hopes go. Delete the FAQ, or mark it superseded and keep it out of the pack. Arguing with the chatbot about which file is newer is how you spend a Tuesday.

Newest is a tie-break, and a bad primary key

Date-aware retrieval recovered my leave number and then handed me a cafeteria page. Recency is a good way to choose between files that actually contain the claim. Used as the first filter, it is a different bug with a nicer name. The test I now apply is: does this file state the number, in a sentence about the entitlement, with a date you would defend to payroll?

Should you fine-tune on the handbook?

No, not for the facts. Fine-tuning is a later move for a style or format job that prompting cannot hold; a 15-day FAQ baked into weights is far harder to retire than a file you can delete.

OpenAI's own split is still the useful one here. Knowledge files for reference. Instructions, or later a fine-tune, for behaviour. If the failure is "it doesn't know the 20-day policy", training on last year's FAQ is how you make the 15-day figure expensive to kill. I have not fine-tuned a handbook in this trial. I have watched enough stale prompts to stop recommending it as a data strategy, and I would want to be shown a handbook that stops changing before I changed my mind.

The three files a new hire would be handed

The average says the setup is usable if you curate. The binding constraint says do not let the dump pick the number, and the scores are my reading of one folder, so weigh them as that.

Run the five packs yourself

You can replay the recorded trial without an API key. You still need your own three files if you want to choose a default for your own folder.

git clone https://github.com/shravan1996/generative-ai-folder-is-not-the-data
cd generative-ai-folder-is-not-the-data
python3 -m unittest discover -s tests -t .
python3 eval.py

No network, no GPU, no third-party package. The full implementation is in the prototype repository.

1. 01

   Name one question you asked the chatbot last week

   Owner

   You

   Artifact

   One sentence: the question, the number or file you needed

   Signal

   If you cannot name it, you are uploading, not using

2. 02

   List every file in the pack that mentions that fact

   Owner

   You

   Artifact

   Filenames, dates, and the number each one states

   Signal

   A second number is the actual problem

3. 03

   Delete or quarantine the superseded copies

   Owner

   You

   Artifact

   A pack whose unique number you would defend to payroll

   Signal

   If you need the old file for history, it does not belong in the Project

4. 04

   Ask the chatbot which file it used, then open that file

   Owner

   You

   Artifact

   A named source, or a refusal

   Signal

   A fluent answer with no file is a skip

5. 05

   Keep the curated pack, or split retrieval onto the leftovers

   Owner

   You

   Artifact

   A one-line default and a date to look again

   Signal

   If a week returns a retired number, the window does not get a vote

Before you trust the pack:

• The pack contains one number for the fact you care about
• You can name the file that number came from
• Anyone else on the team can find that file without a private nickname

Rebuild the pack, or delete a file, if:

• A week of real questions returns a number you know is retired
• The chatbot cannot name a source file
• You needed a second tool to stop two current files arguing

Delete the FAQ, then split the rest

Split for giving a chatbot your documents in August 2026.

As far as I can tell, dumping the folder gives you a search over a dump, and the dump will happily hand you last year's FAQ. I would start a colleague on a curated current pack, the files you would actually hand a new hire. I would retrieve when that pack no longer fits, and I would still open the file. I would not fine-tune to teach a number that lives in a markdown file. I expect Claude Projects, ChatGPT knowledge, and NotebookLM can all host that pack, and I suspect it matters less which one you pick than which three files you put in it.

I would change this verdict if:

1. Your live Project, Custom GPT, or notebook, on three real questions, returns the current file first even with the superseded FAQ still uploaded.
2. A dated retrieval that also requires the file to contain the claim stops missing neighbouring pages, on a folder that is not mine.
3. You have a style or format job that few-shot prompting cannot hold, and the facts are already unique in the pack. Fine-tune that. Leave the handbook.

Three rules I kept from ten files

The result I could not produce is a live Project putting the 2026 policy first with the 2024 FAQ still in the folder. If yours did, show me the pack it built, because that would be a better default than deleting the file, and I would take it over mine.

Acknowledgements

This article builds on public vendor docs and two research results linked in References. Liu et al. and the Chroma Context Rot report are the reason I refused to treat window size as a strategy. OpenAI's knowledge-versus-instructions split is the reason the fine-tuning section is short.

References

Independent results

• Lost in the Middle: How Language Models Use Long Contexts. Nelson F. Liu et al., arXiv 2307.03172, TACL 2024. U-shaped position curve; GPT-3.5-Turbo middle below 56.1% closed-book; 50 vs 20 retrieved documents.
• Context Rot: How Increasing Input Tokens Impacts LLM Performance. Kelly Hong, Anton Troynikov, Jeff Huber, Chroma, 14 July 2025. 18 models; non-uniform use of long context; shuffled haystacks beat coherent ones.
• LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs. Kuan Li et al., ICML 2025. No silver bullet for RAG versus long context.

Vendor posts and product help

• Retrieval augmented generation (RAG) for projects. Anthropic Help Center. Automatic RAG at the context limit; up to 10x capacity; project-knowledge search instead of loading everything.
• What are projects?. Anthropic Help Center. Paid-plan RAG; context windows of 200K, 500K, or 1M depending on the model, per the usage-limits article.
• How do usage and length limits work?. Anthropic Help Center. Per-model window sizes on paid plans.
• Knowledge files for GPTs. OpenAI Help Center. Files for reference material; instructions for behaviour.
• Optimizing File Uploads in ChatGPT Enterprise. OpenAI Help Center. Hybrid search; fewer focused documents; one question at a time.
• Claude vs NotebookLM (2026). Atlas Workspace, July 2026. Source-grounded notebooks versus project workspaces; cited for product roles, not for scores I did not run.

User report, labelled as such

• Project files switch to RAG search at 2% capacity. GitHub issue, 14 February 2026, closed as filed against the wrong repository. About 13 files, 73,000 tokens, 2% displayed capacity. Not an Anthropic spec.

This trial

• Source, corpus, packing strategies, and evaluation
• Exact tested commit
• Ten-file Northwind HR dump, 16 August 2026. Gold: 20 days annual leave, 16 weeks parental leave, $15 meal stipend. Extractive reader, no model API.

Sources last checked: 2026-08-16