RAG vs Cache-Augmented Generation (CAG): Two Ways to Give an AI the Information It Needs

RAG vs Cache-Augmented Generation (CAG): Two Ways to Give an AI the Information It Needs

An AI assistant can write a polished answer and still know nothing about your business.

It was not present when you changed your cancellation policy. It cannot see today's room availability unless you connect it to the booking system. It does not automatically know which version of the staff handbook is current.

So how do you give a large language model, or LLM, the information it needs?

Two approaches come up often: Retrieval-Augmented Generation (RAG) and Cache-Augmented Generation (CAG). The names sound technical. The difference is easier to understand if you picture a filing cabinet and an open binder.

RAG searches the filing cabinet for each question. CAG puts a selected binder on the desk before the questions begin.

The choice affects how much information the system can handle, how quickly it answers, how it stays current and where it can fail.

Here, CAG means Cache-Augmented Generation; the acronym is also used for other techniques.

First, what does the LLM do?

An LLM is the part that reads the question and writes the response. It is very good at working with language, but it should not be treated as a reliable store of your current company knowledge.

RAG and CAG both give the model extra material before it answers. They do not permanently teach the model new facts. They change what is available on its desk for that particular piece of work.

RAG: look it up when someone asks

RAG stands for Retrieval-Augmented Generation. The name and an influential formulation appeared in a 2020 research paper.

Imagine a customer asks a hotel assistant:

“Can I bring my dog, arrive after midnight and leave the car there for two days?”

A RAG system does roughly this:

1. It searches the hotel's approved documents.
2. It selects passages about pets, late check-in and parking.
3. It gives those passages to the LLM with the customer's question.
4. The LLM writes an answer based on what was retrieved.

The model does not read the entire hotel knowledge base. It receives the few pieces the search system believes are relevant.

RAG narrows a large collection for each question. CAG keeps a bounded reference set ready for repeated questions.

Why businesses use RAG

RAG works well when the collection is too large to place in front of the model all at once. A company might have thousands of policies, manuals, product pages, contracts and support conversations. Most questions need only a small portion of that material.

It also gives the system a natural record of what it found. If you preserve the source information, you can show which documents and passages were retrieved and supplied to the model. Correct citations and claim-by-claim evidence checks still need to be built deliberately.

RAG is a strong fit for a large collection, especially when it changes regularly or questions span many subjects. Access checks must happen before restricted material is retrieved and placed in the model's context; RAG is not permission-safe by default.

Where RAG goes wrong

RAG adds a search stage, and that search can fail.

The right document may not have been uploaded. The material may have been split into awkward fragments. The search may choose a passage that contains the same words but answers a different question. The best passage may rank sixth when the system sends only the top five.

Even perfect retrieval does not guarantee a perfect answer. The LLM can still misread a passage, combine sources badly or sound more certain than the evidence allows.

RAG can improve grounding. It does not eliminate mistakes.

CAG: keep the handbook open

Cache-Augmented Generation takes a different route.

Instead of searching for a few passages each time, the system gives the model the same prepared reference pack for every question. The hotel could prepare one approved pack covering matters such as check-in, parking, pets, accessibility and cancellations.

If that pack is small enough, stable enough and shared by the intended users, the assistant can keep it ready. There is no document search before every answer.

What is being cached?

CAG does not save one old answer and repeat it. It combines a prepared reference set with reuse of the model's earlier processing. The question can change, and the model can produce a new answer.

On a hosted model, that reuse usually depends on the reference material and instructions forming the same cacheable beginning each time. If the prefix changes, the cache expires or the provider evicts it, the material may need to be processed again. Provider rules vary.

Prompt caching is related but not identical. A system can cache repeated instructions without being CAG. The paper's CAG method specifically combines a complete working reference set in long context with reuse of its processed state instead of a live retrieval step.

Why CAG can be attractive

CAG removes the live search and ranking step. If an important fact is already in the prepared handbook, it cannot be missed because the search chose the wrong document. The model can still overlook or misread it.

The CAG arXiv preprint reported more accurate answers and faster responses than the particular RAG systems it tested. That result is promising, but it applied to specific models, datasets, corpus sizes and RAG configurations.

CAG is most useful when the same small, stable reference pack serves many questions, the model can use the whole pack reliably, and the cache can be refreshed and separated safely.

The binder has limits

A model may advertise a very large context window, but that does not mean it uses every part equally well.

Research such as “Lost in the Middle” found that models can perform better when relevant information appears near the beginning or end of a long input and worse when it sits in the middle. The RULER benchmark also found that models may struggle to use very long inputs reliably, even when those inputs fit within the published limit.

So “the documents fit” is not enough. You still need to test whether the model can reliably find and use facts across the whole prepared reference set.

CAG becomes a weak fit when the reference set is too large, questions span a broad archive, freshness requirements exceed your ability to rebuild the cache, or one shared cache would mix confidential information. Separate permission-scoped caches are possible, but they add operational cost and complexity.

Caching does not enlarge the context window. It saves repeated processing; it does not create more room.

A hotel assistant usually needs more than one approach

Consider a hotel that wants an AI assistant for guest calls and messages.

• Stable guest guidance → CAG. An approved pack could cover parking, breakfast, pet rules, accessibility, facilities and escalation instructions.
• Large document archive → RAG. The hotel might search years of event menus, procedures, supplier information and corporate-booking documents only when required.
• Availability, prices and bookings → live system. Room availability, today's rate, booking status and payment state belong in the authorised property-management or reservation system.

Neither a cached handbook nor a document index should pretend to be the live source of truth. The assistant should query the authorised system directly, then confirm any consequential action before committing it.

Stable, shared guidance        → CAG
Large document archive         → RAG
Availability, prices, bookings → Live API or database

The useful question is rarely “Which acronym wins?” It is “Which information belongs in each lane?”

How to choose

Ask these questions in order.

1. Can the full working reference set fit comfortably?

Leave room for instructions, conversation history, the new question and the answer. If the material only just squeezes under the model's published maximum, treat that as a no until testing proves otherwise.

2. How often does the information change?

A controlled handbook may suit CAG if you can rebuild and invalidate its cache within the freshness window the business requires. Inventory, appointment slots and prices that change throughout the day should come from live systems. A broad document collection that changes steadily may suit RAG with a reliable way to keep the collection updated. RAG is not automatically current either.

3. Does everyone have the same access?

A shared cache is dangerous if it contains material that some users should not see. RAG that checks access before retrieval, separate permission-scoped caches or direct authorised lookups may be safer. Each option needs explicit testing.

4. Do questions stay within one bounded subject?

Repeated questions against the same policy pack make CAG attractive. Unpredictable questions across a huge archive favour RAG.

5. Can you prove the system works with real questions?

Do not choose from a diagram or a benchmark alone. Build a small comparison using the same model and the same answer instructions.

Start with a representative set of real questions, then expand it as you discover failure cases. Include ambiguous wording, missing answers, conflicting documents, changed policies and revoked permissions. Measure:

• answer correctness;
• whether the right source was used;
• unsupported claims;
• response time;
• cost at realistic volume;
• freshness;
• permission handling.

Many useful assistants combine RAG, CAG and live systems. The labels matter less than giving each kind of information to the right system and testing the answers against questions people actually ask.

Sources and further reading

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, Lewis et al., 2020.
• Don't Do RAG: When Cache-Augmented Generation Can Be All You Need for Knowledge Tasks, Chan et al.
• Lost in the Middle: How Language Models Use Long Contexts, Liu et al.
• RULER: What's the Real Context Size of Your Long-Context Language Models?, Hsieh et al.
• OpenAI prompt caching guide.
• Anthropic prompt caching guide.