Why Chunk-Based RAG Is Ineffective for Constantly Changing Customer Data

Traditional chunk-based RAG fails on dynamic customer conversations because querying raw text repeatedly produces inconsistent answers and cannot aggregate portfolio-wide trends. Shifting comprehension to ingestion time—processing interactions once into structured, evidence-linked facts—solves this. This "compute once, remember" model delivers fast, consistent answers and enables reliable tracking of account risks and trends across an entire client base.

The Velaris Team

November 6, 2026

Building a customer platform that remembers, Part 1

Retrieval is the default answer to "give the AI memory." For customer memory, we changed what gets retrieved: understand every signal once, when it arrives, and retrieve what it means.

The question that broke our first version

A customer success manager opens the assistant on a Monday morning and types: Is Northwind at risk?

Our first version did what most AI assistants do. It searched months of emails, call notes and tickets for passages that looked relevant, put the best matches in front of the model, and asked it to decide. The answer was plausible and well written.

Then the manager asked again an hour later and got a slightly different answer, built from slightly different passages. Again, plausible and well written.

Then she asked the question she actually cared about: Which of my forty accounts are at risk? The assistant had no good way to answer it at all.

Nothing was broken in the usual sense. The search worked, the model reasoned well. The problem was the shape of the approach itself, and it took us a while to see it.

The default everyone reaches for

Retrieval-augmented generation (RAG) is the standard way to give a model knowledge it wasn't trained on. RAG is a pattern (retrieve, then generate), not one implementation. Its classic form is chunk-based: text is split into chunks and indexed by meaning. When a question arrives, the closest chunks are fetched and placed in the prompt, and the model answers from them.

How chunk-based RAG works · prepare the text, then retrieve and generate

For documents, chunk-based RAG works very well. A help centre article or a product manual is written to be read, says the same thing every time, and rarely contradicts itself. The question "how do I reset my password?" has an answer sitting in one place, waiting to be found.

Customer conversations are a different kind of material. They pile up continuously. They contradict each other as situations change. Their meaning is spread across many messages and many weeks. And nobody wrote them to answer the questions we later ask of them.

So the real question isn't whether to retrieve. It's what to retrieve.

Where chunk retrieval breaks for customer memory

When we looked closely at why the Northwind answers wobbled, we found four problems. None of them is fixed by a better embedding model or a smarter ranker, because they come from when the understanding happens.

  1. Every question pays to understand the same history again. Retrieval interprets raw text at question time. Ask about an account ten times and the model reads and re-reads the same months of conversation ten times. That is slow, it is expensive, and each reading can land somewhere slightly different. 
  2. Similar is not the same as relevant. Search finds text that sounds like the question. But the most important evidence for risk is often a message that doesn't mention risk at all: a contact going quiet, a deadline mentioned in passing. Worse, retrieval has no sense of what has been superseded. A complaint from July and the email in August confirming it was fixed look equally relevant.
  3. It can't retrieve data across a client portfolio. "Which accounts are at risk?" and "How many customers raised onboarding concerns this quarter?" are questions about a whole portfolio. Fetching the ten most similar passages cannot answer them, because top results can't be filtered, grouped or counted. So our retrieval became structured: the assistant filters facts by topic, time window and channel, and counts them across a portfolio. 
  4. It is hard to check and harder to track. If an answer is assembled fresh each time from whichever passages ranked highest, there is no stable conclusion to inspect, compare with last month, or correct.

What we tried first

We didn't start here. In early 2025 we ran two experiments that looked like everyone else's. The first embedded conversations into a vector index for semantic search. The second built a knowledge graph of people, companies and topics from email (an approach now often called GraphRAG).

Both experiments taught us something. Semantic search was excellent at finding passages and poor at telling us what they added up to, so it stayed in our toolbox, just not as the memory itself. The knowledge graph captured relationships well, but keeping it accurate as conversations changed was costly, and most of the questions our customers asked didn't need graph traversal. They needed a reliable, current answer to "what do we know about this account?"

The lesson was that our problem wasn't finding text. It was remembering meaning, and then retrieving that.

The principle: compute once, remember

Think about how a great account manager works. She doesn't re-read her entire inbox before every meeting. After each conversation, she notes what mattered: the concern raised, the promise made, the deadline mentioned. Before the next meeting, she reads her notes. When something changes, she updates them.

That is the design we adopted, and we call it compute once, remember. Instead of interpreting raw conversations every time a question is asked, we interpret each signal once, when it arrives. What it means is stored as a structured fact about the customer, linked to the conversation it came from. Every AI experience on the platform then reads the same facts.

The context layer · signals in, facts out

The expensive, variable work of understanding moves from question time to arrival time. Question time still retrieves, but what it retrieves is already understood: a fast, consistent read of what is known.

In AI terms, this moves understanding from query time to ingestion time. It is a form of structured, memory-augmented retrieval: the model is still grounded in retrieved context, but that context is curated facts rather than raw passages. Much of what is now called context engineering is deciding what belongs in the model's context. Our answer is conclusions with evidence, not transcripts.

Chunk retrieval repeats the interpretation for every question.      Compute once does it on arrival, and every question retrieves the result.

Four ideas that make it work

1. Understand at arrival, not at question time

Every email, ticket, note, chat thread and call is processed as it comes in. In our platform, Fact Factory handles written conversations and CallSense handles calls. Each turns raw interaction into a small number of clear statements about the customer: a concern, a commitment, a request, a sign of growth.

2. A shared vocabulary, not just similarity

Customers describe the same experience in different words:

•   "Getting started is taking longer than expected."

•   "Our team still feels lost."

•   "We need more help with onboarding."

A similarity search might connect two of these and miss the third. We classify every fact into a shared, structured vocabulary of customer topics instead, so all three land in the same place. If something doesn't fit the vocabulary, it is filed as "other" rather than inventing a new label. That consistency is what turns individual remarks into patterns a team can count across hundreds of accounts.

3. Every fact keeps its evidence

An AI interpretation should be something a person can check. Each fact links back to the interaction it came from, so a manager who sees "onboarding concern" can open the email behind it and judge for herself. Every answer is grounded and citable: the assistant cites the fact, and the fact cites the email. A concern is evidence of a concern, not a prediction.

4. Facts compound

Because facts are stored, not regenerated, they build on each other. When an AI agent reviews an account and records a conclusion, such as a risk assessment, that conclusion becomes a fact too. Its history becomes a trend: not just "at risk" but "at risk, and getting worse for three weeks." This is long-term memory for agents: what one agent concludes, the next one starts from. It is the foundation for the agents in the rest of this series.

What changed

Go back to Monday morning. The manager asks Is Northwind at risk?, and this time the assistant doesn't search months of conversation. It retrieves what is already known about Northwind, choosing which facts it needs (which topics, which period, which channels), counting across accounts where the question asks for it, and citing the evidence. The model plans its own retrieval with tools rather than taking a fixed top ten.

  • Consistent answers. The same question asked twice reads the same facts, so it gets the same answer unless something has actually changed.
  • Much faster, much lighter. For a single-account question like this one, reading stored conclusions takes a fraction of the tokens and comes back in a fraction of the time.
  • Portfolio questions became possible. "Which of my accounts are at risk?" and "Where are customers struggling with onboarding?" are now questions over structured facts, so they can be filtered, grouped and counted.
  • One memory, many experiences. The assistant, account headlines, our autonomous agents and connected AI tools all retrieve from the same facts. Work done once on arrival is reused everywhere.
  • Trends for free. Because conclusions are stored over time, the question shifts from "is this account at risk?" to "how has this account changed?"

What we gave up

Every architecture is a set of trade-offs, and this one has real costs.

  • Fuzzy recall of anything ever said. If a detail wasn't recognised as meaningful when it arrived, it won't be a fact later. We accept this today because the evidence is still kept, and because a reliable answer to the common questions matters more than a lucky answer to rare ones. It is also exactly where we are adding semantic retrieval: as a fallback over the raw evidence, behind the fact layer, never instead of it.
  • A vocabulary to look after. A shared set of topics only works if it stays current. As customers and products change, the vocabulary has to evolve with them, and that takes ongoing care.
  • Mistakes persist until corrected. A misread email becomes a stored fact, not a one-off bad answer. Linking every fact to its evidence is how people catch these, and it is why we treat evidence as a requirement rather than a feature.
  • Work up front. We process every interaction, including ones nobody will ever ask about. In practice this is cheaper than repeatedly re-reading history at question time, but it is a deliberate choice to spend effort early.

Where classic retrieval fits, and where we're heading

This isn't an argument against RAG. It is an argument about what to retrieve. Classic chunk retrieval has three clear jobs alongside the fact layer, and they are where we are taking it next:

  • Reference knowledge. Product documentation, help articles and policies are written to be read and stay true until someone edits them. Retrieval over them is the right tool.
  • Long-tail recall. When a question needs a detail that was never captured as a fact, semantic search over the original evidence can find it, as a fallback behind the facts.
  • Hybrid retrieval over facts. Filtering by structure first and then ranking by meaning lets the assistant match phrasings the vocabulary hasn't learned yet.

The distinction we draw is simple. Reference knowledge is something you look up. Customer memory is something you build up. The best systems do both: build up memory, then retrieve from it.

Next: from remembering to watching

Once a company remembers what its customers have told it, a new possibility opens up. AI agents can watch that memory for change: a concern that hasn't been resolved, a contact going quiet, a growth signal nobody followed up. In Part 2 we'll look at the first of them, Risk Watchdog, and how it decides what is worth a human's attention.

Your customers remember what they told you. Your AI should too.

Frequently Asked Questions

The Velaris Team

The Velaris Team

A (our) team with years of experience in Customer Success have come together to redefine CS with Velaris. One platform, limitless Success.

Want to see Velaris in action?

Discover the difference it can make for your team.