Our RAG Got Worse Without Any Code Changes

You open your support inbox on a Tuesday and there are three tickets in a row: "the AI gave me the wrong answer," "search stopped working," "it used to find this instantly." You pull up your deployment history. Nothing went out in the last two weeks. You check the model. Same one. You check the API. No errors. You check the retrieval logs. Queries are completing.

The code did not change. The retrieval got worse. Those two things should not be able to coexist, but they do, regularly, in production RAG systems. Understanding why requires looking at what actually changed: not the code, but the data the code runs on.

What Actually Causes This

A RAG pipeline has two moving parts: the retrieval system and the corpus it searches. Most teams monitor the retrieval system obsessively and ignore the corpus entirely. The corpus is where most silent regressions live.

There are three common patterns:

All three of these happen gradually. There is no moment where something breaks. The quality slope is shallow enough that no single day looks like a problem, and by the time the aggregate degradation is obvious, it has been accumulating for weeks.

The Manual Approach: What Catching This Actually Takes

If you want to detect corpus-driven quality regression before your users do, you need three things: a measurement of how your embedding distribution changes over time, a baseline to compare against, and a trigger that runs that comparison automatically.

Here is what building that yourself looks like.

Step 1: Calculate centroid distance

The centroid of your embedding space is the average vector across every document in your corpus. When you add a large batch of new content, the centroid moves. If it moves far enough, the retrieval geometry that worked before is now operating on different assumptions.

To measure this, you compute the mean embedding of your corpus at time A, compute it again at time B, and calculate the cosine distance between those two centroids. A small shift (under 0.05) is normal. A shift above 0.15 or 0.20 usually correlates with a measurable retrieval quality drop on queries tuned to the original corpus.

import numpy as np

def centroid_distance(embeddings_v1, embeddings_v2):
    c1 = np.mean(embeddings_v1, axis=0)
    c2 = np.mean(embeddings_v2, axis=0)
    # cosine distance
    dot = np.dot(c1, c2)
    norm = np.linalg.norm(c1) * np.linalg.norm(c2)
    return 1.0 - (dot / norm)

shift = centroid_distance(baseline_embeddings, current_embeddings)
if shift > 0.15:
    print(f"Centroid shifted {shift:.3f} — retrieval geometry changed")

The problem is that running this requires you to have stored your baseline embeddings somewhere. Most teams do not. When the shift happens, you have no reference point. You know something changed; you cannot measure by how much.

Step 2: Track norm shift

Centroid distance tells you where the average vector went. Norm shift tells you something different: whether the average magnitude of your vectors changed. This matters because if your documents are getting longer on average, or if you changed your preprocessing pipeline, the norms will drift even if the semantic direction of your corpus stays roughly the same.

High norm shift combined with low centroid shift usually means document length or preprocessing changed. High centroid shift with stable norms usually means the semantic content of your corpus changed. Both patterns are worth catching. They point at different problems.

Step 3: Measure distribution divergence

Jensen-Shannon divergence measures how different two probability distributions are. Applied to your embedding space, it tells you whether the overall shape of your corpus distribution changed, not just the center. Two corpora can have the same centroid but very different distributions: one clustered tightly around a single topic, the other spread across ten.

A JS divergence under 0.1 between versions indicates healthy continuity. Above 0.3 typically means the corpus added content that is semantically distant from what existed before. That is not always bad, but it needs to be caught so you can evaluate whether your retrieval config still makes sense.

Step 4: Store versioned baselines

All of the above is only useful if you have something to compare against. That means every time you update your corpus, you need to snapshot your embeddings, store the metadata about that snapshot, and keep the history long enough to be useful.

In practice this means writing a versioning layer on top of your vector database, a storage strategy for the embedding snapshots (which can be large), a retrieval mechanism to pull any historical version on demand, and a comparison pipeline that runs the checks automatically.

Most teams are not going to build this. The ones that do spend two to four weeks on it and end up with something brittle that they stop maintaining after the first major corpus migration. The problem is not that the approach is wrong. It is that building it from scratch is expensive and the maintenance burden compounds.

Step 5: Wire a trigger to run checks automatically

Even if you build all of the above, it only catches regressions if it actually runs. Manually kicking off a comparison after every corpus update requires someone to remember to do it. In practice, nobody does, especially not on a Friday when the document sync happens to run at 11pm.

You need the checks to run automatically on every version change, produce a clear pass or fail, and surface failures somewhere a human will actually see them before the next deployment goes out.

What This Costs to Build and Maintain

How Decompressed Handles This

The approaches described above are the right ones. Centroid distance, norm shift, and JS divergence are the metrics that actually tell you whether your corpus changed in a way that affects retrieval. The problem is not the measurement. It is the infrastructure cost of running it continuously.

Decompressed is built around the assumption that your corpus is a versioned artifact, not a mutable blob. Every time you sync or upload documents, a new version is created automatically. You do not opt into versioning. It happens by default. That means the baseline always exists, going back to the first sync.

The Quality tab

The Quality tab runs the exact checks described above on your actual corpus. Centroid shift is measured against the previous version. Norm shift is compared version over version. JS divergence tells you how the distribution changed. Each check surfaces the raw metric value and the threshold it is measured against, so you can see not just pass or fail but by how much.

When a check fails, you see: what moved, how far it moved, and what version it moved from. That is the full diagnosis in one view. No SQL queries against your vector database. No custom scripts to pull embedding snapshots. The comparison is already done.

Auto-run on version create

Quality checks run automatically every time a new version of your corpus is created. You do not have to schedule them, trigger them manually, or remember to check the results. When the nightly document sync completes and a new version is written, the checks run against it. If something failed, you know before the next morning standup. Not when users start complaining two weeks later.

Version history as a testing surface

Because every version of your corpus is retained, you can test retrieval quality against any of them. If a user reports that search felt better three months ago, you can pull the corpus state from that period and run the same gold set evaluation against it. That comparison either confirms the regression or rules out the corpus as the cause. Either answer is useful.

This also means you can test configuration changes before committing them. Run the same evaluation on two different corpus versions with the same retrieval config. Or run two different configs against the same version. The version history gives you a stable reference point that does not move while you are experimenting.

When You Get the Regression Alert

When a quality check fails, the first question is whether the shift is a problem or expected. A centroid shift of 0.08 after adding 200 new support articles is probably fine: you added content, the distribution moved slightly, and retrieval is still operating in the same general space. A centroid shift of 0.31 after what was supposed to be a routine re-index is not fine.

The threshold values are yours to set. The defaults are conservative starting points. For corpora that change frequently and intentionally (news, support articles, product documentation), you may raise the centroid shift threshold. For corpora that are supposed to stay stable (legal documents, compliance content, certified knowledge bases), you may tighten it.

When a shift is genuinely unexpected, the version history is the starting point. Pull the two versions on either side of the failure. Identify which documents were added or removed. If a single batch introduced the shift, you can evaluate whether those documents belong in the index at all, or whether they need a separate retrieval config.

The Pattern That Keeps Failing

Teams discover RAG regressions one of two ways: from a quality check, or from users. The ones who find out from users spend the next several days in reactive mode: pulling logs, running ad hoc queries, trying to reproduce complaints without a clear baseline.

The ones who find out from a quality check spend twenty minutes looking at what shifted and deciding whether to act. The total calendar time is not that different. The experience is completely different. One is controlled. One is a fire.

Building the monitoring yourself is worth it if you have the engineering capacity and you want to own the instrumentation. If you do not, you will keep finding out from users, and the cost of that is not just the support tickets. It is the trust you spend every time the system behaves worse than expected without warning.