That’s roughly what happened to one platform engineer who deployed an updated retrieval-augmented generation (RAG) pipeline — a system that answers questions using external documents — only to discover that a slight drift in the embedding model had made it prefer stale pricing chunks over fresh ones. No alert fired. No test failed. As the engineer put it, "the dashboard was green, and the outputs were garbage". That single sentence captures something important about the shift happening across software teams right now: the tools that kept traditional applications safe for two decades were never designed to catch a system that fails by slowly getting worse, rather than by breaking outright.
Why "pass" and "fail" stop being useful words
Traditional continuous integration and continuous delivery (CI/CD) — the automated pipeline that tests and ships code — works because software is deterministic. Same input, same output, every time. A test either matches the expected result or it doesn’t, and that binary judgment is the whole point of the gate.
Large language models don’t offer that certainty. They produce probabilistic behavior: plausible, context-dependent, and never guaranteed to repeat itself exactly. As the engineer behind the original incident put it, "traditional CI/CD was built for deterministic software. LLMs are probabilistic. Our gates need to be, too". A relevance score can quietly slide from 0.82 to 0.79 to 0.74 over successive days without ever tripping a fixed threshold — and because no single measurement crosses the line, nobody gets paged while users are already receiving worse answers.
This is not a hypothetical concern reserved for exotic edge cases. It shows up in ordinary production systems in at least three recognizable ways, all of which the underlying tests were technically passing:
- Evaluation drift — the scoring system keeps producing numbers, but those numbers stop tracking real quality. This can happen after a model swap, a product change, or simply a shift in what users are asking.
- Distribution shift — real users type short, messy, typo-ridden questions that bear little resemblance to the clean test dataset a team built during development.
- Context or corpus drift — the documents a RAG system retrieves from change over time, so a system that answered correctly last month may be quietly serving outdated or irrelevant material today, even though nobody touched the underlying model.
Each of these fails the same way: the aggregate score looks fine, but the actual experience underneath has rotted. As the original engineer described it, unhealthy production AI "rots rather than explodes" — a framing borrowed from years of infrastructure work, where a server reporting 99.9% uptime while silently dropping a slice of financial transactions isn’t healthy either — it’s hiding a bug.
Building gates that watch behavior, not just outcomes
If the failure mode is gradual decay rather than a crash, the fix has to be a system that watches for decay, not just for errors. The approach described in the source incident organizes this into four layers that build on each other, each catching a different kind of risk:
Baseline evaluation. Before anything ships, a fixed dataset runs against the candidate system to score relevance, faithfulness, groundedness, and safety. This is the closest analogue to a traditional test suite, but the goal isn’t a perfect score — it’s catching an obvious regression before it becomes a support ticket.
Drift detection. A static pass/fail threshold isn’t enough, because a system can degrade gradually without ever crossing a fixed line. Instead, current scores get compared against a rolling baseline from recent deployments, so a meaningful drop relative to "last known good" — rather than relative to an arbitrary number — becomes the trigger.
Shadow (and mirror) validation. Before a change reaches users, real traffic is duplicated and sent to the candidate system in parallel — the user still receives the original, production answer, while the candidate’s response is scored behind the scenes for comparison. This is different from a canary release, where a small slice of real users is actually served the candidate live, with rollback ready if something goes wrong. One source frames the distinction sharply: "shadow mirrors with no user-visible effect; canary serves live with rollback-ready percentage routing". Confusing the two invites one of two failure modes — either skipping shadow and discovering the candidate’s weaknesses on real users for the first time, or skipping canary and stalling forever on offline numbers that never feel quite "ready enough". A lighter-weight variant called mirror sampling — testing only a fraction of shadow traffic — helps control the cost of this step before scaling it up.
Runtime budgets. A model that scores brilliantly but costs three times more, or adds seconds of latency, isn’t a viable release either — cost and speed are part of what "acceptable behavior" means in production, not an afterthought bolted on later.
Old gates versus new gates, side by side
| Dimension | Traditional CI/CD gate | LLM release gate |
|---|---|---|
| What’s being tested | Deterministic code logic | Probabilistic output behavior |
| Pass/fail signal | Binary: matches expected result or not | Relative: compares against a rolling behavioral baseline |
| Typical failure shape | Sudden break, visible exception | Gradual decay, silent score erosion |
| What can hide the problem | Rarely hides — errors surface | Aggregate scores masking localized quality loss |
| Core question asked | "Did the test pass?" | "Did behavior stay within an acceptable range compared to production and cost/latency norms?" |
How the pieces fit together
None of these gates work well in isolation — a team that only runs baseline evals is still exposed to slow drift; a team that only watches for drift but never validates against real traffic is still guessing at the distribution mismatch problem. The value comes from running them in sequence, each one closing a gap the previous one leaves open.
flowchart TD A[Baseline eval suite] --> B[Drift check vs rolling baseline] B --> C[Shadow / mirror validation] C --> D[Canary: limited live rollout] D --> E[Runtime cost & latency budget] E --> F[Full rollout + ongoing monitoring]
The loop doesn’t end at rollout. One recurring theme across current guidance is that production failures should feed back into the test suite itself: when a live trace reveals a real failure, it gets labeled and turned into a new regression test, so the same mistake can’t slip through unnoticed twice. That habit — sometimes described as a weekly ritual of reviewing the worst-scoring traces from the past week — is what keeps an evaluation system from going stale as usage patterns change. It also underscores why evaluation itself needs monitoring: a scoring system built a year ago, on last year’s model and last year’s user behavior, may no longer measure anything meaningful today.
What this doesn’t solve
It’s worth being honest about the limits here. None of these gates make an AI system’s behavior fully predictable, and no fixed number — a specific drift percentage, a specific latency ceiling — should be mistaken for a universal industry standard; the right thresholds depend heavily on a product’s risk tolerance and traffic patterns. Automated judge models are useful as a signal, but treating any single judge, or any one vendor’s routing scheme, as the definitive authority tends to backfire — the release policy, not the model, should make the final call. And how much of the rollback decision should be automated versus reviewed by a human remains a genuinely open question, not a solved one.
What these gates do offer is more modest and more useful: they replace a single green checkmark with a running, honest picture of how a system is actually behaving — for real users, at real cost, over real time. In a field where the thing being shipped is a probability distribution rather than a fixed set of instructions, that shift in mindset — from "did it pass" to "how is it behaving" — may matter more than any specific tool built to enforce it.
Sources
- Why traditional CI/CD fails for LLMs (and the release gates we built to fix it)
- Eval Drift: Why Your LLM Evaluations Stop Working (And How to Detect It) | Composo Blog
- Why RAG Systems Fail in Production | DigitalOcean
- LLM Eval with Shadow Traffic and Canary Deployment in 2026
- Real-Time Eval Strategies for LLMs | Latitude
