State of AI Ops: what breaks first when you ship an LLM feature
Three failure modes show up before any others, in roughly this order: a silent prompt regression, retrieval drift nobody measured, and a bill that points at a month instead of a feature.
2026-08-04 — AI Ops
The regression nobody saw
A prompt edit that reads like an improvement can make a narrow slice of traffic worse while the average holds steady. Without a fixed evaluation dataset run against both versions, the first signal is usually a support ticket days later, by which point the release that caused it is one of several candidates.
The fix is boring on purpose: replay recorded traffic through both versions and compare, so the answer to "did this get better" is a diff rather than an impression from a handful of manual tries.
Drift with no number on it
A retrieval index that answered well at launch degrades as the underlying documents change — content gets added, corrected, or goes stale — and nothing about the generation step notices, because it only ever sees whatever the retriever handed it. Recall and precision per query cluster, tracked continuously, are what turn "the answers feel worse lately" into a chart with a date on it.
The bill that points at a month
Provider invoices are itemised by model and by day, not by feature, user, or prompt template — so a cost spike is easy to see and hard to attribute. Cost attributed per request at the point it is spent, rather than reconstructed from an invoice at the end of the month, is the difference between "spend went up" and "this one feature costs more than the other four combined."