When the Model Isn't the Problem: Degradation Accounting in Fallback-Enabled Production LLM Pipelines

Summary

Production LLM pipelines frequently use deterministic fallbacks to preserve availability when model calls fail. This creates a measurement problem: benchmark outputs may remain correct even when the model being evaluated did not produce them. We call the resulting instrumentation requirement degradation accounting: run-level recording of fallback, retry-recovery, salvage, transport, and serving events alongside conventional capability scores.

In a production analysis pipeline, a baseline configuration produced near-perfect answer-key scores while only 4 of 10 repetitions reached all primary model agents without fallback. Instrumentation separated three externally similar but mechanistically distinct failure classes, and a pre-registered budget-and-retry intervention raised the agent-clean rate to 10/10. Component ablations attribute prevention of first-attempt starvation to proactive budget headroom, and one pre-registered prediction was falsified by its own registered threshold, revealing that attempt outcomes are not exchangeable: an unexplained provider-side asymmetry we document rather than build on.