Eval suites, judge loops, smart routing, retry/circuit-breaker patterns, and the day-2 ops that keep a production LLM pipeline honest at scale.
The day-2 ops layer of an LLM product — what to evaluate, what to judge in real time, what to retry, and when to fail closed. The components that turn a prototype into something operable.
Ordered by chapter. Each post stands alone but builds on the one before it.
Judge scores below 85? Retry with temperature 0.3, 0.4, 0.5 — three attempts inside a 60-second wall-clock budget. The simple loop that hits 99.5% on-spec output without crossing the latency ceiling.
A pipeline of 12 scorers + 11 analysts does not need to fan out everywhere. Route each chunk to matching analysts and save three quarters of the LLM bill.
A four-agent hiring pipeline is a regulated decision system. Continuous monitoring with alerts at the four-fifths-rule disparity-impact threshold.