Smart Category Routing for Contract Review

Contract review is a domain where the LLMOps routing pattern earns its complexity. A typical contract has 50-100 chunks. Each chunk is potentially relevant to one or two of 11 specialist analysts (compliance, indemnity, IP, payment, termination, etc.). Running every analyst on every chunk is 1,100+ LLM calls. Routing cuts that to ~250.

Playbook-driven configuration

Each customer has a playbook in S3 — categories that matter to them, severity weights, and party-specific clauses (one customer cares deeply about IP indemnity; another cares about data-residency clauses). At job start, the system loads the playbook and configures analysts accordingly.

• Categories: 12 standard, 1-3 customer-specific
• Severity weights: how much to escalate findings in each category
• Party-specific clauses: customer-defined patterns the analyst should specifically look for

Per-chunk scoring

Each chunk runs through 12 category scorers (cheap model, $0.25/M tokens). Each scorer emits a 0-100 score for "is this chunk relevant to my category?" The router selects analysts to run based on the scores: above-threshold categories trigger their analyst; below-threshold categories skip.

HyDE-augmented retrieval

HyDE runs at ingestion, per playbook term. The enrichment agent generates a hypothetical ideal answer for each term — search-optimized text embedded alongside the term itself — so retrieval matches contract chunks against what a relevant clause would say, not against the term's literal wording. Better recall than embedding the label alone, especially when the playbook uses legal jargon and the contract uses plain English (or vice versa).

LLM reranking after HyDE

Vector search returns the top ten candidate chunks per playbook term; an LLM reranker scores each candidate and narrows to the three most relevant, which are deduplicated before anything reaches an analyst. The wide first pass is allowed to be cheap and loose because the reranker pays the precision bill.

Numbers from production

• 22 chunks → 116 LLM calls per chunk (12 scorers + ~3 routed analysts × ~30 LLM calls each) = ~660 calls per chunk on the misleading top-line
• Actually: 22 chunks × 12 scorers + ~3 selected analysts per chunk × 8 calls = 264 + 528 = ~800 calls per contract typical
• P50 review time: 154 seconds end-to-end
• Per-contract cost: $0.50-2.00 depending on contract length and customer playbook
• Reduction vs. naive fan-out: ~75%

Where this fails

Customer playbooks with overlapping categories (the "compliance" category overlaps with "regulatory" and "data-handling" 70% of the time). Routing collapses to "everyone." Mitigation: routing analytics dashboard shows per-category overlap rates; surfaces the problem; encourages playbook tightening.