Routing by query difficulty

Sending every request to the largest available model is the most expensive way to be wrong.

It is also the default, because it is the only configuration that is obviously defensible. Nobody was ever criticised for using the better model. This note is about what happens when the cost constraint is tight enough that the default is not available to you.

That is our situation and it is not a temporary one. Redrob answers questions for an Indian consumer subscription, in twelve languages, at a price set by the Indian market rather than by what inference costs. The router is the component that makes that arithmetic close: every query is scored for difficulty, and the overwhelming majority resolve on a small model and never touch a large one.

The premise

Requests are not uniformly hard. In any real workload there is a long head of queries that a small model answers as well as a large one, a tail that genuinely needs the large one, and a middle where it depends on things you can partly detect in advance.

If you can classify a request into those bands before generating, you can serve most of the volume from a much smaller model and reserve the expensive path for the requests that need it. The saving is not marginal, because the head is large and the cost gap between model sizes is not linear.

The whole problem is that classifying difficulty before you have the answer is not obviously possible.

What difficulty is not

Two proxies that seemed promising and were not.

Length. Long queries are not harder queries. A long query is often a well-specified one, which makes it easier. Some of the hardest requests we see are six words.

Topic. Difficulty does not partition by subject. Within any topic there are trivial requests and hard ones, and a topic classifier mostly learns which topics attract verbose users.

What difficulty looked like

Three signals carried most of the usable information.

Number of reasoning steps implied by the request. A question that can be answered from one retrieved passage is a different class from one that requires combining two and noticing they disagree. This is partly detectable from the syntax of the request, particularly from comparatives, conditionals and questions that contain their own sub-question.

Whether the answer is likely to be in the retrieval corpus at all. If retrieval returns high-scoring, mutually consistent passages, a small model reading them will usually do fine. If retrieval returns weak or contradictory passages, the generation step is doing the work rather than the retrieval step, and a small model will confabulate. Running retrieval first and routing on its output rather than on the raw query turned out to be the single most useful change.

Whether the request is in a language the small model is weak in. Capability is not uniform across the languages we serve, and the gap between a small and a large model is much wider in some of them than in others. Routing has to be language-conditional or it silently degrades service for exactly the users who are already least well served.

Routing on retrieval output

That second signal deserves expanding, because it inverts the obvious pipeline.

The intuitive order is to classify difficulty, choose a model, then retrieve and generate. The order that worked is to retrieve first, then use the retrieval result as the main input to the routing decision, then generate.

Retrieval is cheap relative to generation, so paying for it before the routing decision costs little. And the retrieval result is a far better predictor of whether generation will be hard than the query is, because it tells you whether the information needed is present and coherent.

In effect the router is not asking "is this question hard". It is asking "given what we found, how much work is left".

The failure mode that matters

Misrouting is not symmetric.

Sending an easy request to the large model wastes money and nobody notices. Sending a hard request to the small model produces a confident wrong answer, and the user has no way to tell.

That asymmetry has to be built into the threshold. The router is deliberately biased towards escalation, and it escalates on uncertainty rather than on a difficulty estimate alone. A router calibrated to maximise average accuracy will happily accept a bad tail; a router calibrated on the cost of the worst outcome will not.

We also allow the small model to abstain. If it generates and its own confidence is low, the request is escalated and regenerated. That costs more than getting it right first time and much less than being wrong.

What we would do differently

We built the first version of the router as a classifier over the query. That was several months spent on the wrong input, and the lesson generalises: when a prediction is hard, check whether something cheap and downstream already contains the answer.

We would also have made it language-conditional from the beginning rather than discovering the need later. A single global threshold is a decision to serve some languages worse than others, whether or not anyone intended it.

Where this sits

The router is one of three things holding the cost line, and on its own it is not the largest. Returning structured results with a summary rather than long-form generation puts a hard ceiling on output tokens, and job queries repeat heavily enough that a large share of traffic never needs recomputing at all.

None of the three is a research breakthrough. They compound, and together they are the difference between a product an Indian consumer can afford and one that is subsidised until the subsidy stops.