Routing One Request Across Many Small Models: What the Demo Doesn't Show You

Everyone shows the same demo. A request comes in, it hits a router, the router picks a small specialist model instead of a giant generalist one, and the slide moves on to the next bullet point about cost savings. I've spent most of this year living in the part the demo skips: the arbitration layer that has to decide, for every request, which specialist, if any, is even allowed to touch it, and how the pieces of a multi-step request get stitched back together afterwards.

A single big generalist model doesn't need any of this. It has one context window, one set of weights, and whatever access it's already been granted. The moment you split the work across several smaller, narrower models, you've created a coordination problem that didn't exist before. That problem, not the models themselves, is where most of the engineering effort actually goes.

Arbitration is a permissions question before it's a quality question

The instinct when you first build a router is to treat it like a classifier: look at the request, work out its intent, pick the model that's best at that intent. That's backwards. The first question has to be which models are even permitted to see this particular input and take this particular action, and only then do you ask which of the permitted ones is best.

Keeping these as two separate checks matters. A contract clause extractor might be the best tool in the building for parsing a lease, but if the request includes a field tagged as personal data and that specialist's permission set doesn't cover personal data, it never gets to compete on quality. It's excluded before ranking starts, not after. With three specialists you can eyeball the permission matrix by hand. With thirty, the exclusion check has to be a first-class step in the pipeline, not something a prompt gently asks the model to respect.

A single request is rarely a single task

Once you accept that requests get split across specialists, you notice most real requests were never one task to begin with. "Summarise this contract, flag anything that needs legal review, and draft a holding reply" is three jobs with a dependency order: the flagging step needs the summary, the drafting step needs to know what got flagged. That's a small directed acyclic graph, not a single function call.

We don't author these graphs by hand per request type. The arbitration layer builds the DAG dynamically: the incoming request gets decomposed into sub-tasks, each sub-task declares what it needs as input and what permissions it requires, and the graph only commits to an execution order once every node has a specialist it's actually allowed to run.

What the interface actually looks like

Stripped down to the part that matters, the arbitration interface is smaller than people expect:

interface Specialist {
  id: string
  domain: string          // e.g. "clause-extraction", "pii-redaction", "drafting"
  permissions: string[]   // data classes / actions this model may touch
  costTier: "low" | "medium" | "high"
}

interface SubTask {
  id: string
  intent: string
  dependsOn: string[]        // ids of upstream nodes this needs output from
  requiredPermissions: string[]
}

function arbitrate(task: SubTask, registry: Specialist[]): ArbitrationResult {
  const eligible = registry.filter(s =>
    task.requiredPermissions.every(p => s.permissions.includes(p))
  )
  if (eligible.length === 0) {
    return { subTaskId: task.id, eligible, chosen: null, reason: "no_eligible_specialist" }
  }
  const ranked = eligible
    .filter(s => s.domain === task.intent)
    .sort((a, b) => costWeight(a.costTier) - costWeight(b.costTier))
  return { subTaskId: task.id, eligible, chosen: ranked[0] ?? null, reason: ranked[0] ? "chosen" : "generalist_fallback" }
}

The detail worth noticing is the order of operations inside arbitrate(). Permission filtering happens first and narrows the field before any quality ranking runs. Flip that order, rank everything by quality first and check permissions second, and you'll eventually ship a request where the best model got chosen before anyone checked whether it should have seen the input at all.

The tradeoff nobody puts in the demo

The case for many small specialists is real. Each one is narrow enough to audit properly, you can point at a specific model and say exactly what data classes it's allowed near. Each one is independently updatable, a regulatory change in one domain means retraining one node, not re-evaluating a monolith's behaviour across every domain it touches.

None of that is free. Every hop between specialists is a network call, a permission check, and a schema translation between what one model outputs and what the next one expects as input. A four-node DAG can easily spend close to a second just coordinating, before any model has done its actual work. A single generalist model never pays that coordination tax because it never had to hand off to anyone. You're trading a hard-to-audit black box for several auditable ones, plus a new black box sitting in front of them, the router and arbitration layer itself, which now needs its own logging, its own test suite, and its own failure modes worked out.