Dev.to #systemdesign·October 20, 2026

Designing an Arbitration Layer for Routing Requests to Specialized AI Models

This article discusses the architectural challenges and trade-offs of using an arbitration layer to route requests across multiple smaller, specialized AI models instead of a single generalist model. It highlights the complexities of managing permissions, sequencing multi-step tasks as DAGs, and the hidden coordination costs, offering insights into building robust AI-driven systems with modular components.

AI & ML Infrastructure Distributed Systems API Design

Read original on Dev.to #systemdesign

When moving from monolithic generalist AI models to an ensemble of smaller, specialized models, a critical architectural component emerges: the arbitration layer. This layer is responsible for intelligently routing incoming requests to the appropriate specialist models, managing dependencies, and ensuring compliance. While generalist models simplify the internal workflow, specialized models introduce a coordination problem that requires significant engineering effort to solve effectively.

Permission-First Arbitration

A common mistake when designing a router for AI specialists is to treat it solely as a classifier, prioritizing model quality or accuracy first. However, the most crucial initial step is to determine which models are permitted to access the input data and perform the requested action. This is a security and compliance concern that must precede any quality-based ranking. Separating permission checks from quality ranking prevents scenarios where a high-quality but unauthorized model processes sensitive data.

💡

Design Principle: Explicit Permissioning

Implement permission filtering as a distinct, primary step in the arbitration pipeline. Do not rely on models to self-regulate or assume permission implicitly. Explicitly define and check the data classes and actions each specialist model is authorized to handle before considering its suitability based on intent or quality.

Handling Multi-Step Requests with DAGs

Real-world requests often involve multiple, dependent tasks. For example, 'summarize, flag for review, and draft a reply' is a sequence where each step builds on the previous one. The arbitration layer must be capable of decomposing these complex requests into sub-tasks and dynamically building a Directed Acyclic Graph (DAG) of dependencies. Each sub-task then declares its input needs and required permissions, allowing the system to commit to an execution order only after valid specialists are identified for all nodes. This modularity allows for independent updates and retraining of individual specialists without affecting the entire system.

typescript

interface Specialist {
  id: string
  domain: string // e.g. "clause-extraction", "pii-redaction", "drafting"
  permissions: string[] // data classes / actions this model may touch
  costTier: "low" | "medium" | "high"
}

interface SubTask {
  id: string
  intent: string
  dependsOn: string[] // ids of upstream nodes this needs output from
  requiredPermissions: string[]
}

interface ArbitrationResult {
  subTaskId: string
  eligible: Specialist[] // passed the permission check
  chosen: Specialist | null
  reason: "chosen" | "no_eligible_specialist" | "generalist_fallback"
}

function arbitrate(task: SubTask, registry: Specialist[]): ArbitrationResult {
  const eligible = registry.filter(s => task.requiredPermissions.every(p => s.permissions.includes(p)) )
  if (eligible.length === 0) {
    return { subTaskId: task.id, eligible, chosen: null, reason: "no_eligible_specialist" }
  }
  const ranked = eligible
    .filter(s => s.domain === task.intent)
    .sort((a, b) => costWeight(a.costTier) - costWeight(b.costTier))
  return { 
    subTaskId: task.id, 
    eligible, 
    chosen: ranked[0] ?? null, 
    reason: ranked[0] ? "chosen" : "generalist_fallback",
  }
}

Trade-offs: Modularity vs. Coordination Overhead

While specialized models offer benefits like easier auditing, independent updates, and cost optimization (using cheaper models for common cases), they introduce significant coordination overhead. Each hop between specialists involves network calls, permission checks, and schema translation, potentially adding hundreds of milliseconds per step. A generalist model avoids this "coordination tax." Architects must weigh the benefits of modularity and audibility against the increased latency and complexity of managing an arbitration layer, including its own logging, testing, and failure modes.

AI architectureLLM routingmicroservicespermission managementdynamic task graphsarbitration layersystem design patternsorchestration

Comments

Loading comments...

Architecture Design

Design this yourself

Design an arbitration layer for a system that routes incoming AI inference requests to multiple specialized models. The layer must support permission-first routing, dynamic decomposition of complex requests into directed acyclic graphs (DAGs) of sub-tasks, and efficient execution while managing coordination overhead. Detail the data structures for specialists and sub-tasks, the arbitration logic, and considerations for error handling and observability.

Practice Interview

Focus: arbitration layer for routing requests to specialized AI models

Other design angles

· Design a simplified version of the arbitration layer focused only on single-task routing with strict permission enforcement.· Design an arbitration layer that prioritizes cost optimization, dynamically choosing specialists based on real-time cost and availability, while still meeting latency SLOs.· Extend the arbitration layer to include dynamic retries, fallback mechanisms (e.g., to a generalist model), and circuit breakers for specialist model failures.