Architecting Hybrid Systems: Balancing Deterministic Software and Probabilistic AI
This article explores the architectural considerations and trade-offs when integrating deterministic software with probabilistic AI, specifically LLMs. It highlights the challenges of context management and the often-overlooked computational costs of probabilistic intelligence, advocating for a design philosophy that strategically assigns tasks based on whether they require cheap, predictable procedures or expensive, adaptive judgment.
Read original on Dev.to #systemdesignThe article introduces a hybrid system design paradigm where deterministic software handles well-defined, repeatable tasks, while probabilistic intelligence (like LLMs) manages tasks requiring judgment, interpretation, and adaptation. The core idea is to combine these strengths to create systems capable of performing complex operations previously requiring human intervention.
Challenges with Probabilistic Intelligence at Scale
As the complexity of AI-driven systems grows, several architectural challenges emerge. The primary one is context management. Initially, the instinct is to provide more context to LLMs, but simply making context available doesn't guarantee correct usage. The author experimented with various techniques to improve context handling:
- Task-driven development: Making work explicit and bounded for agents.
- Externalizing context: Building an MCP server to query conversation history instead of stuffing everything into a single context window.
- Sub-agents: Delegating specific jobs to specialized agents to distribute tasks and manage complexity.
- Optimized inter-agent communication: Writing detailed results to disk and passing only small summaries between agents to reduce context size and inference costs.
The Hidden Cost of Autonomy and Inference
Computational Work of Understanding
A crucial realization is that understanding information is itself computational work. Agents require significant processing to acquire, understand, decide upon, execute, inspect, and respond to information, especially when autonomous. This work is multiplied with sub-agents.
The article emphasizes that the probabilistic part (LLM inference) is not free. LLMs generate responses incrementally, meaning more thought, communication, and investigation translates directly into more token generation and higher inference costs. This often leads to unexpected depletion of token allowances, making the economic aspect of AI visible.
Architectural Principle: Cheap Certainty + Expensive Uncertainty
The core architectural principle derived is to leverage the strengths of each component:
- Cheap Certainty (Deterministic Software): Ideal for known procedures, transformations, structured data processing, and repeatable tasks. It's predictable, testable, and fast.
- Expensive Uncertainty (Probabilistic Intelligence): Valuable when procedures are unknown, requiring judgment, interpretation, or generative capabilities (e.g., summarizing, choosing, brainstorming).
- Optimization: If a probabilistic system discovers a repeatable procedure, architecturally, it should be converted into deterministic software to reduce recurring inference costs.
This leads to a mental model: Cheap certainty + expensive uncertainty = powerful system, with the critical asterisk: subject to the cost of state reconstruction and inference. The key engineering problem becomes identifying the optimal boundary between these two types of intelligence, rather than indiscriminately applying AI everywhere.