Designing Real-time AI Interaction Models

The Bottleneck of Traditional AI Interaction

Traditional AI interaction models, like those powering ChatGPT, operate on a turn-based system. A language model waits for a complete input (user finishes typing/speaking), processes it, and then generates a response. During response generation, its perception of new input freezes. This creates a significant bandwidth bottleneck, hindering true human-AI collaboration which is often messy, interruptive, and requires mid-stream corrections. The model forces the human to adapt to its turn-based perception rather than supporting natural, fluid conversation.

Limitations of the 'Harness' Pattern

Many 'real-time' voice AI systems today employ a 'harness' pattern. This involves a stack of simpler helper components (e.g., voice activity detection, speech-to-text, text-to-speech, dialog manager) orchestrating around a core turn-based language model to *simulate* real-time interaction. While providing acceptable latency, this approach has a ceiling because the helpers are significantly less capable than the main language model. They operate on limited signals (e.g., acoustic) and cannot perform complex, context-aware actions like proactive interjections or visual reactions, which require the language model's deep understanding. This represents a hand-crafted heuristic approach, which, as per Rich Sutton's essay, is often outperformed by methods leveraging general computation and learning.

Thinking Machines' Interaction Model Architecture

Thinking Machines proposes an 'interaction model' that integrates interactivity directly into the model's core design. Their first version, TML-Interaction-Small, is a large mixture-of-experts model designed for continuous audio and video input, prioritizing real-time constraints. Key architectural design choices include:

• Time-aligned Micro-turns: Instead of discrete conversational turns, the model processes continuous time in 200-millisecond chunks (micro-turns). Every micro-turn, it takes in multimodal input and decides on output, enabling concurrent speaking while listening, or watching while speaking.
• Lightweight Encoders: Skipping heavy pre-trained encoders, audio and video go through lightweight components trained from scratch, optimizing for the tight real-time constraints.
• Two-Model Coordination: A fast, present 'interaction model' handles real-time conversation, while a slower 'background model' handles deeper reasoning, tool use, and longer-horizon tasks. They share context, and the interaction model weaves asynchronous results from the background model into the fluid conversation.