Thinking Machines Unveils Groundbreaking AI Interaction Models for Real-Time Voice and Video

In a significant leap forward for artificial intelligence, Thinking Machines has announced a research preview of what it calls "interaction models"—a new class of multimodal systems designed to enable near-real-time, fluid conversations across voice, video, and text. Unlike current AI that relies on turn-based exchanges, this innovative approach treats interactivity as a core part of the model architecture, allowing it to simultaneously process input and output in 200-millisecond micro-turns. The result is a system that can listen, talk, and see in real time, opening up possibilities for more natural human-AI collaboration. Below, we explore the key aspects of this breakthrough.

What is the current limitation of AI interaction that Thinking Machines aims to solve?

Today's AI models operate on a turn-based paradigm: the user provides an input (text, image, or voice), waits for the model to process it (which can take milliseconds to hours), and then receives a response. This forces humans to adapt to the machine's rhythm, often phrasing their inputs in awkward, batch-like chunks. For tasks requiring natural interaction—like a real-time conversation or collaborative problem-solving—this delay and lack of fluidity create a "collaboration bottleneck." Thinking Machines recognized that for AI to truly assist in dynamic environments, it must respond more like a human: listening while speaking, and processing while generating, without freezing perception during output.

What are "interaction models" and how do they differ from standard AI models?

Interaction models are a new class of native multimodal systems where interactivity is baked into the model architecture itself, rather than being an external software layer. Standard models treat input and output as separate, sequential steps—like a chatbot waiting for a query before replying. In contrast, Thinking Machines' approach uses a multi-stream, micro-turn design that processes input and output simultaneously in 200ms chunks. This enables the model to engage in full-duplex communication: it can talk while the user is still speaking, backchannel with acknowledgments, or interject based on visual cues. The result is a more natural, human-like interaction that adapts to the flow of conversation rather than requiring strict turn-taking.

How does the full-duplex architecture work in Thinking Machines' new models?

The full-duplex architecture replaces the standard alternating token sequence with a simultaneous processing system. Instead of waiting for a complete user input before generating a response, the model operates in micro-turns of 200 milliseconds. These short processing cycles allow the system to continuously handle both incoming and outgoing information. For example, while a user is speaking, the model can simultaneously process the speech, detect visual cues from a camera feed, and produce vocal acknowledgments or questions. This eliminates the "dead time" where current models pause their perception during response generation, enabling real-time, back-and-forth interaction similar to human conversation.

What technical innovations enable near-real-time voice and video conversation?

Thinking Machines achieved this breakthrough through encoder-free early fusion. Rather than relying on large, standalone encoders like Whisper for audio or separate vision models, their system takes raw audio signals (as dMel spectrograms) and image patches (40x40 pixels) directly into a lightweight embedding layer. All components—audio, visual, and language processing—are co-trained from scratch, ensuring seamless integration. This approach significantly reduces latency because there's no need to convert audio to text or images to descriptions before reasoning. The model processes multimodal inputs as a unified stream, allowing it to react to a user's tone of voice, facial expression, or background events in real time.

When will these models be available to the public?

Currently, the interaction models are only in the research preview stage. Thinking Machines plans to open a limited research preview in the coming months to gather feedback from select users. A wider release is expected later this year, though the company has not specified a precise timeline. Initially, access will be restricted to researchers and developers, with enterprise and general public availability following after further testing. This cautious rollout aims to refine the model's performance and safety before it becomes widely accessible.

What are the potential applications of such natural interaction AI?

The ability to conduct real-time, fluid conversations with AI opens up numerous applications. In customer service, an AI could handle complex inquiries with natural back-and-forth, improving satisfaction. In education, it could act as a tutor that responds to a student's immediate questions and body language. For creative work, designers could verbally iterate on visual content while the AI updates a canvas in real time. Telepresence and virtual meetings could benefit from an AI participant that listens, interjects, and reacts naturally. Even in healthcare, a system that perceives verbal and nonverbal cues could assist in diagnostics or therapy. Essentially, any scenario where seamless human-AI collaboration is desired could be transformed.

How does the new system handle simultaneous input from multiple modalities?

The interaction models use a multi-stream processing design that ingests audio, video, and text streams concurrently. Each 200ms micro-turn, the model receives chunks of raw audio (dMel), visual patches, and any text tokens, all fused at an early stage through a shared embedding space. This allows the model to prioritize and integrate information across modalities without bottlenecks. For instance, if a user speaks while writing on a whiteboard, the model can simultaneously process the speech and the visual changes, responding with both verbal feedback and visual annotations. The system is designed to handle overlapping inputs naturally, much like a human would manage multiple conversation threads.