Identify Speakers in Real Time with NVIDIA Streaming Sortformer

TL;DR

• NVIDIA Streaming Sortformer is an open, production-grade diarization model designed for low-latency, real-time multi-speaker scenarios.
• It sorts speakers by their arrival order, maintains consistent labels across a live stream using an Arrival-Order Speaker Cache (AOSC), and can be integrated into transcription pipelines and voice apps.
• The model uses a convolutional pre-encode module plus conformer and transformer blocks and processes audio in small, overlapping chunks for live use.

Context and background

In every meeting, call, crowded room, or voice-enabled app, a core technical question is simple: who is speaking, and when? For decades, answering that question in real-time transcription was almost impossible without specialized equipment or offline batch processing. Real-time diarization—identifying speaker boundaries and assigning persistent labels during a live stream—has been constrained by latency, robustness to overlapping speech, and the need to maintain consistent speaker identities over time. NVIDIA introduces Streaming Sortformer as a response to these challenges, positioning it as an open, production-ready solution intended for realistic, multi-speaker environments. The model is designed to be dropped into live transcription pipelines, voicebot orchestration, and enterprise meeting analytics.

What’s new

Streaming Sortformer brings a set of capabilities aimed specifically at making real-time diarization practical:

• Open and production-grade: built for deployment in real applications rather than just research experiments.
• Low latency: optimized to operate in realistic, live conditions where timely speaker labels matter.
• Arrival-order sorting: the model uniquely sorts speakers based on when they first appear in the audio stream, enabling consistent labeling.
• Integration-ready: made to work with NVIDIA NeMo and NVIDIA Riva and to be deployed in transcription, voicebot, and analytics pipelines. These characteristics make the model suitable for a broad set of real-time multi-speaker scenarios, from remote meeting transcription to voice-enabled applications that must manage multiple participants.

Why it matters (impact for developers/enterprises)

Real-time diarization unlocks capabilities that matter to both developers and enterprises:

• Improved meeting analytics: enterprises can create transcripts with speaker-attributed content in real time, enabling live summaries, role-based insights, or action-item extraction during a call.
• Better live voicebots and orchestration: voice services that must detect turns and address the right participant can use real-time speaker IDs to route requests and preserve conversational context.
• Reduced infrastructure complexity: by avoiding offline batch processing or specialized capture hardware, organizations can deploy diarization directly in streaming pipelines.
• Faster product iteration: having an open, production-grade model lets teams prototype and iterate quickly without building diarization from scratch. Because Streaming Sortformer is designed to operate with low latency and to maintain stable speaker labels as participants join and speak, it supports practical deployments where immediate speaker information is required.

Technical details or Implementation

Streaming Sortformer is architected to handle live audio and to keep speaker labels consistent over time. Key implementation points described by NVIDIA include:

• Encoder pipeline: the model starts with a convolutional pre-encode module that processes and compresses raw audio before feeding subsequent blocks.
• Conformer and transformer blocks: a series of conformer and transformer layers analyze conversational context and enable the sorting behavior.
• Arrival-Order Speaker Cache (AOSC): a memory buffer that tracks all speakers previously detected in the audio stream. The AOSC lets the model compare current-chunk speakers with previous speakers so that the same person receives the same label across the live stream.
• Chunked processing: to handle continuous live audio, Streaming Sortformer processes sound in small, overlapping chunks. This supports low-latency output while preserving temporal continuity and enabling comparisons across chunks. These elements work together so the model can sort speakers by when they first appear and then maintain speaker identity as the conversation continues. NVIDIA demonstrates arrival-order behavior with multi-speaker examples (e.g., three- and four-speaker scenarios) to show how the AOSC retains speaker ordering and labels across a live stream. Comparison of key model components and their roles:

Key takeaways

• Streaming Sortformer provides an open, production-ready option for real-time speaker diarization.
• It uniquely sorts speakers by arrival order and keeps labels consistent using an Arrival-Order Speaker Cache.
• The architecture combines convolutional pre-encoding with conformer and transformer blocks, and it processes small overlapping audio chunks for low latency.
• The model integrates with NVIDIA NeMo and NVIDIA Riva and is intended for transcription pipelines, voicebot orchestration, and meeting analytics.

References

• Original announcement and details: https://developer.nvidia.com/blog/identify-speakers-in-meetings-calls-and-voice-apps-in-real-time-with-nvidia-streaming-sortformer/
• Offline Sortformer research: available on arXiv (referenced in the original announcement)