NVIDIA's Speech Stack: Parakeet, Canary, Nemotron Speech, and Magpie

The Whisper piece served speech recognition with vLLM, and the TTS piece surveyed speech synthesis. NVIDIA publishes its own family of open speech models, under names that are hard to keep apart: Parakeet, Canary, Nemotron Speech, Sortformer, Magpie. This piece sorts them by job, measures them on the same audio as the Whisper piece, and shows how NVIDIA assembles them into a voice agent.

Five families, five jobs

From each model card, 2026-10-06. All five allow commercial use.

Two more are worth knowing. Canary-Qwen 2.5B puts Canary's audio encoder in front of a Qwen3-1.7B language model, for English transcription that can also be summarized or questioned. NemotronLabs VoiceChat 11B (July 2026) is not a building block at all: it listens and speaks at the same time, in one model (more below).

Why they are fast

Whisper's decoder is an autoregressive transformer that writes one token per step, attending to the whole encoded window each time. NVIDIA's recognizers keep a large encoder, a FastConformer that downsamples audio 8 times to one frame every 80 ms, and replace the decoder with something much lighter:

• Transducers (RNN-T) walk along the encoder frames with a small prediction network, emitting a token or moving to the next frame. The work per output token is tiny compared with a transformer decoder.
• Token-and-duration transducers (TDT), Parakeet's decoder, also predict how many frames to skip after each token, so decoding jumps over silence and the middle of long sounds instead of visiting every frame.
• Cache-aware streaming, in Nemotron 3.5 ASR, trains the encoder to see only a fixed amount of future audio and keeps what it computed for past audio in a cache. Each new chunk is processed once, never re-encoded with overlap, which is what makes a stream cheap.

Measured: against Whisper on the same audio

We ran each model on the 200 LibriSpeech test-clean utterances of the Whisper piece, on the same GPU (an NVIDIA GB10), scored the same way: Parakeet and Canary with NeMo 3.1, NVIDIA's own toolkit, and Nemotron 3.5 ASR with Hugging Face Transformers 5.18, all in batches of 16.

What it shows:

1. Parakeet is the fast transcriber. Slightly more accurate than Whisper on this set, at almost twice the throughput, with a model of 627 million parameters against Whisper's 809 million. It covers 25 European languages, against Whisper's 99.
2. Canary is the accurate, slower one, and the one that translates: 1.51%, the best here, at 41 times real time. Its decoder is a transformer, like Whisper's.
3. Long files depend on the runtime. vLLM cut the 9-minute file into 30-second chunks and ran them in parallel (2.3 s); NeMo gave Parakeet the whole file at once (7.5 s). Both are fast; for batch jobs on long recordings, split them yourself and batch the pieces.
4. Nemotron 3.5 ASR pays a little accuracy for streaming: 3.21% offline. It is built for the next section.

The runtimes differ (vLLM, NeMo, Transformers), so speed compares setups as much as models. None of these NVIDIA models is served by vLLM 0.31.0: its Parakeet code is the audio encoder of a Nemotron vision-language model. In production, NVIDIA serves them through Riva and its NIM containers.

Streaming at four latencies

Nemotron 3.5 ASR's latency is a run-time setting, not a property of the checkpoint: the amount of future audio the encoder may see, in 80 ms frames. We streamed the first 100 utterances chunk by chunk at the four supported settings:

Accuracy barely moves between 80 ms and 1.12 s on clean read speech; harder audio, with noise or accents, is where a longer view helps, as the model card's multilingual results show. The compute is small: about 20 ms of GPU time for each chunk of one stream, so one GPU can carry many streams at once. Compare the Whisper piece, where a streaming-native model kept text half a second behind the voice and Whisper itself needed chunks of seconds.

A practical note from our first run: our loop dropped the last, incomplete chunk of each utterance, and accuracy collapsed as chunks grew (10.6% at 1.12 s), because a longer chunk drops more of the last word. Pad the final chunk with silence and send it.

Who spoke when, in real time

Sortformer labels speakers frame by frame, up to four, and its streaming version keeps a cache of each speaker's voice so labels stay consistent through a long call. On the two-speaker recording of the Whisper piece:

Sortformer matched the one-model route and beat the pipeline, and it did so in streaming mode with a one-second buffer. It produces labels, not text: pair it with Parakeet or Nemotron timestamps, cutting at Sortformer's speaker changes rather than at the recognizer's segments.

Speech out: Magpie

Magpie TTS Multilingual is a 364-million-parameter model with 12 languages and five fixed voices; it does not clone voices, by design. With NeMo's single-utterance call it produced speech at 0.42 to 0.46 seconds per second of audio, similar to Qwen3-TTS in the TTS piece; that call returns the whole utterance, while NVIDIA's NIM container streams.

On the same three texts as the TTS piece, two of three samples were transcribed back word for word. On the third, Whisper and Parakeet both heard "the margin fell from A to E percent" where the text said "from 31 to 28 percent", with Magpie's text normalization turned on. One sentence is not a benchmark, but it is the kind of failure a voice agent reading prices or dates cannot afford: test your own numbers, or normalize them to words before synthesis.

Assembling a voice agent

NVIDIA's voice-agent examples wire these models with an LLM through Pipecat, an open-source framework for real-time voice pipelines: streaming ASR, an LLM, streaming TTS, and the logic that decides when the user has finished speaking. The reply time a user perceives is the sum of what happens after they stop:

With streaming ASR, recognition is not the bottleneck: it finishes inside the wait that decides the user is done, and that wait, a setting, is the largest single item. Shorten it and the agent interrupts people; lengthen it and every reply feels slow. The LLM's first token comes next, which is why the prefix caching of a stable agent prompt matters as much here as in a chat product.

Full duplex removes the cascade. NemotronLabs VoiceChat 11B listens and speaks at the same time, with a speech encoder, a Nemotron Nano 9B language model, and a speech decoder in one network, so it can be interrupted, backchannel, and reply without waiting for an end-of-turn decision. NVIDIA reports about 450 ms of turn-taking latency and tool calling during conversation; we did not measure it. The trade is control: a cascade lets you swap the LLM, inspect the transcript, and enforce what is said, which many products need.

When to choose them

• Batch transcription in European languages: Parakeet. It was the fastest model we ran, at Whisper's accuracy.
• Transcription plus translation, or the best accuracy in those 25 languages: Canary.
• A voice agent: Nemotron 3.5 ASR for streaming recognition, with the latency set per deployment, and Sortformer if you need speakers.
• Speech output with fixed brand voices and a commercial license: Magpie, after testing numbers and names; for cloning or more languages, see the TTS landscape.
• Stay with Whisper for languages outside these lists, for a single engine (vLLM) across LLM and ASR, or when the runtime matters more than the last bit of speed: NVIDIA's models mean NeMo, Riva, or NIM.

Next

This closes the speech series. The pieces that follow turn to the labs behind today's open language models, starting with DeepSeek.