Speculative Decoding to Reduce Latency in AI Inference: EAGLE-3, MTP, and Draft-Target Approaches

TL;DR

• Speculative decoding reduces latency in autoregressive AI inference by proposing multiple next tokens and verifying them in a single forward pass, increasing throughput without sacrificing accuracy.
• The classic draft–target approach uses a small, fast draft model to propose several tokens; the large target model verifies them in batches and keeps the longest accepted prefix.
• EAGLE-3 attaches a lightweight drafting head to the target model itself, extrapolating from hidden states to propose multiple tokens without a separate draft model.
• Multi-Token Prediction (MTP) offers a related approach with dedicated multi-token heads, removing the need for a separate drafting model in some configurations.
• NVIDIA provides paths to apply speculative decoding via the TensorRT-Model Optimizer API, including steps to convert a Hugging Face model to use EAGLE-3. This technique preserves output quality through verification and can significantly boost throughput.

Context and background

Autoregressive generation with large language models (LLMs) is fundamentally sequential: each token typically requires a full forward pass, reloading weights, and memory synchronization. This sequential dependency drives latency, leaves hardware underutilized, and constrains system efficiency. Speculative decoding addresses this bottleneck by running a lightweight draft mechanism in parallel with the target model, proposing several possible continuations and then validating them in a single extended forward pass. Verification ensures that the final output matches what the baseline, high-quality model would generate, preserving accuracy while reducing the number of sequential steps. In this framework, the draft–target method is a two-model system. The target is the large, high-quality model whose output you want to accelerate, and the draft is a smaller, faster model trained on the same data distribution. The two models work in tandem: the draft rapidly proposes several candidate tokens, and the target verifies and decides which tokens to accept, continuing generation from the accepted prefix. The approach aims to maximize acceptance rates—the fraction of draft tokens that the target model accepts—as higher rates translate to greater speedups. The mechanism leverages a cache of key-value (KV) states so that only the new, draft tokens incur computation during verification. A broader family of speculative techniques also exists beyond the draft–target pair, including EAGLE-3 and related methods. This family centers on the idea that you can dramatically reduce the number of sequential steps by offloading part of the drafting work, either to a lighter drafting head attached to the target model (EAGLE) or to specialized multi-token heads (MTP). Crucially, all approaches rely on a verification step that discards any draft results that diverge from what the target model would generate, ensuring the final output is indistinguishable in accuracy from standard autoregressive decoding.

What’s new

A core advancement described in the NVIDIA discussion is EAGLE-3, the third version of Extrapolation Algorithm for Greater Language-Model Efficiency. EAGLE-3 builds on the principles of speculative decoding by operating at the feature level rather than relying on a separate drafting model. Specifically, it attaches a lightweight drafting component—the EAGLE head—to the internal layers of the target model so it can draw from low-, mid-, and high-level feature representations and produce multiple candidate tokens. This approach eliminates the overhead of training and running a second model while still enabling the target model to verify several token candidates in parallel. Key elements of EAGLE-3 include:

• A multi-layer, fused feature representation that feeds a drafting head attached to the target model.
• A context-aware, dynamic draft tree that proposes multiple chained hypotheses, enabling longer, more predictable generation paths where the model is confident.
• Parallel tree attention used by the target model to verify candidate tokens generated by the EAGLE head, pruning invalid branches efficiently.
• An instance-adaptive drafting process that can stop drafting when confidence thresholds are not met, ensuring runtime efficiency. In addition to EAGLE-3, the landscape includes Multi-Token Prediction (MTP). MTP uses multiple specialized token-drafter heads, each predicting a future token, with the main model validating these drafts in order and keeping the longest matching prefix. This style of speculation is conceptually aligned with EAGLE techniques—both aim to produce multiple candidate tokens for verification instead of a strictly single next-token draft. The article also outlines practical deployment paths. You can apply speculative decoding using the NVIDIA TensorRT-Model Optimizer API to convert a model for EAGLE-3-based speculative decoding. The process described includes:
• Step 1: Load the original Hugging Face model.
• Step 2: Import the default EAGLE-3 config and convert it using the mtsp tool. A hands-on tutorial expands this demo into an end‑to‑end speculative decoding fine‑tuning pipeline in the TensorRT-Model-Optimizer GitHub repository. A helpful, intuitive example frames the latency improvement: if a single forward pass (including loading weights and computing a token) takes 200 milliseconds, generating three tokens with standard autoregressive decoding would take 600 milliseconds. Speculative decoding aims to cut that path to results by performing fewer sequential steps through drafting and verification.

Why it matters (impact for developers/enterprises)

For developers building AI-powered products and services, speculative decoding offers a practical route to faster, more responsive inference without sacrificing quality. The approach shakes out two substantial benefits:

• Latency reduction and throughput gains: by generating multiple tokens per forward pass and verifying them efficiently, systems can deliver results faster and process more requests per unit of time.
• Hardware utilization and scalability: speculative decoding helps alleviate memory bandwidth bottlenecks and makes fuller use of GPU compute by avoiding full forward passes for every single token. From an enterprise perspective, these improvements translate into lower end-to-end latency for user-facing AI features, improved service quality under heavy load, and potential cost efficiency through better hardware utilization. Because the verification step discards divergent drafts, there is no loss in the final output quality compared with standard autoregressive generation.

Technical details or Implementation

Draft-target approach (two-model system)

• A smaller, more efficient mechanism generates a sequence of candidate tokens (typically 3 to 12).
• The target model processes the input sequence and all draft tokens in a single forward pass, computing probability distributions for each position.
• Thanks to the KV Cache, only the new, speculated tokens incur a computational cost during verification.
• Rejection sampling provides the decision logic. If P(Draft) is lower than P(Target) for a given token, the draft token and all subsequent tokens are discarded, and the process reverts to standard autoregressive generation from the last accepted token.
• The final output matches what the target model would have produced, because only tokens that survive the acceptance logic are kept.
• The acceptance rate—tokens accepted from the draft relative to total generated—measures the speedup potential.

EAGLE-3 (feature-level extrapolation with an EAGLE head)

• EAGLE-3 attaches a lightweight drafting component to the internal layers of the target model, creating an “EAGLE head.”
• The EAGLE head uses a miniature, streamlined Transformer decoder block followed by a final linear layer and can generate an entire tree of candidate tokens rather than a single token.
• It leverages multi-layer, fused feature representations (low, middle, high) and uses a context-aware dynamic draft tree to propose multiple chained hypotheses.
• The target model performs verification with parallel tree attention to prune invalid branches, improving acceptance rate and throughput.
• The drafting process is instance-adaptive: the head evaluates its own confidence and stops drafting when confidence falls below a threshold, enabling longer branches for simple parts and shorter branches for complex sections.
• Importantly, this approach requires only a forward pass of the target model for verification, not a separate drafting model, which reduces overhead.

Multi-Token Prediction (MTP)

• MTP is a related speculative technique used in several DeepSeek iterations, where the model learns to predict several future tokens at once using specialized multi-token heads.
• Each head acts as a token drafter; the main model then checks those guesses in order and keeps the longest prefix that matches.
• This method removes the separate drafting model in many cases and behaves similarly to EAGLE-style speculative decoding in practice, though the proposal mechanism differs: MTP uses multi-token heads rather than extrapolating internal features.

Implementation notes and steps

• You can apply speculative decoding to your own models using the NVIDIA TensorRT-Model Optimizer API.
• Practical steps described include:
• Step 1: Load the original Hugging Face model.
• Step 2: Import the default EAGLE-3 config and convert it using the mtsp tool.
• NVIDIA provides a hands-on tutorial that expands the demo into an end‑to‑end speculative decoding fine‑tuning pipeline in the TensorRT-Model-Optimizer repository.

A compact performance intuition

The core latency bottleneck in standard autoregressive generation is the fixed, sequential cost of each step. If a single forward pass takes 200 ms, generating three tokens would take 600 ms in a purely sequential regime. Speculative decoding reduces the effective number of sequential steps by generating and validating multiple token candidates in parallel, thereby shrinking the total wall-clock time to results while preserving the final output quality through rigorous verification.

Table: Draft-target vs EAGLE-3 vs MTP (characteristics at a glance)

| Approach | Key Idea | Token proposals per forward pass | Model requirements | Accuracy impact |---|---|---|---|---| | Draft-target | Small draft model proposes tokens; target verifies | Typically 3–12 tokens | Requires training/running a separate draft model | Accuracy preserved via verification |EAGLE-3 | EAGLE head attached to the target, feature-level extrapolation | Multiple tokens via an internal drafting head | No separate draft model; uses target’s internal features | Accuracy preserved via verification |MTP | Multi-token heads propose several tokens | Several tokens from dedicated heads | Requires multi-token prediction heads | Accuracy preserved via verification |

Practical deployment notes

• The EAGLE-3 approach emphasizes integrating a lightweight drafting component into the target model to maximize efficiency while maintaining accuracy.
• The acceptance logic and verification step are central to ensuring that speculative results do not deviate from the baseline model’s output.
• The TensorRT-Model Optimizer API provides a concrete path for practitioners to adapt their Hugging Face models to EAGLE-3 speculative decoding workflows.

Key takeaways

• Speculative decoding accelerates AI inference by enabling the target model to verify multiple token candidates in parallel, reducing sequential steps.
• EAGLE-3 represents a pragmatic evolution by embedding a drafting head within the target model and leveraging internal feature representations to draft several tokens in one forward pass.
• MTP offers an alternative that uses dedicated multi-token heads, potentially removing the need for a separate draft model.
• Acceptance rate and robust verification are critical to ensuring that speedups do not come at the expense of accuracy.
• Deployment is supported by NVIDIA tools, including TensorRT-Model Optimizer API, with example steps to convert a Hugging Face model to use EAGLE-3.

References

• NVIDIA: An Introduction to Speculative Decoding for Reducing Latency in AI Inference. https://developer.nvidia.com/blog/an-introduction-to-speculative-decoding-for-reducing-latency-in-ai-inference/