Multi-Token Prediction (MTP) is a speculative decoding technique designed to accelerate language model inference by generating and verifying multiple tokens in parallel. Rather than generating tokens sequentially—the standard approach in autoregressive language models—MTP drafts multiple candidate tokens simultaneously and validates them against a target model, achieving significant speedup in decoding operations while maintaining output quality.

Multi-Token Prediction operates as a variant of speculative decoding, a category of inference optimization techniques that address the latency bottleneck inherent in autoregressive text generation. In conventional autoregressive decoding, each token generation step requires a full forward pass through the model, creating a sequential dependency chain that prevents parallelization across token positions.

MTP mitigates this bottleneck by employing a lightweight draft model to propose multiple candidate tokens in parallel. These draft tokens are then verified against a target model—typically the larger, higher-quality base model—in a single batched verification pass. Tokens that match the target model's predictions are accepted and advanced to the output sequence, while rejected tokens trigger resampling. This approach transforms a sequential bottleneck into a batch verification problem, enabling parallelization gains. Speculative decoding as an architectural approach decouples token generation from verification to improve inference efficiency, effectively utilizing idle compute resources ¹⁾.

The drafters used in speculative decoding are specialized models that predict several future tokens at once in less time than the target model requires to process a single token, enabling parallel verification and significant latency improvements ²⁾.

The technique typically achieves 2-3× speedup in decoding latency while preserving the output quality and behavior of the target model ³⁾—a critical requirement for production deployments where output consistency is essential. Speculative decoding, the foundation for MTP implementations, uses a smaller draft model to propose multiple tokens that are then verified by a larger target model in parallel ⁴⁾.

MTP has been integrated into multiple inference optimization frameworks and language models. Gemma 4 incorporates MTP drafters as a core component of its inference architecture, enabling efficient deployment across various hardware configurations. Support has been extended to popular open-source inference libraries including llama.cpp and vLLM, democratizing access to this optimization technique for practitioners running locally or in custom deployment scenarios. Tooling integration across these platforms has demonstrated 2× token-generation throughput improvements, making speculative decoding a practical optimization layer for production deployments ⁵⁾.

The implementation in these frameworks typically abstracts away complexity from users, allowing MTP to function as a transparent optimization layer that reduces latency without requiring changes to model architecture or downstream applications. This accessibility has made MTP a practical tool for scaling inference across edge devices, consumer hardware, and cloud infrastructure.

The primary advantage of MTP is its substantial reduction in time-to-first-token and overall decoding latency. By enabling parallel token generation and batch verification, MTP addresses a fundamental constraint in autoregressive models: the sequential nature of token generation. For applications requiring low-latency responses—such as real-time chatbots, interactive systems, and streaming applications—this speedup represents a significant practical improvement.

Additionally, MTP preserves the quality and distributional properties of the target model's outputs. Unlike some inference optimizations that trade accuracy for speed, MTP maintains fidelity because rejected draft tokens simply trigger resampling from the target model's distribution. This characteristic makes MTP particularly suitable for applications where output quality cannot be compromised. MTP differs from alternative speculative decoding methods such as EAGLE-3, DFlash, DTree, and N-gram approaches, which vary in their draft model requirements, context reuse strategies, and suitability for different architectures including dense and mixture-of-experts (MoE) models ⁶⁾.

While MTP offers substantial practical benefits, ongoing research explores extensions and improvements to the core technique. The effectiveness of MTP depends on the quality of the draft model and the alignment between draft and target model distributions. Misalignment can lead to high rejection rates, reducing the practical speedup achieved ⁷⁾.

Recent work investigates adaptive draft model sizing, ensemble draft strategies, and hybrid approaches that combine MTP with other inference optimizations such as quantization and KV-cache optimization ⁸⁾. These directions aim to further improve the practical speedup while expanding MTP's applicability to longer-context generation scenarios.

• DTree
• N-gram Speculative Decoding
• Gemma 4 Multi-Token Prediction Drafters
• DFlash
• Qwen3 Models