Mamba Explained: A State-Space Model Alternative to Transformers for Million‑Token Contexts

TL;DR

• Mamba replaces Transformer attention with a State Space Model (SSM) for inter-token communication. The Gradient
• It achieves linear scaling in sequence length, enabling long contexts up to around 1 million tokens. The Gradient
• Inference can be up to 5x faster than Transformer-based equivalents, with competitive language performance. The Gradient
• The Mamba-3B model reportedly outperforms Transformers of the same size and matches Transformers twice its size in pretraining and downstream evaluation. The Gradient
• The approach uses a discrete-time SSM with Zero-Order Hold (ZOH) discretisation and Mamba blocks built from A, B, C, D matrices, while preserving traditional MLP-like computation and local convolutions. The Gradient

Context and background

Transformers have been the dominant paradigm in recent AI breakthroughs, largely thanks to attention mechanisms that let tokens look back over long histories. A core limitation of this approach is the quadratic bottleneck in attention: during training, the forward pass scales roughly as O(n^2) with context length n, and autoregressive generation scales at O(n) per token. This makes very long contexts expensive in time and memory, leading to concerns about practical limits on context window size. In the article describing Mamba, the authors argue that to push beyond these limits, one must substitute the attention-based communication channel with an alternative mechanism, while retaining the efficient, non-attention-based computation path (MLPs and similar projections). The Gradient Mamba is positioned within the broader class of models known as State Space Models (SSMs). Rather than using dot-product attention to mediate token-to-token communication, Mamba employs an SSM as the communication backbone and keeps MLP-style projections for computation. This design aims to deliver fast inference, linear scaling in sequence length, and competitive performance across modalities such as language, audio, and genomics. The authors report that Mamba’s language-modeling results—specifically a 3B-parameter variant—outperform Transformers of the same size and match Transformers roughly twice its size in both pretraining and downstream evaluation. The Gradient What makes this approach noteworthy is the framing of time and state: the hidden state h carries the world’s relevant information, and new observations x update the state through a state-space dynamic. This is formalized with continuous-time dynamics that are discretised for practical, discrete-time computation. The paper emphasizes that, in theory, once you have a compact and informative state, you can predict the next action through a compact representation, avoiding the need to retain all past tokens. The Gradient

What’s new

• Mamba replaces the traditional attention mechanism with a Control Theory–inspired State Space Model (SSM) to mediate inter-token communication within a stacked Mamba-block architecture. This substitution aims to retain the benefits of attention (effective communication) while removing its quadratic bottleneck. The Gradient
• To enable discrete-time operation, Mamba uses a discretisation strategy known as Zero-Order Hold (ZOH), transforming continuous-time state dynamics into a practical discrete-time update. This yields a forward path that mirrors RNN-like recurrence while leveraging an SSM for communication. The Gradient
• The approach demonstrates linear scaling in sequence length, with performance claims extending to contexts as long as 1 million tokens. The authors report that Mamba runs faster than Transformer-based models by up to roughly 5x in suitable configurations. The Gradient
• The Mamba-3B model is cited as outperforming Transformers of the same size and matching Transformers twice its size on language modeling benchmarks, both during pretraining and downstream evaluation. The Gradient
• The Mamba framework retains traditional computational components (MLP-style projections, nonlinearities, and local convolutions) while reorganising how communication happens across tokens. This positions Mamba as a potential backbone for diverse modalities beyond language. The Gradient

Why it matters (impact for developers/enterprises)

If Mamba’s claims hold broadly, long-context language models could become more practical for real-world applications: chat systems that remember long histories, document-level reasoning, and multi-modal pipelines that require extended context windows. Linear scaling in sequence length reduces memory pressure and can shorten inference times, potentially lowering operational costs and enabling deployment on hardware with tighter memory constraints. The prospect of achieving competitive performance at very long context lengths also broadens the design space for building agents, assistants, and AI services that rely on long-range dependencies. While still early, the Mamba approach represents a concrete alternative path in the ongoing exploration of scalable sequence models. The Gradient

Technical details or Implementation

• The core substitution is: instead of the Transformer’s Attention (communication) module, Mamba uses a State Space Model (SSM) for communication, while keeping MLP-style projection-based computation. This mirrors the broad architecture of stacked blocks, with the SSM occupying the communication role. The Gradient
• The SSM formulation centers on a hidden state h, its evolution h’(t) = A h(t) + B x(t), and an output equation y(t) = C h(t) + D x(t). The state captures the essential information needed to predict the next token, and the observation x updates the state via a linear transformation. The Gradient
• To apply these dynamics in discrete time, Mamba uses a discretisation step Δ with the Zero-Order Hold (ZOH) method. The discrete update for the hidden state becomes h_{t+1} ≈ (I + Δ A) h_t + (Δ B) x_t, enabling stepwise progression through tokens akin to RNN-like recurrence but with a state-space communication backbone. The Gradient
• The A, B, C, D matrices define how the hidden state evolves and how observations are translated into outputs, making the SSM a direct, mechanistic substitute for attention in the Mamba block. The remaining backbone computation—nonlinearities, MLP projections, and local convolutions—remains in the computational path. The Gradient
• The discussion contrasts the efficiency of attention with the compactness of recurrent-like state representations. Transformers provide broad recall across long contexts but are hampered by quadratic memory/time costs; Mamba seeks a Pareto-frontier approach that is closer to the efficiency of RNNs while offering strong performance. The Gradient

Key takeaways

• Mamba introduces a State Space Model as the communication backbone to replace attention in transformers. The Gradient
• It targets linear scaling in context length, enabling practical long-context usage up to about 1 million tokens. The Gradient
• Discretisation via Zero-Order Hold is central to translating continuous-time SSM dynamics into discrete-time updates used for token-by-token processing. The Gradient
• Mamba-3B is reported to outperform Transformers of the same size and match larger Transformers in language modeling benchmarks. The Gradient
• The approach preserves MLP-based computation paths while replacing the communication mechanism, offering a novel design space for scalable sequence modeling. The Gradient

References

• The Gradient