Seq vs Seq: Ettin — Paired Encoders and Decoders Redefine Open-Data LLM Benchmarks

TL;DR

• Ettin introduces the first state-of-the-art paired encoder-only and decoder-only models trained with identical data, model shapes, and training recipes source.
• The suite covers six scales from 17M to 1B parameters for both encoders and decoders source.
• Training follows a three-phase recipe: Pre-training on 1.7T tokens with shorter contexts, Context Extension to 8K tokens on 250B tokens, and a final Decay phase with 100B tokens from premium sources source.
• All training data is public and reproducible, enabling apples-to-apples comparisons; encoders excel at classification and retrieval, while decoders lead in generation source.
• In benchmarks, Ettin models beat or match baselines like Llama 3.2 1B and SmolLM2 on key tasks, with strong gains on knowledge-intensive tasks such as SciQ source.

Context and background

The LLM community has largely converged on decoder-only models (GPT, Llama, Qwen) for generation, while encoder-only models (like BERT) remain central to production tasks such as classification, retrieval, and embeddings. Encoders are typically faster and more memory-efficient for discriminative tasks, but historically lagged in generative capability compared to decoders. Ettin situates itself as a controlled, apples-to-apples comparison by training encoder-only and decoder-only architectures under the same data, the same model shapes, and identical training recipes; the only differences are attention patterns and the training objectives. The project builds on the ModernBERT recipe, which adapted modern techniques from decoder training to encoder training, providing a strong, shared foundation for both architectures. The work is named after the two-headed Norse giant Ettin to reflect its paired-architecture approach. All training data used in Ettin is public and reproducible, reinforcing the emphasis on open science and reproducibility within the community. You can continue to train these models on new data or propose a new recipe to push results further source.

What’s new

Ettin presents a coherent suite of six models for both encoders and decoders, ranging from 17M up to 1B parameters. The same data, model shapes, and training recipes are used across architectures to enable fair comparisons of how attention patterns and objectives shape learning outcomes. Specifically:

• Two training objectives are explored in a controlled setting: masked language modeling (MLM) for encoders and causal language modeling (CLM) for decoders. The same data and recipe are applied to both, allowing apples-to-apples analysis of architecture vs objective.
• The data regime is designed to maximize realism and openness: training data is public and reproducible, enabling users to reproduce results or extend training with new data.
• Ettin’s three-phase training approach mirrors the ModernBERT lineage while extending it to cover longer contexts and higher-quality data filters. Phase 1 pre-trains on a diverse mix of sources with 1.7T tokens and short 1024-token contexts to establish foundational knowledge. Phase 2 extends context length to 8K tokens using higher-quality filtered data. Phase 3 completes with 100B tokens drawn from premium sources (including scientific papers and textbooks) while gradually decaying the learning rate source.
• The encoded and decoded models deliver state-of-the-art performance for open-data models across tasks and sizes. On multiple benchmarks, Ettin encoders outperform ModernBERT, while Ettin decoders match or exceed established baselines such as Llama 3.2 and SmolLM2, with pronounced gains on knowledge-intensive tasks like SciQ.
• The results emphasize fundamental architectural advantages that persist when data, recipes, and scales are controlled: encoders excel at classification and retrieval, while decoders maintain an edge in generation as model size grows. A surprising note is that near-identical data and recipes can reveal performance gaps attributable primarily to architectural choices rather than solely the training objective.
• The project also experiments with switching objectives after initial training (continuing with the opposite objective for an additional 50B tokens) to probe how architecture choice influences learning beyond standard metrics and even bias metrics such as WinoGender. This demonstrates that the architecture matters as a fundamental factor in learning, not just the chosen objective source.
• For practitioners and researchers, Ettin provides practical boilerplates and a clear path to reproduce or build upon the work. The authors explicitly invite users to try the models and consider downstream tasks such as classification, retrieval, and generation using encoder-based or decoder-based workflows source.

Why it matters (impact for developers/enterprises)

The core contribution of Ettin is to enable fair, apples-to-apples comparisons between encoder and decoder architectures under identical data and training recipes. This clarity helps teams choose the right backbone for a given application, whether it is a fast, on-device discriminative model or a capable generative system that can handle long-context tasks. The open-data nature of Ettin lowers the barrier to entry for research and production teams seeking to adapt, reproduce, or extend high-quality models without relying on proprietary data. In practice, enterprises can assess: (1) which architecture best suits their primary tasks (classification, retrieval vs. generation), (2) how scale influences performance within a fixed data regime, and (3) how training objectives might shape model behavior beyond accuracy, including bias-related aspects that are increasingly scrutinized in deployment scenarios source.

Technical details or Implementation

The Ettin suite comprises six model scales for each architecture (encoder and decoder), spanning 17M to 1B parameters. Across both families, training uses an identical data mix, identical model shapes, and the same training recipe to guarantee apples-to-apples comparisons. The three-phase training regimen is a central feature: 1) Pre-training on 1.7T tokens with shorter 1024-token contexts to establish robust foundations; 2) Context Extension to 8K tokens using higher-quality filtered data to capture longer-range dependencies; and 3) Decay phase with 100B tokens from premium sources, while progressively reducing the learning rate. A key experimental twist involved continuing training for 50B tokens with the opposite objective (MLM on encoders, CLM on decoders) to study how objectives shape learning under the same architecture and data regime. The open nature of Ettin’s data and training ingredients means researchers can explore architecture-specific advantages and bias dynamics with transparent inputs source.

Key takeaways

• Ettin establishes a fair apples-to-apples framework for comparing encoder and decoder architectures, using identical data, shapes, and recipes.
• Encoders tend to dominate classification and retrieval tasks, even at smaller sizes, while decoders maintain advantages in generation, particularly as scale increases.
• The six-scale coverage (17M–1B) for both encoders and decoders enables a wide range of deployment options, from fast on-device models to larger, high-performance systems.
• All training data for Ettin is public and reproducible, lowering the barrier to replication, extension, and community-driven improvements.
• A deliberate experiment swapping objectives after initial training highlights that architecture choice matters beyond the objective alone, with measurable effects on learning dynamics and behavior such as bias benchmarking.
• The Ettin suite provides practical boilerplates and a clear path for practitioners to try open-data models across tasks, including classification, retrieval, and generation pipelines source.

References

• https://huggingface.co/blog/ettin