Vision-Language Model Alignment in TRL: GRPO, GSPO, MPO and SFT Support

Vision-Language Models (VLMs) are growing stronger, but aligning them to human preferences remains essential. In TRL, we already showed how to post-train VLMs with Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO). This update goes further by introducing three new multimodal alignment methods—Group Relative Policy Optimization (GRPO), its variant Group Sequence Policy Optimization (GSPO), and Mixed Preference Optimization (MPO)—along with native SFT support for vision-language models and training scripts and demo notebooks to help you get started. These methods are designed to extract more signal from preference data and scale better with modern VLMs, improving alignment quality without sacrificing training stability. This overview draws on the Hugging Face TRL VLM alignment post. Context and background Vision-Language Models blend visual and textual reasoning, and aligning them to human preferences helps ensure useful and safe behavior across tasks. Historically, practitioners would take a base model, apply SFT to teach instruction-following behavior, and then apply DPO to align it to preference data. In the VLM setting, this pipeline was adapted and validated on IDEFICS2, showing improvements in model responses. DPO works by learning from pairwise preferences between a chosen and a rejected response using a contrastive loss; the model is updated to prefer the preferred option. Over the past year, multimodal alignment methods such as GRPO and MPO have gained traction for their ability to push VLM performance further by leveraging preference signals more robustly and scalably. TRL now includes native support for these methods in addition to existing SFT and DPO tooling, enabling researchers and engineers to mix and match signals from multiple losses and reward structures to better supervise multimodal reasoning. What’s new This update introduces three major multimodal alignment methods in TRL, along with enhanced SFT support and end-to-end tooling. A quick tour of each method follows. For reference, the blog notes that a table at the end compares model outputs across methods.

Group Relative Policy Optimization (GRPO)

GRPO extends an approach originally introduced for large-scale policy optimization by applying updates over groups of trajectories (batches of dialogue runs) rather than single episodes. In TRL, this grouping helps average reward noise within each group, making the learning signal more stable and robust. The result is a model that learns a broader, more robust sense of what constitutes a good response rather than chasing a few high-reward samples. In TRL, GRPO support for vision-language models is introduced, with guidance focused on high-level concepts rather than a complete end-to-end script in this post. To use GRPO effectively, the TRL team proposes defining two reward functions, constructing a GRPOConfig and a GRPOTrainer, and then calling train() to begin the alignment process. A full notebook example is available to explore the workflow in practice. The emphasis is on validating the answer format and ensuring the solution aligns with the completed portions of your data and dataset.

Group Sequence Policy Optimization (GSPO)

GSPO is a refinement of GRPO designed to address some limitations by computing importance sampling weights at the sequence level instead of per-token. This sequencing-level perspective can be particularly valuable for mixture-of-experts (MoE) style models, where the distribution over tokens can be highly multiplexed. TRL now includes GSPO support for multimodal models, following the same trainer-configuration approach as GRPO but with additional parameters drawn from the original paper.

Mixed Preference Optimization (MPO)

MPO targets multimodal models by combining multiple losses into a single objective: the standard DPO preference loss (sigmoid), a quality loss from Binary Classifier Optimization (BCO), and the generation loss from SFT. This hybrid objective is designed to address weaknesses seen when relying on a single signal, such as inconsistent rationale or repetitive responses. In published work, switching to this combined loss yielded notable improvements (e.g., a 6.2-point gain on MathVista), illustrating the potential of MPO to enhance multimodal reasoning and generation. Alongside MPO, TRL adds the necessary loss-combination support to the DPOTrainer class, enabling researchers to configure mixed losses without replacing the core DPO workflow. The post also points to a complete notebook example to illustrate how MPO and the other methods fit into a practical workflow. Why it matters (impact for developers/enterprises) The introduction of GRPO, GSPO, and MPO broadens the toolkit for aligning VLMs with human preferences. The group-based approach of GRPO helps dampen reward noise and stabilizes training by learning from broader, contextual signals rather than single samples. This leads to models that generalize better across diverse prompts and environments, which is critical for real-world deployments. GSPO’s sequence-focused importance sampling is particularly well-suited to MoE architectures, where importance weights at the sequence level can reduce variance and improve convergence in complex, large-scale models. This can translate into more reliable training dynamics and better resource utilization when scaling up VLMs in enterprise contexts. MPO explicitly addresses known challenges in multimodal alignment by combining multiple losses—DPO, BCO, and SFT—into a single objective. This approach can yield stronger overall performance and more coherent multimodal behavior, including more reliable reasoning and fewer issues with repetitive or off-topic responses. The reported improvements in the cited work (e.g., 6.2 points on MathVista) underscore the practical value of this approach when evaluation datasets stress multimodal reasoning. For developers and organizations, the availability of these methods in TRL, along with native SFT support for VLMs and ready-to-run training scripts and notebooks, lowers the barrier to experimenting with advanced alignment techniques. It enables faster iteration cycles, easier ablation studies, and more robust deployment pipelines that can adapt to evolving preference data and task requirements. The TRL release also emphasizes accessibility: you can configure, train, and evaluate these methods using standard TRL APIs, with reference notebooks guiding the workflow. If you rely on TRL for VLM work, these additions provide a more scalable path to high-quality multimodal alignment, as described in the accompanying Hugging Face blog TRL VLM Alignment. Technical details or implementation (high level)

• MPO details: MPO extends DPO with a multi-loss objective that combines the DPO preference loss (sigmoid), BCO quality loss, and SFT generation loss. The combined objective has been associated with improved performance on multimodal benchmarks and is enabled via the DPOTrainer in TRL by adding the combined loss capability.
• MPO usage: To employ MPO, initialize the DPOConfig as described in the TRL docs and switch to the combined loss path within the DPOTrainer. The blog notes that this is achieved by integrating the mixed losses into the trainer without removing the DPO base.
• GRPO usage: GRPO requires configuring a GRPOConfig and a GRPOTrainer, defining two reward functions, and running train(). The approach is designed to be robust to reward noise through grouping of trajectories. A complete notebook example demonstrates the workflow end to end.
• GSPO usage: GSPO shares the same trainer setup as GRPO but adds parameters to support sequence-level importance sampling and MoE-friendly training. The process remains aligned with the DPO/GRPO workflow, ensuring a consistent toolchain for multimodal alignment.
• SFT and native VLM support: TRL adds native SFT support for vision-language models, enabling a more streamlined post-training pipeline that can be combined with DPO, MPO, GRPO, or GSPO. The combination of SFT with these advanced alignment methods promises to improve reasoning and generation quality in VLMs.
• DPO background: DPO remains a core component for aligning VLMs to human preferences by optimizing pairwise comparisons between a chosen and a rejected response using a contrastive loss. MPO extends this by incorporating additional signals to address the multimodal setting.
• Practical notes: The blog emphasizes that full GRPO training scripts are not provided in-line, but the key components and training flow are described, with a full notebook available to explore the practicalities of GRPO in TRL. The emphasis is on validating answer formats and ensuring alignment signals reflect completed data.
• DPOTrainer and configuration: TRL has updated the DPOTrainer to support the combined loss. Users can instantiate DPOConfig and DPOTrainer to leverage MPO’s multi-loss setup and begin experimentation with multimodal alignment signals. The referenced notebook serves as a practical guide. Key takeaways
• TRL now supports GRPO, GSPO, MPO for Vision-Language Model alignment, in addition to native SFT and DPO.
• GRPO offers group-based policy updates that dampen reward noise and promote a broader view of good responses.
• GSPO computes sequence-level importance sampling weights, with benefits for MoE-style architectures.
• MPO combines DPO, BCO, and SFT losses to improve multimodal alignment, with examples showing notable gains on benchmark tasks.
• TRL provides training scripts and notebooks to help you implement these methods in practice, along with guidance on how to configure the trainer and reward functions.
• A table in the original post highlights the differences among model outputs under these methods, underscoring practical distinctions for evaluation.
• The approach builds on prior TRL workflows (SFT followed by DPO) while expanding the signal sources and robustness of multimodal alignment. See the Hugging Face post for details TRL VLM Alignment. FAQ
• Q: What is MPO in TRL? A: MPO is Mixed Preference Optimization, an extension of DPO for multimodal models that combines the DPO preference loss, the BCO quality loss, and the SFT generation loss.
• Q: How do I use GRPO in TRL? A: Define two reward functions, create a GRPOConfig and GRPOTrainer, and call train() to start learning from grouped trajectories.
• Q: What is GSPO and when is it advantageous? A: GSPO is Group Sequence Policy Optimization, a variant of GRPO that computes importance sampling weights at the sequence level, which is especially beneficial for MoE-style architectures and multimodal training.
• Q: Is SFT still supported for VLMs in TRL? A: Yes, native SFT support for vision-language models is added, enabling a straightforward post-training workflow alongside DPO, MPO, GRPO, or GSPO.
• Q: Where can I find examples or notebooks to get started? A: The TRL blog post highlights notebooks and examples to illustrate the workflows and how to configure the trainer and losses; a full notebook accompanies the release. References
• Hugging Face blog: TRL VLM Alignment (https://huggingface.co/blog/trl-vlm-alignment)