Items tagged with “Benchmark”.
A detailed look at how NVIDIA Run:ai Model Streamer lowers cold-start times for LLM inference by streaming weights into GPU memory, with benchmarks across GP3, IO2, and S3 storage.
This post explains how msg automated data harmonization for msg.ProfileMap using Amazon Bedrock to power LLM-driven data enrichment, boosting HR concept matching accuracy, reducing manual workload, and aligning with EU AI Act and GDPR.
Explores quantization aware training (QAT) and distillation (QAD) as methods to recover accuracy in low-precision models, leveraging NVIDIA's TensorRT Model Optimizer and FP8/NVFP4/MXFP4 formats.
NVIDIA’s RTX PRO 6000 Blackwell Server Edition dramatically speeds protein-structure inference, enabling end-to-end GPU-resident workflows with OpenFold and TensorRT—achieving up to 138x faster folding than AlphaFold2.
OpenAI explains that hallucinations in language models stem from evaluation incentives that favor guessing over uncertainty. The article outlines how updated scoring and uncertainty-focused benchmarks can reduce confident errors and improve reliability.
Guide to fine-tuning gpt-oss with SFT + QAT to recover FP4 accuracy while preserving efficiency, including upcasting to BF16, MXFP4, NVFP4, and deployment with TensorRT-LLM.
A new RL approach uses instruction-derived checklists to guide alignment, outperforming fixed-criteria reward models across multiple benchmarks on Qwen2.5-7B-Instruct, presented at ICLR 2025.
Apple ML Research introduces SlowFast-LLaVA-1.5 (SF-LLaVA-1.5), a family of token-efficient video LLMs designed for long-form video understanding. It leverages SlowFast two-streams and public data to achieve state-of-the-art results at 1B–7B scales, with mobile-friendly implications.
David Luan, head of Amazon's AGI Lab, argues that solving AI agents is the next major frontier, outlining a factory-like approach to building smarter models and stressing real-world task completion beyond chat.
A benchmarking study shows how Amazon Nova Pro on Bedrock enables high-accuracy document field localization using multimodal models, with FATURA benchmarking and practical implementation guidance.
An in-depth look at intersectional bias in LLMs through a new benchmark and a confidence-based fairness metric, revealing reliability gaps in decision-support scenarios.
NVIDIA Research introduces ProRL v2, the latest evolution of Prolonged Reinforcement Learning for LLMs. It explores thousands of extra RL steps, new stabilization techniques, and broad benchmarking to push sustained improvements beyond traditional RL schedules.
CUDA-QX 0.4 advances quantum error correction workflows with automated detector error-model generation, a tensor-network decoder, improved BP+OSD decoding, and a Generative Quantum Eigensolver (GQE) in the Solvers library, accelerating end-to-end QEC development.
A deep dive into In-Context Retrieval and Reasoning (ICR2) for long-context LLMs, including benchmarks, methods, and implications for retrieval-augmented generation (RAG).
TextQuests is a benchmark testing LLM agents in 25 classic text-based Infocom games, emphasizing long-context reasoning and autonomous exploration.
NVIDIA Cosmos Reason is an open, fully customizable reasoning vision-language model for physical AI and robotics. It enables step-by-step multimodal reasoning and boosts robotics performance through post-training refinements.
OpenAI unveils GPT-5, its most capable AI system yet, featuring unified thinking, a real-time router, reduced hallucinations, and strong performance in coding, writing, health, and more.
OpenAI releases GPT-5 in the API with enhanced reasoning, new developer controls, and three serviceable sizes to optimize coding, agentic tasks, and tool use.
OpenAI unveils gpt-oss-120b and gpt-oss-20b—two open-weight LLMs designed for strong real-world performance at low cost. Licensed under Apache 2.0, they emphasize reasoning, tool use, and efficient on-device deployment across consumer hardware.
Open-source TimeScope tests long-video understanding by inserting short needle clips into base videos (1 minute to 8 hours), evaluating retrieval, synthesis, localization, and motion analysis.
TimeScope introduces a new open-source benchmark to measure how vision-language models process long videos by inserting short needles and evaluating retrieval, synthesis, localization, and motion analysis.
A deep dive into FutureBench, a benchmark that tests AI agents on forecasting future events using real-world data sources, with a focus on reasoning under uncertainty and verifiable outcomes.
A deep dive into Consilium, the multi-LLM platform that enables models to discuss, debate, and reach consensus via MCP servers and a visual Gradio roundtable.
Ettin introduces the first state-of-the-art paired encoder-only and decoder-only models trained with identical data and recipes, measuring apples-to-apples performance across tasks and scales.
Explores why purposeful, multi round dialogue matters for LLM chatbots beyond one shot prompts, and outlines training, evaluation, and implementation challenges for engineers and enterprises.
StrongREJECT advances jailbreak evaluation by pairing a high-quality forbidden-prompt dataset with automated evaluators aligned to human judgments, delivering more reliable measurements of jailbreak effectiveness against frontier LLMs.
Overview of a high-quality jailbreak benchmark with dual automated evaluators, a 313-prompt dataset, and findings that many jailbreaks underperform claims from earlier work.
A new MIQA benchmark tests Large Multimodal Models on visual retrieval and reasoning across 1–10K images, revealing key limitations and introducing MIRAGE, a single-stage approach to scale LMMs.
Benchmark for long-context visual reasoning across large, uncorrelated image sets; introduces MIRAGE to extend LMMs beyond single-image VQA.
Curated overview of key studies showing how AI systems reproduce and amplify gender bias, with concrete measures, benchmarks, and mitigations across embeddings, vision, NLP, and generative models.
Survey of key work measuring gender bias in AI, across word embeddings, coreference, facial recognition, QA benchmarks, and image generation; discusses mitigation, gaps, and the need for robust auditing.