Train with Terabyte-Scale Datasets on a Single NVIDIA Grace Hopper Chip Using XGBoost 3.0

TL;DR

• XGBoost 3.0 introduces External Memory Quantile DMatrix to scale TB-scale datasets on a single GH200 Grace Hopper Superchip, removing the need for distributed multi-node clusters.
• The Grace Hopper architecture (72-core Grace CPU + Hopper GPU with NVLink-C2C at 900 GB/s) streams data from host RAM to the GPU, enabling 1 TB models to train in minutes and up to 8x faster than a 112-core CPU box.
• GPU-based XGBoost delivers substantial performance benefits and cost savings; RBC reports up to 16x end-to-end speedup and ~94% reduction in training TCO in their testing.
• Training remains with familiar XGBoost calls; the external-memory path reduces complexity while preserving accuracy, thanks to pre-binned features and the standard hyper-parameters.
• Practical guidance includes grow_policy=‘depthwise’, CUDA 12.8+, HMM-enabled drivers for Grace Hopper, and RAPIDS Memory Manager integration.

Context and background

Gradient-boosted decision trees (GBDTs) power a wide spectrum of real‑world applications, from fraud filters to petabyte-scale demand forecasts. The XGBoost open source library has long been favored for its state‑of‑the‑art accuracy, SHAP-ready explainability, and flexibility to run on laptops, multi‑GPU nodes, or Spark clusters. With XGBoost 3.0, scalability is the guiding principle, prioritizing a regime where throughput and simplicity meet very large datasets. A key limitation historically was fitting TB-scale datasets in GPU memory alone. NVIDIA’s GH200 Grace Hopper Superchip—a combination of a 72‑core Grace CPU and a Hopper GPU connected by NVLink C2C with 900 GB/s bandwidth—offers a unique external-memory path. The architecture enables streaming of data from host RAM to the GPU at every iteration, which makes TB-scale training feasible on a single chip rather than requiring a large distributed cluster. This approach leverages the new External Memory Quantile DMatrix, built atop the existing Data Iterators that manage dataset memory and feed the booster object with unchanged hyper-parameters. XGBoost’s GPU histogram method previously accelerated training versus CPU implementations, but XGBoost 3.0 expands capability further by introducing a dedicated external-memory pathway that fits TB-scale datasets within the Grace Hopper ecosystem. This aligns with the broader goal of making accelerated data science more accessible and cost-efficient for organizations that require rapid model iteration on massive data volumes. Source coverage: NVIDIA’s XGBoost 3.0 on GH200 Grace Hopper.

What’s new

• External-Memory Quantile DMatrix: A third mechanism for scaling beyond GPU RAM, built on top of Data Iterators. It pre-bins features (like QuantileDMatrix) but streams data from host RAM to the GPU every iteration, enabling TB-scale training on a single GH200 without distributed frameworks.
• TB-scale on a single GH200: XGBoost 3.0 enables processing terabyte-scale GBDT training with the same XGBoost calls you’ve used before, leveraging the ultrafast 900 GB/s NVLink-C2C between the Grace CPU and Hopper GPU.
• External memory streaming: The data sits in host RAM and streams to the GPU in each iteration, with performance advantages driven by the GPU’s ability to handle dense tables and reduced bus traffic.
• Performance and API improvements: Beyond the external-memory overhaul, XGBoost 3.0 includes API cleanups and broader GPU-memory-efficiency improvements to push external memory toward a default workflow when GPU memory cannot hold the dataset.
• Start-up guidance and requirements: Use grow_policy=‘depthwise’ to build trees layer by layer; run on CUDA 12.8+ with an HMM-enabled driver for Grace Hopper; leverage RAPIDS Memory Manager (RMM) pools when using RAPIDS. Key technical note: The GH200 chip packages a 72‑core Grace CPU and Hopper GPU, linked by NVLink C2C, delivering 900 GB/s bidirectional bandwidth—roughly seven times the bandwidth of x16 PCIe Gen 5, with lower latency. For reference, a 1 TB training job typically requires either a CPU box with around 2 TB of DRAM or a small GPU cluster with 8–16 H100 GPUs; the single-GH200 path replaces the RAM‑monster server and multi‑GPU pod with streaming from host memory.

Why it matters (impact for developers/enterprises)

The ability to train TB-scale models on a single Grace Hopper chip dramatically reduces architectural complexity and operational overhead. By removing the need for large multinode GPU clusters, organizations can simplify deployment while preserving, and often increasing, training throughput and explainability. In practical terms, this approach enables faster iteration on model tuning and feature engineering for high‑volume, real‑time, or batch pipelines. RBC, one of the world’s largest banks by market capitalization, uses XGBoost for predictive lead scoring at scale. As Christopher Ortiz, RBC’s Director of Gen AI Planning and Valuation, noted: “We’re confident that XGBoost, powered by NVIDIA GPUs, will make our predictive lead scoring model possible for the data volumes we’re projecting.” He adds: “We’ve seen up to a 16x end-to-end speedup by leveraging GPUs, and for our pipeline testing, we’ve seen a remarkable 94% reduction in TCO for model training.” This reflects the potential cost-to-performance gains for enterprise-grade ML pipelines.

Technical details or Implementation

• Hardware and bandwidth: A GH200 Grace Hopper Superchip combines a 72‑core Grace CPU and a Hopper GPU, connected via NVLink C2C, delivering about 900 GB/s bidirectional bandwidth. This bandwidth is roughly 7x the bandwidth of x16 PCIe Gen 5, with lower latency, enabling efficient streaming of large data from host memory to the GPU.
• External Memory Quantile DMatrix: This new data structure sits atop existing Data Iterators, managing dataset memory and interfacing with the XGBoost booster while preserving existing hyper-parameters. The data remains in host RAM and streams to the GPU at each iteration.
• Data shape sensitivity: While GPUs excel on dense (or near-dense) tables due to compression and reduced bus traffic, the External Memory Quantile DMatrix is sensitive to dataset shape. Specifically, when training with a feature matrix (x) and labels (y), only the feature matrix is paged according to row count, not the labels. The total data size can stay constant while varying the number of rows and columns, affecting fit on a single GH200.
• Memory and on-chip considerations: The GH200 configuration referenced includes 80 GB of HBM3 and 480 GB of LPDDR5X, both fed by 900 GB/s NVLink-C2C. This setup is designed to support streaming TB-scale workloads without resorting to large external clusters.
• Software and workflow recommendations: Start with grow_policy=‘depthwise’ to build trees layer by layer. When working with RAPIDS, run in a fresh RAPIDS Memory Manager (RMM) pool, and use CUDA 12.8+ with an HMM-enabled driver for Grace Hopper. External-memory workflows are designed to align with the familiar XGBoost API, so existing users can adopt TB-scale training with minimal changes.
• Practical implications: The external-memory approach reduces the need for complex distributed frameworks and can deliver significant speedups for end-to-end pipelines, including model training, feature iteration, and deployment readiness.

Tables: key facts at a glance

| Configuration | Data scale | Training time vs CPU | Notes |---|---|---|---| | Single GH200 Grace Hopper chip | 1 TB model | Up to 8x faster than a 112-core CPU box | External-memory streaming from host RAM; TB-scale capability |

Key takeaways

• XGBoost 3.0 enables TB-scale training on a single GH200 Grace Hopper Superchip using External Memory Quantile DMatrix.
• The streaming data path leverages 900 GB/s NVLink-C2C and the Grace-Hopper pairing to minimize data movement and maximize throughput.
• Training a 1 TB dataset can complete in minutes, with performance advantages up to 8x over a high-end CPU box and substantial cost reductions observed in enterprise pilots.
• The approach preserves familiar XGBoost APIs and hyper-parameters, reducing the need to redesign pipelines for TB-scale workloads.
• Practical guidance emphasizes growth policy, driver and CUDA requirements, and RAPIDS integration for an optimal external-memory workflow.

References

• NVIDIA: Train with Terabyte-Scale Datasets on a Single NVIDIA Grace Hopper Superchip Using XGBoost 3.0