How to Improve CUDA Kernel Performance with Shared Memory Register Spilling (CUDA 13.0)

TL;DR

• CUDA 13.0 adds an optimization that spills registers to shared memory first, using local memory only if shared memory is insufficient.
• When available, on‑chip shared memory lowers latency and reduces pressure on the L2 cache compared with traditional local memory spills.
• Prior to CUDA 13.0, all spills went to local memory; this could cause cache eviction and performance loss in high‑pressure regions.
• Enable this feature with a PTX pragma inside a function: enable_smem_spilling. The pragma is valid only within the function scope.
• Real‑world results from QUDA lattice QCD workloads show typical gains of about 5–10% in kernels with substantial register pressure.

Context and background

A CUDA kernel can require more hardware registers than are available on a streaming multiprocessor (SM). When this happens, the compiler spills the excess variables into local memory, which is physically located in off‑chip global memory. Spilling increases memory traffic and latency because spilled data must be read from and written to local memory. Before CUDA 13.0, register spills were always directed to local memory. Even with larger L1 caches, spilled data could still miss and be written to the L2 cache, potentially evicting useful cache lines and degrading performance in loops and other hot code paths that are highly register‑pressured. A further contributor to inefficiency was that, in many workloads, a sizable portion of per‑block shared memory remained unused at runtime, especially when launch bounds or register pressure limited occupancy or when the kernel did not maximize shared memory usage. The CUDA developer blog discusses how spilling behavior interacts with shared memory and occupancy, and how limiting factors like launch bounds can influence kernel performance. The post uses a kernel design that intentionally pushes register pressure to illustrate the overhead of spills and the inefficiency of not using shared memory when possible. The key takeaway is that keeping spills on‑chip, when feasible, helps bring spilled data closer to the SM and reduces costly off‑chip memory traffic.

What’s new

CUDA 13.0 adds a PTXAS optimization that enables spilling registers to on‑chip shared memory for CUDA kernels. When the feature is enabled, the compiler prioritizes spilling registers into available shared memory. If there isn’t enough space, the remaining spills fall back to local memory, preserving correctness. This change brings a clear performance advantage by leveraging the lower‑latency, on‑chip memory for spilled values whenever possible. Compared with earlier toolkits, where spilled data resided in global memory and could substantially affect performance, the shared memory spilling pathway reduces latency and alleviates pressure on the L2 cache in many scenarios. The optimization was demonstrated on CUDA kernels derived from the QUDA library (used for lattice QCD calculations on GPUs). In that work, activation of shared memory spilling typically yielded performance gains in the 5–10% range, driven by reduced or eliminated local spills. To opt in, developers targeting CUDA 13.0+ must insert a PTX pragma inside the function, immediately after the function declaration:

// inside the function body after the declaration
.pragma_enable_smem_spilling

In inline assembly form, the directive is expressed as:

#pragma enable_smem_spilling

The feature is valid only within a function scope. It should not be used when launch bounds are not explicitly specified; if launch bounds are unknown, PTXAS may assume the maximum possible number of threads per block, which can misestimate shared memory usage and reduce occupancy. For more predictable behavior and better performance, use this feature only when launch bounds are explicitly defined. The feature is only available in CUDA 13.0 and later; it was not present in earlier toolkit versions. When spills are redirected to shared memory, the kernel’s per‑block shared memory usage may reflect visible spills as in the example where 46080 bytes of shared memory were allocated (indicating spilled data used on‑chip memory).

Why it matters (impact for developers/enterprises)

For developers building performance‑critical GPU kernels, especially those with high register pressure, this optimization offers a practical path to improving throughput without changing algorithmic design. By directing spills to on‑chip shared memory first, workloads can see lower latency for spilled data and reduced contention in the L2 cache, which can translate into meaningful execution time improvements in tight loops and hot paths. However, the benefits are not universal. The gains depend on having well‑defined launch bounds and consistent shared memory utilization so that a kernel can actually take advantage of the on‑chip spill path. If the shared memory budget per block is over‑provisioned or misestimated, occupancy can suffer, offsetting potential gains. Consequently, a cautious, data‑driven approach is recommended, evaluating performance with and without the opt‑in path on representative workloads. For enterprises running large CUDA deployments, the feature provides another knob to tune kernel performance, especially in libraries and applications where register pressure is common. The 5–10% gains observed in QUDA indicate a tangible improvement for lattice QCD kernels under high register pressure, and similar gains may be realized in other workloads with comparable characteristics.

Technical details or Implementation (how it works)

• The core idea is to prioritize spills into on‑chip shared memory (smem) rather than local memory whenever there is available space per thread block.
• If shared memory space is insufficient, the compiler gracefully falls back to spilling into local memory, preserving correctness.
• The optimization is introduced as a PTXAS feature in CUDA 13.0; enabling it requires an explicit opt‑in inside the function scope using the inline PTX pragma enable_smem_spilling.
• The feature has constraints: you should specify launch bounds to ensure accurate memory budgeting; otherwise, the compiler may assume the maximum threads per block, potentially limiting occupancy if actual launches are smaller.
• The performance impact is workload dependent. In the QUDA evaluation, typical improvements were in the 5–10% range, driven by reduced or eliminated local spills when shared memory could accommodate spilled data.
• The mechanism maintains correctness by falling back to local memory when needed and keeping spilled data close to the SM when possible.

Table: spill targets and implications

| Aspect | Before CUDA 13.0 | With Shared Memory Spilling (CUDA 13.0) |---|---|---| | Spill target | Local memory (off‑chip) | Shared memory (on‑chip) when space is available; fallback to local memory else |Latency | Higher due to off‑chip access | Lower when spills stay in shared memory |L2/cache pressure | Possible eviction of useful lines | Reduced pressure when spills reside in shared memory |Occupancy risk | Not directly tied to spills | Requires explicit launch bounds to avoid occupancy regressions |

Example reference from the NVIDIA post

The article notes a kernel with 46080 bytes of shared memory usage when spilling to shared memory, illustrating on‑chip spill usage in a practical context. This example underscores how the feature can reflect on the per‑block shared memory footprint when enabled.

Key takeaways

• Shared memory spilling is an opt‑in feature in CUDA 13.0 via a function‑scope pragma.
• Spills are redirected to on‑chip shared memory first, with a fallback to local memory if space is insufficient.
• Real‑world tests show typical performance gains around 5–10% for register‑limited kernels, as observed in QUDA benchmarks.
• To avoid occupancy and budgeting issues, define explicit launch bounds when using this feature.
• The optimization is designed to maintain correctness while improving access latency for spilled data.

References

• https://developer.nvidia.com/blog/how-to-improve-cuda-kernel-performance-with-shared-memory-register-spilling/