Enabling Kotlin Incremental Compilation on Buck2 with the Build Tools API KEEP

TL;DR

• Kotlin incremental compilation is now integrated with Buck2 to speed up builds by focusing only on changed files.
• The approach uses the Build Tools API KEEP for a stable, future-proof integration, addressing issues around shading and classpath conflicts.
• Buck2’s incremental actions enable preserving outputs between runs, using hashes to detect changes and classpath snapshots to account for dependency changes.
• Some critical modules now build up to 3x faster, highlighting meaningful productivity gains at scale.
• Plugins and annotation processing (KSP2) were adapted to work with incremental rounds, ensuring consistent results across multiple evaluation passes. Source: Meta’s engineering blog on enabling Kotlin incremental compilation in Buck2. See https://engineering.fb.com/2025/08/26/open-source/enabling-kotlin-incremental-compilation-on-buck2/ for the detailed write‑up.

Context and background

Kotlin’s incremental compiler has long been a boon for developers seeking faster feedback by recompiling only the changed portions of a codebase. Buck2, Meta’s build system, also champions small modules to achieve fast builds through parallelism and incremental execution. As codebases grow and teams evolve, some modules inevitably gain size, which can impact overall build times. To explore whether incremental compilation would unlock measurable gains in Buck2, Meta evaluated the Kotlin incremental compiler within the Android toolchain. The core benefit of incremental compilation is straightforward: instead of rebuilding an entire module, the compiler focuses on changed sources and the files that depend on them. Meta observed that several critical modules could benefit significantly, with some builds accelerating by as much as threefold. This finding motivated a deeper integration effort that balanced long-term stability with the need for speed. From a tooling perspective, the team faced a few hurdles. As of Kotlin 2.2.0, the only guaranteed public contract for the compiler is via the command-line interface (CLI). The CLI does not support incremental compilation, which pushed Meta toward alternative integration points. Internal compiler APIs are unstable and not intended for public use, so relying on them would risk fragility across Kotlin updates. The Build Tools API (KEEP), introduced with Kotlin 1.9.20, emerged as the official and more sustainable integration point that includes incremental compilation support. The decision was to adopt KEEP, with the intent to influence its direction while benefiting from a stable, future-proof path. Another practical challenge came from classpath conflicts between shaded and unshaded Kotlin compiler jars. The Build Tools API depends on the shaded kotlin-compiler-embeddable, while the Android toolchain historically used an unshaded kotlin-compiler. To resolve this, Meta first unshaded the Build Tools API for a working prototype, then migrated the entire toolchain to the shaded compiler to align with the API’s expectations and ensure long-term stability. This staged approach avoided large-scale, risky refactors while validating the incremental path before a full migration. To enable incremental compilation, Buck2 must have access to the previous build’s outputs. Buck2’s default behavior deletes outputs before rebuilding a module, which would break incremental compilation. The team introduced a mode in Buck2’s incremental actions that skips automatic cleanup, allowing the compiler to reuse prior results. This comes with a tradeoff: developers must manually prune outputs that are no longer needed. The change was essential to unlock the incremental path while maintaining control over cache hygiene. A broader goal was to support distributed builds—Buck2 can run builds remotely and aggregate results. A relocatable compiler cache is important in such setups to avoid path conflicts and ambiguity across machines. Meta explicitly configured the root project directory and the build directory within the incremental compilation settings to preserve a stable cache across workers and environments.

What’s new

• Adoption of the Build Tools API KEEP as the official integration point for Kotlin’s incremental compilation within Buck2, providing a more sustainable and future-proof approach than internal APIs.
• A practical workaround was used during prototyping to unshade the Build Tools API (with jarjar) before migrating to the fully shaded Kotlin compiler, ensuring a working prototype before a full migration.
• Buck2 incremental actions preserve outputs from the previous run and produce hash digests for every action input, enabling accurate change detection and selective recompilation.
• The compiler now relies on two primary change-tracking mechanisms: automatic detection of source changes via the Build Tools API and explicit lists of modified files for Kotlin versions earlier than 2.1.20. Even with newer versions, feeding the tool a change list can optimize the process.
• A dedicated Buck2 action generates classpath snapshots from library outputs. These ABI snapshots enable dependency change detection and help determine which module files are affected when dependencies change. The snapshot artifact can be cached or executed remotely, avoiding heavy local work.
• If the toolchain already analyzes dependencies and detects changes, snapshot comparison can be skipped, allowing performance optimizations to be applied with minimal friction.
• Integrating the incremental compiler with custom compiler plugins required careful handling: plugins must be incremental and able to cope with partial inputs, as the incremental compiler can operate on subsets of files and in multiple rounds.
• The Kotlin incremental compiler can run in multiple rounds within a single build to determine the full set of affected files. This behavior introduces the potential for multiple plugin invocations with different inputs, so Meta updated plugins to accumulate results across rounds instead of replacing outputs.
• Kotlin Symbol Processing (KSP2) was leveraged for annotation processing in this workflow. KSP2 is designed as a standalone tool that uses the Kotlin Analysis API to analyze source code, which helped compatibility with incremental compilation.

Why it matters (impact for developers/enterprises)

The integration delivers tangible productivity gains by reducing work for developers when rebuilding large Android codebases. By reusing outputs and focusing on changed files, builds can complete faster, which translates into shorter iteration cycles and faster feedback. In distributed build environments, a stable, relocatable cache is essential to avoid conflicts and ambiguity arising from differing paths across machines. This work also helps teams scale their toolchains as modules grow and teams expand, maintaining build happiness and developer efficiency.

Technical details or Implementation

• The Kotlin incremental compiler requires access to previous outputs, which Buck2 normally cleans up by default. The team configured Buck2’s incremental actions to preserve previous outputs, with the caveat that developers must manually clean up outputs that are no longer useful. This enables the incremental compiler to analyze the prior state and perform selective recompilation.
• Change detection can be automatic or manual. Automatic detection depends on the Kotlin version; for Kotlin versions earlier than 2.1.20, a manual list of modified files must be provided. Even when automatic detection is available, supplying the change list can improve performance.
• Buck2’s incremental actions capture hash digests for every action input to determine what changed since the last build. This enables precise identification of the affected files and minimizes unnecessary work.
• Classpath snapshots capture the ABI state of libraries. By comparing current snapshots with previous ones, the compiler can determine which files in a module are affected by library changes. A separate action generates these snapshots, and the resulting artifact can be cached or executed remotely to avoid heavy local processing.
• If dependency analysis conducted by the build tool already identifies the relevant changes, the snapshot comparison step can be skipped to streamline the build further.
• Integrating incremental compilation with custom compiler plugins posed notable challenges, as these plugins were not originally designed for partial inputs. The team made the plugins incremental so they can handle partial inputs without producing incomplete results. Most annotation processing tasks rely on Kotlin Symbol Processing (KSP2), which helps maintain compatibility with the incremental workflow. KSP2 operates as a standalone tool using the Kotlin Analysis API to analyze source code, which reduces coupling with the core compiler.

Change-tracking at a glance

| Change-tracking option | What it does |

Key takeaways

• The Build Tools API KEEP enables a sustainable path for Kotlin incremental compilation in Buck2.
• Preserving previous outputs and using input hashes are central to accurate and efficient incremental builds.
• Classpath snapshots add a powerful dependency-change filter, reducing unnecessary recompilation caused by library updates.
• Plugins and KP2-based processing must be evolving to align with incremental compilation, including multi-round execution and result accumulation.
• Buck2’s distributed build support benefits from a stable, relocatable cache to avoid cross-machine conflicts.

References

• https://engineering.fb.com/2025/08/26/open-source/enabling-kotlin-incremental-compilation-on-buck2/