Hi all,
Almost 4 1/2 years ago, we open sourced llbuild alongside Swift and the Swift package manager. At the time, llbuild was still fairly new, but it's design has largely stood the test of time, powering SwiftPM builds as well as other products like Swift playgrounds and Xcode's XCBuild build system. And in that time, llbuild has driven a lot of builds!
In the meantime, a lot of other things have changed in the world. Swift's ecosystem has grown considerably, including substantial packages like Swift NIO giving us far easier access to sophisticated network technology from Swift. The world outside of Swift has changed too, with Bazel's Remote Execution API becoming something of a de facto standard with many backend implementations (like buildbarn, buildfarm, BuildGrid, and scoot) and multiple frontends (like Bazel, BuildStream, and recc). We've been pondering on how best to make forward progress on all the other things
we'd like to do with llbuild, while also taking into account some of the existing layering challenges in llbuild.
About a month ago, @David_M_Bryson started playing around with a concept of doing a new implementation of llbuild's API, but sitting atop NIO's Futures. We've since written a number of experiments atop it which make us think this is now the direction we'd like to go for a more modern API. At the same time, we've been having a lot of discussions with @Aciid and @sergiocampama about how we should go about supporting Swift "explicit modules" in the build system. One thing that quickly emerged in those discussions is there is an opportunity to get much more alignment between llbuild, SwiftPM, swift-driver, and the Swift compiler.
With that in mind, we are going to try something new... llbuild2! This an experiment, we don't yet know exactly how it will pan out, but our intention is to try it for a bit and see where it takes us. If we like it, we will most likely fold it back into the primary llbuild repository as the future direction for its API. Here is what we are planning:
- We've created a new
swift-llbuild2repository to hold this work. This will be a pure Swift package for the time being (no CMake, and not worrying about bootstrap concerns yet).- We will most likely use the existing
llbuildas a dependency, and for sinking performance critical code into C++ (where necessary). - We've moved the
cevobuildexperiment out intollbuild2, and will begin to build up more, better, higher-level build system layers using the lessons we've learned. - Unlike
llbuild, which tried very hard to have minimal dependencies, we can be much more open on taking on Swift package dependencies to let us build up functionality faster or in a more maintainable way. - In particular, we'd like to adopt Bazel's Remote Execution API so we can begin supporting some forms of automatic distributed builds on top of
llbuild2. We also intend to adoptgRPC-Swiftfor RPC support, when useful inother contexts. - We will also be creating a
swiftpm-on-llbuild2repository (wow, this is a mouthful), which will extend the existingswift-package-managerto usellbuild2for its build system. - Our intention is to use that repository to develop SwiftPM-specific build rules on top of the new llbuild2 model, including work to develop the explicit modules support in
swift-driverto make that performant. (Fast dependency scanning for Swift by DougGregor · Pull Request #28515 · apple/swift · GitHub)- When combined with
llbuild2support for Bazel's Remote Execution API, this should allow us to start exploring fully distributed/cached builds for SwiftPM packages (when a suitable backend is available). - As with
llbuild2, if those experiments work out, we intend to fold that back into the primary repository.
- When combined with
- We will most likely use the existing
We are pretty excited about this new direction... if you are interested in contributing, come join us in #llbuild-dev on the SwiftPM slack.
Cheers,
- Daniel
for that.