Embedding a dylib in a Swift Package

Saturday, February 12, 2022

I have recently been working on building my own Markdown editor for both macOS and iOS so that I can use it to write the articles for this blog.

The project is inspired by objc.io’s concept of Markdown Playgrounds, which is a Markdown editor tailored to writing Swift development articles, with features such as syntax highlighting and integration with the Swift REPL to execute code snippets.

This project was published in 2019, which means that, while the content is still valid, I wanted to take the opportunity and make my own version of it with the newer APIs such as the swift-syntax package from Apple.

In this article, I will go through one of the issues I had when trying to use the swift-syntax package and how I went about solving it.

I am planning on writing a follow-up article on how swift-syntax works and how I designed my CodeHighlighter module in more detail as it has been a great learning experience.

The Problem

Swift-syntax is a collection of Swift bindings for the libSyntax library in the swift toolchain. This library, under the hood, depends on a dynamic library called lib_InternalSwiftSyntaxParser.dylib, which is also part of the Swift toolchain.

What this means is that, to make an application that uses the swift-syntax package, we need to find a way of embedding the .dylib file into our application, as Apple recommend in this package’s Readme file.

In the next few sections, I’ll explain the process I followed, from getting it up and running quickly for a single platform to making it available on all platforms I wanted my app to run on eventually.

Running on macOS

My initial intention was to only build my new markdown editor for macOS, which meant that I was happy to use the toolchain version of the library, with no need of building the iOS equivalents from source, as the documentation suggests.

It is important to understand some of the choices taken later on that my application had a modular architecture and swift-syntax would be used from within a swift package itself.

Embedding the .dylib directly

To achieve this, I decided to embed the library directly in the app target instead of attempting to ship it with the Swift Package itself - I actually had no idea how I would go about this 😅:

As you can see, this can be done directly by going to the Frameworks, Libraries, and Embedded Content section in the target’s General settings tab.

The rpath solution

While my app was building and running fine at this stage, it required the .dylib file to be embedded in the target’s side. This was hurting the portability of my new CodeHighlighter package because, if I were to move it to a separate project, it would not run out of the box. I would need to copy the .dylib required file from the toolchain and embed it in the app’s target.

Ideally, what I wanted to achieve, was that a single import was enough and that the Swift Package would ship with the library embedded and linked dynamically. But how do you link a library in a Swift Package?

Setting a linker setting

The first thing we need to understand is that for the linker to find the dynamic library, we need to set its Runtime Search Path (rpath). When working with xcodeproj targets, this is usually set for you to some default search paths and it is very easily customisable by appending paths to the LD_RUNTIME_SEARCH_PATH setting.

I am not going to go into much detail on how rpaths works (Marcin Krzyzanowski has an amazing article on this if you want to read more about it) but, to get it to work, we need to pass a path of the directory in which to find this .dylib to the linker.

This is usually set by using passing an -rpath flag to the linker’s command and, in my case, I am going to make this an absolute path, but this could be based on the @executable_path or @loader_path based on your needs.

Luckily, it is not too hard to do this for Swift Packages. Each target can be passed a set of linkerSettings and, by using the .unsafeFlag method, we can funnel in any flag-value combination we like. In the snippet below, it is sufficient to set the rpath to the path where the Package.swift lives, which happens to be where the .dylib file is also located.

// swift-tools-version:5.5
import PackageDescription
import Foundation

let package = Package(
		// ...
    targets: [
        .target(name: "CodeHighlighter",
                dependencies: [
		                .product(name: "SwiftSyntax", package: "swift-syntax")
		            ],
                linkerSettings: [
		                .unsafeFlags([
				                "-rpath",
					             URL(fileURLWithPath: #file).deletingLastPathComponent().path
					         ])
		            ]
		    ),
    ]
)

A ‘universal’ approach

I quickly reached a point where I wanted to try and get it to build on iOS devices and this is where the headaches really started 😅.

Neither of the approaches above were sufficient, because the .dylib was built for the wrong host (it was built for macosx when I also wanted iphonesimulator and iphoneos) and, as soon as I ran it, I was prompted with the following error:

I needed to do a couple things in order to get the app building on multiple platforms:

• Build the iOS libraries, as stated by SwiftSyntax’s README.md. This is because in the toolchain, only the macOS binaries are provided.
• Find a way to bundle multiple dylibs for different platforms in a single framework and distribute it with my CodeHighlighter package.

Creating the iOS libraries

I started by creating a convenience make-ios-syntax-parser.sh bash script which took care of building the multiple libraries. I followed the instructions from the Readme file and made it call the following commands:

#!/usr/bin/env bash
set -euo pipefail
set -x

# Checkouts
if [ ! -d ".checkouts" ]; then
  mkdir .checkouts && pushd .checkouts
  git clone https://github.com/apple/swift.git
  ./swift/utils/update-checkout --clone --scheme release/5.5
  popd
fi

# Clean before building
rm -rf build && mkdir build && pushd build

# Build Simulator dylib
../.checkouts/swift/utils/build-parser-lib --release --no-assertions --build-dir /tmp/parser-lib-build-iossim --host iphonesimulator --architectures x86_64
# Build Device dylib
../.checkouts/swift/utils/build-parser-lib --release --no-assertions --build-dir /tmp/parser-lib-build-ios --host iphoneos --architectures arm64
# Build macOS dylib
../.checkouts/swift/utils/build-parser-lib --release --no-assertions --build-dir /tmp/parser-lib-build --architectures x86_64

The script above clones the Swift repo and uses the update-checkout script to check out the current version I will be building with: release/5.5. This is very important as the libraries need to be built with the same version as the toolchain the client will be building with.

After this, it is a matter of compiling all of the libraries, both for device and simulator, with the correct hosts and architectures - note that the current machine architecture (in my case it is arm64) always gets appended to the list you pass in, so for the simulator, I just had to pass x86_64 to build both.

The output of the commands above will have created a .dylib with the two simulator slices (x86_64 for intel-based macOS computers and arm64 for Apple Silicon computers), a separate .dylib with the arm64 slice built for the iphoneos host and, lastly, a .dylib with the arm64 and x86_64 slices for the macosx host. How do we put all these together into something we can use in our app?

Bundling all architectures in an xcframework

The answer is to use a xcframework! It is a feature that was announced in WWDC2019 which put an end to the hacky fat frameworks approach that consisted in bundling all slices for both device and simulator architectures into a single binary.

In fact, using these also called universal binaries becomes an issue when trying to build for simulators running on Apple Silicon machines, as these have the same architecture as devices (arm64) and two slices built for the same architecture can not be included in the same binary.

xcframeworks, on the other hand, work in such a way that we can bundle multiple frameworks or libraries that serve different purposes and platforms (one for macOS, one for iOS devices and one for iOS simulators in my case) and import them directly into our applications. Then Xcode will do the magic of deciding which library to use based on the device the application will be running on.

xcrun xcodebuild -create-xcframework \
  -library artifacts/simulator/lib_InternalSwiftSyntaxParser.dylib -headers artifacts/include \
  -library artifacts/device/lib_InternalSwiftSyntaxParser.dylib  -headers artifacts/include \
  -library artifacts/macosx/lib_InternalSwiftSyntaxParser.dylib  -headers artifacts/include \
  -output InternalSwiftSyntaxParser.xcframework

Running xcodebuild with the -create-framework flag and making use of the -library flags to pass in any libraries we want to bundle and a path to the headers is sufficient, and outputs an .xcframework file that can be directly embedded into apps.

Binary Targets

In Xcode projects, it is enough to drag and drop the generated .xcframework file but, as I mentioned earlier, I had to find a way of adding it to the Swift Package that I was planning on distributing to meet my self-imposed requirements.

Thankfully, this is pretty easy to do and all that is required is creating a .binaryTarget, which is provided a path to the xcframework file and then can then be added as a dependency to the CodeHighlighter Swift Package like so:

// swift-tools-version:5.5
// The swift-tools-version declares the minimum version of Swift required to build this package.

import PackageDescription

let package = Package(
    name: "CodeHighlighter",
    products: [
        .library(
            name: "CodeHighlighter",
            targets: ["CodeHighlighter"]),
    ],
    dependencies: [
        .package(url: "https://github.com/apple/swift-syntax.git", revision: "593d01f4017cf8b71ec28689629f7b9a6739df0b")
    ],
    targets: [
        .target(
            name: "CodeHighlighter",
            dependencies: [.product(name: "SwiftSyntax", package: "swift-syntax"), "lib_InternalSwiftSyntaxParser"]),
        .testTarget(name: "CodeHighlighterTests", dependencies: ["CodeHighlighter"]),
        .binaryTarget(name: "lib_InternalSwiftSyntaxParser", path: "InternalSwiftSyntaxParser.xcframework")
    ]
)

This solution allowed me to import the package client-side with no need of embedding the library manually and get to use it in both macOS and iOS devices.

Testing the end result

The last thing to do was to check that the approach worked. In order to do so, I created a test SwiftUI app, with a single Text field and a code snippet that can be highlighted with the brand new CodeHighlighter package.

I created the project as a multi-platform SwiftUI app so that I could run it for both macOS and iOS and this is the result I got! It ran without any issues and worked as expected! 🎉