Xcode 15 and Swift 5.9 are finally here, and that means that one of the most celebrated new features of Swift is available to us all: macros!

Macros are little compiler plugins that allow us to extend the Swift language in ways that do not require going through Swift evolution, or waiting for a Swift core team member to implement the feature, or require us learning C++ so that we can try implementing the feature ourselves. It is an incredibly powerful new feature, and we’re excited to finally get our hands on it.

Macros are going to play a huge role in how we evolve some of our most popular open source libraries, such as the Composable Architecture, Case Paths and more, but today we want to talk about something more foundational. And that is, how do you debug and test macros?

Because macros are compiler plugins that are run while your app is compiling, you can’t simply put a breakpoint in the macro code and step through it line-by-line. And this can be incredibly frustrating since SwiftSyntax, which is important for writing macros, is a complex beast that can take a lot of trial-and-error to get right.

And it’s for this reason that SwiftSyntax provides a tool for testing macros. It allows you to provide a string with some sample code that uses a macro, and then you provide another string with the code that the macro expands to. If the expanded code you specified doesn’t match the true expanded code you get a test failure, and the failure message is even nicely formatted to show you exactly which lines didn’t match.

So, that’s great that there’s a tool provided for testing, but it does have some problems. And I think it gives us an opportunity to rethink how one tests macros in Swift. We will show we can leverage one of our first ever open source libraries to provide a much nicer tool than the default tool that SwiftSyntax ships, and we’ll even add a few bells and whistles along the way.

Let’s first start by seeing what tool SwiftSyntax gives us out of the box.

Let’s start by creating a new Swift package and using Xcode’s “Macro” template, and we will just name it “Experimentation.”

This sets up all the basic infrastructure one needs in order to write a macro in Swift, because there are quite a few initial steps to get started.

The first interesting thing in the template is the Package.swift file:

// swift-tools-version: 5.9
// The swift-tools-version declares the minimum version
// of Swift required to build this package.

import PackageDescription
import CompilerPluginSupport

let package = Package(
  name: "Experimentation",
  platforms: [
    .macOS(.v10_15),
    .iOS(.v13),
    .tvOS(.v13),
    .watchOS(.v6),
    .macCatalyst(.v13)
  ],
  products: [
    // Products define the executables and libraries a
    // package produces, making them visible to other
    // packages.
    .library(
      name: "Experimentation",
      targets: ["Experimentation"]
    ),
    .executable(
      name: "ExperimentationClient",
      targets: ["ExperimentationClient"]
    ),
  ],
  dependencies: [
    // Depend on the latest Swift 5.9 prerelease of
    // SwiftSyntax
    .package(
      url: "https://github.com/apple/swift-syntax.git",
      from: """
        509.0.0-swift-5.9-DEVELOPMENT-SNAPSHOT-2023-04-25-b
        """
      ),
  ],
  targets: [
    // Targets are the basic building blocks of a package,
    // defining a module or a test suite.
    // Targets can depend on other targets in this package
    // and products from dependencies.
    // Macro implementation that performs the source
    // transformation of a macro.
    .macro(
      name: "ExperimentationMacros",
      dependencies: [
        .product(
          name: "SwiftSyntaxMacros",
          package: "swift-syntax"
        ),
        .product(
          name: "SwiftCompilerPlugin",
          package: "swift-syntax"
        )
      ]
    ),
    
    // Library that exposes a macro as part of its API,
    // which is used in client programs.
    .target(
      name: "Experimentation",
      dependencies: ["ExperimentationMacros"]
    ),
    
    // A client of the library, which is able to use the
    // macro in its own code.
    .executableTarget(
      name: "ExperimentationClient",
      dependencies: ["Experimentation"]
    ),
    
    // A test target used to develop the macro
    // implementation.
    .testTarget(
      name: "ExperimentationTests",
      dependencies: [
        "ExperimentationMacros",
        .product(
          name: "SwiftSyntaxMacrosTestSupport",
          package: "swift-syntax"
        ),
      ]
    ),
  ]
)

In particular, there is a new kind of import:

import CompilerPluginSupport

…which gives you access to a new kind of target called a .macro target:

.macro(
  name: "ExperimentationMacros",
  dependencies: [
    .product(
      name: "SwiftSyntaxMacros",
      package: "swift-syntax"
    ),
    .product(
      name: "SwiftCompilerPlugin",
      package: "swift-syntax"
    )
  ]
),

This special kind of target is what allows you to define a compiler plugin so that Swift can execute your macro while it is compiling the rest of your application.

And there’s a dependency on Apple’s SwiftSyntax project already added:

dependencies: [
  // Depend on the latest Swift 5.9 prerelease of
  // SwiftSyntax
  .package(
    url: "https://github.com/apple/swift-syntax.git",
    from: """
      509.0.0-swift-5.9-DEVELOPMENT-SNAPSHOT-2023-04-25-b
      """
    ),
],

While it is technically possible to implement macros without depending SwiftSyntax, it is quite difficult. The SwiftSyntax package gives you a very high level tool for analyzing and iterating over Swift code.

However, depending on SwiftSyntax does have its downsides. It is a super heavyweight dependency. A clean build from scratch takes over 30 seconds on an M1 MacBook Pro in DEBUG mode, but it can take over 4 minutes for release builds. This is likely to massively bloat your builds, even if you follow the highly modular feature approach that we have discussed many times on Point-Free. Because if you have a nice, slim feature module that compiles super quickly, the moment you decide to use a macro in that module you may immediately incur a 30 second build penalty. And that’s a real bummer.

SwiftSyntax also doesn’t have a great story for bringing in libraries that depend on different versions of SwiftSyntax. It is going to be very complicated to depend on or manage a library that depends on SwiftSyntax while supporting multiple versions of Swift, and it will be quite easy for different libraries to conflict in their usage of SwiftSyntax. We have opened a discussion on the Swift forums about these issues, so Apple is aware of the problems but there are no solutions as of the recording of this episode.

But moving on, in addition to this “ExperimentationMacros” target there is also a “ExperimentationClient” client, which is an executable. This is a completely optional target and not actually needed to define a macro. Its only purpose is to show off the usage of the macro.

For example, the macro that the template provides as a sample is called #stringify. It can take any Swift expression and turn it into a tuple that holds the result of the expression, as well as a string representation of the expression:

import Experimentation

let a = 17
let b = 25

let (result, code) = #stringify(a + b)

print(
  """
  The value \(result) was produced by the code "\(code)"
  """
)

Running this will print:

The value 42 was produced by the code "a + b”

…to the console. After about 30 seconds of compiling SwiftSyntax for the first time, that is.

While not exactly an amazingly useful macro, it does show the beginnings of what macros are capable of achieving. This was just not possible in Swift prior to macros. The macro gets access to the actual expression that generates a value so that it can turn it into a string, whereas typical Swift code only gets access to the final, evaluated value.

The way this works is that the #stringify macro is actually a whole Swift program in its own right that is provided the expression inside the parentheses as an abstract syntax tree, and then it can return a whole new syntax tree to be substituted into the final compiled program.

You can even right click #stringify and click “Expand Macro” to see what the macro evaluates to:

 let (result, code) = #stringify(a + b)
+(a + b, "a + b")

So, how is this macro actually defined? Well, there are two separate modules to accomplish this: one for defining the signature of the macro and another for defining the implementation.

There is a target provided by the template that matches the package name, in this case “Experimentation”, and it is a plain library that defines the signatures of the macro you want to vend to the public. This is where you get to decide what type of macro you are exposing, such as a freestanding macro, or an extension macro, et cetera. And you define what parameters it takes as arguments, as well as what it returns, if relevant.

By default, the Xcode template starts us off with a “freestanding” expression macro called stringify:

@freestanding(expression)
public macro stringify<T>(_ value: T) -> (T, String) =
  #externalMacro(
    module: "ExperimentationMacros",
    type: "StringifyMacro"
  )

There’s a lot going on in this signature.

First, there’s this @freestanding(expression). This determines what kind of macro is being defined. There are currently two main buckets of macros: freestanding and attached. And more may be coming soon.

And within each of those buckets there is even more variety. We aren’t going to get into all of those details right now because we are not trying to teach macros from first principles, but instead trying to only explore how one debugs and tests macros.

Next there is a public macro declaration, which can be thought of as being similar to public func, except it defines a macro instead of a function. It even behaves a lot like a function signature, in that you can specify arguments and a return type and can even introduce generics. But, unlike functions, it does not have a body enclosed by curly braces. Instead is assigned to this #externalMacro thing, which allows you to describe where the implementation of the macro is housed.

You do not put the implementation of the macro in this module. This module is only for the signature. Instead, the “ExperimentationMacros” module has a type called StringifyMacro, and that is where the actual macro implementation lives.

If we hop over to the file in that module we will see the following:

import SwiftCompilerPlugin
import SwiftSyntax
import SwiftSyntaxBuilder
import SwiftSyntaxMacros

public struct StringifyMacro: ExpressionMacro {
  public static func expansion(
    of node: some FreestandingMacroExpansionSyntax,
    in context: some MacroExpansionContext
  ) -> ExprSyntax {
    guard
      let argument = node.argumentList.first?.expression
    else {
      fatalError(
        """
        compiler bug: the macro does not have any arguments
        """
      )
    }

    return """
      (\(argument), \(literal: argument.description))
      """
  }
}

@main
struct ExperimentationPlugin: CompilerPlugin {
  let providingMacros: [Macro.Type] = [
    StringifyMacro.self,
  ]
}

There are two main things accomplished in this file. First, the StringifyMacro type is defined that conforms to the ExpressionMacro protocol, and it provides the implementation of the #stringify macro.

It is handed a value that represents the syntax tree provided to the macro, and its job is to return a new syntax tree that it wants to insert back into the source code:

public static func expansion(
  of node: some FreestandingMacroExpansionSyntax,
  in context: some MacroExpansionContext
) -> ExprSyntax {

And it does the simplest thing possible, which is return a string that holds the contents of the code that the #stringify macro should be expanded to. In this case, it’s a tuple with the original argument interpolated in, as well as a second component with the description of the argument interpolated in as a literal. This is what allows us to expand #stringify into a tuple that holds exactly what was provided to the macro, alongside a string representation of that argument.

Then right below the StringifyMacro type we have the second responsibility of this module, which is to describe the compiler plugin that powers the macro:

@main
struct ExperimentationPlugin: CompilerPlugin {
  let providingMacros: [Macro.Type] = [
    StringifyMacro.self,
  ]
}

We do this by having a @main entry point type that conforms to the CompilerPlugin protocol and provides a list of all the macros that are vended by this module.

This is what allows the Swift compiler to discover what macros are available and then execute them while compiling your project. If we had more macros in this library we would just add more entries to this array.

And that’s all it takes to provide the implementation of the macro. Now you may wonder why is the signature of the macro put in a completely separate module from the implementation? When Swift was first released 9 years ago it made the very conscious decision to get rid of signatures being defined separately from implementation, like was common in Objective-C.

Well, the reason for this is that the implementation of the macro does not need to be included in your application binary at all. Its sole purpose is to transform your program’s syntax as Swift is compiling your application, but the implementation code of the macro of itself does not need any representation in your binary. And that’s a good thing because one needs to compile SwiftSyntax for macros, and that is a very hefty dependency. You wouldn’t want all that bloat added to your application’s binary just because you are using a macro. So, the only thing your app needs to use the macro is just the signature, and so that is why the signature is separated from the implementation.

So, this is looking great, but how does one debug and test a macro? When I look at the StringifyMacro type’s method:

public struct StringifyMacro: ExpressionMacro {
  public static func expansion(
    of node: some FreestandingMacroExpansionSyntax,
    in context: some MacroExpansionContext
  ) -> ExprSyntax {
    …
  }
}

…it looks pretty intimidating! There are 3 protocols involved, ExpressionMacro, FreestandingMacroExpansionSyntax, and MacroExpansionContext, as well as a struct ExprSyntax, and they are pretty abstract concepts. I’m not really sure how to analyze or manipulate them. I’d love if I could put a breakpoint in here:

guard let argument = node.argumentList.first?.expression else {

…so that I could print things to the console and explore what I have access to.

However, how could this breakpoint ever catch? This code is not run when we run our application. It runs when our application is compiled. And if I try compiling the ExperimentationClient executable we will see that the breakpoint does not catch. Nor does it catch if I run the executable.

Well, luckily Apple and the SwiftSyntax library provides a tool to help with this. There is a way to execute your macro’s code directly without it being apart of the compiler plugin system. You do so in a test, and the template even comes with a test for the #stringify macro to show you how it’s done:

import SwiftSyntaxMacros
import SwiftSyntaxMacrosTestSupport
import XCTest

// Macro implementations build for the host, so the
// corresponding module is not available when
// cross-compiling. Cross-compiled tests may still make
// use of the macro itself in end-to-end tests.
#if canImport(ExperimentationMacros)
  import ExperimentationMacros

  let testMacros: [String: Macro.Type] = [
    "stringify": StringifyMacro.self
  ]
#endif

final class ExperimentationTests: XCTestCase {
  func testMacro() throws {
    #if canImport(ExperimentationMacros)
      assertMacroExpansion(
        """
        #stringify(a + b)
        """,
        expandedSource: """
          (a + b, "a + b")
          """,
        macros: testMacros
      )
    #else
      throw XCTSkip(
        """
        macros are only supported when running tests for \
        the host platform
        """
      )
    #endif
  }

  func testMacroWithStringLiteral() throws {
    #if canImport(ExperimentationMacros)
      assertMacroExpansion(
        #"""
        #stringify("Hello, \(name)")
        """#,
        expandedSource: #"""
          ("Hello, \(name)", #""Hello, \(name)""#)
          """#,
        macros: testMacros
      )
    #else
      throw XCTSkip(
        """
        macros are only supported when running tests for \
        the host platform
        """
      )
    #endif
  }
}

There’s a lot going on here, but before even describing it all, let’s just run the test suite.

We instantly get caught in the breakpoint we created a moment ago. And now we can use lldb to explore all the arguments we have access to. Well, at least theoretically. If we try to print out one of the arguments, such as the node, it does not seem to work:

(lldb) po node 
error: <EXPR>:8:1: error: cannot find 'node' in scope
node 
^~~~

Nor does it work for the context.

Turns out this is just a bug in Xcode and lldb when it comes to opaque types, in particular the some types we see in the signature of expansion. If we convert those to simple generics:

public static func expansion<
  F: FreestandingMacroExpansionSyntax,
  C: MacroExpansionContext
>(
  of node: F,
  in context: C
) -> ExprSyntax {
  …
}

…and run again, then it works just fine to po the node and context:

(lldb) po node 
MacroExpansionExprSyntax
├─pound: pound
├─macroName: identifier("stringify")
├─leftParen: leftParen
├─arguments: LabeledExprListSyntax
│ ╰─[0]: LabeledExprSyntax
│   ╰─expression: SequenceExprSyntax
│     ╰─elements: ExprListSyntax
│       ├─[0]: DeclReferenceExprSyntax
│       │ ╰─baseName: identifier("a")
│       ├─[1]: BinaryOperatorExprSyntax
│       │ ╰─operator: binaryOperator("+")
│       ╰─[2]: DeclReferenceExprSyntax
│         ╰─baseName: identifier("b")
╰─rightParen: rightParen
(lldb) po context
<BasicMacroExpansionContext: 0x600001750600>

SwiftSyntax provides a nicely formatted description of the syntax node that we can use to guide us to what is available. And we can even do so right in lldb.

For example, if we want to get the name of the macro we can do:

(lldb) po node.macro.text
"stringify"

Or if we want to explore the arguments supplied to the macro:

(lldb) po node.argumentList
LabeledExprListSyntax
╰─[0]: LabeledExprSyntax
  ╰─expression: SequenceExprSyntax
    ╰─elements: ExprListSyntax
      ├─[0]: DeclReferenceExprSyntax
      │ ╰─baseName: identifier("a")
      ├─[1]: BinaryOperatorExprSyntax
      │ ╰─operator: binaryOperator("+")
      ╰─[2]: DeclReferenceExprSyntax
        ╰─baseName: identifier("b")

And we can take the first argument:

(lldb) po node.arguments.first
▿ Optional<LabeledExprSyntax>
  - some : LabeledExprSyntax
╰─expression: SequenceExprSyntax
  ╰─elements: ExprListSyntax
    ├─[0]: DeclReferenceExprSyntax
    │ ╰─baseName: identifier("a")
    ├─[1]: BinaryOperatorExprSyntax
    │ ╰─operator: binaryOperator("+")
    ╰─[2]: DeclReferenceExprSyntax
      ╰─baseName: identifier("b")

And further the expression of the first argument:

(lldb) po node.arguments.first?.expression
▿ Optional<ExprSyntax>
  - some : SequenceExprSyntax
╰─elements: ExprListSyntax
  ├─[0]: DeclReferenceExprSyntax
  │ ╰─baseName: identifier("a")
  ├─[1]: BinaryOperatorExprSyntax
  │ ╰─operator: binaryOperator("+")
  ╰─[2]: DeclReferenceExprSyntax
    ╰─baseName: identifier("b")

The type returned here, ExprSyntax, is pretty opaque on its own, and doesn’t have much more we can journey to through autocomplete. We can see, though, that it’s actually a SequenceExprSyntax, so let’s cast it that using cast or as:

(lldb) po node.arguments.first?.expression.as(SequenceExprSyntax.self)
▿ Optional<SequenceExprSyntax>
  - some : SequenceExprSyntax
╰─elements: ExprListSyntax
  ├─[0]: DeclReferenceExprSyntax
  │ ╰─baseName: identifier("a")
  ├─[1]: BinaryOperatorExprSyntax
  │ ╰─operator: binaryOperator("+")
  ╰─[2]: DeclReferenceExprSyntax
    ╰─baseName: identifier("b")

And we can dive further into this type’s elements and maybe pluck out the first one:

(lldb) po node.arguments.first?.expression.as(SequenceExprSyntax.self).elements.first
▿ Optional<ExprSyntax>
  - some : DeclReferenceExprSyntax
╰─baseName: identifier("a")

We’re seeing how SwiftSyntax provides everything you need to traverse through the syntax tree.

And you can also print the description of a node in order to see the exact code that it represents:

(lldb) po node.description
"#stringify(a + b)"
(lldb) po node.arguments.first?.expression.as(SequenceExprSyntax.self).elements.first.description 
▿ Optional<String>
  - some : "a "

Exploring SwiftSyntax nodes in this way can be necessary to figure out what is going on unfortunately. The documentation for SwiftSyntax is a bit meek, but hopefully that will be improved as macros become more popular.

If we click continue to let the test finish we will see that the whole suite passes. But what is the test actually testing?

Well, there are two tests, and the first tests its most basic functionality:

func testMacro() throws {
  #if canImport(ExperimentationMacros)
    assertMacroExpansion(
      """
      #stringify(a + b)
      """,
      expandedSource: """
        (a + b, "a + b")
        """,
      macros: testMacros
    )
  #else
    throw XCTSkip(
      """
      macros are only supported when running tests for \
      the host platform
      """
    )
  #endif
}

The interesting part is this right here:

assertMacroExpansion(
  """
  #stringify(a + b)
  """,
  expandedSource: """
    (a + b, "a + b")
    """,
  macros: testMacros
)

This assertMacroExpansion helper is provided by the SwiftSyntax library and it allows you to describe a string of Swift source code that uses a macro, and then assert on what the expanded code looks like after the macro executes. You must also provide this macros argument:

macros: testMacros

And the value provided is defined at the top of the test:

let testMacros: [String: Macro.Type] = [
  "stringify": StringifyMacro.self
]

This is necessary to let SwiftSyntax’s testing tool to know which macros it should try to expand while analyzing the string you provide.

If you make a small change to the expected output, like say leave off the closing quote:

expandedSource: """
  (a + b, "a + b)
  """

Then the test will fail, and it will even provide a helpful failure message

−(a + b, "a + b)
+(a + b, "a + b")

(a + b, "a + b")

So that’s pretty cool.

It’s so easy to write tests that there really isn’t any reason to not write a whole bunch to make sure your macro behaves the way you expect. For example, the template includes another test to prove that it handles more complex expressions correctly, such as when string interpolation is involved:

assertMacroExpansion(
  #"""
  #stringify("Hello, \(name)")
  """#,
  expandedSource: #"""
    ("Hello, \(name)", #""Hello, \(name)""#)
    """#,
  macros: testMacros
)

If the macro naively turned the expression into a string it would have looked like this:

("Hello, \(name)", "Hello, \(name)")

Meaning the stringified expression would have actually been interpolated. But really the macro wants to prevent interpolation and instead show the exact expression that was provided, hence it should expand to:

#""Hello, \(name)""#

The hash marks at either end of the string prevent \(…) from interpolating values.

It can be really important to get test coverage on these edge cases of your macro. After all, at the end of the day we are just manipulating one abstract syntax tree into a whole new one, and there is a very good chance it will not be done correctly, leaving behind syntax that does not compile.

So, this all seems great!

Macros are an amazing new tool for analyzing a piece of Swift syntax before it is compiled and then performing a transformation on the syntax so that something slightly different is actually compiled into the user’s application. And there is even a tool that allows one to test macros in a super convenient manner, and it even has pretty good failure messages when the expanded macro code doesn’t match our expectation.

However, there are some problems with these tools. Don’t get us wrong, it’s a fantastic start, and we are thrilled Apple provided any testing tools at all, but that doesn’t mean there isn’t room for improvement.

Before making a better macro testing tool, let’s quickly see what the problems are with the existing tool.

Let’s perform a seemingly innocuous change to the macro’s output where we put each tuple element on a new line:

return """
  (\n\(argument),\n\(literal: argument.description)\n)
  """

We haven’t changed anything about the actual expanded Swift code other than it’s formatting. Functionally it should behave exactly the same.

But, perhaps we do want to format the code like this so that when one expands the macro in their app code, they see something a little nicer:

 let (result, code) = #stringify(a + b)
+(
+    a + b,
+    "a + b"
+)

This becomes a lot more helpful the more complex the expression becomes:

 let (result, code) = #stringify(
   (a + b) * (a - b) * (b - a)
 )
+(
+    (a + b) * (a - b) * (b - a),
+    "(a + b) * (a - b) * (b - a)"
+)

Comparing this expanded macro to what it would have been before:

((a + b) * (a - b) * (b - a), "(a + b) * (a - b) * (b - a)")

…shows that a little bit of code formatting in the macro can go a long way.

But what did this do to tests?

Well, if we run the test suite we will see that both tests fails:

−(a + b, "a + b")
+(
+    a + b,
+    "a + b"
+)

(
    a + b,
    "a + b"
)

−("Hello, \(name)", #""Hello, \(name)""#)
+(
+    "Hello, \(name)",
+    #""Hello, \(name)""#
+)

(
    "Hello, \(name)",
    #""Hello, \(name)""#
)

And this is to be expected. We did change the formatting, and so our expectation of what the expanded source looks like is wrong. And so we should have to fix this test to correct the expectation and make sure that our formatting didn’t accidentally introduce a bug anywhere.

So, how do we fix these tests? Well, I suppose you could just manually apply the changes:

assertMacroExpansion(
  """
  #stringify(a + b)
  """,
  expandedSource: """
    (
      a + b,
      "a + b"
    )
    """,
  macros: testMacros
)

 (
−  a + b,
−  "a + b"
+    a + b,
+    "a + b"
 )

(
    a + b,
    "a + b"
)

But we can get failures if we don’t get it just right, like matching all the indentation.

assertMacroExpansion(
  """
  #stringify(a + b)
  """,
  expandedSource: """
    (
        a + b,
        "a + b"
    )
    """,
  macros: testMacros
)

It’s easy enough to fix, but did require some iteration.

We have our test passing, but also it’s not going to scale. If you have a very complex macro, then manually updating the expandedSource string is going to be very difficult.

For example, a seemingly simple class with the @Observable macro applied:

import Observation
@Observable
class Model {
  var count = 0
  var isLoading = false
  var title = ""
}

…expands to 20+ lines of new code:

 import Observation
 @Observable
 class Model {
   var count = 0
   var isLoading = false
   var title = ""
+  @ObservationIgnored
+  private let _$observationRegistrar =
+    Observation.ObservationRegistrar()
+  
+  internal nonisolated func access<Member>(
+    keyPath: KeyPath<Model, Member>
+  ) {
+    _$observationRegistrar.access(self, keyPath: keyPath)
+  }
+  
+  internal nonisolated func withMutation<
+    Member, MutationResult
+  >(
+    keyPath: KeyPath<Model, Member>,
+    _ mutation: () throws -> MutationResult
+  ) rethrows -> MutationResult {
+    try _$observationRegistrar.withMutation(
+      of: self, keyPath: keyPath, mutation
+    )
+  }
+  
+  @ObservationIgnored private var _count  = 0
+  
+  @ObservationIgnored private var _isLoading  = false
+  
+  @ObservationIgnored private var _title  = ""
 }

And there’s even more macro expansion to be performed for each field in the class.

Manually maintaining such an expandedSource string is going to be very difficult and error prone.

Well, another option is to just grab the final expanded source string from the test failure message itself:

−("Hello, \(name)", #""Hello, \(name)""#)
+(
+    "Hello, \(name)",
+    #""Hello, \(name)""#
+)

(
    "Hello, \(name)",
    #""Hello, \(name)""#
)

We can see the final expanded source string right in the message, and so we can copy and paste it:

expandedSource: #"""
  (
      "Hello, \(name)",
      #""Hello, \(name)""#
  )
  """#,

If we’re lucky enough, it will paste correctly and we will have a passing test, but if we don’t paste in just the right way, the string will lose its indentation formatting and either immediately cause a compiler error or later cause a test failure.

Well, if done so naively you lose the indentation formatting in Xcode:

expandedSource: #"""
  (
  "Hello, \(name)",
  #""Hello, \(name)""#
  )
  """#,

So that’s actually a pretty big pain. Not just to format the string to match the expected source, but even copying the final expanded source from the test failure message can be a pain. We were lucky that the expanded source is just a few lines, but in practice it can be dozens or even hundreds of lines, as we see with the fully-expanded @Observable macro. And Xcode currently has a bug in its display of failure messages where if it is too long it does not even show in this little popover. You will instead have to hunt the failure message down in the console or in the report navigator.

And just to show what a pain all of this is, let’s make another small change to the macro by giving names to the tuple elements:

return """
  (
  value: \(argument),
  string: \(literal: argument.description)
  )
  """

This of course fails the test suite because now the expanded source has named tuple elements:

 (
−    a + b,
−    "a + b"
+    value: a + b,
+    string: "a + b"
 )

(
    value: a + b,
    string: "a + b"
)

 (
−    "Hello, \(name)",
−    #""Hello, \(name)""#
+    value: "Hello, \(name)",
+    string: #""Hello, \(name)""#
 )

(
    value: "Hello, \(name)",
    string: #""Hello, \(name)""#
)

To fix this we can make the change to the expanded source string directly, or we can copy and paste the contents of the failure message into the test.

So, while it was nice that SwiftSyntax gave us some ability to test our macros, the ergonomics are not quite there. We have to go through many laborious steps if we want to just update our test to have the freshest expanded source code from the macro.

But that’s not the only problem. The assertMacroExpansion tool unfortunately doesn’t make it super easy to assert against any diagnostics your macro emits while executing. This is actually a really powerful feature of macros, but the macro that comes with the template doesn’t really show off this power.

While executing the logic of your macro to transform some existing syntax tree into a new one, you can emit diagnostic errors, warnings, notes or even fix-its. This allows you to detect potential problems in the code that the macro is applied to and give the user a specific message of what they can do to fix it.

For example, right now the StringifyMacro does do a little bit of validation logic before returning a new piece of syntax by making sure that at least one argument is provided to the macro:

guard
  let argument = node.argumentList.first?.expression
else {
  fatalError(
    """
    compiler bug: the macro does not have any arguments
    """
  )
}

However, the macro has decided to handle this validation error by performing a fatalError. And this actually makes sense in this case because it should be absolutely impossible for this macro to start executing with no arguments. After all, the signature of the macro requires exactly one argument to be provided:

public macro stringify<T>(_ value: T) -> (T, String) = …

…and so the Swift compiler can catch if you don’t do this before even trying to execute the macro:

let (result, code) = #stringify()

This produces a compiler error without ever even touching your macro code:

So the stringify macro doesn’t really have a lot of need for diagnostics. But that isn’t always the case. More complicated macros do need this kind of functionality.

For example, the Observation macro is only allowed to be applied to classes. It cannot be applied to structs, enums or even actors:

import Observation
@Observable
struct Model {}

This produces the compiler error:

And it’s the @Observable macro that is producing this diagnostic, not the Swift compiler itself. The @Observable macro performs some validation logic to make sure that the thing the macro is attached to is a class, and if not it throws a diagnostics error, which bubbles up as a compiler error in Xcode.

Since the @Observation protocol is a part of the public, open source Swift code base we can even hop over to the Swift project on GitHub, navigate to ObservableMacro.swift, and find the exact place this validation happens:

if declaration.isEnum {
  // enumerations cannot store properties
  throw DiagnosticsError(
    syntax: node,
    message: """
      '@Observable' cannot be applied to enumeration \
      type '\(observableType.text)'
      """,
    id: .invalidApplication
  )
}
if declaration.isStruct {
  // structs are not yet supported;
  // copying/mutation semantics tbd
  throw DiagnosticsError(
    syntax: node,
    message: """
      '@Observable' cannot be applied to struct \
      type '\(observableType.text)'
      """,
    id: .invalidApplication
  )
}
if declaration.isActor {
  // actors cannot yet be supported for their isolation
  throw DiagnosticsError(
    syntax: node,
    message: """
      '@Observable' cannot be applied to actor \
      type '\(observableType.text)'
      """,
    id: .invalidApplication
  )
}

These diagnostics are an intrinsic part of the macro itself, not some aberration. They give very useful information to the user on what went wrong and how to fix it. So you will want test coverage on these diagnostics in order to make sure that you are communicating properly to the user what went wrong and what they can do to fix it.

How can that be done?

Well, first we need a diagnostic to actually test. Let’s add a very silly diagnostic. Let’s say that if the expression the user provides to the macro is “too long” then we will throw an error and not allow the macro to be used. We of course would never want to do such a thing in real life, but we need some way of seeing a diagnostic.

The easiest way to do this is to just throw an error in the expansion method:

public struct StringifyMacro: ExpressionMacro {
  public static func expansion<
    F: FreestandingMacroExpansionSyntax,
    C: MacroExpansionContext
  >(
    of node: F,
    in context: C
  ) throws -> ExprSyntax {
    …

    guard argument.description.count < 20
    else {
      throw SomeError()
    }

    return """
      (\n\(argument),\n\(literal: argument.description)\n)
      """
  }
}
struct SomeError: Error {}

So if the expression provided is more than 20 characters, we will fail to compile the macro.

And in fact, our ExperimentationClient is already failing to build:

let (result, code) = #stringify(
  (a + b) * (a - b) * (b - a)
)

The error message is very basic, but with a bit more work you can be very specific with the message, and can even point to an exact line and column number, highlight a range of characters in the source code, and more. But we don’t need any of that power right now.

So, how can we write a test or this?

Well, let’s write a new test that asserts against a usage of the macro that is provided too long of an expression:

func testMacro_LongArgumentDiagnostic() throws {
  assertMacroExpansion(
    """
    #stringify((a + b) * (a - b) * (b - a))
    """,
    expandedSource: """
      #stringify((a + b) * (a - b) * (b - a))
      """,
    macros: testMacros
  )
}

I am asserting that the expandedSource is the same as the input source because we don’t expect there to be any expansion performed.

If we run this it fails letting us know that a diagnostic was emitted that we did not assert against:

This is a great test failure to have, and it’s great that assertMacroExpansion does this. It makes it so that we have to assert on everything the macro is doing, including its diagnostics. And the way you do this is by providing the diagnostics argument to the assertMacroExpansion helper:

diagnostics: <#[DiagnosticSpec]#>

This takes an array of DiagnosticSpecs, each of which describes the exact diagnostic emitted by the macro.

A DiagnosticSpec can be created by specify all the various properties of a diagnostic, such as its message, line, number, severity, and more:

DiagnosticSpec(
  id: <#MessageID?#>, 
  message: <#String#>, 
  line: <#Int#>, 
  column: <#Int#>, 
  severity: <#DiagnosticSeverity#>, 
  highlight: <#String?#>, 
  notes: <#[NoteSpec]#>, 
  fixIts: <#[FixItSpec]#>
)

Our macro emits a single error diagnostic, and we saw exactly what it looks like a moment ago in the client executable, so perhaps it’s as simple as:

diagnostics: [
  DiagnosticSpec(
    message: "SomeError()", line: 1, column: 1
  ),
],

Well, unfortunately that still fails:

Currently there is a bug in SwiftSyntax that causes this diagnostic to be emitted twice, but there is a proposed fix on the repo so hopefully it will be merged and available soon. But in the meantime we can fix our test by just specifying two diagnostic specs:

diagnostics: [
  DiagnosticSpec(
    message: "SomeError()", line: 1, column: 1
  ),
  DiagnosticSpec(
    message: "SomeError()", line: 1, column: 1
  ),
],

And now the test passes.

So, it’s nice that we can get some basic test coverage on these diagnostics of our macro, but also this isn’t a very friendly way to do it. The assertion of diagnostics is a completely separate process from the source code that is specified even though diagnostics are intimately related to an example line and column in the source code. Sure we get to specify line 1, column 1 in the assertion, but it’s hard to visualize where those abstract numbers correspond to in the source, especially for a large chunk of code.

So, we’ve see over and over again that while the assertMacroExpansion test helper is powerful, and we are thankful that Apple provided it, it does have quite a bit of room for improvement. When macros change we have to manually copy-and-paste code from a test failure message and reformat it in order to get the test passing again. And testing diagnostics is a little abstract and hard to visualize the exact line and column the diagnostic is associated with.

There’s also some difficulty in testing fix-its, which we haven’t looked into at all yet. A fixit is a kind of diagnostic, either an error or warning, that allows the user to automatically apply a change to their source code to fix the diagnostic. Testing these with Apple’s tool is also a little strange, but fix-its are a pretty advanced topic of macros so we aren’t going to worry about that for now. We will be getting into those details a bit later in this series though.

And so how can we improve on Apple’s macro testing helper? Well, we think we have just the tool for massively improving the testing of macros. In fact, in unison with the release of this episode we have released a brand new open source project that provides a more powerful assertMacro test helper, and surprisingly it is actually built on the back of our very popular snapshot testing library.

So, let’s bring in that library, and see how it improves testing this very simple stringify macro.

We are going to add a dependency to our package on our swift-macro-testing package:

dependencies: [
  // Depend on the latest Swift 5.9 prerelease of
  // SwiftSyntax
  .package(
    url: "https://github.com/apple/swift-syntax.git",
    from: """
      509.0.0-swift-5.9-DEVELOPMENT-SNAPSHOT-2023-04-25-b
      """
  ),
  .package(
    url: """
      https://github.com/pointfreeco/swift-macro-testing
      """,
    from: "0.1.0"
  ),
],

And we are going to have our test target depend on the “MacroTesting” product in the swift-macro-testing package:

.testTarget(
  name: "ExperimentationTests",
  dependencies: [
    "ExperimentationMacros",
    .product(
      name: "SwiftSyntaxMacrosTestSupport",
      package: "swift-syntax"
    ),
    .product(
      name: "MacroTesting",
      package: "swift-macro-testing"
    )
  ]
),

And with that we can already import the library into our test file:

import MacroTesting

The library comes with basically one testing tool, and it’s called assertMacro:

func testMacro_Improved() throws {
  assertMacro(
    <#macros: [String : Macro.Type]?#>,
    applyFixIts: <#Bool#>,
    record: <#Bool?#>,
    of: <#() throws -> String#>,
    matches: <#(() -> String)?#>
  )
}

It has 5 arguments, only one of which is required. The first argument is the collection of macros that will be used to expand the given source string. This is the same as the macros provided to Apple’s assertMacroExpansion tool:

assertMacro(
  testMacros,
  applyFixIts: <#Bool#>,
  record: <#Bool?#>,
  of: <#() throws -> String#>, 
  matches: <#(() -> String)?#>
)

This argument is optional because we provide an alternative way to specify the macros in a more global fashion.

If you didn’t know this already, there is a method on XCTestCase that can be overridden called invokeTest:

override func invokeTest() {
}

This method is invoked right when it is time to run a particular test method, and in fact, super.invokeTest() is precisely the test method being currently executed:

override func invokeTest() {
  super.invokeTest()
}

This allows you to surround the work of your test with some additional work. That may sound similar to setUp and tearDown:

override func setUp() {
}
override func tearDown() {
}

…but it is subtly different. The setUp and tearDown methods do not work when you need to surround your test in a scoped unit of work, which is exactly the kind of tool our library provides. It’s called withMacroTesting and can be used to specify the test macros a single time for an entire test:

override func invokeTest() {
  withMacroTesting(macros: testMacros) {
    super.invokeTest()
  }
}

In fact, because we no longer have to do this repetitive work, we can ditch the global and inline the macro configuration directly:

macros: ["stringify": StringifyMacro.self]

The library also does a little bit of work for us to map the macro type to the string, by trimming any trailing Macro from the type name, and by lowercasing expression macros, so we can even simplify this further:

macros: [StringifyMacro.self]

With that put into place we no longer need to specify the test macros in each test:

assertMacro(
  applyFixIts: <#Bool#>,
  record: <#Bool?#>,
  of: <#() throws -> String#>,
  matches: <#(() -> String)?#>
)

The next argument will optionally apply any fix-its diagnostics provide when expanding the macro. We won’t worry about this for now, so we’ll remove this argument as well.

assertMacro(
  record: <#Bool?#>,
  of: <#() throws -> String#>, 
  matches: <#(() -> String)?#>
)

The third optional argument is whether or not we’re recording, which we’ll get to in a moment.

assertMacro(
  of: <#() throws -> String#>, 
  matches: <#(() -> String)?#>
)

The next argument is required, and it’s a trailing closure that specifies the string of the source code containing a macro that you want to test. So, let’s keep things simple for right now and just test the stringification of “a + b”:

assertMacro(
  of: { "#stringify(a + b)" }, 
  matches: <#(() -> String)?#>
)

The last argument is a trailing closure that returns a string also, and it represents the expanded macro source code that we want to assert against. This is quite similar to Apple’s assertMacroExpansion helper, but unlike their helper it is optional in our library. So even just this:

assertMacro {
  "#stringify(a + b)"
}

…is already compiling.

How can this be? How is this an assertion at all if we haven’t even specified what we are asserting against?

Well, let’s run the test and see what happens.

And almost as if by magic the source code updated right in Xcode with the fully expanded macro generated Swift code:

func testMacro_Improved() throws {
  assertMacroSnapshot {
    """
    #stringify(a + b)
    """
  } matches: {
    """
    (
        a + b,
        "a + b"
    )
    """
  }
}

And there was a test failure:

And here we are getting a little bit of insight into how our macro testing library is built under the hood. Secretly we are using our snapshot testing library. If you didn’t know already, nearly 5 years ago we released a highly flexible and customizable snapshot testing library.

Snapshot testing is a style of testing where you don’t explicitly provide both values you are asserting against, but rather you provide a single value that can be snapshot into some serializable format. When you run the test the first time, a snapshot is recorded to disk, and future runs of the test will take a new snapshot of the value and compare it against what is on disk. If those snapshots differ, then the test will fail.

Perhaps the most canonical example of this is snapshot testing views into images. This is because testing views can be quite difficult in general. You can sometimes perform hacks to actually assert on what kinds of view components are on the screen and what data they hold, but this often feels like testing an implementation detail. And it’s also possible to perform UI tests, but those are very slow, can be flakey, and test a wide range of behavior that you may not really care about.

Snapshot testing of views allows you to test just the very basics of what a view looks like. For example, we could test a very small, simple SwiftUI view by assert its snapshot as an image like this:

assertSnapshot(
  of: Text("Bye")
    .frame(width: 200, height: 200)
    .background(Color.yellow.opacity(0.2)),
  as: .image
)

But we need to import SwiftUI and SnapshotTesting:

import SnapshotTesting
import SwiftUI

The first time we run this the test fails because there was is no snapshot of this view already on disk:

open "…/__Snapshots__/ExperimentationTests/testMacro_Improved.2.png"

And it even helpfully let’s us know where the new snapshot was recorded so that we can easily preview it.

The next time we run this test it passes because it made a new snapshot of the image and compared it to the previously recorded snapshot. Since nothing changed in the view, the test passes.

But, if we change something in the view:

assertSnapshot(
  of: Text("Hi")
    .frame(width: 200, height: 200)
    .background(Color.yellow.opacity(0.2)),
  as: .image()
)

…then the test fails:

@-
"file:///…/__Snapshots__/ExperimentationTests/testMacro_Improved.2.png"
@+
"file:///…/tmp/ExperimentationTests/testMacro_Improved.2.png"

SnapshotTesting.diffTool = "ksdiff"

And we helpfully get easy links to the expected and actual images so that we can see the difference. Or, if we have an application on our computers that can do image diffing, such as Kaleidoscope, then we can use it:

diffTool = "ksdiff"

Now the test fails with a command that we can copy-and-paste into terminal to open Kaleidoscope and show us a very nice diff of the images:

ksdiff \
  "…/__Snapshots__/ExperimentationTests/testMacro_Improved.2.png" \
  "…/tmp/ExperimentationTests/testMacro_Improved.2.png"

So, this is pretty great, but snapshot testing goes well beyond just snapshotting views into images. You can snapshot any Swift data type into any kind of format you want.

For example, if you have very custom JSON encoding and decoding logic in one of your models, then you probably want to write a test to make sure you are getting everything right. But doing so can be quite onerous. You have to assert against a big, hardcoded JSON string, and you can easily get a test failure if your formatting is slightly wrong.

Well, snapshot testing makes this incredibly easy. You can instantly test any data type that is Codable by turning it into a JSON file:

struct User: Codable {
  let id: Int
  var name: String
}
assertSnapshot(of: User(id: 42, name: "Blob"), as: .json)

Running this fails letting us know that a new file was saved to disk:

open "…/__Snapshots__/ExperimentationTests/testMacro_Improved.1.json"

And that file contains the JSON representation of the data type:

{
  "id" : 42,
  "name" : "Blob"
}

And of course if this data type was a lot more complicated we would have a lot more JSON here.

So this is great, but also sometimes it can be a bit of a pain to have the snapshot stored in an external file, especially for text-based snapshot formats.

Well, our library has another snapshotting tool that makes this a lot nicer, and it is called “inline” snapshots. This was actually a tool first contributed to the library by a Point-Free viewer, Robert J. Chatfield, over 4 years ago.

You can assert an inline snapshot by changing assertSnapshot to assertInlineSnapshot:

assertInlineSnapshot(
  of: User(id: 42, name: "Blob"),
  as: .json
)

And we also have to change import SnapshotTesting to import InlineSnapshotTesting:

import InlineSnapshotTesting

It’s a separate library because it depends on SwiftSyntax, and as we discussed previously SwiftSyntax is a pretty heavyweight dependency that we would not want to force on everyone that uses snapshot testing.

Running this causes the library to see that you are not currently asserting against a particular snapshot, and so generates a fresh one for you in a trailing closure:

assertInlineSnapshot(
  of: User(id: 42, name: "Blob"),
  as: .json
) {
  """
  {
    "id" : 42,
    "name" : "Blob"
  }
  """
}

And you can run this over and over and it will pass, but now the snapshot lives right alongside the value you are snapshotting.

There is one downside to inline snapshots, and it’s that you lose your undo history after the snapshot is recorded. If I type cmd+Z right now we will see that nothing happens. We’re not sure how to preserve the undo history, but hopefully this is just a limitation of Xcode that can be fixed someday.

So now it is clear why our macro assertion tool behaved the way it did. It is technically using the inline snapshotting tool under the hood in order to automatically record the expanded macro when none is provided. And if we make a small change to the expected expansion, such as removing the comma:

assertMacroSnapshot {
  """
  #stringify(a + b)
  """
} matches: {
  """
  (
      a + b
      "a + b"
  )
  """
}

…then we get a test failure with a nicely formatted failure message letting us know exactly what went wrong:

 (
−    a + b,
+    a + b
     "a + b"
 )

We can fix this by adding the comma back.

So, this is looking much better, but things get even better.

Remember those diagnostics? Well our assertMacro helper can make testing those much better with a lot more contextual information. For example, if we test the #stringify macro with an expression that is longer than 20 characters, we should get an error:

func testMacro_LongArgument_Improved() throws {
  assertMacro {
    """
    #stringify((a + b) * (a - b) * (b - a))
    """
  }
}

Running this expands the matches trailing closure the show a nicely formatted error messaging pointing to the exact line and column where the error occurred:

func testMacro_LongArgument_Improved() throws {
  assertMacro {
    """
    #stringify((a + b) * (a - b) * (b - a))
    """
  } matches: {
    """
    #stringify((a + b) * (a - b) * (b - a))
    ┬──────────────────────────────────────
    ├─ 🛑 SomeError()
    ╰─ 🛑 SomeError()
    """
  }
}

Again we are not sure why there are two errors here, but that seems to have to do with something inside SwiftSyntax.

And to see how helpful this context can be, let’s modify the source a bit and re-record the diagnostics:

assertMacro {
  """
  let (result, string) = #stringify((a + b) * (a - b) * (b - a))
  """
} matches: {
  """
  let (result, string) = #stringify((a + b) * (a - b) * (b - a))
                         ┬──────────────────────────────────────
                         ╰─ 🛑 SomeError()
  """
}

We’ll see the error associated with the exact code that caused it. No need to manually track down the error’s location from some cryptic line and column.

Similar annotations will show up in the expanded source for warnings, notes and even fix-its.

So, now this is already kinda incredible. But things get even more incredible.

Let’s see what happens when we make a change to our macro. Let’s go back to the original form of the macro, where there are no diagnostics and we don’t do any tuple formatting or element labels:

) throws -> ExprSyntax {
  guard
    let argument = node.argumentList.first?.expression 
  else {
    fatalError(…)
  }
  return "(\(argument), \(literal: argument.description))"
}

And let’s run the test suite.

Well, we of course get test failures. In fact, every test that doesn’t have diagnostics fails because its expanded source has changed slightly.

Well, thanks to the fact that our assertMacro helper is built on the back of the snapshot testing library, it is incredibly easily to quickly re-record all of our macro expansions to get the freshest output. This can be done with the withAssertMacroConfiguration method we previously used for setting the test macros once and for all:

override func invokeTest() {
  withMacroTesting(
    isRecording: true,
    macros: [StringifyMacro.self]
  ) {
    super.invokeTest()
  }
}

Now we run tests and they fail again, but this time each of the tests that use our test helper have automatically updated their expanded strings. And if we turn recording off:

withAssertMacroConfiguration(
  // isRecording: true,
  macros: [StringifyMacro.self]
) {
  …
}

…and run tests again, our new tests now pass, whereas all the old tests fail.

It’s pretty incredible how easy it is to force all macro expansions to automatically update in a test suite.

This can also be super handy when updating to newer versions of SwiftSyntax, which for whatever reason can cause whitespace to slightly change in expanded macros. We can easily update all macro tests simultaneously, and then review the diff in git to make sure that only whitespace changed.

But we still have all the old tests failing.

The only way to get those old tests to pass is to manually copy-and-paste the expected expansion from each test failure message, format the pasted text, and even update diagnostics. Only after all of that work do we get a passing test:

So, this is pretty incredible. We have now explored our new assertMacro testing tool for testing macros, and shown that it fixes basically every shortcoming of Apple’s native testing tool:

• We get to capture the expanded macro directly inline in the test without resorting to copying-and-pasting.

• We can update macro tests by just flipping an isRecording boolean to true and running the test suite again.

• And we can even assert on diagnostics directly inline in the the source code. No need to mentally try to visualize where column 20 in line 7 is. It’s just rendered right at the offending character of the input source code.

So, this is all looking pretty great, but so far we have only given this tool a spin for the most basic kind of macro, which is the #stringify macro that comes by default with the macro template in SPM.

Let’s quickly see how the tool fares when testing larger, more complicated macros…next time!