Reliable Async Tests: 😳

Stephen: So, that was a pretty deep overview of the current state-of-the-art when it comes to testing async code in Swift. We’re not going to mince words here: it can be incredibly frustrating to try to test features that make use of any asynchrony beyond some very simple constructs. And unfortunately we’re not really sure there is much work being done from the core Swift team to improve the situation.

And while what we have built is a pretty silly toy application, we promise that your application has these exact shapes somewhere. You are certainly making async network requests, and maybe you are juggling some state to let the view know when that request is in flight. And maybe you have some additional logic somewhere that determines when one should cancel that request if it is still in flight. And further, as async sequences become more and more ubiquitous in Apple’s APIs, and as Combine begins its slow march towards deprecation, you will be using async sequences a lot more.

As soon as you have some complex and nuanced logic involving asynchronous code you are in for a world of hurt when it comes to testing. You have no choice but to either not test all of your code, or sprinkle in some mega yields or sleeps just to get things passing, and hope it doesn’t flake on your CI server.

Brandon: So, everything we have encountered so far is what pushed us to start a discussion on the Swift forums asking others how they are going about testing complex, async code. And unfortunately it seems that pretty much everyone is in the same boat. They either don’t test their complex async code, or at least not as deeply as they could, or they insert lots of yields and sleeps to force the concurrency runtime to push forward.

We wanted to find a better way, and around that time Apple had just opened sourced their new “async algorithms” package, and it had a couple of interesting advanced usages of Swift’s underlying concurrency runtime. We dug in a bit deeper and we saw that there was maybe something we could leverage to bend the will of the concurrency runtime to our advantage in tests.

So, let’s take a look at Apple’s code and see what clues it gives us in how we can better predict how Swift’s concurrency tools will behave at runtime.