Parallelizing UI Tests for iOS Applications
As an iOS Engineer part of developing applications is defining processes and tools that help us ensure a certain quality baseline. One of the main ways in which this is done is through automation, unit testing an UI testing.
Xcode provides us with theXCTest framework which allows us to write Unit and UI tests that we can run locally and on CI/CD servers.
Xcode 10 introduced Parallel UI Testing and Xcode 11 introduced Test Plans, features that greatly improve the flexibility and speed of our tests.
Testing in our development machine is straightforward, however in the CI/CD environment we have to manage paths, environment, configuration and build artifacts a bit more carefully.
Moreover as our test suite grows the time it takes to run from start to finish also increases, this leads to developers waiting on builds and tests lengthening the development cycle of our app.
The goal of this article / tutorial is to take advantage of test parallelization both locally and on CI/CD servers, so without further ado let's jump in!
Local UI Testing
To get started with local parallel UI testing I’ve prepared a sample project available on Github so that you can poke around into the project’s final configuration, it is meant to be a good starting point to expand your tests or to adapt into your project.
We’ll use the same project for CI/CD testing and the application is really simple; it’s a “Super Car Market” app in which you view listed cars for sale, you select one by taping it to view the details and then you can optionally contact the seller.
It’s an iOS 13+ application with local data as assets that uses Combine, UIKit, storyboards and segue navigation (yikes!), really simple straightforward code, some of it includes hardcoded values that is definitively non-production worthy but our focus is having an app with enough UI and transitions that we can write and run enough UI tests for.
Below a short GIF of the full app:
In Xcode 11.7 the default new project does not use test plans, we have to opt-in explicitly, to do so, edit the project’s scheme and in the “Test” section click “Convert to use Test Plans …” and select an existing test plan (if any).
If you haven’t created any test plans you can do so by creating a new file in the desired group and searching for “Test Plan”. I’ve explicitly created a “Test Plans” folder:
With tests plans and some tests then we can enable Parallel testing, to do so, open the “Test Navigator” (cmd+6) and select your test plan and select “Edit Test Plan …”, you should see an “options” button that opens up and allows you to check the “Execute in parallel (if possible)” option (see screenshots):
After that, you can simply run all the tests in the ParalleleUITestsUITests target and Xcode should automatically create multiple clones of the selected simulator destination and distribute any classes (test suites) in the target to each of the clones. It is important to note that Xcode will run classes in parallel, not individual test functions when distributing the work amongst the worker clones. Xcode also won’t use the original run destination, rather it creates 3 (or more) new clones of the original. Our parallel run should look like this, often some of the clones will be used to run a single test suite, while others will run many test suites.
CI/CD Parallel UI Testing
With our tests running locally, we should start thinking of running them in our CI/CD servers.
When running in servers where there’s no UI to click “Run” or to view the execution of the tests in the simulator, instead we have to rely on xcodebuild and on archiving the *.xcresult file that contains the aggregated results of the parallel tests, we also might run into the issue of the “Hardware Keyboard Connect” setting of the simulator blocking input to Secure TextFields (password) or simply messing around with the keyboard hide/show behavior.
When running in a CI/CD env, my recommendation is to manage the devices explicitly and to define a special *.xctestplan for a CI/CD environment. This specially so because we can include environment variables and launch arguments as part of the *.xctestplan file further simplifying our UI or Unit tests by predefining app state.
With that in mind our parallel test runner script will have to do the following:
- Create a new device.
- Disable the device’s “Connect Hardware Keyboard Setting”.
- Build the project for testing.
- Pre-boot the new device.
- Run the tests in parallel.
- Archive the
*.xcresultfile.
This will be a step-by-step guide but you can find the final z-shell script in the Github gist below, the script assumes a clean VM on each run (so there’s no need to delete devices):
Creating a new device
Here we create the device and store it’s UDID, we will use the UDID as an argument to future xcodebuild invocations, so let’s keep it handy:
device_name="CI UI Parallel"
device_udid=$(xcrun simctl create "$device_name" "iPhone 8")Disable the device’s “Connect Hardware Keyboard Setting”
This setting is what allows you to use your computer’s keyboard to type text into the app when running in the simulator, unfortunately it has the side effects of preventing input to Secure TextFields (passwords) and it also prevents the on-screen iOS keyboard from appearing when enabled, this setting will be passed onto the clones when they’re created and are prone to cause failures on tests if you rely on keyboard visibility or interacting with password fields.
Extending our previous script with the following:
alias plist=/usr/libexec/PlistBuddy
simprefs="$HOME/Library/Preferences/com.apple.iphonesimulator.plist"plist -c "Set :DevicePreferences:$device_udid:ConnectHardwareKeyboard true" "$simprefs" \
|| plist -c "Add :DevicePreferences:$device_udid:ConnectHardwareKeyboard bool true" "$simprefs";
The previous will disable the keyboard in the new device and the setting should be copied over to the clones.
Build the project for testing
With our new device setup now we can build the project, do so by using the following xcodebuild command. We want to use a well known build directory and to build it for our new device so we have to pass the $device_udid as part of the -destination parameter:
BUILD_DIR=$(mktemp -d)
SYMROOT="$BUILD_DIR"xcodebuild build-for-testing \
-project ./ParallelUITests.xcodeproj -target ParallelUITests \
-destination "platform=iOS Simulator,id=$device_udid" \
SYMROOT=$BUILD_DIR \
BUILD_DIR=$BUILD_DIR
Many of Xcode’s variables and settings are documented here:
Pre-boot the new device
Since our new device will be cloned and because it’s never been booted this means that when our tests run the cloned devices will have to do a cold boot (20secs+), to avoid this we will first boot the source device and wait for it to idle at the home screen, this will make our clones start at the home screen too!, do so by adding the following to your script:
# Boot the device
xcrun simctl boot $device_udid# Block until it's fully booted and add 2s buffer because this
# reports boot completion a bit early
xcrun simctl bootstatus $device_udid && sleep 2
Run the tests
We’re finally ready to run the tests, we’ll use the *.xctestrun file produced by our previous xcodebuild command, we’ll also specify the location of the collected test results.
Since the xctestrun file will include the name of the Test Plans along with platform and scheme name we can find it with this command:
testrun=$(find "$BUILD_DIR" -name '*Parallel UI Tests*.xctestrun')Next we run the actual tests in parallel:
result_bundle="$BUILD_DIR/$device_name.xcresult"xcodebuild test-without-building \
-destination "platform=iOS Simulator,id=$device_udid" \
-xctestrun "$testrun" \
-resultBundlePath "$result_bundle"
If you need a bit more control over the clones xcodebuild has the following CLI flags:
-parallel-testing-enabledAYES|NOoverrides the per-target setting in the scheme.-parallel-testing-worker-countANUMBERthe exact number of test runners that will be spawned during parallel testing.-maximum-parallel-testing-workersANUMBERthe maximum number of test runners that will be spawned during parallel testing.
xcodebuild --help provides that information and much more!
Archive the results
Last but not least is archiving the *.xcresult bundle file since we’re looking to download from the CI/CD server and inspect the results using Xcode’s bundle explorer we append this final part to the script:
bundle_name=$(basename "$result_bundle")
cd $(dirname "$result_bundle")
zip -r "$bundle_name.zip" "$bundle_name"We cd into the parent directory of the bundle because we want to avoid the full directory path that the zip command includes, after archiving if you open the *.xcresult file you should see the following:
Here you can find the full script, it includes some extra commands for readability, you can run this script in your CI/CD servers to take advantage of parallel execution.
Note: The specifics of how the script is run will depend on your CI/CD provider.
