Advanced UI Testing: Strategies, Hidden Gems and Limitations

July 5, 2015
xcode icon derivative

Image by Yung-Luen Lan

Previously I’ve introduced using UI Testing and discussed the differences between UI Testing in Xcode and the prior UI Automation Instrument. However, there is a lot more to learn about UI style testing. In this article I’ll speak about how to think about the end-end life-cycle of tests, some limitations of the Xcode solution and some undiscussed functionality.

Improve Test Robustness by Managing Application State

As I discussed previously, to write robust tests you should access elements through queries least likely to be affected by future changes. However, another aspect of UI testing that greatly affects test robustness is managing your app state affectively.

For example, imagine a Photos Application that applies effects to photos in an album and flags photos that are modified. Now imagine two tests. Test A modifies the sharpness of a photo in the album and checks that a photo in the album has been modified. Test B adds a filter to a photo and checks the that a photo in the album has been modified. Sometimes this tests passes, sometimes it fails. Why?

Consider the following test order:

  1. Run Test A
  2. Add sharpness to random image
  3. Check that number of modified images == 1
  4. Run Test B
  5. Add filter to random image
  6. Check that number of modified images == 1
If you haven’t noticed already, sometimes #2 and #5 will modify the same image and the check in #6 will succeed. However, if the random image in #2 and #5 is different then the number of modified images in #6 will equal two. This simple example demonstrates why it is so important to tear down application state after modifying it in your test.

Writing Efficient Tests

As you write more and more tests you’ll realize that all this setup and tear down of state can really take a toll on your test suite run time. A complex way of solving this in the past has been issuing network calls to your application code to that modify the application state programmatically. However, a more reasonable approach is to optimize the flow of your tests to limit repeating view transitions within your app.

Limitations of UI Testing

  • No ability to simulate physical buttons, but from looking at the code it looks like me may get this in a later beta.
  • There are no hooks to build in testing under different wireless and cellular conditions automatically. It might be possible if you can run a script to manipulate the network link conditioner dynamically
  • No multi-device support for testing device-device interaction
  • No clear solution for dealing with adding sample data for tests. However, you may get somewhere using App Restoration.
  • No ability to access the app code to programmatically modify app state. This is essential for speeding up large scale testing.

Hidden Features of UI Testing

  • Users can obtain size class information of individual items
  • The orientation can be gathered from the SharedDevice object, but physical buttons appear not to be implemented
  • You access / set launchArguments and launchEnvironment on XCUIApplication which may help with setting up test data.

Send me a tweet if you see others!


This is part 3 of a 3-part series on Xcode UI Testing…

Read Part 1: How to Use iOS UI Testing in Xcode

Read Part 2: Xcode UI Testing vs. The UI Automation Instrument