Advanced UI Testing: Strategies, Hidden Gems and Limitations

As I discussed previously, to write robust tests you should access elements through queries least likely to be affected by future changes. However, another aspect of UI testing that greatly affects test robustness is managing your app state affectively.

For example, imagine a Photos Application that applies effects to photos in an album and flags photos that are modified. Now imagine two tests. Test A modifies the sharpness of a photo in the album and checks that a photo in the album has been modified. Test B adds a filter to a photo and checks the that a photo in the album has been modified. Sometimes this tests passes, sometimes it fails. Why?

Consider the following test order:

If you haven’t noticed already, sometimes #2 and #5 will modify the same image and the check in #6 will succeed. However, if the random image in #2 and #5 is different then the number of modified images in #6 will equal two. This simple example demonstrates why it is so important to tear down application state after modifying it in your test.

As you write more and more tests you’ll realize that all this setup and tear down of state can really take a toll on your test suite run time. A complex way of solving this in the past has been issuing network calls to your application code to that modify the application state programmatically. However, a more reasonable approach is to optimize the flow of your tests to limit repeating view transitions within your app.

Limitations of UI Testing

• No ability to simulate physical buttons, but from looking at the code it looks like me may get this in a later beta.
• There are no hooks to build in testing under different wireless and cellular conditions automatically. It might be possible if you can run a script to manipulate the network link conditioner dynamically
• No multi-device support for testing device-device interaction
• No clear solution for dealing with adding sample data for tests. However, you may get somewhere using App Restoration.
• No ability to access the app code to programmatically modify app state. This is essential for speeding up large scale testing.

• Users can obtain size class information of individual items
• The orientation can be gathered from the SharedDevice object, but physical buttons appear not to be implemented
• You access / set launchArguments and launchEnvironment on XCUIApplication which may help with setting up test data.

Send me a tweet if you see others!