Designing For Test

Date first published: 22nd April 2020

Introduction

Mobile apps are difficult, expensive, and time consuming to test made worse by the large number of different mobile devices available that they must work on. The right design approach can vastly reduce this cost and effort by making it much easier to test effectively, and giving a high degree of confidence in the finished product.

My name is Graeme Clarke and I’ve been developing software for nearly 30 years. Although by day I’m a manager, I still write code at home for fun. In this article, I’ll show example code from my latest mobile game: ‘Otherworld: Epic Adventure’ which is currently live in production and on sale in Amazon, Apple and Google stores.

Otherworld is a murder mystery adventure game where the player walks around a world interacting with items to solve puzzles and uncover the plot. Its a huge game with over 200 locations and 100 items to interact with. It comprises nearly ten thousand lines of code accompanied by a further thousand resource files including images, sounds, string tables and configuration files. This represents a massive testing burden especially as the game targets all major mobile platforms. Despite being a sole developer I've been able to use the approach described here to help build a high quality product currently being enjoyed by many customers.

Otherworld is written in Microsoft Xamarin which is a cross-platform development framework built on Microsoft .Net. This means the code only needs to be written once and it will run on all mobile platforms.

Without further ado, let’s dive right in and start by examining how a Xamarin solution is organised:

Xamarin Project Structure

Otherworld has been developed using Microsoft Visual Studio. Visual Studio allows us to create a ‘solution’ containing all the libraries and projects required for the app:

Highlighted in green is the Portable User Interface project. This project will contain all our UI code and will work on all target platforms

In orange are separate projects for each platform:

Here there are projects for Android, IOS, Windows, and Windows Phone. These projects will only contain code specific to each platform which will normally be configuration or build options or specific requirements for the target app stores. These projects may also contain user interface code but as we’ll find out later in this article, any such code should be minimised as much as possible.

Outlined in red is the ‘domain’ or ‘business logic’. We are allowed to create a re-usable library that doesn’t contain user interface and can be shared by all projects. Its here all the business logic for the app should be put.

This diagram shows a high-level structure of our application:

The solution contains other projects including UML design, UI and Unit Testing projects and a supporting image conversion tool.

Testing the User Interface

Testing the app’s user interface requires the application to be running which typically means that a physical mobile device must be connected to the computer. Visual Studio also provides an emulator, or, there are several providers that make devices available via the cloud.

Although the App must be manually tested, it makes sense to make use of test automation as much as possible. Test automation allows a suite of tests to be executed that quickly sanity check the application every time something changes. This gives a high degree of confidence that there are no bugs before we start to play with it ourselves.

UI tests come in at the very top level of the diagram and aside from testing features of the user interface they can also effectively test end to end ‘business flows’ or ‘user journeys’ through the application.

Automated user interface tests for a mobile app typically run slowly and only limited diagnostic information may be available from the device. In my experience, the user interface will behave differently on different devices. and although Apple devices are reasonably consistent, there are a huge number of Android phones all of which may behave differently.

Here is an example of a simple Otherworld UI test that checks the operation of the ‘Learn How to Play’ button presented on the loading screen. When the button is pressed a 'help guide' will be displayed and the test must check that this has been done. It verifies that the correct type of screen has been displayed and that it shows the right title and description.

Here is what happens on the screen:

Here is the code for one of the automated tests that check this functionality.

Although the test itself is short and concise it takes a long time to execute on my Samsung Galaxy – nearly 20 seconds! I’ll also need to run this test on my IPhone, Amazon Fire and as many other devices as I can get my hands on. Unfortunately, I only own 3 devices so I won’t have a high degree of confidence that the code works correctly on all platforms.

The Otherworld UI test automation suite contains nearly 200 tests in total and some of these take much longer to run than 20 seconds. The whole test suite normally takes a couple of hours to execute and sometimes will be left running overnight.

If a test fails then the code on the device can be debugged directly but again this is slow. All the necessary debugging information may not be available and the device settings must be changed to allow debugging, meaning it may no longer be representative of a typical customer device.

While this is not great, it's certainly much better than manually running tests on each device and UI test automation is an essential element of our overall test strategy. Techniques and technologies can improve the situation, but these typically come at a price. More on UI testing later in this article.

Testing the Business Logic

Fortunately testing the business logic is much easier and can be done using 'Unit Tests'. Unit Tests run inside the development environment on the local computer and don’t require a physical device to be present. They are easy to write, quick to execute and can be used to effectively debug the code. Unit Tests can also run in parallel which vastly reduces the time to execute the overall suite.

Unit Tests only test the business logic code and come in below the User Interface layer:

In Otherworld, the player can move through gates provided that they have been unlocked first:

Here is a unit test that checks that the player can’t move through a locked gate.

This test is short, running in only 1 millisecond and if a bug is found then it’s easy to put in a break-point, dive in and debug the code directly. The full range of debugging tools including in-memory variable values, memory maps, threading and immediate code execution are all available as the code is running on the local computer. These tools can be invaluable for tracking down the source of any problem.

In fact, the entire Otherworld Unit Test suite which has over 200 tests takes less than 2 seconds to run and Visual Studio has been configured to execute these tests on every build operation. This will give a high degree of confidence in the stability of the code as changes are made.

The important point to note is that we don’t need to run these tests on each individual device as the Business Logic code is going to operate exactly the same on every platform – it only needs to be tested once.

For these reasons, we should put as much code as possible within the Business Logic with only the absolute minimum within the User Interface. Doing so will ensure most of the application is tested before it is even deployed onto a device. Additionally, we won’t have to wait overnight to know if the tests have all passed or not.

Bulking out the Business Layer

Techniques of Object Orientated Design can help move more code out of the User Interface and into the Business Logic.

In this example from Otherworld, the player moves into an old Greenhouse and picks up a pair of gloves which are then added to their bag:

All this logic, including the location of the player, the set of items contained within the bag, and the list of items still available in the Old Greenhouse are contained within the Business Layer of the application.

Below are 3 small pieces of code taken from the game that illustrate how this is done. First is a code snippet from the Avatar class which describes the current location of the player plus the list of items in their bag:

Secondly, this next snippet shows the Location class, which is instantiated for each location. It contains a list of the items at the location plus the directions that the player can take to move away from that location.

Thirdly, the InputController object handles all player interaction with the controls.

• Whenever the player picks an item up it is added to the bag.
• The location is refreshed in the UI to remove the item from the display.
• If no more items can be carried a message must be displayed to the player.

Here is the code from the ‘Pick Up Item’ event handler:

In object orientated programming we can use the Visitor pattern to add new operations or functionality to an object without modifying the structure of the object. The visitor can be defined within the Business Logic as an abstract class to provide additional functionality but can then be implemented higher up in the User Interface code.

In Otherworld, I’ve used a Visitor – the ‘LocationVisitor’ to provide the ability for code in the Business Layer to display user interface including messages or user selection screens.

In the code sample below, the ‘VisitLocation’ function refreshes the User Interface if information about the location or its list of available items changes. The code displays the details of the location on-screen, the various items that are available (which has just changed in our example) and the list of movement options the player has. Finally, it also plays any background music relevant for the location.

Here is the code for the VisitLocation function together with the DisplayMessage function referenced in the last code snippet.

Those functions that directly update the User Interface are marked ‘abstract’ and cannot be implemented with the Business Logic. However, thanks to the properties of an abstract class, implementations of key, non-abstract, functions can be added. In the code example above you can hopefully see how the VisitLocation function makes use of other abstract calls to display the various properties of the Location object.

A concrete implementation of the LocationVisitor has been added higher up in the 'green' Portable User Interface layer:

This UI class derives directly from the Business Logic class LocationDisplayVisitor and it contains concrete implementations that display user interface.

For example, the DisplayMessage is implemented by simply using the Xamarin platform DisplayAlert function:

The concrete UI implementation is as simple as possible with the minimum of code or any type of program flow or logic.

While of course this is a very straightforward example the other concrete implementations in the LocationDisplayVisitor follow the same guidelines. Remember, there should be the least amount of code possible within the User Interface Layer as it is more difficult to test.

This implementation of the LocationVisitor is great for the UI but it isn’t going to help unit testing the Business Logic as it's defined too high up - in the portable UI layer.

Instead, I’ve created an alternative 'test' implementation within the UnitTest project itself that will record any messages within a buffer instead of writing to the screen.

Its DisplayMessage implementation just adds the string message to the buffer:

To see how this is used we must look at a test class so here is the unit test class that exercises the input controller:

Some points to note about the code:

1. The test class derives from a base class ‘TestBase’. Even though this is a unit test class we are still allowed to use all the features of object orientated programming including inheritance.The TestBase class contains re-usable objects that will be needed by all tests including the Player object (that contains the ‘bag’ with the list of items held by the player), as well as the map of all possible locations.
2. The object under test is the input controller which is instantiated in the TestInitialize function along with our TestLocationVisitor. TestInitialize will be called before every single test inside this class is executed.Both the input controller and our TestLocationVisitor are therefore cleaned up in the TestUninitialize function.
3. When the InputController is instantiated ‘nulls’ are being passed for a lot of the parameters. The point of these of tests is to exercise the InputController object in isolation rather than a business flow through the application. Some of the parameters aren’t required for the specific tests to be run so nulls are passed into the constructor.

Now let’s look at a test in this class that examines the pick-up functionality. This test will check that a message is displayed if the player has too many items and can’t pick any more up.

When the input controller calls the LocationDisplayVisitor to display the error message, control goes to this test implementation which then adds the message string to the buffer.

The unit test can now check the buffer to verify that the right message has been displayed:

If an error occurs, then the test can be easily debugged including stepping into the code. Visual Studio is one of the best development environments available and provides a comprehensive set of debugging and diagnostic information.

What has just been demonstrated is that by using simple object orientated principals, classes in the Business Logic that need to display and interact with the user interface can be tested effectively. Without this approach additional code would need to go into the User Interface layer, bulking it out and almost certainly requiring UI test automation to be written to effectively test it.

Some important things to note about unit tests

Parallel Execution

When running on the build server these tests will all be executed in parallel. For each test the system takes the following actions:

• Allocate a new test class
• Call the TestInitialize function
• Execute the test
• Call TestUninitialize to clean up

That means that if the test class contains two Unit Tests then the system takes the following actions at the same time:

There will also be two instances of the test class active in memory at once. Of course, it there are more than two tests in the suite then the system will have many instances in memory at the same time.

When the test class is allocated it will get a new copy of all the member variables, but static variables will be shared amongst all the running instances. This means that objects or data that will change can’t be saved within static variables without providing thread synchronisation. Only use static variables or thread synchronisation if they are absolutely necessary.

Otherworld has approximately 250 unit tests, all of which are executed in parallel with the entire suite taking less than 2 seconds to run meaning that there will be a lot of tests running in parallel. The computer will choose the optimum number of concurrent instances based on the system resources available with the intention of running the whole suite as quickly as possible.

If a static variable has been used, then tests which pass on the local machine will start failing with spurious errors whenever the code is run on the build server. The error messages returned may not have any bearing on the actual cause of the problem making race conditions of this type very difficult to track down.

Non-functional Testing and Memory Leak Detection

Although this article has focused on functional testing, it is also important to conduct non-functional tests including performance and memory leak testing. Performance profiling code running on physical devices is once again slow and the device may not be able to provide enough detailed information to effectively pinpoint errors.

As with functional testing, having a comprehensive set of unit tests that cover most of the code without needing the physical devices present will make profiling the application a lot easier. I’ll be writing more on this in a future article.

Summary

Bringing together all these thoughts:

1. The use of Xamarin means there is only one code-base for all platforms which increases the confidence in our testing and reduces the amount of testing needed.
2. Code in the Business Logic can be unit tested, which is faster, easier and doesn’t require the app to be deployed to a physical device.
3. Object Orientated techniques can be used to help shift as much code as possible out the UI and into the Business Logic.
4. Object Orientated techniques can also be applied to the unit test classes allowing us to re-use code across the suite.

Earlier on I painted a bleak picture of the problems associated with UI testing, but it’s not all bad and in the next part of this article I’ll examine it in depth and describe how once again Object Orientated programming techniques can make the job easier.

I’ll also look at how the main app stores like Apple, Amazon and Google Play as well as continuous integration platforms like Microsoft App Center, can help UI test our application without incurring any expense.

To keep you entertained in the meantime why not play Otherworld:

Currently available on Apple, Google Play and Amazon or find out more on our web-site.

I hope you've found this article informative and that you enjoy playing Otherworld: Epic Adventure. Stay safe and well!

Graeme

• Check out more technical articles in my Streetwise series right here.
• Watch one of my conference presentations.
• Have a laugh, get one of my Charlie Bluster books for free.