This article aims to offer insights and awareness with regards to testing in a mobile environment.

We should focus our attention on gaining more confidence that our system works as a whole before merging our code. On top of that, we oftentimes can get a false sense of confidence when manually testing our apps.

No quick rollbacks

Unlike web or backend engineers, we don’t have the luxury of quick rollbacks and auto-deploying multiple times within a day. Having to roll back is not great, but at least there is that option that we mobile engineers don’t have.

We tend to painstakingly do the following: make a build, run a ton of tests, (for some teams) that includes slow UITests, make a build, have the department spend a week to check the build, maybe apply hot-fixes on the branch and recheck everything again, and then we submit the build, only for it to be reviewed by the fruit overlords, and then with a bit of luck they approve.

After all that we may not even release fully, we may choose to release gradually using a staged rollout mechanism. If there is a major bug or crash then that’s unquestionably a /big/ problem for the mobile department, or even the entire company.

Now we’re in damage control mode.

We can stop a staged rollout if we’re on time, but that might be too late. One option is hot-fixes, but hot-fixes are more like lukewarm-fixes in the world of mobile. We can’t quickly merge code and get it deployed right away. We need to make new builds and for iOS we have to go through an expedited review by Apple. All the above is an absolute pain.

One way to get more control in the release procedure is by implementing feature flags, which is a system where we put new code behind a backend-powered flag — like a boolean. This allows us to gradually release features, or even remotely turn them off in case of an incident.

Sometimes even if do everything right, our builds still might break. For instance, an OS update changes something fundamental underwater, meaning that that once stable build now suddenly has a high crash rate.

It’s imperative to have strong release processes and damage control, but on the flipside of that coin, there is opportunity for damage prevention before going live. Which is arguably even more important to focus our efforts on.

Damage prevention by testing early

Even when your team has fantastic damage control, it could be advantageous to still improve upon damage prevention. To gain more confidence that things work well before merging can be a lifesaver for a mobile app.

If you’re writing unit tests all the time, kudos to you, sincerely. That’s already a great step in the right direction. But let’s bring up two challenges that come with unit testing in a mobile context:

First, with unit tests we tend to test code in isolation, not the system as a whole. Normally this is exactly the point, we are testing “units” after all. However, that does mean we don’t know the system works as a whole until after merging.

But what if we can verify our code works together as a component or system before merging? We could consider testing units grouped together as a system, which would be the opposite of an isolated unit test, which suits a mobile environment.

A second challenge with isolated unit tests is that it’s easy to keep adding interfaces just to make your code testable. In Swift terms, that would be adding numerous protocols all over the code-base with the sole purpose of testing, thus making the production code base more porous which we could consider a code smell.

One way to combat these two challenges is by testing a system or components as a whole. You can further explore the shift-left testing article article for further insight on specific approaches.

What solutions can you think of that gives you more confidence before hitting that merge button?

Be sure to test build-to-build updates

Another easily and often overlooked check is to test updating the build from an older version to the latest version.

For instance, developers may give a testing team the latest build to test and it may all work great. But, an obscure crash may occur when the latest build has to perform a local database migration from the previous build, which could affect most users.

Maybe it works fine generally, but perhaps a slower device may have trouble finishing a database migration on time during startup and now the initial app load time is negatively affected.

These issues tend to slip through the cracks easily. Because when testing locally, we are constantly overriding and updating builds on our devices, thus making it harder to detect a real issue in regular day-to-day work. Even when distributing builds, a single crash may only appear once and never show up again, thus making serious issues appear as a fluke.

With all of this in mind, be sure to incorporate build updates in your testing scenarios. This should preferably be carried out on the slowest device you can get your hands on.

Manual testing is a safety net

Unfortunately, we can’t rely on manual tests on our own. They are more a last resort safety net. It’s therefore essential to identify and contemplate all the possible combinations in which your app builds may run.

Assuming you’re able to acquire all the devices you’d support, then:

multiply that with all supported builds
multiple that with all supported OS versions
multiply that with low disk space devices
multiply that with dark mode/light mode
multiply that with all supported screen sizes
multiply that with running your app in a a multi-tasked environment
multiply that with all screens in landscape/portrait mode
multiply that with your app being backgrounded at each screen
multiply that with your app being interrupted at each screen (e.g. a phone-call)
etc etc.

The list goes on… But, it’s a combinatory explosion of environments in which your build would run.

With all these combinations in mind, it may almost sound silly to give a device to a tester after which they say “Yup, looks good, ship it!”

Of course it helps to have someone check a build. But, we have to be aware that by manually testing we may only check a handful of variations, as opposed to the thousands that are out there in our user’s hands.

What exacerbates the problem is when people test with only a few devices. Make sure your team tests with enough variants to be confident. It doesn’t make sense when everyone is sporting the latest iPhone 21 with 20TB of disk space with a processor faster than a quantum computer.

Finding fun in writing tests

Unrelated to mobile testing, but still a point worth sharing is: Try to find fun in writing tests!

Not all engineers are fans of writing tests, and who can blame them?

Testing can get a lower priority for numerous reasons: Time pressure, hard to test code, having to merge quickly before a weekly build cutoff, etc. We’ve all been there.

A little mindset-shift that can make testing more enjoyable is figuring out new ways to help the team you’re in. Maybe you can think of an interesting way to offer stubs. Or maybe figure out a way to generate testing code. Or maybe you’ll be the one that finds a good way to speed up UI-tests.

Tests can be a nice experimental playground for code without going into production, yet effectively helping others. As a result, this is a low-pressure way to experiment with some code that won’t just stay on your laptop.

So if you catch yourself saying, “I don’t like writing tests,” consider making a little library to help you and others write tests a little faster and easier.

Learn more

If you want to learn more about testing techniques, please check out the upcoming Mobile System Design book and the Shift-left testing approaches article.