10 points to keep in mind when choosing the right mobile testing tool(s)

March 30, 2013

How do you choose the right tool? Which tool is the one to deal with the platform fragmentation that just keeps getting worse? Which tool gives you most bang for the buck? Which tool saves you time? What is a smart selection of tools and devices?

There are a lot of question marks and it’s hard to know what approach you should take on mobile testing and tooling. Mobile testing is still so fresh that we don’t have any guidelines written in stone or extremely intelligent and well-made comprehensive tools right now.

There are not many companies which have a structured approach to mobile testing and the requirements are usually deficient. I don’t see many specialized mobile testers out there either. I sometimes go on events and hear about the companies’ unstructured approach on mobile testing within all sectors. That scares me half to death.

As I have evaluated a quite a few tools over my time as a mobile tester I’ve made a little list in order to specify some questions I think everyone should have in mind when they are choosing which mobile testing tool they would go for. Of course it will depend on what you will be using it for, sometimes an open source product like Selenium or Robotium is enough. But what if it isn’t? And what if we don’t have the scripting experience or time requested in order to fully utilize these tools? Then we need to take a look at alternatives

These are 10 questions to help you to be critical on mobile testing tools and to not get fooled by a cool logo or a nice sales person.

Which OS does the testing tool support?
Will it run on both Native and Web applications?
Is there a recorder or only scripting possibilities?
Are there possibilities to use different scripting languages?
Is it possible to run tests on multiple devices in parallel and how many devices can a decent server machine handle at the same time?
Do they provide a server solution with clustering possibilities?
Do you have to Jailbreak or Root your devices in order for them to run with the solution?
Will you be able to schedule test runs and jack them in to a Continuous integration environment?
Can you integrate with the device hardware? Mock GPS-signals, Turn On/off 4G,3G, Edge, Wi-Fi, use OCR (object character recognition), Get the device log, Send Key commands change orientation on the device in the script, monitor CPU, memory leakage and is there a screenshot and/or baseline feature to easily detect possible deviations?
Can you switch between applications in the script to simulate user behavior and try out third party browser?

Written by Viktor Johansson a QA professional and Wheelchair athlete, living in Stockholm, Sweden. Feel free to reach out on Twitter or email.