Remote UX Testing

In March this year we received a usability (UX) testing project brief: Moderated sessions in our awesome usability lab. As a UX agency, it was typical. We were confident that the tests and analysis would go smoothly. After all, our lab gives us a lot of control and allows our client to observe the session, adding to a good client experience. Fast forward a few weeks, the President of South Africa announces a total lockdown of the country.

Although we had done many remote usability tests before, this meant adapting the test, from live moderated to remote unmoderated. We would have to change the task instructions – guiding the users in a much clearer and concise way than originally designed. This was a challenge, but light in comparison to what we experienced with each user profile during the tests. The purpose of this article is to take you through some of the new experiences we had in this season of remote testing and what we have learnt.

Types of Remote Testing

There are two types of remote usability testing, moderated and unmoderated. Each one has its advantages and disadvantages, and it is important to choose one that will meet your testing objectives as well comfortably allow the user to participate.

Before, I explain how each type works, let’s define remote testing: Remote usability refers to tests conducted in different physical locations. This means that “the participant interacts with the design in his own home, office or other location while the expert watches remotely” (Schade, A Remote Usability tests: Moderated and unmoderated Nielsen Norman group Oct 2013. Because of the ongoing lockdown restrictions, we knew that we would be conducting more remote tests, and we would be able to test users in their real and authentic environments, which is not often experienced by researchers, as there is usually a preference to bring the user into a test environment where the researcher has more control and resources to conduct the tests. Remote testing, brings to the fore user struggles, hurdles and successes when engaging with technology and connectivity. They would not need to verbalize their experience, we experienced it with them, we overcame with them.

Moderated UX Testing

Virtual testing and viewing the session live as the user goes through each task, scenario and activity. Here the user can hear the moderator, who in turn guides the discussion as they typically would in a live in-person test.

Moderated tests are often longer than unmoderated tasks and work well for usability tests that require a guided experience and comprise several tasks. This way of testing also allows the moderator to immediately probe user actions and any observations that they pick up. Anxious users or those who are uncomfortable with technology, have the added support of someone on the other end of engagement that offers guidance and reassurance.

Advantages:

• Real-time support and engagement
• Asking detailed questions – probing
• The research team can view user actions in real-time
• User feedback in real-time
• Favourable for a more qualitative researcher – fewer users, deeper insights

Unmoderated UX Testing

Tests that are designed for users to complete independently of a moderator in a familiar environment. The tasks are pre-determined, and there is no room to ask additional questions because the researcher will only see the recordings and results post-test. In this testing method, the experience is reliant on the user understanding of the task instructions and their perception if they have completed the task or not. This is a real-world experience of how users experience products and services, unassisted and without influence. The user is also not pressured to ‘pass’ the test because there are no observers. Researchers have the tricky task of ensuring all instructions are clear, simple to understand and direct. Applying user-friendly language instead of industry jargon that may confuse or alienate the user is important.

Advantages:

• Quick tests and access to results immediately
• Less expensive
• Results are more reliable and natural because there is no chance of bias
• Consumes less data for the user
• Favourable for quantitative studies – more users can be tested, and results can be quantified

Software to Conduct Remote Testing

Choosing a tool to conduct the tests is important, and they range from the basic and inexpensive to the more complex tools that offer various behavioural analysis like click-throughs. Here is a list of factors to consider when choosing a tool and designing your research approach

• Screen recording
• Face recording
• Ability to perform more than one task without stopping the session
• Uploading results immediately to the cloud
• Ability to test relevant tasks i.e. card sorting or tree jacking
• Capability with mobile devices, tablets, and desktops
• Analytics capabilities to track user activity and task success

Basic and inexpensive tools can include familiar video conferencing tools like Skype and Zoom. These have been very effective because of their ease of use, ability to share the user screen and obviously, their call capabilities. More complex tools allow recruitment on the platform, click tests, card sorts and surveys.

Dos and Don’ts for Conducting Your Remote Tests

Do you feel ready to run your tests, here is a checklist of things to consider before you begin?

• Brief users before the test. This can be a short telephonic or text brief where you introduce the research and its purpose. This allows the researcher to chat with the user, set the scene and answer any questions that the user may have. I take this opportunity to download any additional testing software that is needed and check if the software works on the user’s phone.

• Create a short mock test for the user to complete. The mock test does not alert them on how the actual test will work but allows them to become familiar and comfortable with the testing platform. Test platforms can sometimes be a challenge to users, and it helps to remove that navigation out of the way so that users do not believe they are being tested on the test platform.

• Ask users to clear space on their phones to download apps, one would think this is not a problem, but it can become a major time-consuming hurdle and impact the functionality of the device.

• Get ready to troubleshoot users’ devices. Yes!, you may have to help users figure out their phones especially for more technically challenged users. They may have a smartphone but do not freely explore it as most usability researchers explore tech. Be prepared to assist older users to download apps, create space, authorize device permissions and even set the correct time.

• Over recruit. Have more users on standby to participate especially for the longer more in-depth tests. You want to be able to achieve your sample and still have quality feedback. No researcher wants to choose between sample size and quality.

• In moderated tests, be patient with the users. Some may experience connection interruptions and may need to restart the tests. Speak slowly, clearly and be prepared to repeat yourself. Give instructions in a step by step manner to ensure the user has understood and heard everything, and expect a connection lag.

• Use platforms like Whatsapp in unmoderated tests to communicate with the user in-between tasks. This is an efficient method as it allows the users and moderator to stay in contact so the user can report issues experienced like connection breaks or language misunderstandings.

• Don’t be hard on yourself, we are testing in challenging times, the test may take longer than expected, you may have had to run your household while running the test. Be kind and patient with yourself. You are doing your best

Challenges in SA

Every researcher knows that each testing country has its unique challenges, South Africa is no different. Although the market is adapting to digital experience, there are still some factors to consider.

• Testing with low-income users
  • Empathy, patience and understanding are key here. These users have real daily challenges that are often not documented or communicated in the industry because it simply does not sound appealing.
  • They struggle with unreliable network speeds and signals – testing platforms need to be low bandwidth use.
  • Users tend to be more self-conscious of their technological abilities, they need to know that the session is not testing their capabilities but exploring a product or service.
  • Wide range of low-end smartphones that may be old or unable to support new testing platforms. This results in the phone freezing or closing spontaneously.
  • Airtime or data is often seen as a good incentive for shorter tests, however, in the current economic conditions, retail store vouchers can be more useful and needed.
  • Low-income users are generally more patient and willing to go the extra mile to complete the session, be empathic to their situations, and always respectful, they may have to complete the test in a noising house of family members. We conducted a test while a mom sat in her car because that was the only quiet place she could get to during the lockdown.

• South Africa has many languages and interpretations of phrases – the language used needs to be simple, clear and direct. The researcher may need to change the language according to the profile.

• Moderated remote testing sessions may have to exclude additional observers because of the unreliable connection speeds. This puts pressure on the user’s connection as they may be connected to mobile data which is much slower than ADSL or fibre.

• Unreliable networks means there may be no provision for the rest of the team to view the test because of the network using so much bandwidth

There are still some real challenges when it comes to running remote tests in South Africa, especially during the lockdown. This however does not mean that brands and companies should stay away from testing their digital products and services. It is crucial in these times to create and improve the user experience because actions like social distancing and self-isolation have become the new normal and have resulted in new needs and new ways of doing things . Brands and companies that can create experiences that are immersive, useful and easy, are the brands that users will engage with on an ongoing basis in this new uncharted world in which we now find ourselves.

Author: Katlego Hlongwane from Interact RDT

Follow us on LinkedIn to get daily insights, download some of our amazing e-books HERE or simply Give us a call.