With traditional usability testing, 2 people are usually sitting in a room — one in front of the computer trying to complete a set of given tasks, and one moderating the test. Often, there will also be another group of people observing and taking notes remotely from another room. Without going into the ins and outs of usability testing, we’ll just say that this is a tried and true method for testing your solutions with the target audience (or as called in this article, “Participants”).
However, with XR, this proven method is challenged since the dynamics of the solution have changed. Participants are no longer sitting in front of a flat, framed screen — and depending on which technology you’re testing, they’re not even able to see the moderator or anything else in the real world. Even still, this can be tested in a lab for VR — just with an additional set of considerations and rules.
However, with AR or MR, it may be best to test the design out in the field — where they actually do their jobs — if you want a successful measure of whether the solution will work as intended.
This article will focus on usability testing for VR applications and specifically the areas where VR testing is different than traditional usability testing.
Things to test
Each application will have specific interactions and experiences you will want to test, but the factors listed are a general representation of questions you will want answered for all VR applications. You will get some of these answers by observing Participants during testing and some through interview questions.
Speed
- Do they feel like they have control over their speed of movement?
Accuracy of locomotion
- How accurately are they able to arrive at a desired target location when moving around?
- How often do they overshoot the target location?
Spatial awareness
- Do they know their position and orientation within the environment during and after travel?
- Did they get disoriented during travel?
- Do they have a good sense of scale and space?
- Do they feel like they know where they are within the world?
Ease of learning
- How long does it take to get the hang of interacting with the world?
- How easy is the solution to use once they’ve learned how to interact with the world?
Comfort
- Did they feel any physical discomfort or illness at any time during or after testing?
Information gathering
- Can they remember what they saw while they were in-world?
- Were they able to complete the tasks that were given to them?
Plausibility
- Was it engaging?
- Did they feel like they were somewhere else?
Roles
These are the only people who should be in the room during a VR usability test. Any other Observers will need to watch remotely from another room.
- Moderator: Interviews the Participant and keeps the test on track. Works with the Observers to summarize the results of the test and writes a report on the findings.
- Safety lead/spotter: Ensures the Participant’s safety during the test by helping them put on and take off the VR headset and equipment and keeping any cables out of the way. They also spot for the Participant to ensure they don’t fall during the test. After the test, they will work with the Moderator and any other observers to summarize the results.
- Observer: If at all possible, it is best to have the observers in a separate room watching remotely. However, when that is not an option, keep the Observers down to one maximum in the room. During the test they will take notes and then work with the Moderator to summarize the results after the test.
- Participant: This person is participating in the usability test as a user.
Rules of Engagement
In order to get the most value out of your usability test, these listed rules should be followed…
- This is not a demo — it’s a test. You want to know how well Participants will be able to use the application without someone helping them or telling them how to complete tasks.
- Anyone who is involved in the project or creation of the prototype or solution is not considered a “Participant” and shouldn’t be included in the test results since it would skew the outcome.
- Anyone who has had a preview of the test or prototype should not be considered for testing since that’s basically like looking at the answers before the exam.
- No one who will be taking the test should observe the test.
- Only one Participant per test session.
- There should be no side conversations taking place during the test. The Moderator and the Participant should be the only ones talking.
- If Observers have questions, they should write a note to the Moderator or wait until prompted by the Moderator.
- Observation notes should be taken on paper since typing can be disconcerting to someone in a VR headset — unless the Observers will be watching remotely from another room.
Supplies needed
These are the supplies you will need to conduct a VR usability test.
- Clipboards for notes
- Print-outs of the test script
(number of Participants x (Observers + Moderator)) - Projector or big screen monitor for observing what the Participant sees
- Functional prototype or software selected
- Wipes for sanitation between Participants (if sharing a headset)
- VR disposable hygiene face shields for Participants (if sharing a headset)
- Stopwatch or app to track time on task
- Video camera for reviewing tests during post-test analysis (with consent from Participant)
- Desktop screen recording software to record what is taking place in the application
- Fan for fresh air circulation (to reduce the possible effects of motion sickness)
You may also want to consider using a heart rate monitor, Fit Bit or EEG monitor (with consent from the Participant) to test stress levels depending on the subject matter of your prototype. It could also be a way to measure trends for areas that induce stress where they shouldn’t.
Have cold water close at hand. To account for extreme cases of motion sickness, you may also want to have vomit bags close. Just keep them out of sight so that they don’t influence the Participant’s feeling of wellness.
User screeners
When choosing who from your target audience will participate in your usability test, it is important to get a diverse sample of users to make sure you get input from the various types of people who will be using your application. With VR in particular, it’s important to take into account their previous experience with VR applications and gaming since this has a direct impact on their expectations for how VR should work.
When screening for users, you will want a diversified sample across the following groups:
- Age range
- Gaming experience (non-gamer, occasional, heavy)
- VR experience (none, occasional, heavy)
It is also important to screen out those with a history of epilepsy during this stage of the screening process. This can be easily done while still respecting personal privacy by adding a question to the beginning of the screener before any personal data is captured.
Sample digital survey question:
It is recommended that anyone with a history of epilepsy should not use VR. Would you like to continue with the screener?
Yes/No
If no, end the survey and do not collect any more information. If yes, continue to the survey questions.
Pre-test setup
Before the test, you will want to set up early to make sure everything is functioning properly and that the room is a good environment for testing (e.g. not a highly-trafficked area where lots of people will be disrupting the test). If at all possible, it is also a good idea to do a dry run to ensure that there are no unforeseen issues with your test script, equipment, or prototype.
Before each individual test…
- Ensure mirroring is set up on the desktop for use with the big monitor or projector.
- Wipe down the headset and controllers between tests.
- If using a FitBit or other type of health monitor, save or record the results, wipe down the health monitor, and set up for the next Participant
- Reset the prototype to the right screen
- If recording the screen on the desktop, save the recording and set up for the next recording
- If video recording (by consent), save the recording and set up for the next recording
- Have surveys and scripts ready to go for the next test
- Have the stopwatch/app reset and ready to go
Pre, mid, and post-test surveys
Pre-test baseline
As a part of the test, you will want to measure any physical discomfort the Participant may feel during or after the VR test. In order to do that, you will need to set a baseline for how the Participant is feeling before the test begins.
Have them answer some general questions without divulging too much personal information in order to establish pre-test eyestrain, and other feelings of sickness or discomfort. These questions can be asked verbally, or through a quick digital survey.
Mid-test break and wellness status
Since headsets can be physically tiring, it is highly recommended to have a break during the test so that the Participant can take off the headset. This will give them time to rest and it will allow you double check how they are feeling.
When conducting an experimental test while creating these UX best practices, we had a mid-point break to allow the Participant to rest while preparing the next simulation. I used that time to ask them health status questions.
They expressed their appreciation for the break outside of the headset and gave me some good insights into transitioning from virtual worlds back to the real world.
Post-test surveys and questionnaires
Then, once the test is over, follow up with a more detailed survey of how they feel in addition to the standard SUS and follow-up questions. You might also consider conducting the survey and asking your follow-up questions in an area with lots of natural sunlight to help the Participant’s eyes readjust and give them breathing room.
Also, consider how the Participant feels and recommend they wait an hour or so before driving a vehicle if they seem unwell or disoriented.
Learn more
- Do’s and Don’ts of Usability Testing by Demetrius Madrigal and Bryan McClain
- Conducting Qualitative, Comparative Usability Testing by Jim Ross
- Lessons in Mobile VR Usability Testing by Chloe Ng
- Virtual Reality Specialization on Coursera by Dr Sylvia Xueni Pan, Goldsmiths, University of London
If you would like to learn more about field testing for AR, check out the following article by Steven Hoober from UX Matters:
Succeeding with Field Usability Testing and Lean Ethnography
Also, check out the sample surveys and usability testing script from The VR Book by Jason Jerald, PhD.
If you enjoy these articles, consider supporting me on Patreon.
I’m an Immersive Tech UX Design Professional with over 22 years of experience designing for kiosks, websites, mobile apps and desktop software for many well-known and not-so-well-known companies.
I’m not speaking on behalf of or representing any company. These are my personal thoughts, experiences and opinions.
