Input Methods in XR: Gestures

Gesture-based inputs allow you to use your hands to manipulate and interact with objects without the use of controllers. There are two types of gesture-based interactions.

Direct manipulation

You interact with the environment the way you would in real life by picking up objects.

Formed gestures

You use your hand to form shapes that then interact with the environment.

---

Pros

• It’s a more natural interaction when designed and implemented well.
• It gives people a better sense of presence since their hands are represented in-world.
• It frees your hands from the use of controllers.
• The direct manipulation type means no learning curve since you’re interacting the way you would in real life.

Cons

• Your hands have to be within line of sight of the camera or sensor to work.
• Holding your hands out far enough to be picked up by the sensors can cause fatigue (gorilla arm).
• There’s no sense of touch or tactile feel to give haptic feedback.
• Formed gesture types require you to memorize hand shapes to interact with the environment.

---

TL;DR Do’s and Don’ts

Do

• Ensure longer duration interactions are within the participant’s physical comfort zone. 
• Follow physical ergonomic design standards.
• Use tooltips and memory cues to remind people how to use any formed gestures.
• Minimize the amount of formed gestures used or required in your experiences.

Don’t

• Don’t use overly-complicated gestures that people will have to memorize.

---

Recommendations

• Ergonomic fatigue can be mitigated by using digital adaptations of grabbers and other extender tools for longer reach interactions. 
• Since tactile senses aren’t available with gesture-based input, give visual feedback such as glows or highlights along with subtle audio so that the participant will know when an object is being touched. 

---

Example to try

Oculus Hand Tracking

Oculus Quest headsets use a mix of direct manipulation and formed gestures. They use formed gestures on their menus and UI panels in the home world, but provide the technology for developers to create experiences that use direct manipulation.

---

Learn more

• Designing for Hands by Oculus
• Hand Coach by Microsoft Mixed Reality
• Hand Menu by Microsoft Mixed Reality
• Ultraleap Docs by Leap Motion, UltraLeap
• The VR Book by Jason Jerald PhD, Input Devices: Bare Hands, pg 317, 27.2.5
• Virtual Reality Specialization by Goldsmiths, University of London on Coursera

---

If you enjoy these articles, consider supporting me on Patreon.

I’m an Immersive Tech UX Design Professional with over 22 years of experience designing for kiosks, websites, mobile apps and desktop software for many well-known and not-so-well-known companies.

I’m not speaking on behalf of or representing any company. These are my personal thoughts, experiences and opinions.

---