How Important are Detailed Hand Motions for Communication for a Virtual Character Through the Lens of Charades?

Detailed hand motions play an important role in face-to-face communication to emphasize points, describe objects, clarify concepts, or replace words altogether. While shared virtual reality (VR) spaces are becoming more popular, these spaces do not, in most cases, capture and display accurate hand motions. In this paper, we investigate the consequences of such errors in hand and finger motions on comprehension, character perception, social presence, and user comfort. We conduct three perceptual experiments where participants guess words and movie titles based on motion captured movements. We introduce errors and alterations to the hand movements and apply techniques to synthesize or correct hand motions. We collect data from more than 1000 Amazon Mechanical Turk participants in two large experiments, and conduct a third experiment in VR. As results might differ depending on the virtual character used, we investigate all effects on two virtual characters of different levels of realism. We furthermore investigate the effects of clip length in our experiments.

Amongst other results, we show that the absence of finger motion significantly reduces comprehension and negatively affects people's perception of a virtual character and their social presence. Adding some hand motions, even random ones, does attenuate some of these effects when it comes to the perception of the virtual character or social presence, but it does not necessarily improve comprehension. Slightly inaccurate or erroneous hand motions are sufficient to achieve the same level of comprehension as with accurate hand motions. They might however still affect the viewers' impression of a character. Finally, jittering hand motions should be avoided as they significantly decrease user comfort.

• Alex Adkins
• Aline Normoyle
• Lorraine Lin
• Yu Sun
• Yuting Ye
• Massimiliano Di Luca
• Sophie Jörg

Alex Adkins, Aline Normoyle, Lorraine Lin, Yu Sun, Yuting Ye, Massimiliano Di Luca, and Sophie Jörg. How Important are Detailed Hand Motions for Communication for a Virtual Character Through the Lens of Charades? ACM Transactions on Graphics, June 2023, Volume 42, Issue 3, Article No. 27, pp 1-16. doi.org/10.1145/3578575 [bibtex]

This material was supported in part by Oculus VR, LLC. Alex Adkins and Sophie Jörg's contribution is furthermore based in part upon work supported by the National Science Foundation under Grant Number IIS-1652210. The authors thank Adam Wentworth for spending many hours on post-processing motion capture files and Christian Sharpe for fitting our character model to our skeleton.