Developing and Testing Photorealistic Avatar with Body Motions and Facial Expressions for Communication in Social Virtual Reality Applications

Developing and Testing Photorealistic Avatar with Body Motions and Facial Expressions for Communication in Social Virtual Reality Applications

Abstract

Providing effective communication in social virtual reality (VR) applications requires a high level of avatar representation realism and body movement to convey users’ thoughts and behaviours. In this research, we investigate the influence of avatar representation and behaviour on communication in an immersive virtual environment (IVE) by comparing video-based versus model-based avatar representations. Additionally, we introduce a novel VR communication system that mimics users’ movements, facial expressions, and speech in order to render these capture data into different types of photorealistic avatar representations in real-time.

These techniques could be beneficial for any type of VR application that requires avatar-based communication.

Keywords

Virtual reality; avatar; realism; video see-through; social presence; social communication.

Virtual Reality technology, a three-dimensional (3D) graphical system that simulates reality in different contexts, has provided great potential for supporting communication in recent years by enabling two or more groups of people to communicate together in a social immersive virtual environment. These techniques could be beneficial in numerous fields such as medicine, architecture, gaming, education, and others.

However, creating an effective social virtual environment that accurately simulates a real social environment for VR applications remains one of the most difficult problems in VR. What is the best way to capture human social interaction through the movement of facial expression, body gestures, and eye gaze? How can we make avatars, which are 3D virtual characters, look real in terms of visual appearance and physical movement, to deliver the best experience of social VR? These difficulties demand an understanding of human interactions so that we can implement communication cues, such as verbal and non-verbal cues, onto the avatars. Additionally, we must develop techniques to enhance the realism of representation and behaviour of embodiment avatars in a 3D virtual environment, thereby providing users with effective communication with other users in VR applications that heightens the users' sense of inhabiting the virtual world.

In this research, we are tackling some of these problems by developing and evaluating techniques that could enhance users’ experiences for social VR applications. After reviewing prior research, we found that most focuses either on delivering a photorealistic avatar representation with limited body motion and facial expressions or on facilitating an abstract avatar with limited body motion for communication in VR applications. Our research's primary contribution is the application of techniques to enhance the representation of photorealistic avatars and behaviour in social IVEs. We also introduce a novel VR communication system that mimics users’ movements, facial expressions, and speech in order to render these captured data into different types of photorealistic avatar representations in real-time. Additionally, we will conduct human subject experiments to examine our research methodology and collect data for subjective and objective measurements.

One aspect of my research focuses on the influence of photorealistic avatar representation and behavior in social IVE by comparing different types of photorealistic avatar representations: video see-through, scanned realistic avatar, and no-avatar representations (see Figure 1). One research question is: To what extent, and under what conditions, does photorealistic avatar representation and behavior influence communication in an IVE? Given the choice of developing technologies to better achieve video-based versus model-based avatar representations for applications like spatial tasks for architectural design reviews, which approach should we choose? Previous research suggests that photorealistic avatar representation and behavior could positively influence communication during interactions in an IVE, thereby increasing the sense of presence. However, how to most effectively achieve a suitably photorealistic avatar representation remains an open question.

Figure 1: Comparing different types of photorealistic avatar representations including video see-through, scanned avatar, and no-avatar representations in an IVE.

A related study by Roth et al. (2016), explored the impact of reduced social behavior and information channels in an IVE. They focused on how a lack of avatar realism, including realistic appearances and behavioral cues, could influence interpersonal interactions and co-presence in a virtual environment. Their experiment compared verbal and physical social interactions in the real and virtual worlds, as represented by abstract avatars. The scientists measured simulator sickness, networked mind subscale, presence, attention focus, and task performance. The findings demonstrated a considerable difference between interactions in the real and virtual worlds. The authors concluded that the absence of certain gestures and facial expressions affected users' physical performance and feeling of presence. Similarly, we investigated these ideas but used photorealistic avatars obtained via technologies that aim to support a higher level of visual and behavioral realism.

We will also discuss the impact of embodied VR avatars on communication behavior in a social context. We are interested in understanding how different avatar visual appearances—including gender, age, skin color, and body shape—can influence the perception of others in the virtual environment. IVE provides individuals with the opportunity to communicate with an embodied virtual avatar in real time, using a variety of appearances and scenarios to enhance user performance or alter user perspectives. Many VR applications support low-fidelity embodied avatars for interaction and communication. However, limitations in avatar communication, such as a lack of facial expression or body movement, can affect users' perception and the immersion and sense of presence in a 3D IVE. Smith and Neff (2018) found a significant social presence with communication behavior and conversation when using a low-fidelity embodied avatar in VR for interaction, compared to not having an embodied avatar. Although excluding a discussion of facial expressions due to technology limitations, they noted that introducing facial expressions into a realistic embodied avatar could enhance the sense of presence of others, leading to more effective communication in VR.

Thus, we propose a novel VR communication system that supports embodied virtual avatars in VR by capturing different types of data in real-time (see Figure 2). These data are a combination of a user’s body movement, facial expressions, and streaming audio that are applied onto a 3D virtual avatar to support communication in VR in real-time. The system would be beneficial for creating a 3D interactive environment for different types of VR applications that are based on communication with a virtual avatar. For example, one application could be job interview training with a virtual interviewer in real-time. Another application could be reducing implicit biases of protected groups through various interventions (e.g., stereotype replacement, counter-stereotypic imaging, or partnership building) in an IVE.

Figure 2: An embodied virtual avatar with realistically rendered user movements and facial expressions for VR real-time communications.

My future plan for research contribution is to focus on gaining a better understanding of people's needs for social VR applications that require virtual avatars. This will be achieved by finding robust solutions and conducting experiments to collect subjective and objective data for publications. Additionally, I aspire to apply the outcomes of my research in a way that makes it accessible to others in the VR community. My career goal is to remain in academia and secure a post-doctoral position in the USA or Germany, eventually working as a professor and leading a lab that focuses on utilizing VR in various aspects of life. As part of my professional agenda, I hope to keep expanding my network field by attending, organizing, and chairing different prestigious conferences like IEEE VR. By doing so, I can pave the way for future research collaborations, as I am currently doing with colleagues from different departments at the University of Minnesota. I have already made strides towards these goals by assisting with VR venue organizational tasks and I plan to continue attending these meetings to stay abreast of the newest advances in the field. Serving the VR community while remaining current with my field of knowledge is a practice I find thoroughly enjoyable.

As an international student, I am eager to bring such events to my home country by inviting speakers to local venues and helping with the hosting of international conferences. This aligns with the 2030 mission of my country, which has already begun to focus more on VR at universities and within the context of Smart Cities. I plan to involve myself in initiatives focused on encouraging other women to consider computer education and careers in my homeland, utilizing American strategies I have learned from the conferences I have attended and will attend in the future.

Did you like this example?

316

17