Developing and Testing Photorealistic Avatar with Body Motions and Facial Expressions for Communication in Social Virtual Reality Applications

Category: Technology
Date added
Pages:  5
Words:  1450
Order Original Essay

How it works

Developing and Testing Photorealistic Avatar with Body Motions and Facial Expressions for Communication in Social Virtual Reality Applications


Providing effective communication in social virtual reality (VR) applications requires a high realism of avatar representation and body movement to transmit users’ thoughts and behavior. In this research, we investigate the influence of avatar representation and behavior on communication in an immersive virtual environment (IVE) by comparing video-based versus model-based avatar representations. Furthermore, we introduce a novel VR communication system that mimics users’ movements, facial expressions, and speech in order to render these capture data into different types of photorealistic avatar representations in real-time. All these techniques could be beneficial for any type of VR application that requires avatar-based communication.


Virtual reality; avatar; realism; video see-through; social presence; social communication.

Virtual Reality technology, which is a three-dimensional (3D) graphical system that simulates reality in different contexts, has provided great potential techniques that support communication in recent years by enabling two or groups of people to communicate together in a social immersive virtual environment. These techniques could be beneficial in numerous fields such as training in medicine, viewing 3D design in architecture, playing in games, teaching in education, and others.

However, Creating an effective social virtual environment that simulates a real social environment for VR applications remains one of the most difficult problems in VR. One of these problems is how to capture human social interaction in the movement of facial expression, body gestures, and eye gaze? How to make the avatar, which is a 3D virtual character, look real in terms of visual appearance and physical movement, to deliver the best experience of social VR? These difficult problems require us to understand the nature of human interactions so we can apply communication cues on the avatars such as verbal and nonverbal cues. Also, develop some techniques that help to increase the realism of representation and behavior of embodiment avatars in a 3D virtual environment. This could provide users with effective communication with other users for VR applications that increases users’ sense of inhabiting the virtual world.

In this research, we are tackling some of these problems by developing and evaluating some techniques that could enhance users’ experience for social VR applications. Looking at previous work we found that most research focuses on either delivering a photorealistic avatar representation with a lack of body motion and facial expressions or having an abstract avatar with limited body motion for communication in VR Applications. This research’s main contribution is applying some techniques to enhance the representation of avatar photorealistic and behavior on communication in social IVEs. Also, introduce a novel VR communication system that mimics users’ movements, facial expressions, and speech in order to render these capture data into different types of avatar photorealistic avatar representations in real-time. Moreover, conducting some human subject experiments to examine our research methodology and collect data for subjective and objective measurements.

One aspect of my research is to focus on the influence of the photorealistic avatar representation and behavior in social IVE by comparing different types of photorealistic avatar representations: video see-through scanned realistic avatar, and no-avatar representations (see figure 1). One research question is: to what extent, and under what conditions, does photorealistic avatar representation and behavior influence the communication that takes place in an IVE? Given the choice of developing technologies to better achieve video-based versus model-based avatar representations for collaborative applications like spatial tasks for architectural design reviews, which approach should we choose? Previous research suggests that photorealistic avatar representation and behavior could have a positive influence on communication while people interact in the IVE, and this could increase the sense of being present. However, it remains an open question of how best to achieve a suitably photorealistic avatar representation.

Figure 1: Comparing different types of photorealistic avatar representations including video see-through, scanned avatar, and no-avatar representations in IVE.

A related prior study, by Roth et al. (2016), investigated the impact of reduced social-behavior and information channels in an IVE. They focused on how a lack of avatar realism—including a lack of realistic appearances and behavioral cues—could influence interpersonal interactions and co-presence in a virtual environment. Their experiment was accomplished by representing people by abstract avatars and comparing their verbal and physical social interactions in the real and virtual worlds. The scientists measured simulator sickness, networked mind subscale, presence, attention focus, and task performance. The findings illustrated a considerable difference between the interactions in the real and virtual worlds. The authors concluded that the absence of certain gestures and facial expressions affected the users’ physical performance and feelings of presence. We have investigated similar ideas but by using photorealistic avatars obtained via technologies that aim to support a higher level of visual and behavioral realism.

Another aspect that we are going to discuss is the impact of embodied VR avatar on communication behavior in a social context. We would like to see how different avatar visual appearance gender, age, skin color, and body shape could make a positive or negative judgment in other people? IVE gives the opportunity for individuals to experience communication with an embodied virtual avatar in real time with various kinds of appearances and different types of scenarios to enhance user performance or change user perspectives. There are many VR applications that support a low-fidelity embodied avatar for interaction and communication. However, these applications have some limitations in communication with the avatar due to a lack of facial expression or body movement. This problem could affect users’ perception and the immersion and sense of presence in a 3D IVE. Smith and Neff (2018) discussed the fact that having a low-fidelity embodied avatar in VR for interaction resulted in a significant social presence with communication behavior and conversation compared to not having an embodied avatar. They excluded a discussion of facial expressions due to the limitations of the technology, but they mentioned that adding facial expressions into a realistic embodied avatar could provide a much greater sense of the presence of other people; this, in turn, could deliver more effective communication in VR.

Thus, we propose a novel VR communication system that supports embodied virtual avatars in VR by capturing different types of data in real-time (see figure 2). These data are a combination of a user’s body movement, facial expressions, and streaming audio that are applied onto a 3D virtual avatar to support communication in VR in real-time. The system would be beneficial for creating a 3D interactive environment for different types of VR applications that are based on communication with a virtual avatar. For example, one application could be job interview training with a virtual interviewer in real-time. Another application could be reducing implicit biases of protected groups through various interventions (e.g., stereotype replacement, counter-stereotypic imaging, or partnership building) in an IVE.

Figure 2: An embodied virtual avatar with realistically rendered user movements and facial expressions for VR real-time communications.

My future plan for research contribution is to focus on having a better understanding of people’s needs for social VR applications that require virtual avatars by finding robust solutions and conducting some experiments to collect some subjective and objective data for publications. Also, applying the outcomes of my research and make it more accessible by other people in the VR community. My goal for a future career is to stay in academia and find a position for a post-doc here in the USA or Germany and later work as a professor and have my own lab to lead teams that work on utilizing VR in many fields of life aspects. Future professional goals are to continue expanding my network in the field by attending, organizing, and working as a committee chair in different prestigious venues like IEEE VR to have a good start for future research collaborations, which is what I am doing now with other collaborators from different departments at the University of Minnesota. I have already started fulfilling these goals in helping with VR venue organizational tasks and will continue attending these meetings to keep updated with the latest advances in the field as I personally enjoy both serving the VR community and keeping in touch with my field of knowledge.

Eventually, as an international student, I can bring such meetings to my home country by inviting speakers at local venues and by helping with hosting such international venues aligning with the 2030 mission of my country that inaction had started to give more attention to VR at universities and for Smart Cities. I intend to engage in initiatives related to helping other women to consider computer education and careers in my homeland applying the American approaches learned from the conferences I have participated in and expect others to come.

Did you like this example?

Cite this page

Developing and Testing Photorealistic Avatar with Body Motions and Facial Expressions for Communication in Social Virtual Reality Applications. (2021, Jun 01). Retrieved from

The deadline is too short to read someone else's essay

Hire a verified expert to write you a 100% Plagiarism-Free paper