Virtual Reality-based Collaborative Activities Simulator (ViRCAS)

Key personnel: Ashwaq Zaini Amat, Deeksha Adiani, Mahrukh Tauseef, Michael Breen, Nilanjan Sarkar

Goals/ objectives:

1. Create an interactive, engaging, and easy-to-access simulator that can support teamwork skills training for two or more individuals in a collaborative virtual environment.
2. Provide a quantitative measurement and evaluation of teamwork skills through the analysis of multimodal data.

Outline:

The employment settings for autistic individuals in the USA is grim. As more children are diagnosed with ASD, the number of adolescents and young adults with ASD will increase as well over the next decade. Based on reports, one of the main challenges in securing and retaining employment for individuals with ASD is difficulty in communicating and working with others in workplace settings. Most vocational trainings focused on technical skills development and very few address teamwork skills development. Motivated by these, we designed and developed a Collaborative Activities Simulator (CAS) that consists of three team-based activities in a collaborative virtual environment (CVE). CAS is designed with a wide range of collaboration features that are aimed to nurture and encourage autistic individuals to practice their teamwork skills by working together with a partner in a shared virtual space. CAS comprises a network communication layer that can support two or more users to work together in the virtual environment, a video and audio streaming service embedded within the virtual environment that allows users to see and hear each other in real-time while they are performing the task together, and a multimodal data capture module that integrates multimodal data such as speech, eye gaze, controller manipulation data, and task performance data from all users within the same repository that can provide researchers with a quantitative observation of the teamwork interactions. We are currently working on expanding the simulator with an intelligent agent that can measure and assess teamwork skills using a probabilistic predictive model such as Hidden Markov Model (HMM).

 

We have so far completed 1 pilot study and 1 experiment planned.

Video:

Related Publications:

• Design of a Virtual Task to Understand the Nature of Collaboration Between Autistic and Neurotypical Adults in Workplace Using Multimodal Data
• Collaborative Virtual Environment to Encourage Teamwork in Autistic Adults in Workplace Settings
• Design of a Virtual Reality-based Collaborative Activities Simulator (ViRCAS) to Support Teamwork in Workplace Settings for Autistic Adults (Under Review)
• Automated Behavior Labeling During Team-based Activities involving Neurodiverse and Neurotypical Partners using Multimodal Data (Pending Publication)