OTTRS talk: Snehesh Shrestha (CS)
Register here for Zoom link: https://umd.zoom.us/webinar/register/WN_KosvhRxuTEuIybXNl7Wkag#/registra...
Multimodal Human-AI Interaction
Abstract: People communicate through verbal and non-verbal cues. AI and ML have made tremendous progress in language understanding. Audio tone, gestures, gaze, and touch, along with speech, offer new challenges and opportunities. My work dissects multimodal human expression, focusing on Human-AI interaction in Robotics and Music. In the first part, I discuss creating a robot capable of understanding natural commands, emphasizing multimodal repair mechanisms. I’ll briefly share data collection challenges, which greatly impact data quality and validity. We used a Wizard-of-Oz setup, deceiving participants into believing we had a human-level AI robot, to capture ‘natural’ interactions. Verbal and non-verbal strategies were studied to train machine learning algorithms for multi-modal commands, highlighting the importance of combining gestures with speech. In the second part, I explore AI-mediated Student-Teacher Interaction systems towards violin education. I will discuss challenges in remote music lessons, which became particularly pronounced during the COVID-19 pandemic. I will discuss data collection challenges for precise motion capture, especially with young students. I share insights into using audio to enhance pose estimation algorithms for 3D player visualization. Lastly, I introduce a novel haptic band designed for remote feedback, prompts, and metronome functions, enhancing online music education experiences.
Bio: Snehesh Shrestha is a Ph.D. candidate at the University of Maryland College Park. He works in the Perception and Robotics Group (PRG) lab in the Department of Computer Science under the guidance of Prof. Yiannis Aloimonos (CS), Dr. Cornelia Fermüller (UMAICS), Dr. Ge Gao (INFO), and Dr. Irina Muresanu (School of Music). He has also worked with Dr. Michelle Gelfand (Department of Psychology) in the Culture Lab. Additionally, he works at NIST, developing new standards towards recommended practices for the design of human subject studies in human-robot interaction. His research is at the intersection of robotics, artificial intelligence, human factors, arts, and culture. He is interested in multidisciplinary research aimed at building rich and intuitive experiences that ‘amplify human abilities, empowering people and ensuring human control’ inspired from Dr. Ben Shneiderman’s Human-Centered AI book. His recent work has focused on human-robot interaction and AI for music education.
The Organizational Teams and Technology Research Society (OTTRS) advances research and collaboration on multiple aspects of the study of teams relevant to technology and information, increasing relevant work both within UMD and in the Technology and Teaming community outside of UMD.
