Multimodal Interactive Designs, Algorithms and User Studies

My research in multimodal interactive designs and human studies investigates the development and evaluation of interactive systems that leverage multiple modalities to enhance user experience and performance. Through collaborative projects, we have developed recurrent-based vision-based and depth-sensor hand gesture recognition system, natural language processing, and robot control to enable seamless sign language interaction between humans and social robots. This research addresses the need for inclusive communication interfaces that cater to individuals with hearing impairments, promoting accessible human-robot interaction.

Electromyography (EMG) signals also play a crucial role in my research. Several studies have been published on EMG signal processing and recognition, such as time-frequency feature transform for sEMG signals and an incremental learning framework for multimodal sensor fusion.

We have also introduced novel machine learning approaches that improves gesture recognition performance by leveraging cross-modal transfer learning using a fast incremental learning algorithm. This work builds upon the growing body of research on transfer learning in multimodal interaction, showcasing the potential of adaptive learning techniques in creating personalized and efficient interfaces. These studies have been extended in our recent project funded by PolyU RIAIoT, where control interfaces for disabled people (including EMG, depth camera, and joystick) are tested, and feedback is collected from users in real-world settings.

To address real-world challenges, these studies consistently employ data collected from real-world scenarios, and usability, acceptance, and other metrics are typically examined. My work demonstrates expertise in designing multimodal interfaces that adapt to user needs and preferences. By leveraging state-of-the-art machine learning techniques and conducting rigorous human studies, I aim to advance the field of multimodal interaction and create inclusive, personalized, and efficient interfaces for a wide range of applications.