Multimodal Synchronization for Human-Machine Interaction in Virtual Environment

Abstract

Human-machine interfacing technologies lead the human to communicate with virtual environment in various effective format and more interactive manner. With the combination of keyboard, mouse, haptic, gesture, speech, and vision modes, the interfaces allow human to be involved with the virtual environment in multimodal form, as if they are embedded as a part of simulation. However, with different mechanism and computing method in multimode interface, they lack of synchronization to support human interaction in the virtual environment. This paper explained human-machine interfacing techniques and proposed multimodal synchronization mechanism to control the tempo and to solve the problems of multimodal interaction. The experimental set to verify human perception with multimode human-machine interface in distributed virtual environment was described at the end. The result gave some figure of how human could adjust their perception to be synchronized with the interactive event in the virtual environment through multimode interface.