Multimodal interactions with an instrumented shelf

In this paper, we describe the initial implementation of our application demonstrator called ‘ShopAssist’. This application will aid users in product queries within a shopping scenario context. We describe the wide range of input modalities that our application supports such as speech, handwriting, intraand extra gestures, and the mixed modality combinations that promote advanced user interaction with real-world and virtual objects.