Multimodal Dialog with the MALACH Audiovisual Archive

In this paper, we present a multimodal dialog system capable of information retrieval from the large audiovisual archive MALACH of Holocaust testimonies. The users can use spoken natural language queries to search the archive. A graphical user interface allows the users to quickly view footage with the answers and explore their context. The dialog was deployed in two languages — English and Czech. The system uses automatic speech recognition and natural language processing for knowledge base construction and for processing of the user’s input.