From Speech Science to Language Transparence

Breaking down language barriers has been a dream of centuries. Seemingly unsolvable, we are now lucky to live in the one generation that makes global communication a common reality. Such global transformation was not thought to be possible, and has only become possible through revolutionary advances in AI, language and speech processing. Indeed, the challenges of processing spoken language have required, caused, guided and motivated the most impactful advances in AI. During a time of knowledge-based speech and language processing, I became convinced that only data-driven machine learning can reasonably be expected to handle the complexities, the uncertainty, and variability of communication, and that only latent learned representations would be able to abstract and fuse new and complementary knowledge. It turned out to work beyond our wildest expectations. Starting with small shift-invariant time-delay neural networks (TDNN’s) for phonemes, we would eventually scale neural systems to massive speech, language and interpretating systems. From small vocabulary recognition, we could advance to simultaneous interpretation, summarization, interactive dialog, multimodal systems and now automatic lip-synchronous dubbing. Despite the data-driven machine learning, however, speech science was necessary to inspire the models, and observing human communication continues to motivate our ongoing work in AI. In the first part of my talk, I will revisit some of our earliest prototypes, demonstrators, and their transition into start-up companies and products in the real world. I will highlight the research advances that took us there from poorly performing early attempts to human parity on popular performance benchmarks and the lessons learned. In the second part I will discuss current research and a roadmap for the future: the dream of a language-barrier free world between all the peoples on the planet has not yet been reached. What is the missing science and how can we approach the