Deb Roy

My passion is to understand how language works, and to leverage that understanding toward building machines that use language in fluid, meaningful, human-like ways.

How do we learn to behave intelligently? What is the role of language in perception, planning, and memory? How can we build machines that learn, act, and communicate naturally? These broad questions set the stage for our research program.

People use language to exchange ideas and influence the actions of others through shared conceptions of word meanings, and through a shared understanding of how word meanings are combined. Under the surface form of words lie complex networks of mental structures and processes that give rise to the richly textured semantics of natural language. Machines, in contrast, are unable to use language in human-like ways due to fundamental limitations of current computational approaches to representing the meaning of words.

To explore alternative models of language processing and the computational nature of semantics, we build conversational robots. Endowing robots with language highlights the impossibility of isolating language from other cognitive processes. We embrace a holistic approach in which various non-linguistic elements of perception, action, and memory provide the foundations for grounding word meaning.

A central theme of our work is to resolve the circularity of "dictionary definitions" inherent in today's language-processing systems. While some words may be fully defined in terms of other words, others—for example "blue," "heavy," "want," or "I"—cannot. To understand these words, one needs to have experienced the underlying concepts. It is only when words can be related to experiences that symbols can truly be hooked into the world.

The potential payoff to understanding the language-grounding process is immense. I envision a range of human-machine interfaces from conversational robotic assistants to multimodal information retrieval. More fundamentally, I hope to gain new insights into how people represent, learn, and use language, and the nature of conceptual knowledge that underlies language.

> Full Essay