Links:

Welcome to the Multimodal Speech Processing (MSP) Laboratory

The Multimodal Speech Processing (MSP) laboratory is part of the Language Technologies Institute at Carnegie Mellon University. Our research program is on human-centered multimodal machine intelligence and applications for speech-based interfaces. We are dedicated to advancing technology to solve theoretical problems with practical applications. Our goal is to develop methods, algorithms, and models to recognize and synthesize human verbal and non-verbal communication behaviors to improve human-machine interaction.

Human communication involves a complex orchestration of cognitive, physiological, physical, cultural, and social processes. As a result, spoken language carries essential information beyond the verbal message that a robust natural human-computer interface should capitalize on, and be able to detect and engage. This includes not only subtle changes in vocal quality and articulation as well as modulation in linguistic and discourse structure, but also complex ways in which spoken language interplays with other modalities such as facial expressions and body posture. Interactions with other interlocutors also determine the behaviors since we tend to synchronize our verbal and non-verbal behaviors, a phenomenon known as entrainment. Our team has studied this complex interplay between communicative goals and behaviors expressed through multiple channels, which can bring insights not only to derive better theoretical computational models for human interaction but also to advance the capabilities of human-computer interfaces for applications with clear societal benefits. Our human-centered research program informs, designs, and develops digital software relevant to scholarly disciplines in computational psychology, linguistics, and computational social sciences.

In this website, you will find an overview of the exciting activities that are happening at MSP lab. It also introduces the faculty and students involved in the lab.

Spotlight

You can click on the image to learn more about the study. You can use the arrows to navigate among the teaser images.

❮ ❯

Research Interests

The research at the MSP laboratory focuses on human-centered multimodal machine intelligence and its applications in speech-based interfaces. The work explores cutting-edge topics, including:

• Speech and multimodal processing
• Affective computing
• Social signal processing
• Multimodal machine learning
• Multimodal foundational models
• Language technology for healthcare and mental health
• Multimodal behavior generative models

The outcomes of his research have direct implications in many practical domains, including national security, health care, entertainment, transportation systems, and education.

About the MSP laboratory

The MSP lab was established by Prof. Carlos Busso in August 2009. He is also the director of the group. The laboratory was originally established at the University of Texas at Dallas. In January 2025, the laboratory was relocated to Carnegie Mellon University.