Research

Audiovisual Modeling: I develop audiovisual models for human communication and interaction, exploring diffusion models to integrate speech and visual modalities — enhancing both the understanding of human interactions and the generation of natural facial expressions and body motions.

• Embodied AI Agents: Modeling the World
• Seamless Interaction: Dyadic Audiovisual Motion Modeling and Large-Scale Dataset.
• [ICCV’25] AV-Flow: Transforming Text to Audio-Visual Human-like Interactions.

Multimodal Language Model: This line of research focuses on language models that integrate text, speech, and visual signals to advance multimodal understanding and generation. My work explores modality fusion approaches and multi-task training at scale, enabling capabilities such as spoken dialogue generation and cross-modal translation.

• [ACL’26] AV-Dialog: Spoken Dialogue Models with Audio-Visual Input
• SeamlessExpressiveLM: Speech Language Model for Expressive Speech-to-Speech Translation with Chain-of-Thought.
• [EMNLP’24] Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agents
• [Interspeech’24] Investigating Decoder-only Large Language Models for Speech-to-text Translation
• MSLM-S2ST: A Multitask Speech Language Model for Textless Speech-to-Speech Translation with Speaker Style Preservation

Cross-modal Translation: This research addresses core challenges in translation across speech and text modalities, focusing on multilingual data mining, massive multilinguality, and multi-task training.

• [Nature’25] Joint Speech and Text Machine Translation for Up To 100 Languages
• [ACL’23] SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations
• [ICML’23] Pre-training for Speech Translation: CTC Meets Optimal Transport
• Multilingual Speech-to-Speech Translation into Multiple Target Languages
• [EMNLP’22] T-Modules: Translation Modules for Zero-Shot Cross-Modal Machine Translation
• [NAACL’22] Textless Speech-to-Speech Translation on Real Data
• [ACL’22] Unified Speech-text Pre-training for Speech Translation and Recognition

Multilingual Modeling: This research focuses on building adaptive model architectures, robust representations, and scalable training recipes across diverse languages and domains.

• [NeurIPS’21] Pay Better Attention to Attention: Head Selection in Multilingual and Multi-Domain Sequence Modeling
• Adaptive Sparse Transformer for Multilingual Translation
• [NeurIPS’21] Multimodal and Multilingual Embeddings for Large-Scale Speech Mining
• [NeurIPS’21] Robust Optimization for Multilingual Translation with Imbalanced Data