Publications

(* denotes equal contribution)

Technical Report

• Embodied AI Agents: Modeling the World
  

• Seamless Interaction: Dyadic Audiovisual Motion Modeling and Large-Scale Dataset
  

2026

• AV-Dialog: Spoken Dialogue Models with Audio-Visual Input
  Tuochao Chen, Bandhav Veluri, Hongyu Gong, Shyamnath Gollakota.
  In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (ACL 2026).

2025

• Joint Speech and Text Machine Translation for Up To 100 Languages
  In Nature 2025.

• AV-Flow: Transforming Text to Audio-Visual Human-like Interactions
  Aggelina Chatziagapi, Louis-Philippe Morency, Hongyu Gong, Michael Zollhöfer, Dimitris Samaras, Alexander Richard.
  In International Conference on Computer Vision (ICCV 2025).

• DC-Spin: A Speaker-invariant Speech Tokenizer for Spoken Language Models
  Heng-Jui Chang, Hongyu Gong, Changhan Wang, James Glass, Yu-An Chung
  In Interspeech 2025.

2024

• Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agents
  Bandhav Veluri, Benjamin N Peloquin, Bokai Yu, Hongyu Gong, Shyamnath Gollakota
  In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024).

• Enhancing Language Models with Idiomatic Reasoning
  Jianing Zhou, Ziheng Zeng, Hongyu Gong, Suma Bhat
  In COLM 2024.

2023

• SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations
  Paul-Ambroise Duquenne*, Hongyu Gong*, Ning Dong, Jingfei Du, Ann Lee, Vedanuj Goswani, Changhan Wang, Juan Pino, Benoît Sagot, Holger Schwenk.
  In The 61st Annual Meeting of the Association for Computational Linguistics (ACL 2023).

• Pre-training for Speech Translation: CTC Meets Optimal Transport
  Phuong-Hang Le, Hongyu Gong, Changhan Wang, Juan Pino, Benjamin Lecouteux, Didier Schwab.
  In 2023 International Conference on Machine Learning (ICML 2023).

• Speech-to-Speech Translation For A Real-world Unwritten Language
  Peng-Jen Chen, Kevin Tran, Yilin Yang, Jingfei Du, Justine Kao, Yu-An Chung, Paden Tomasello, Paul-Ambroise Duquenne, Holger Schwenk, Hongyu Gong, Hirofumi Inaguma, Sravya Popuri, Changhan Wang, Juan Pino, Wei-Ning Hsu, Ann Lee.
  In Findings of the Association for Computational Linguistics (ACL Findings 2023).

• Improving Speech-to-Speech Translation Through Unlabeled Text
  Xuan-Phi Nguyen, Sravya Popuri, Changhan Wang, Yun Tang, Ilia Kulikov, Hongyu Gong.
  In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2023).

• Named Entity Detection and Injection for Direct Speech Translation
  Marco Gaido, Yun Tang, Ilia Kulikov, Rongqing Huang, Hongyu Gong, Hirofumi Inaguma.
  In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2023).

• A Holistic Cascade System, benchmark, and Human Evaluation Protocol for Expressive Speech-to-Speech Translation Wen-Chin Huang, Benjamin Peloquin, Justine Kao, Changhan Wang, Hongyu Gong, Elizabeth Salesky, Yossi Adi, Ann Lee, Peng-Jen Chen.
  In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2023).

2022

• T-Modules: Translation Modules for Zero-Shot Cross-Modal Machine Translation
  Paul-Ambroise Duquenne, Hongyu Gong, Benoît Sagot, Holger Schwenk.
  In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP) (EMNLP 2022).

• Unified Speech-Text Pre-training for Speech Translation and Recognition
  Yun Tang, Hongyu Gong, Ning Dong, Changhan Wang, Wei-Ning Hsu, Jiatao Gu, Alexei Baevski, Xian Li, Abdelrahman Mohamed, Michael Auli, Juan Pino.
  In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL 2022).

• Textless Speech-to-Speech Translation on Real Data
  Ann Lee, Hongyu Gong, Paul-Ambroise Duquenne, Holger Schwenk, Peng-Jen Chen, Changhan Wang, Sravya Popuri, Yossi Adi, Juan Pino, Jiatao Gu, Wei-Ning Hsu
  In Proceedings of NAACL-HLT 2022 .

• Idiomatic Expression Paraphrasing without Strong Supervision
  Jianing Zhou, Ziheng Zeng, Hongyu Gong, Suma Bhat.
  In Proceedings of the AAAI Conference on Artificial Intelligence 2022 (AAAI 2022).

• Contrastive Clustering to Mine Pseudo Parallel Data for Unsupervised Translation
  Xuan-Phi Nguyen, Hongyu Gong, Yun Tang, Changhan Wang, Philipp Koehn, Shafiq Joty.
  In International Conference on Learning Representations 2022 (ICLR 2022).

• Incremental Speech Synthesis For Speech-To-Speech Translation
  Danni Liu, Changhan Wang, Hongyu Gong, Xutai Ma, Yun Tang, Juan Pino.
  In Interspeech 2022.

2021

• Pay Better Attention to Attention: Head Selection in Multilingual and Multi-Domain Sequence Modeling
  Hongyu Gong, Yun Tang, Juan Pino, Xian Li.
  In Advances in Neural Information Processing Systems (NeurIPS 2021).

• Multimodal and Multilingual Em- beddings for Large-Scale Speech Mining
  Paul-Ambroise Duquenne, Hongyu Gong and Holger Schwenk.
  In Advances in Neural Information Processing Systems (NeurIPS 2021).

• Robust Optimization for Multilingual Translation with Imbalanced Data
  Xian Li, Hongyu Gong.
  In Advances in Neural Information Processing Systems (NeurIPS 2021).

• Abusive Language Detection in Heterogeneous Contexts: Dataset Collection and the Role of Supervised Attention
  Hongyu Gong, Alberto Valido Delgado, Katherine Ingram, Giulia Fanti, Suma Bhat and Dorothy Espelage.
  In AAAI 2021 (AI for Social Impact track).

• Self-Supervised Euphemism Detection and Identification for Content Moderation
  Wanzheng Zhu, Hongyu Gong, Rohan Bansal, Zachary Weinberg, Nicolas Christin, Giulia Fanti and Suma Bhat.
  Accepted by IEEE Symposium on Security and Privacy 2021.

• FST: the FAIR Speech Translation System for the IWSLT21 Multilingual Shared Task
  Yun Tang*, Hongyu Gong*, Xian Li, Changhan Wang, Juan Pino, Holger Schwenk, Naman Goyal.
  In Proceedings of the 18th International Conference on Spoken Language Translation (IWSLT 2021).
  

• WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia
  Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, Francisco Guzmán.
  In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2021).
  

• PIE: Parallel Idiomatic Expression Corpus for Idiomatic Sentence Generation and Paraphrasing
  Jianing Zhou, Hongyu Gong and Suma Bhat.
  In the 17th Workshop on Multiword Expressions (ACL Workshop 2021).
  

2020

• Recurrent Chunking Mechanisms for Long-Text Machine Reading Comprehension
  Hongyu Gong, Yelong Shen, Dian Yu, Jianshu Chen and Dong Yu.
  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL 2020).
  slides

• Enriching Word Embeddings with Temporal and Spatial Information
  Hongyu Gong, Suma Bhat and Pramod Viswanath.
  In The SIGNLL Conference on Computational Natural Language Learning (CoNLL 2020).
  slides

• Rich Syntactic and Semantic Information Helps Unsupervised Text Style Transfer
  Hongyu Gong, Linfeng Song, and Suma Bhat.
  In International Conference on Natural Language Generation (INLG 2020).

• IlliniMet: Illinois System for Metaphor Detection with Contextual and Linguistic Information
  Hongyu Gong, Kshitij Gupta, Akriti Jain and Suma Bhat.
  In Proceedings of the Second Workshop on Figurative Language Processing (Fig-Lang@ACL 2020).

• FUSE: Multi-Faceted Set Expansion by Coherent Clustering of Skip-grams
  Wanzheng Zhu, Hongyu Gong, Jiaming Shen, Chao Zhang, Jingbo Shang, Suma Bhat Jiawei Han
  In The European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD 2020).

2019

• PaRe: A Paper-Reviewer Matching Approach Using a Common Topic Space
  Omer Anjum*, Hongyu Gong*, Suma Bhat, Jinjun Xiong and Wen-mei Hwu.
  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP 2019).

• Reinforcement Learning Based Text Style Transfer without Parallel Training Corpus
  Hongyu Gong, Suma Bhat, Lingfei Wu, Jinjun Xiong and Wen-mei Hwu.
  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL 2019).
  slides

• Context-Sensitive Malicious Spelling Error Correction
  Hongyu Gong, Yuchen Li, Suma Bhat and Pramod Viswanath.
  In World Wide Web Conference (WWW 2019).

• Equipping Educational Applications with Domain Knowledge
  Tarek Sakakini, Hongyu Gong, Jong Yoon Lee, Robert Schloss, Jinjun Xiong and Suma Bhat.
  In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications (ACL Workshop 2019).

2018

• Preposition Sense Disambiguation and Representation
  Hongyu Gong, Jiaqi Mu, Suma Bhat and Pramod Viswanath.
  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP 2018).

• Document Similarity for Texts of Varying Lengths via Hidden Topics
  Hongyu Gong, Tarek Sakakini, Suma Bhat and Jinjun Xiong.
  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL 2018).

• Embedding Syntax and Semantics of Prepositions via Tensor Decomposition
  Hongyu Gong, Suma Bhat and Pramod Viswanath.
  In Proceedings of the 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics, (NAACL 2018).

2017

• Geometry of Compositionality
  Hongyu Gong, Suma Bhat and Pramod Viswanath.
  In Proceedings of the Thirty-First Conference on Artificial Intelligence, (AAAI 2017).

• Distributed Multicast Tree Construction in Wireless Sensor Networks
  Hongyu Gong, Luoyi Fu, Xinzhe Fu, Lutian Zhao, Kainan Wang and Xinbing Wang.
  In IEEE Transactions on Information Theory, 2017.

2015

• A Distributed Algorithm to Construct Multicast Trees in WSNs: An Approximate Steiner Tree Approach
  Hongyu Gong, Lutian Zhao, Kainan Wang, Xinbing Wang, and Weijie Wu.
  In Proceedings of the 16th ACM International Symposium on Mobile Ad Hoc Networking and Computing, (MobiHoc 2015).

• A Distributed Algorithm to Construct Multicast Trees in Wireless Multi-hop Networks
  Hongyu Gong, Lutian Zhao, Kainan Wang, Weijie Wu and Xinbing Wang.
  In IEEE International Conference on Communications, (ICC 2015)