Publications

2024

1. EMNLP

   Enhancing Reinforcement Learning with Dense Rewards from Language Model Critic

   Meng Cao ,  Lei Shu ,  Lei Yu , and 4 more authors

   arXiv preprint arXiv:2401.07382, 2024
2. EMNLP

   Mechanisms of non-factual hallucinations in language models

   Lei Yu ,  Meng Cao ,  Jackie Chi Kit Cheung , and 1 more author

   arXiv preprint arXiv:2403.18167, 2024
3. ICML

   Successor Features for Efficient Multisubject Controlled Text Generation

   Meng Cao ,  Mehdi Fatemi ,  Jackie Chi Kit Cheung , and 1 more author

   In International Conference on Machine Learning , 2024

2023

1. ICLR

   Systematic Rectification of Language Models via Dead-end Analysis

   Meng Cao ,  Mehdi Fatemi ,  Jackie CK Cheung , and 1 more author

   In International Conference on Learning Representations , 2023
2. EMNLP

   Responsible AI Considerations in Text Summarization Research: A Review of Current Practices

   Yu Lu Liu ,  Meng Cao ,  Su Lin Blodgett , and 3 more authors

   In Findings of the Association for Computational Linguistics: EMNLP 2023 , Dec 2023

   Abs

   AI and NLP publication venues have increasingly encouraged researchers to reflect on possible ethical considerations, adverse impacts, and other responsible AI issues their work might engender. However, for specific NLP tasks our understanding of how prevalent such issues are, or when and why these issues are likely to arise, remains limited. Focusing on text summarization—a common NLP task largely overlooked by the responsible AI community—we examine research and reporting practices in the current literature. We conduct a multi-round qualitative analysis of 333 summarization papers from the ACL Anthology published between 2020–2022. We focus on how, which, and when responsible AI issues are covered, which relevant stakeholders are considered, and mismatches between stated and realized research goals. We also discuss current evaluation practices and consider how authors discuss the limitations of both prior work and their own work. Overall, we find that relatively few papers engage with possible stakeholders or contexts of use, which limits their consideration of potential downstream adverse impacts or other responsible AI issues. Based on our findings, we make recommendations on concrete practices and research directions.

2022

1. EMNLP

   Learning with Rejection for Abstractive Text Summarization

   Meng Cao ,  Yue Dong ,  Jingyi He , and 1 more author

   In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , Dec 2022

   Abs

   State-of-the-art abstractive summarization systems frequently hallucinate content that is not supported by the source document, mainly due to noise in the training dataset.Existing methods opt to drop the noisy samples or tokens from the training set entirely, reducing the effective training set size and creating an artificial propensity to copy words from the source. In this work, we propose a training objective for abstractive summarization based on rejection learning, in which the model learns whether or not to reject potentially noisy tokens. We further propose a regularized decoding objective that penalizes non-factual candidate summaries during inference by using the rejection probability learned during training.We show that our method considerably improves the factuality of generated summaries in automatic and human evaluations when compared to five baseline models, and that it does so while increasing the abstractiveness of the generated summaries.

2. ACL

   Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarization

   Meng Cao ,  Yue Dong ,  and  Jackie Cheung

   In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , May 2022

   Abs

   State-of-the-art abstractive summarization systems often generate hallucinations; i.e., content that is not directly inferable from the source text. Despite being assumed to be incorrect, we find that much hallucinated content is actually consistent with world knowledge, which we call factual hallucinations. Including these factual hallucinations in a summary can be beneficial because they provide useful background information. In this work, we propose a novel detection approach that separates factual from non-factual hallucinations of entities. Our method is based on an entity’s prior and posterior probabilities according to pre-trained and finetuned masked language models, respectively. Empirical results suggest that our method vastly outperforms two baselines in both accuracy and F1 scores and has a strong correlation with human judgments on factuality classification tasks.Furthermore, we use our method as a reward signal to train a summarization system using an off-line reinforcement learning (RL) algorithm that can significantly improve the factuality of generated summaries while maintaining the level of abstractiveness.

3. arXiv

   A Survey on Neural Abstractive Summarization Methods and Factual Consistency of Summarization

   Meng Cao

   arXiv preprint arXiv:2204.09519, May 2022

2020

1. EMNLP

   Factual Error Correction for Abstractive Summarization Models

   Meng Cao ,  Yue Dong ,  Jiapeng Wu , and 1 more author

   In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , Nov 2020

   Abs

   Neural abstractive summarization systems have achieved promising progress, thanks to the availability of large-scale datasets and models pre-trained with self-supervised methods. However, ensuring the factual consistency of the generated summaries for abstractive summarization systems is a challenge. We propose a post-editing corrector module to address this issue by identifying and correcting factual errors in generated summaries. The neural corrector model is pre-trained on artificial examples that are created by applying a series of heuristic transformations on reference summaries. These transformations are inspired by the error analysis of state-of-the-art summarization model outputs. Experimental results show that our model is able to correct factual errors in summaries generated by other neural summarization models and outperforms previous models on factual consistency evaluation on the CNN/DailyMail dataset. We also find that transferring from artificial error correction to downstream settings is still very challenging.

2. EMNLP

   TeMP: Temporal Message Passing for Temporal Knowledge Graph Completion

   Jiapeng Wu ,  Meng Cao ,  Jackie Chi Kit Cheung , and 1 more author

   In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , Nov 2020

   Abs

   Inferring missing facts in temporal knowledge graphs (TKGs) is a fundamental and challenging task. Previous works have approached this problem by augmenting methods for static knowledge graphs to leverage time-dependent representations. However, these methods do not explicitly leverage multi-hop structural information and temporal facts from recent time steps to enhance their predictions. Additionally, prior work does not explicitly address the temporal sparsity and variability of entity distributions in TKGs. We propose the Temporal Message Passing (TeMP) framework to address these challenges by combining graph neural networks, temporal dynamics models, data imputation and frequency-based gating techniques. Experiments on standard TKG tasks show that our approach provides substantial gains compared to the previous state of the art, achieving a 10.7% average relative improvement in Hits@10 across three standard benchmarks. Our analysis also reveals important sources of variability both within and across TKG datasets, and we introduce several simple but strong baselines that outperform the prior state of the art in certain settings.

2019

1. EMNLP

   Referring Expression Generation Using Entity Profiles

   Meng Cao ,  and  Jackie Chi Kit Cheung

   In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , Nov 2019

   Abs

   Referring Expression Generation (REG) is the task of generating contextually appropriate references to entities. A limitation of existing REG systems is that they rely on entity-specific supervised training, which means that they cannot handle entities not seen during training. In this study, we address this in two ways. First, we propose task setups in which we specifically test a REG system’s ability to generalize to entities not seen during training. Second, we propose a profile-based deep neural network model, ProfileREG, which encodes both the local context and an external profile of the entity to generate reference realizations. Our model generates tokens by learning to choose between generating pronouns, generating from a fixed vocabulary, or copying a word from the profile. We evaluate our model on three different splits of the WebNLG dataset, and show that it outperforms competitive baselines in all settings according to automatic and human evaluations.