results: 研究结果表明，本方法能够减少翻译语言的影响，保持内容完整性和目标风格流畅性。Abstract
Translated texts exhibit systematic linguistic differences compared to original texts in the same language, and these differences are referred to as translationese. Translationese has effects on various cross-lingual natural language processing tasks, potentially leading to biased results. In this paper, we explore a novel approach to reduce translationese in translated texts: translation-based style transfer. As there are no parallel human-translated and original data in the same language, we use a self-supervised approach that can learn from comparable (rather than parallel) mono-lingual original and translated data. However, even this self-supervised approach requires some parallel data for validation. We show how we can eliminate the need for parallel validation data by combining the self-supervised loss with an unsupervised loss. This unsupervised loss leverages the original language model loss over the style-transferred output and a semantic similarity loss between the input and style-transferred output. We evaluate our approach in terms of original vs. translationese binary classification in addition to measuring content preservation and target-style fluency. The results show that our approach is able to reduce translationese classifier accuracy to a level of a random classifier after style transfer while adequately preserving the content and fluency in the target original style.
摘要
文本翻译后会显示系统性的语言差异，这些差异称为翻译语言（translationese）。这些语言差异会影响跨语言自然语言处理任务的结果，可能导致偏向结果。在这篇论文中，我们探索了一种新的方法来减少翻译语言：翻译样式传递。由于没有同语言的人工翻译和原始数据，我们使用了一种自动学习的方法，可以从相似的原始和翻译数据中学习。然而， même 这种自动学习方法需要一些平行数据来验证。我们可以消除平行验证数据的需求 by combining the self-supervised loss with an unsupervised loss。这种无supervised loss 利用了原始语言模型的损失 sobre la output de estilo transferido y una pérdida de similitud semántica entre la entrada y la output de estilo transferido。我们按照原始vs. 翻译语言二分类、内容保持和目标风格流畅来评估我们的方法。结果表明，我们的方法可以在style transfer后减少翻译语言分类器的准确率到随机分类器的水平，同时保持内容和目标风格的流畅。
Are NLP Models Good at Tracing Thoughts: An Overview of Narrative Understanding
results: 本研究发现，通过扩展 modularized LLM 的能力，可以解决一些新的 narative understanding 任务。此外，通过将 narative understanding 定义为捕捉作者的想象创作灵感的问题，本研究提出了一新的视角，以增强 narative comprehension 能力。Abstract
Narrative understanding involves capturing the author's cognitive processes, providing insights into their knowledge, intentions, beliefs, and desires. Although large language models (LLMs) excel in generating grammatically coherent text, their ability to comprehend the author's thoughts remains uncertain. This limitation hinders the practical applications of narrative understanding. In this paper, we conduct a comprehensive survey of narrative understanding tasks, thoroughly examining their key features, definitions, taxonomy, associated datasets, training objectives, evaluation metrics, and limitations. Furthermore, we explore the potential of expanding the capabilities of modularized LLMs to address novel narrative understanding tasks. By framing narrative understanding as the retrieval of the author's imaginative cues that outline the narrative structure, our study introduces a fresh perspective on enhancing narrative comprehension.
摘要
narrative understanding 涉及捕捉作者的认知过程，提供作者的知识、意图、信仰、愿望等信息的启示。虽然大语言模型（LLM）在生成 grammatically coherent text 方面表现出色，但它们对作者的思想真实理解仍存在uncertainty。这种限制阻碍了 narraitve understanding 的实际应用。在这篇论文中，我们进行了全面的 narrative understanding 任务调查，详细检查了这些任务的关键特征、定义、分类、相关数据集、训练目标、评价指标以及局限性。此外，我们还探讨了扩展 modularized LLM 的能力，以解决新的 narrative understanding 任务。我们通过将 narrative understanding 定义为捕捉作者的想象力cue 的抽象，从新的角度增强了 narrative comprehension。
ProMap: Effective Bilingual Lexicon Induction via Language Model Prompting
results: 在评估多种单词翻译方法，包括静态单词表示的方法，ProMap consistently achieve 状态的 лучResults ，并且在少数例示enario 下（ fewer than 10 个训练示例）也能够达到良好的性能。Abstract
Bilingual Lexicon Induction (BLI), where words are translated between two languages, is an important NLP task. While noticeable progress on BLI in rich resource languages using static word embeddings has been achieved. The word translation performance can be further improved by incorporating information from contextualized word embeddings. In this paper, we introduce ProMap, a novel approach for BLI that leverages the power of prompting pretrained multilingual and multidialectal language models to address these challenges. To overcome the employment of subword tokens in these models, ProMap relies on an effective padded prompting of language models with a seed dictionary that achieves good performance when used independently. We also demonstrate the effectiveness of ProMap in re-ranking results from other BLI methods such as with aligned static word embeddings. When evaluated on both rich-resource and low-resource languages, ProMap consistently achieves state-of-the-art results. Furthermore, ProMap enables strong performance in few-shot scenarios (even with less than 10 training examples), making it a valuable tool for low-resource language translation. Overall, we believe our method offers both exciting and promising direction for BLI in general and low-resource languages in particular. ProMap code and data are available at \url{https://github.com/4mekki4/promap}.
摘要
百度 Lexicon 推理 (BLI), 将词语翻译 между两种语言，是 NLP 任务中的一项重要任务。虽然在使用静态词嵌入的情况下，在丰富资源语言中已经取得了可注目的进步，但词语翻译性能可以通过使用语言模型的上下文化词嵌入进一步改进。在这篇论文中，我们介绍了 ProMap，一种新的 BLI 方法，利用预训练的多语言多方言语言模型的力量，解决这些挑战。为了超越使用子词 токен，ProMap 利用有效的补充提示语言模型的方法，并使用种子词典 achieve 好的性能。我们还 demonstarte ProMap 可以在其他 BLI 方法的结果中进行排名，如采用静态词嵌入的方法。当评估在丰富资源语言和低资源语言上时，ProMap consistently 取得了状态的艺术结果。此外，ProMap 可以在少量示例下进行几极enario （即使使用 less than 10 个训练示例），这使其成为低资源语言翻译中的有价值工具。总之，我们认为我们的方法对 BLI 和低资源语言来说是一种激动人心的和有前途的方向。ProMap 代码和数据可以在 \url{https://github.com/4mekki4/promap} 上找到。
Crossing the Aisle: Unveiling Partisan and Counter-Partisan Events in News Reporting
results: 研究发现，新闻媒体通过事件包容或排除来影响公众意见，并且这种影响可以通过语言模型更好地理解事件在更广泛的上下文中。同时，研究也发现了新闻媒体的选择性报道可能会影响公众意见的方向性。Abstract
News media is expected to uphold unbiased reporting. Yet they may still affect public opinion by selectively including or omitting events that support or contradict their ideological positions. Prior work in NLP has only studied media bias via linguistic style and word usage. In this paper, we study to which degree media balances news reporting and affects consumers through event inclusion or omission. We first introduce the task of detecting both partisan and counter-partisan events: events that support or oppose the author's political ideology. To conduct our study, we annotate a high-quality dataset, PAC, containing 8,511 (counter-)partisan event annotations in 304 news articles from ideologically diverse media outlets. We benchmark PAC to highlight the challenges of this task. Our findings highlight both the ways in which the news subtly shapes opinion and the need for large language models that better understand events within a broader context. Our dataset can be found at https://github.com/launchnlp/Partisan-Event-Dataset.
摘要
新闻媒体应该保持不倚于任何政治立场的报道，但它们可能仍然影响公众意见通过选择性地包括或排除支持或反对其政治立场的事件。在这篇论文中，我们研究了新闻报道是如何帮助或妨碍公众意见的。我们首先介绍了检测政治立场事件的任务，包括支持和反对作者政治立场的事件。为了进行这项研究，我们在304篇来自不同政治立场的新闻媒体发布的文章中标注了8511个（Counter-)政治立场事件。我们使用PAC数据集进行测试，以高亮这个任务的挑战。我们的发现表明新闻可以在不显着的方式下形成公众意见，同时也表明需要更好地理解事件在更广泛的上下文中。我们的数据集可以在GitHub上找到：https://github.com/launchnlp/Partisan-Event-Dataset。
results: 实验表明，TLM可以在4种不同的自然语言处理任务上提高性能，比如GLUE、ChineseGLUE、中文语法错误修复和数据到文本生成等，并且可以超越DropHead和注意力遮盾。例如，使用BERT-large模型，TLM在GLUE上提高了0.5点相对于DropHead。此外，TLM在Rotowire上达到了18.93 BLEU的新纪录。Abstract
Structured dropout approaches, such as attention dropout and DropHead, have been investigated to regularize the multi-head attention mechanism in Transformers. In this paper, we propose a new regularization scheme based on token-level rather than structure-level to reduce overfitting. Specifically, we devise a novel Token-Level Masking (TLM) training strategy for Transformers to regularize the connections of self-attention, which consists of two masking techniques that are effective and easy to implement. The underlying idea is to manipulate the connections between tokens in the multi-head attention via masking, where the networks are forced to exploit partial neighbors' information to produce a meaningful representation. The generality and effectiveness of TLM are thoroughly evaluated via extensive experiments on 4 diversified NLP tasks across 18 datasets, including natural language understanding benchmark GLUE, ChineseGLUE, Chinese Grammatical Error Correction, and data-to-text generation. The results indicate that TLM can consistently outperform attention dropout and DropHead, e.g., it increases by 0.5 points relative to DropHead with BERT-large on GLUE. Moreover, TLM can establish a new record on the data-to-text benchmark Rotowire (18.93 BLEU). Our code will be publicly available at https://github.com/Young1993/tlm.
摘要
“structured dropout方法，如注意力Dropout和DropHead，已经用来规化Transformer中的多头注意力机制。在这篇论文中，我们提出了一新的规化方案，基于Token Level而不是结构 Level，以减少过拟合。 Specifically, we develop a novel Token-Level Masking（TLM）训练策略 дляTransformer，以规化自我注意力的连接，这包括两种遮盾技术，它们是有效且易于实现。 The underlying idea is to manipulate the connections between tokens in the multi-head attention via masking, where the networks are forced to exploit partial neighbors' information to produce a meaningful representation。”“我们透过广泛的实验评估TLM的通用性和效果，包括18个不同的自然语言处理任务和4个测试集。结果显示，TLM可以较DropHead和注意力Dropout表现出色，例如，与BERT-large在GLUE上的结果提高0.5分。此外，TLM可以创下Rotowire（18.93 BLEU）中的新纪录。我们将代码公开在https://github.com/Young1993/tlm。”
When Reviewers Lock Horn: Finding Disagreement in Scientific Peer Reviews
for: 本研究旨在Automatically identifying contradictions among reviewers on a given article.
methods: 我们提出了一种基本模型，可以从open review-based ICLR和NeurIPS会议的 around 8.5k paper中检测出对 противоречи的评论。
results: 我们创建了一个包含around 28k review pair中nearly 50k review pair comment的comprehensive review-pair contradiction数据集，并提出了一种基本模型可以自动检测评论中的对 противоречи。Abstract
To this date, the efficacy of the scientific publishing enterprise fundamentally rests on the strength of the peer review process. The journal editor or the conference chair primarily relies on the expert reviewers' assessment, identify points of agreement and disagreement and try to reach a consensus to make a fair and informed decision on whether to accept or reject a paper. However, with the escalating number of submissions requiring review, especially in top-tier Artificial Intelligence (AI) conferences, the editor/chair, among many other works, invests a significant, sometimes stressful effort to mitigate reviewer disagreements. Here in this work, we introduce a novel task of automatically identifying contradictions among reviewers on a given article. To this end, we introduce ContraSciView, a comprehensive review-pair contradiction dataset on around 8.5k papers (with around 28k review pairs containing nearly 50k review pair comments) from the open review-based ICLR and NeurIPS conferences. We further propose a baseline model that detects contradictory statements from the review pairs. To the best of our knowledge, we make the first attempt to identify disagreements among peer reviewers automatically. We make our dataset and code public for further investigations.
摘要
ASTormer: An AST Structure-aware Transformer Decoder for Text-to-SQL
results: 对五个文本到SQL benchmark进行了广泛的实验，并证明了ASTormer比基于RNN的竞争对手更有效和高效。Abstract
Text-to-SQL aims to generate an executable SQL program given the user utterance and the corresponding database schema. To ensure the well-formedness of output SQLs, one prominent approach adopts a grammar-based recurrent decoder to produce the equivalent SQL abstract syntax tree (AST). However, previous methods mainly utilize an RNN-series decoder, which 1) is time-consuming and inefficient and 2) introduces very few structure priors. In this work, we propose an AST structure-aware Transformer decoder (ASTormer) to replace traditional RNN cells. The structural knowledge, such as node types and positions in the tree, is seamlessly incorporated into the decoder via both absolute and relative position embeddings. Besides, the proposed framework is compatible with different traversing orders even considering adaptive node selection. Extensive experiments on five text-to-SQL benchmarks demonstrate the effectiveness and efficiency of our structured decoder compared to competitive baselines.
摘要
文本到SQL的目标是生成基于用户语音和相应的数据库架构的可执行SQL程序。为保证输出SQL的正确性，一种广泛采用的方法是使用 grammar-based 回归decoder生成相应的SQL抽象语法树（AST）。然而，前一代方法主要采用 RNN 序列decoder，这些方法有以下两点缺陷：1）时间consuming 和不效率，2）不提供多少结构偏好。在这项工作中，我们提议一种AST结构意识的Transformer decoder（ASTormer）来取代传统的RNN细胞。在decoder中，结构知识，如树中节点类型和位置，通过绝对和相对位置嵌入被灵活地嵌入。此外，我们提出的框架可以与不同的搜索顺序一起使用，包括自适应节点选择。我们对五个文本到SQLbenchmark进行了广泛的实验，并证明了我们的结构化decoder与其他基准值比较有效和高效。
Personalised Distillation: Empowering Open-Sourced LLMs with Adaptive Learning for Code Generation
results: 研究表明，个性化储存方法在代码生成任务中表现出色，可以在只有一 third的数据量下达到传统方法的水平。在HumanEval中，通过使用2.5-3K个性化示例，可以提高CodeGen-mono-16B的表现，从33.6%提高到36.4%，并提高StarCoder的表现，从39.5%提高到45.8%。Abstract
With the rise of powerful closed-sourced LLMs (ChatGPT, GPT-4), there are increasing interests in distilling the capabilies of close-sourced LLMs to smaller open-sourced LLMs. Previous distillation methods usually prompt ChatGPT to generate a set of instructions and answers, for the student model to learn. However, such standard distillation approach neglects the merits and conditions of the student model. Inspired by modern teaching principles, we design a personalised distillation process, in which the student attempts to solve a task first, then the teacher provides an adaptive refinement for the student to improve. Instead of feeding the student with teacher's prior, personalised distillation enables personalised learning for the student model, as it only learns on examples it makes mistakes upon and learns to improve its own solution. On code generation, personalised distillation consistently outperforms standard distillation with only one third of the data. With only 2.5-3K personalised examples that incur a data-collection cost of 4-6$, we boost CodeGen-mono-16B by 7% to achieve 36.4% pass@1 and StarCoder by 12.2% to achieve 45.8% pass@1 on HumanEval.
摘要
Inspired by modern teaching principles, we designed a personalized distillation process in which the student attempts to solve a task first, and then the teacher provides an adaptive refinement for the student to improve. Instead of feeding the student with the teacher's prior knowledge, personalized distillation enables personalized learning for the student model, as it only learns from examples it makes mistakes on and learns to improve its own solutions.On code generation, personalized distillation consistently outperforms standard distillation with only one-third of the data. With only 2,500 to 3,000 personalized examples that incur a data-collection cost of $4 to $6, we boosted CodeGen-mono-16B by 7% to achieve 36.4% pass@1 and StarCoder by 12.2% to achieve 45.8% pass@1 on HumanEval.
paper_authors: Chonggang Lu, Richong Zhang, Kai Sun, Jaein Kim, Cunwang Zhang, Yongyi Mao
for: DocRE document-level relation extraction
methods: Anaphor-Assisted (AA) framework
results: new state-of-the-art performanceAbstract
Document-level relation extraction (DocRE) involves identifying relations between entities distributed in multiple sentences within a document. Existing methods focus on building a heterogeneous document graph to model the internal structure of an entity and the external interaction between entities. However, there are two drawbacks in existing methods. On one hand, anaphor plays an important role in reasoning to identify relations between entities but is ignored by these methods. On the other hand, these methods achieve cross-sentence entity interactions implicitly by utilizing a document or sentences as intermediate nodes. Such an approach has difficulties in learning fine-grained interactions between entities across different sentences, resulting in sub-optimal performance. To address these issues, we propose an Anaphor-Assisted (AA) framework for DocRE tasks. Experimental results on the widely-used datasets demonstrate that our model achieves a new state-of-the-art performance.
摘要
文档级关系EXTRACTION（DocRE）涉及到在文档中多个句子中Identifying关系 между实体。现有方法是建立不同类型的文档图来模型实体的内部结构和实体之间的外部互动。然而，现有方法存在两点缺陷。一方面，Anaphora在理解关系 identification 中发挥重要作用，但这些方法忽略了它。另一方面，这些方法通过使用文档或句子作为中间节点来实现跨句sentenceEntity interaction，这会增加学习细化实体之间的交互问题，导致性能下降。为了解决这些问题，我们提出了Anaphor-Assisted（AA）框架来Address DocRE任务。实验结果表明，我们的模型在广泛使用的数据集上达到了新的状态的艺术性能。
Accelerating LLM Inference by Enabling Intermediate Layer Decoding
results: 在Alpaca数据集上进行了广泛的实验，并对四个不同的人工指令测试集进行了总体评估：Vicuna、WizardLM、Koala和Self-Instruct。结果表明，“动态早退”可以实现37.86%的平均成本改进，保持生成质量。进一步的分析结果表明，输出的语义相似性和生成的数量均有改进。Abstract
Large Language Models (LLMs) have achieved remarkable performance across a wide variety of natural language tasks; however, their large size makes their inference slow and computationally expensive which poses a practical challenge for resource constrained real-world applications. Focusing on this problem, we propose to instruction tune LLMs in a way that enables intermediate layer decoding for efficiently generating text, but importantly without compromising the quality of the generation. Specifically, we instruction tune LLMs with additional explicit Losses from the InTermediate layErs (LITE) and show that it enables these layers to acquire 'good' generation ability without affecting the generation ability of the final layer. We perform 'dynamic confidence-based early exiting' at token level from the intermediate layers which improves the efficiency of inference while maintaining the generation quality. We conduct comprehensive experiments by instruction tuning LLaMA-2 models on the widely used Alpaca dataset and holistically evaluate on four different human-instruction test sets: Vicuna, WizardLM, Koala, and Self-Instruct. We show that 'dynamic early exiting' achieves consistent and considerable cost improvements (37.86% on average) while maintaining the generation quality of the responses. We further conduct a thorough analysis of the results over several important aspects, such as comparing the semantic similarity of the outputs and dissecting the efficiency improvements by comparing the number of tokens generated in the output. In summary, our work contributes to improving the efficiency of LLM inference while maintaining the generation quality, a crucial step en route to enabling their widespread adoption.
摘要
大型自然语言模型（LLM）在各种自然语言任务上达到了很高的表现水平，但是它们的大小使得其推理慢且计算成本高，这成为了实际应用中的实际挑战。在这个问题上，我们提出了在LLM中 instrucion 优化，以实现中间层解码，以便高效地生成文本，而不会影响最终层的生成质量。我们在LLM中添加了额外的明确损失（LITE），使中间层学习“好”的生成能力，而不会影响最终层的生成能力。我们在中间层进行“动态信息确定早退”，从而提高推理效率，保持生成质量。我们对 LLamA-2 模型进行了广泛的实验，并对四个不同的人工指令测试集进行了总体评估：Vicuna、WizardLM、Koala 和 Self-Instruct。我们发现，“动态早退”可以具有重要的成本改善（37.86% 的平均提高），同时保持生成质量。我们进一步进行了详细的分析结果，包括比较输出的semantic相似性和生成量的比较分析。总之，我们的工作为LLM推理效率的提高，并保持生成质量，是实际应用中的关键一步。
Identifying Conspiracy Theories News based on Event Relation Graph
results: 实验结果表明，基于事件关系图的方法可以提高阴谋理论检测的准确率和受检测率，并且能够在新的媒体源上进行检测。Abstract
Conspiracy theories, as a type of misinformation, are narratives that explains an event or situation in an irrational or malicious manner. While most previous work examined conspiracy theory in social media short texts, limited attention was put on such misinformation in long news documents. In this paper, we aim to identify whether a news article contains conspiracy theories. We observe that a conspiracy story can be made up by mixing uncorrelated events together, or by presenting an unusual distribution of relations between events. Achieving a contextualized understanding of events in a story is essential for detecting conspiracy theories. Thus, we propose to incorporate an event relation graph for each article, in which events are nodes, and four common types of event relations, coreference, temporal, causal, and subevent relations, are considered as edges. Then, we integrate the event relation graph into conspiracy theory identification in two ways: an event-aware language model is developed to augment the basic language model with the knowledge of events and event relations via soft labels; further, a heterogeneous graph attention network is designed to derive a graph embedding based on hard labels. Experiments on a large benchmark dataset show that our approach based on event relation graph improves both precision and recall of conspiracy theory identification, and generalizes well for new unseen media sources.
摘要
《刺激论题》是一种不合理或恶意的含义，用于解释事件或情况。而大多数前期工作都是通过社交媒体短文来研究刺激论题，却受到了长文报道的限制。在这篇论文中，我们想要判断一篇新闻文章是否包含刺激论题。我们发现，刺激故事可以通过将不相关的事件混合起来，或者通过事件之间的不寻常的关系分布来构成。为了检测刺激论题，我们需要了解事件的上下文知识。因此，我们提议使用事件关系图来识刺刺激论题。每篇文章都有一个事件关系图，其中事件是节点，而四种常见的事件关系，核心引用、时间关系、 causal 关系和 subsequential 关系，被视为边。然后，我们将事件关系图 интеグриinto刺激论题标识中两种方式：首先，我们开发了一个事件意识语言模型，以提高基本语言模型的知识水平，并通过软标签将事件和事件关系传递给模型；其次，我们设计了一个多类Graph注意力网络，以生成基于硬标签的图 embedding。实验表明，我们基于事件关系图的方法可以提高刺激论题标识的精度和准确率，并在新的媒体来源上具有良好的泛化性。
results: 实验结果表明，通过利用教师预测概率或知识储存框架来汇集 discourse structures 可以显著提高宣传内容识别的精度。Abstract
Propaganda is a form of deceptive narratives that instigate or mislead the public, usually with a political purpose. In this paper, we aim to identify propaganda in political news at two fine-grained levels: sentence-level and token-level. We observe that propaganda content is more likely to be embedded in sentences that attribute causality or assert contrast to nearby sentences, as well as seen in opinionated evaluation, speculation and discussions of future expectation. Hence, we propose to incorporate both local and global discourse structures for propaganda discovery and construct two teacher models for identifying PDTB-style discourse relations between nearby sentences and common discourse roles of sentences in a news article respectively. We further devise two methods to incorporate the two types of discourse structures for propaganda identification by either using teacher predicted probabilities as additional features or soliciting guidance in a knowledge distillation framework. Experiments on the benchmark dataset demonstrate that leveraging guidance from discourse structures can significantly improve both precision and recall of propaganda content identification.
摘要
宣传是一种欺骗性的叙述，通常有政治目的。在这篇论文中，我们目标是在新闻文本中发现宣传。我们发现宣传内容更容易在归因或者评价附近的句子中出现，以及在评价、推测和未来预测中出现。因此，我们建议使用本地和全局文本结构来发现宣传。我们设计了两种教师模型，一个用于确定邻近句子之间的 PDTB 风格的语言关系，另一个用于确定新闻文本中句子的常见语言角色。此外，我们还提出了两种方法来结合这两种文本结构来发现宣传内容，一种是使用教师预测概率作为额外特征，另一种是在知识填充框架中寻求指导。实验表明，通过使用文本结构的指导，可以大幅提高宣传内容发现的精度和准确性。