Advancing multilingual reasoning research through large scale human annotated Natural Language Inference data.
We are building NewsNLI, a large scale multilingual NLI dataset designed to advance reasoning capabilities across diverse languages. Our goal is to carefully annotate the dataset by native speakers who not only validate the semantic relationship between sentence pairs but also provide spoken reasoning explaining their decisions.
Rather than simply scaling the number of examples, NewsNLI focuses on reasoning quality and cultural authenticity. By collecting annotations directly from native speakers, we aim to build a dataset that better reflects how people naturally reason in their own languages.
Natural Language Inference (NLI) is one of the fundamental tasks in NLP in which you are given two sentences (premise and hypothesis) and the objective is to determine whether the hypothesis is: (Entailment: supported by the premise), (Contradiction: inconsistent with the premise), (Neutral: neither supported nor contradicted by the premise).
Despite the simplicity of NLI, it measures a model's ability to perform semantic understanding, contextual interpretation, and logical inference. Most existing NLI benchmarks have been developed primarily for high resource languages. While multilingual datasets such as XNLI have expanded language coverage, many languages remain severely underrepresented. As a result, today's multilingual language models often exhibit substantially weaker reasoning capabilities outside English.
In our previous research on NLI, we showed that multilingual reasoning does not improve by scaling data alone and that carefully designed prompting strategies significantly enhance reasoning in low-resource languages . NewsNLI builds on these findings by moving that same emphasis on structured reasoning from prompt engineering to dataset construction, embedding it into annotation guidelines, quality control, and native speaker reasoning rather than relying on inference time prompts or expanding dataset size unecessarily.
Read the guidelines and understand the project.
Fill the Application Form
Receive onboarding and begin contributing.
Both. The dataset will be released on Hugging Face. The accompanying research paper will be submitted to an ACL-family venue. The publication priority is ACL/NeurIPS D&B Track > EMNLP/NAACL/EACL > LREC/COLING.
Language assignment depends on project needs and annotator availability.
Multiple annotators support quality assurance through triangulation, since comparing annotations improves reliability, and distribute the workload so the project does not depend on a single annotator.
No. But for very low-resource languages, help from existing annotators is appreciated and if you already know qualified speakers, you are welcome to invite them.
The test verifies that annotators understand the guidelines and can perform the task correctly.
It checks that you understand the annotation guidelines, distinguish between the labels, and provide appropriate reasoning that you feel is right for that sample in the native language. Carefully read the premise, hypothesis and record your natural reasoning explaining why the relationship between the premise and hypothesis is entailment, contradiction, or neutral. Focus on your reasoning even if the sample is incorrectly labelled, your reasoning should provide that clarity.
No. The reasoning is more important than simply focusing on the label.
Yes. Don't look for external context.
You might encounter samples where the premise may not explicitly support every detail in the hypothesis. Therefore, focus on overall alignment rather than strict fact-by-fact evidence. If the hypothesis is largely aligned with what the premise conveys, label it Entailment; if it contradicts the premise, label it Contradiction; otherwise, label it Neutral. This broader alignment based approach is used because traditional strict NLI definitions that earlier datasets used were mainly designed for synthetic datasets, whereas news based dataset in our case consist of premise that is made-up using Headlines of news articles and hypothesis using summarized news article content. So, naturally for many cases premise will be shorter, and hypothesis will be longer and it won't be possible for a premise to provide enough content to support every claim made in hypothesis that's why our approach used an adapted form of NLI. Therefore, reasoning and overall semantic alignment are more important than requiring the premise to support every individual claim.
Your audio reasoning will enable research into how native speakers reason compared to AI models, whether reasoning patterns transfer across languages, what information is preserved or lost when speech is converted into text, and whether native spoken reasoning captures cultural or linguistic nuances beyond written transcriptions.
There is no fixed weekly quota. The expectation is consistent progress according to your availability and commitment. The project intentionally avoids assigning a minimum number of samples so contributors do not treat that number as a stopping point. However, you are expected to communicate the progress atleast once in every 15 days.
The original target is approximately 10,000 annotated samples per language. This may not be achievable for every language, especially low-resource ones. The objective is to get as close to that target as possible or, at minimum, achieve reasonably balanced coverage across all supported languages.
There is no fixed duration. Annotation continues until sufficient high-quality data has been collected for the project's goals. The expected timeline is the last week of November but it may take even longer.
Potentially, yes. Contributors who make substantial, high-quality contributions and take responsibility for their language throughout the project will be included as co-authors on the research paper and contributors on the dataset card. Smaller but valuable contributions will instead be recognized in the paper's acknowledgements.
No, it is not a funded project and the appointed annotators and contributors are volunteerly contributing to it to advance research in this area. People working on it are affiliated with various institutions including ML Collective, Masakhane, EleutherAI, and etc.