Call for Main Conference Papers

Important Notice

The direct submission link is available. Authors will use the same link to submit the abstract first, then the full paper. For abstract submissions, there is no need to upload a pdf paper. Authors will have until the submission date to submit the full manuscript. Once a submission has been made, the same submission can be revised until the paper deadline by going to the author console. After the abstract submission deadline, the form will change and some fields will become mandatory.

The ARR commitment link is available.

Overview

The 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023) invites the submission of long and short papers on substantial, original, and unpublished research on empirical methods for Natural Language Processing. As in recent years, some of the presentations at the conference will be for papers accepted by the Transactions of the ACL (TACL) and Computational Linguistics (CL) journals.

EMNLP 2023 will follow EMNLP 2022 and ACL 2023 and go with a hybrid format with respect to ARR. This means that while EMNLP will accept ARR-reviewed papers, it will also accept submissions directly to EMNLP through the OpenReview system.

However, in order to keep the review load on the community as a whole manageable, we need to ask authors to decide up-front if they want to be reviewed through ARR or EMNLP.

Review Process:

Papers submitted directly to EMNLP will have the “regular” review process: paper reviewed by 3 reviewers, authors are invited to write an author response and revise their paper before the camera ready deadline, if accepted. ARR papers committed to EMNLP will be handled by the Senior Area Chairs. For these papers, the authors may provide an author response but not revise their paper (with the exception of adding the required “limitations” section, if it was missing from the ARR submission; see below).

Open Papers/Reviews/Metareviews/Rebuttal via OpenReview Platform

This year, some of the reviews, author responses and meta reviews will be made publicly available. Our motivation is to provide increased transparency in the review process, to foster more accountability for reviewers and higher quality reviews as well as enabling peer review research by providing an open collection of papers and reviews. We will follow a process where only reviews, author responses and meta reviews of accepted papers and opt-in rejected papers (where opting in is done by authors) will be made publicly available after the acceptance notifications. We will use the OpenReview platform, which is also currently used by the ACL Rolling Review as well as other related conferences and is well suited to this type of process.

Cross Submission Policy with ARR

Any ARR-reviewed paper that got all of its reviews and meta-reviews available by the ARR-to-conference commitment deadline (August 22, 2023), can be committed to EMNLP 2023.
- Notes:
  - The manuscript submission from ARR that is committed to EMNLP cannot be modified (with the exception of adding the required “limitations” section, if it was missing from the ARR submission), but the authors can add an author response that will be seen by the EMNLP programme committee.
  - EMNLP will consider any ARR paper that has been fully reviewed by the August 22 2023 commitment deadline. The regular ARR timeline suggests that ARR submissions should be made by June 15th. Consequently, care must be taken in deciding whether a submission should be made to ARR or EMNLP directly if the work has not been submitted anywhere before the call. Plan accordingly.
  - The OpenReview system deadline for direct submission papers, namely non-ARR submission papers, is June 23, 2023.
Papers submitted to ARR before May 27, 2023, can be withdrawn and submitted to EMNLP 2023.
- Notes:
  - In order for a paper to be submitted directly to EMNLP 2023, it must be inactive in the ARR system. This means that the submission must either be explicitly withdrawn by the authors, or the ARR reviews are finished and shared with the authors before May 27, and the paper was not re-submitted to ARR.
  - The authors can withdraw from ARR by May 27, 2023, regardless of how many reviews they have received.
Papers that are in the ARR system after May 27, 2023, either submitted after or submitted before and not withdrawn, cannot be submitted to EMNLP 2023.
Papers submitted to EMNLP 2023 may not be submitted for review elsewhere (including ARR) while being under review at EMNLP 2023.

Major differences from “standard” recent conferences include:

Hybrid ARR + EMNLP models.
Mandatory discussion of limitations.
Theme: Large Language Models and the Future of NLP

Important Dates

All deadlines are 11:59 pm UTC -12h (“anywhere on Earth”).

Anonymity period begins	May 23, 2023
Abstract deadline for OpenReview direct submissions for direct submissions	June 16, 2023
Direct paper submission deadline (long & short papers)	June 23, 2023
Commitment deadline for ARR papers	August 22, 2023
Author response period	Aug 22 – Aug 28, 2023
Notification of acceptance (long & short papers)	Oct 6, 2023
Camera-ready papers due (long & short papers)	Oct 20, 2023
Workshops & Tutorials & Conference	December 6-10, 2023

Mandatory abstract submission

The paper title, author names, contact details, and a brief abstract must be submitted electronically through the EMNLP 2023 paper submission site by the abstract submission deadline (June 16). It will be possible to make minor edits to the title and abstract until the full paper submission deadline, but you cannot change ~~authors and~~ subject areas. As this is the first time the main conference is using OpenReview as platform, which requires to create an account prior to submission, we will exceptionally allow modifications to the list of authors between the abstract deadline and the paper submission deadline. No further changes to the author list will be allowed after the paper submission deadline. Submissions with “placeholder” abstracts will be removed without consideration;

Important: if you miss the abstract submission deadline, then you cannot submit the full paper.

Submissions

EMNLP 2023 has the goal of a broad technical program. Relevant topics for the conference include, but are not limited to, the following areas (in alphabetical order):

Commonsense Reasoning
Computational Social Science and Cultural Analytics
Dialogue and Interactive Systems
Discourse and Pragmatics
Efficient Methods for NLP
Ethics in NLP
Human-Centered NLP
Information Extraction
Information Retrieval and Text Mining
Interpretability, Interactivity and Analysis of Models for NLP
Language Grounding to Vision, Robotics and Beyond
Language Modeling and Analysis of Language Models
Linguistic Theories, Cognitive Modeling and Psycholinguistics
Machine Learning for NLP
Machine Translation
Multilinguality and Linguistic Diversity
Natural Language Generation
NLP Applications
Phonology, Morphology and Word Segmentation
Question Answering
Resources and Evaluation
Semantics: Lexical, Sentence level, Document Level, Textual Inference, etc.
Sentiment Analysis, Stylistic Analysis, and Argument Mining
Speech and Multimodality
Summarization
Syntax, Parsing and their Applications
Theme Track

Paper Submission Information

Paper Submission and Templates

Submission is electronic, using the OpenReview conference management system. Both long and short papers must follow the EMNLP 2023 two-column format, using the supplied official style files. The templates can be downloaded in Style Files and Formatting. Please do not modify these style files, nor should you use templates designed for other conferences. Submissions that do not conform to the required styles, including paper size, margin width, and font size restrictions, will be rejected without review.

Verification

To guarantee conformance to publication standards, we will be using the ACL Pubcheck tool (https://github.com/acl-org/aclpubcheck). The PDFs of camera-ready papers must be run through this tool prior to their final submission, and we recommend its use also at submission time.

Long Papers

Long paper submissions must describe substantial, original, completed and unpublished work. Wherever appropriate, concrete evaluation and analysis should be included. Review forms will be made available prior to the deadlines. Long papers may consist of up to 8 pages of content, plus unlimited pages for references and appendix; final versions of long papers will be given one additional page of content (up to 9 pages) so that reviewers’ comments can be taken into account.

Short Papers

Short paper submissions must describe original and unpublished work. Please note that a short paper is not a shortened long paper. Instead short papers should have a point that can be made in a few pages.

Short papers may consist of up to 4 pages of content, plus unlimited references and appendix. Upon acceptance, short papers will be given 5 content pages in the proceedings. Authors are encouraged to use this additional page to address reviewers’ comments in their final versions.

Contributions

EMNLP 2023 welcomes the following kinds of contributions:

Computationally-aided linguistic analysis (of either models or data resources)
NLP engineering experiment
Reproduction study
New data resources, particularly for low-resource languages
Approaches for data- and compute efficiency
Position papers
Surveys
Publicly available software and pre-trained models

While there is no direct mapping between types of contributions and paper length, some kinds of papers naturally gravitate towards a certain length: e.g. surveys are more likely to be long rather than short papers. One paper can make more than one contribution of different types.

Presentation Mode

Long and short papers will be presented orally or as posters as determined by the program committee. The decisions as to which papers will be presented orally and which as poster presentations will be based on the nature rather than the quality of the work. While short papers will be distinguished from long papers in the proceedings, there will be no distinction in the proceedings between papers presented orally and as posters.

Authorship

The author list for submissions should include all (and only) individuals who made substantial contributions to the work presented. Each author listed on a submission to EMNLP 2023 will be notified of submissions, revisions and the final decision. No changes to the order or composition of authorship may be made to submissions to EMNLP 2023 after the abstract submission deadline.

Citation and Comparison

You are expected to cite all refereed publications relevant to your submission, but you may be excused for not knowing about all unpublished work (especially work that has been recently posted and/or is not widely cited). While not citing such unpublished works upon submission is not sufficient grounds for paper rejection, you are expected to cite such relevant work in camera ready, if notified about it by reviewers.

In cases where a preprint has been superseded by a refereed publication, the refereed publication should be cited instead of the preprint version. Papers (whether refereed or not) appearing less than 3 months before the submission deadline are considered contemporaneous to your submission, and you are therefore not obliged to make detailed comparisons that require additional experimentation and/or in-depth analysis. However, you are expected to mention such works in your submission, and list their published results if they are directly relevant.

For more information, see the ACL Policies for Submission, Review, and Citation.

Multiple Submission Policy

EMNLP 2023 will not consider any paper that is under review in a journal or another conference at the time of submission, and submitted papers must not be submitted elsewhere during the EMNLP 2023 review period. This policy covers all refereed and archival conferences and workshops (e.g., NeurIPS, ACL workshops), as well as ARR. In addition, we will not consider any paper that overlaps significantly in content or results with papers that will be (or have been) published elsewhere. Authors submitting more than one paper to EMNLP 2023 must ensure that their submissions do not overlap significantly (>25%) with each other in content or results.

EMNLP 2023 will also accept submissions of ARR-reviewed papers, provided that the ARR reviews and meta-reviews are available by the ARR-to-conference submission deadline. However, EMNLP 2023 will not accept direct submissions that are actively under review in ARR, or that overlap significantly (>25%) with such submissions.

Optional: Sticky Reviews

The papers previously reviewed at other *ACL venues (but not through ARR) have the option to submit the paper together with information about their previous submission, from which the track chairs will be able to access the old reviews. They will also be able to submit a short (~ 1 page) explanation of how the paper was changed in response to the old reviews. This option could be beneficial for the authors who have addressed the problems identified before, and can argue strongly for how the paper has been improved. The prior reviews will not be seen by the new reviewers, but they may be used by the area chairs, senior area chairs and program chairs in review quality control, resolving disagreements between reviewers, and in deciding borderline papers. All the information submitted should be anonymous.

Mandatory Discussion of Limitations

We believe that it is also important to discuss the limitations of your work, in addition to its strengths. EMNLP 2023 requires all papers to have a clear discussion of limitations, in a dedicated section titled “Limitations”. This section will appear at the end of the paper, after the discussion/conclusions section and before the references, and will not count towards the page limit. Papers without a limitation section will be automatically rejected without review.

ARR-reviewed paper that did not include “Limitations” section in their prior submission, should submit a PDF with such a section together with their EMNLP 2023 submission.

While we are open to different types of limitations, just mentioning that a set of results have been shown for English only probably does not reflect what we expect. Mentioning that the method works mostly for languages with limited morphology, like English, is a much better alternative. In addition, limitations such as low scalability to long text, the requirement of large GPU resources, or other things that inspire crucial further investigation are welcome.

Theme Track: Large Language Models and the Future of NLP

We are happy to announce that EMNLP 2023 will have a new theme with the goal of stimulating discussion around Large Language Models and the Future of NLP. While the new generation of Large Language Models such as GPTX, LLAMA, BLOOM etc. claim to perform at unprecedented levels for generation and understanding, we are in unexplored territory on many aspects of such LLMs, including performance on various NLP tasks and languages, data sovereignty, fairness, interpretability, ethics, transparency, NLP applications, etc.

The theme track invites empirical and theoretical research, as well as position and survey papers on the ways in which such LLMs perform on NLP tasks and applications, and what this means for the future of NLP as a field. The possible topics of discussion include (but are not limited to) the following:

How reliably do the current generation of LLMs perform on NLP tasks and applications?
How is linguistic diversity covered by these LLMs?
What are the different systemic failures of such LLMs and recovery strategies and methodologies?
Do these models enhance scientific understanding (of language, cognition, or deep learning technology)? In what ways?
What are the different ethical and FATE-related considerations regarding the design and use of such models?
What are the opportunities LLMs offer to NLP research ?
How do LLMs capture world knowledge?
How can we incorporate existing knowledge bases effectively into LLMs?
How can we evaluate the performance of such models intrinsically (with no downstream application involved)
How can such models influence how NLP research is done in the future?
How replicable is the performance of these models in both NLP research and real-life applications?

The theme track submissions can be either long or short. We anticipate having a special session on this theme at the conference and a Thematic Paper Award in addition to other categories of awards.

Ethics Policy

Authors are required to honor the ethical code set out in the ACL Code of Ethics. The consideration of the ethical impact of our research, use of data, and potential applications of our work has always been an important consideration, and as artificial intelligence is becoming more mainstream, these issues are increasingly pertinent. We ask that all authors read the code, and ensure that their work is conformant to this code. Where a paper may raise ethical issues, we ask that you include in the paper an explicit discussion of these issues, which will be taken into account in the review process. We reserve the right to reject papers on ethical grounds, where the authors are judged to have operated counter to the code of ethics, or have inadequately addressed legitimate ethical concerns with their work.

Authors will be allowed extra space after the 8th page (4th for short papers) for an optional broader impact statement or other discussion of ethics. The EMNLP review form will include a section addressing these issues and papers flagged for ethical concerns by reviewers or ACs will be further reviewed by an ethics committee. Note that an ethical considerations section is not required, but papers working with sensitive data or on sensitive tasks that do not discuss these issues will not be accepted. Conversely, the mere inclusion of an ethical considerations section does not guarantee acceptance. In addition to acceptance or rejection, papers may receive a conditional acceptance recommendation. Camera-ready versions of papers designated as conditional accept will be re-reviewed by the ethics committee to determine whether the concerns have been adequately addressed. Please read the ethics FAQ for more guidance on some problems to look out for and key concerns to consider relative to the code of ethics.

Optional Supplementary Materials

Appendices, Software and Data

Each EMNLP 2023 submission can be accompanied by an appendix, which will appear in the main paper’s PDF, after the bibliography. A submission may also be accompanied by one .tgz or .zip archive containing software, and one .tgz or .zip archive containing data. EMNLP 2023 encourages the submission of these supplementary materials to improve the reproducibility of results, and to enable authors to provide additional information that does not fit in the paper. For example, anonymised related work (see above), preprocessing decisions, model parameters, feature templates, lengthy proofs or derivations, pseudocode, sample system inputs/outputs, and other details that are necessary for the exact replication of the work described in the paper can be put into the appendix. However, the paper submissions need to remain fully self-contained, as these supplementary materials are completely optional, and reviewers are not even asked to review or download them. If the pseudo-code or derivations or model specifications are an important part of the contribution, or if they are important for the reviewers to assess the technical correctness of the work, they should be a part of the main paper, and not appear in the appendix. Supplementary materials need to be fully anonymized to preserve the double-blind reviewing policy.

Anonymity Period

The following rules and guidelines are meant to protect the integrity of double-blind review and ensure that submissions are reviewed fairly. The rules make reference to the anonymity period, which runs from 1 month before the submission deadline (starting May 23, 2023) up to the date when your paper is accepted or rejected (Oct 6, 2023). Papers that are withdrawn during this period will no longer be subject to these rules.

You may not make a non-anonymized version of your paper available online to the general community (for example, via a preprint server) during the anonymity period. Versions of the paper include papers having essentially the same scientific content but possibly differing in minor details (including title and structure) and/or in length.
If you have posted a non-anonymized version of your paper online before the start of the anonymity period, you may submit an anonymized version to the conference. The submitted version must not refer to the non-anonymized version, and you must inform the programme chairs that a non-anonymized version exists.
You may not update the non-anonymized version during the anonymity period, and we ask you not to advertise it on social media or take other actions that would further compromise double-blind reviewing during the anonymity period.
You may make an anonymized version of your paper available (for example, on OpenReview), even during the anonymity period.
For arXiv submissions, May 23, 2023 11:59pm UTC-12h (anywhere on earth) is the latest time the paper can be uploaded.

As reviewing will be double blind, papers must not include authors’ names and affiliations. Furthermore, self-references or links (such as github) that reveal the author’s identity, e.g., “We previously showed (Smith, 1991) …” must be avoided. Instead, use citations such as “Smith previously showed (Smith, 1991) …” Papers that do not conform to these requirements will be rejected without review. Papers should not refer, for further detail, to documents that are not available to the reviewers. For example, do not omit or redact important citation information to preserve anonymity. Instead, use third person or named reference to this work, as described above (“Smith showed” rather than “we showed”). If important citations are not available to reviewers (e.g., awaiting publication), these paper/s should be anonymised and included in the appendix. They can then be referenced from the submission without compromising anonymity. Papers may be accompanied by a resource (software and/or data) described in the paper, but these resources should also be anonymized.

Reproducibility Criteria

Reviewers will be asked to assess the reproducibility of the work as part of their reviews. The following are the criteria that reviews will take under consideration.

For all reported experimental results:

A clear description of the mathematical setting, algorithm, and/or model
Submission of a zip file containing source code, with specification of all dependencies, including external libraries, or a link to such resources (while still anonymized) Description of computing infrastructure used
The average runtime for each model or algorithm (e.g., training, inference, etc.), or estimated energy cost
Number of parameters in each model
Corresponding validation performance for each reported test result
Explanation of evaluation metrics used, with links to code

For all experiments with hyperparameter search:

The exact number of training and evaluation runs
Bounds for each hyperparameter
Hyperparameter configurations for best-performing models
Number of hyperparameter search trials
The method of choosing hyperparameter values (e.g., uniform sampling, manual tuning, etc.) and the criterion used to select among them (e.g., accuracy)
Summary statistics of the results (e.g., mean, variance, error bars, etc.)

For all datasets used:

Relevant details such as languages, and number of examples and label distributions
Details of train/validation/test splits
Explanation of any data that were excluded, and all pre-processing steps
A zip file containing data or link to a downloadable version of the data
For new data collected, a complete description of the data collection process, such as instructions to annotators and methods for quality control.

This list is based on Dodge et al, 2019 and Joelle Pineau’s reproducibility checklist.

Presentation Requirement

All accepted papers must be presented at the conference—either on-line or in-person—in order to appear in the proceedings. Authors of papers accepted for presentation at EMNLP 2023 must notify the program chairs by the camera-ready deadline if they wish to withdraw the paper.

At least one author of each accepted paper must register for EMNLP 2023 by the early registration deadline.

More information can be found in the Committee blog. If you have questions that are not answered there, please email the program co-chairs at emnlp-2023-pc@googlegroups.com