Communications Chairs 2025

About Communications Chairs 2025

Posts by Alex Lu:

March 10 2025

Self-nomination for reviewing at NeurIPS 2025

Communications Chairs 2025 2025 Conference

Authors:
Program Chairs: Razvan Pascanu, Nancy Chen, Marzyeh Ghassemi, Piotr Koniusz, Hsuan-Tien Lin;
Assistant Program Chairs: Elena Burceanu, Junhao Dong, Zhengyuan Liu, Po-Yi Lu, Isha Puri

NeurIPS relies on the active participation of the research community to evaluate numerous submissions and uphold scientific quality. Being involved in the reviewing process provides an opportunity to engage with the NeurIPS community, while contributing to the advancement of research excellence. Starting this year, we are aiming to enhance transparency and fairness of the NeurIPS reviewing process by introducing a self-nomination system, in addition to the usual reviewer invitation procedures. The main track and the D&B track decided to recruit reviewers and area chairs together this year, and the selected ones will be allocated to either the main track or the D&B track afterwards.

Criteria

We encourage anyone who wants to be part of the NeurIPS community and believes has the experience to act as a reviewer or area chair to apply. Good research cannot happen without fair and careful review, and being a reviewer or an area chair is an integral part of being a researcher and will help you hone vital skills that will help you in your career.

To provide guidance of whether you are ready to be a reviewer or an area chair, we provide a few criteria. Note that you should not take these criteria in a very strict sense, and you do not need to meet all of them, but rather a subset of them to qualify.

Also, as our goal is to increase fairness, we understand that individuals may come from different contexts, and due to the large diversity of backgrounds in the field, many might qualify as strong reviewers without fitting the criteria below. In that sense, we provide a free-form text box that allows self-nominations to justify their readiness to be a reviewer if their background does not fit the criteria below.

As this initiative is intended to extend the list of reviewers from previous years, depending on the volume of self-nomination applications and the needs of the conference, we may invite only a random subset of eligible candidates. So please do not treat not being selected as not being qualified for the role you applied for and do consider re-applying in future years, as the pool of reviewers changes from year to year.

If you have accepted an invitation to be a reviewer, but want to self-nominate as an AC, please note that in the form you should indicate that you previously accepted being a reviewer. This will allow us to remove you from the double role in case you are selected. Please understand that the most crucial role in the whole reviewing pipeline is that of the reviewer, and is where NeurIPS needs most support. So in case you are not selected as an AC, please consider staying as a reviewer providing your expertise in judging submitted works.

Self-nominating reviewers

For each self-nomination application for reviewing at NeurIPS 2025, we will consider the following criteria as relevant. You do not need to meet all of them, but should ideally meet several of them to qualify.

Hold a Ph.D. in a relevant field
Published at least 2 papers as first author in the past 5 years in any conference/journal from the following list: REVIEWER_VENUE_LIST
Received at least 10 citations on your first-authored papers (in relevant fields)
Served as a reviewer at any conference/journal in REVIEWER_VENUE_LIST, for at least 2 times in a row
Be an author on at least 4 peer-reviewed papers (this can include non-archival venues like workshops) on relevant topics

REVIEWER_VENUE_LIST: NeurIPS, ICLR, ICML, CVPR, ICCV/ECCV, ACL, EMNLP, WACV, COLM, CoLLAs, RLC, AISTATS, KDD, IJCAI, AAAI, COLT, ML4H, CHIL, TMLR, JMLR

If you are willing to self-nominate to serve as a reviewer for NeurIPS 2025, please fill in this form.

Self-nominating ACs

For each self-nomination application for being an AC at NeurIPS 2025, we will consider the following criteria as relevant. You do not need to meet all of them, but should ideally meet several of them to qualify.

At least 5 last-authored papers accepted at AC_CONF_LIST in the past 5 years
At least 10 papers accepted at AC_CONF_LIST in the area of expertise
Served as an AC at any conference in the AC_CONF_LIST at least 2 times in a row
Served as a reviewer at AC_CONF_LIST at least 5 times in a row

AC_CONF_LIST: NeurIPS, ICLR, ICML, CVPR, ICCV/ECCV, ACL, EMNLP, KDD, COLT

If you are willing to self-nominate to serve as an AC for NeurIPS 2025, please fill in this form.

March 10 2025

NeurIPS Datasets & Benchmarks: Raising the Bar for Dataset Submissions

Communications Chairs 2025 2025 Conference

Authors:
DB Track chairs: Lora Aroyo, Francesco Locatello, Konstantina Palla,
DB Track resource and metadata chairs: Meg Risdal, Joaquin Vanschoren

The NeurIPS Datasets & Benchmarks Track exists to highlight the crucial role that high-quality datasets and benchmarks play in advancing machine learning research. While algorithmic innovation often takes center stage, the progress of AI depends just as much on the quality, accessibility, and rigor of the datasets that fuel these models. Our goal is to ensure that impactful dataset contributions receive the recognition and scrutiny they deserve.

This blog post accompanies the release of the Call for Papers for the 2025 Datasets & Benchmarks Track (https://neurips.cc/Conferences/2025/CallForDatasetsBenchmarks), outlining key updates to submission requirements and best practices. Please note that this year Datasets & Benchmarks Track this year will follow the NeurIPS2025 Main Track Call for Papers, with the addition of three track-specific points: (1) Single-blind submissions, (2) Required dataset and benchmark code submission and (3) Specific scope for datasets and benchmarks paper submission. See the Call for Papers for details.

The Challenge of Assessing High-Quality Datasets

Unlike traditional research papers that have well-established peer review standards, dataset and benchmark papers require unique considerations on their review evaluation. A high-quality dataset must be well-documented, reproducible, and accessible while adhering to best practices in data collection and ethical considerations. Without clear guidelines and automated validation, reviewers face inconsistencies in their assessments, and valuable contributions risk being overlooked.

To address these challenges, we have developed submission and review guidelines that align with widely recognized frameworks in the research community and the open-source movement. For instance, in 2024, we encouraged authors to use established documentation standards such as datasheets for datasets, dataset nutrition labels, data statements for NLP, data cards, and accountability frameworks. By promoting these frameworks, we aim to ensure that dataset contributions are well-documented and transparent, making it easier for researchers to assess their reliability and impact.

Raising the Bar: Machine-Readable Metadata with Croissant

A persistent challenge has been the lack of a standardized, reliable way for reviewers to assess datasets against industry best practices. Unlike the main track, which has commonly accepted standards for paper submissions, dataset reviews still have to mature in this respect.

In 2024, we took a significant step toward improving dataset review by encouraging authors to generate a Croissant machine-readable metadata file to document their datasets. Croissant is an open community effort created because existing standards for dataset metadata lack ML-specific support and lag behind AI’s dynamically evolving requirements. Croissant records ML-specific metadata that enables datasets to be loaded directly into ML frameworks and tools, streamlines usage and community sharing independent of hosting platforms, and includes responsible AI metadata. At that time, Croissant tooling was still in its early stages, and many authors found the process burdensome. Since then, Croissant has matured significantly and gained industry and community adoption. Platforms like Hugging Face, Kaggle, OpenML, and Dataverse now natively support Croissant, making metadata generation effortless.

Making High-Quality Dataset Submissions the Standard

With these improvements in tooling and ecosystem support, we are now requiring dataset authors to ensure that their datasets are properly hosted. That means that they release their datasets via a data repository (e.g., Hugging Face, Kaggle, OpenML, or Dataverse) or provide a custom hosting solution that supports and ensures long-term access and includes a croissant description. We also provide detailed guidelines for authors to make the process as smooth as possible. This requirement ensures that:

Datasets are easily accessible and discoverable through widely used research platforms over long periods of time.
Standard interfaces (e.g., via Python client libraries) simplify dataset retrieval for both researchers and reviewers.
Metadata is automatically validated to streamline the review process.

By enforcing this requirement, we are lowering the barriers to high-quality dataset documentation while improving the overall transparency and reproducibility of dataset contributions.

Looking Ahead

The NeurIPS Datasets & Benchmarks Track is committed to evolving alongside the broader research community. By integrating best practices and leveraging industry standards like Croissant, we aim to enhance the visibility, impact, and reliability of dataset contributions. These changes will help ensure that machine learning research is built on a foundation of well-documented, high-quality datasets that drive meaningful progress.

If you are preparing a dataset submission for NeurIPS, we encourage you to explore Croissant-integrated repositories today and take advantage of the powerful tools available to streamline your metadata generation. Let’s work together to set a new standard for dataset contributions

December 12 2024

NeurIPS 2024 Experiment on Improving the Paper-Reviewer Assignment

Communications Chairs 2025 2024 Conference

By Yixuan Even Xu, Fei Fang, Jakub Tomczak, Cheng Zhang, Zhenyu Sherry Xue, Ulrich Paquet, Danielle Belgrave

Overview

Paper assignment is crucial in conference peer review as we need to ensure that papers receive high-quality reviews and reviewers are assigned papers that they are willing and able to review. Moreover, it is essential that a paper matching process mitigates potential malicious behavior. The default paper assignment approach used in previous years of NeurIPS is to find a deterministic maximum-quality assignment using linear programming. This year, for NeurIPS 2024, as a collaborative effort between the organizing committee and researchers from Carnegie Mellon University, we experimented with a new assignment algorithm [1] that introduces randomness to improve robustness against potential malicious behavior, as well as enhance reviewer diversity and anonymity, while maintaining most of the assignment quality.

TLDR

How did the algorithm do compared to the default assignment algorithm? We compare the randomized assignment calculated by the new algorithm to that calculated from the default algorithm. We measure various randomness metrics [1], including maximum assignment probability among all paper-reviewer pairs, the average of maximum assignment probability to any reviewer per paper, L2 norm, entropy, and support size, i.e., the number of paper-reviewer pairs that could be assigned with non-zero probability. As expected, the randomized algorithm was able to introduce a good amount of randomness while ensuring the overall assignment quality is $98 %$ of the default assignment. We show the computed metrics in the table below. Here, ↑ indicates that a higher value is better, and ↓ indicates that a lower value is better.

	Default	New
Quality (↑)	100%	98%
Max Probability (↓)	1.0	0.90
Average Maxprob (↓)	1.0	0.86
L2 Norm (↓)	250.83	199.50
Entropy (↑)	0	40,678.48
Support Size (↑)	62,916	191,266

Also, one key takeaway from our analysis is that it is important for all the reviewers to complete their OpenReview profile and bid actively to get high-quality assignments. In fact, among all reviewers who bid “High” or “Very High” for at least one paper, $95.2 %$ of them got assigned a paper that they bid “High” or “Very High” on.

In the rest of this post, we introduce the details of the algorithm, explain how we implemented it, and analyze the deployed assignment for NeurIPS 2024.

The Algorithm

The assignment algorithm we used is Perturbed Maximization (PM) [1], a work published at NeurIPS 2023. To introduce the algorithm, we first briefly review the problem setting of paper assignment in peer review as well as the default algorithm used in previous years.

Problem Setting and Default Algorithm

In a standard paper assignment setting, a set $P$ of $n_{p}$ papers needs to be assigned to a set $R$ of $n_{r}$ reviewers. To ensure each paper receives enough reviewers and no reviewer is overloaded with papers, each paper $p$ in $P$ should be assigned to $ℓ_{p}$ reviewers and each reviewer $r$ in $R$ should receive no more than $ℓ_{r}$ papers. An assignment is represented as a binary matrix $x$ in ${0, 1}^{n_{p} \times n_{r}}$ , where $x_{p, r} = 1$ indicates that paper $p$ is assigned to reviewer $r$ . The main objective of paper assignment is usually to maximize the predicted match quality between reviewers and papers [2]. To characterize the matching quality, a similarity matrix $S$ in $R_{\geq 0}^{n_{p} \times n_{r}}$ is commonly used [2-8]. Here, $S_{p, r}$ represents the predicted quality of a review by reviewer $r$ for paper $p$ and is generally computed from various sources [9], e.g., reviewer-selected bids and textual similarity between the paper and the reviewer’s past work [2, 10-13]. Then, the quality of an assignment can be defined as the total similarity of all assigned paper-reviewer pairs, i.e., $Quality (x) = \sum_{p, r} x_{p, r} S_{p, r} .$ One standard approach for computing a paper assignment is to maximize quality [2, 5-8, 14], i.e., to solve the following optimization problem:

$\begin{array}{lll} Maximize & Quality (x) = \sum_{p, r} x_{p, r} S_{p, r} \\ Subject to & \sum_{r} x_{p, r} = ℓ_{p} & \forall p \in P, \\ \sum_{p} x_{p, r} \leq ℓ_{r} & \forall r \in R, \\ x_{p, r} \in {0, 1} & \forall p \in P, r \in R . \end{array}$

The optimization above can be solved efficiently by linear programming and is widely used in practice. In fact, the default automatic assignment algorithm used in OpenReview is also based on this linear programming formulation and has been used for NeurIPS in past years.

Perturbed Maximization

While the deterministic maximum-quality assignment is the most common, there are strong reasons [1] to introduce randomness into paper assignment, i.e., to determine a probability distribution over feasible deterministic assignments and sample one assignment from the distribution. For example, one important reason is that randomization can help mitigate potential malicious behavior in the paper assignment process. Several computer science conferences have uncovered “collusion rings” of reviewers and authors [15-16], in which reviewers aim to get assigned to the authors’ papers in order to give them good reviews without considering their merits. Randomization can help break such collusion rings by making it harder for the colluding reviewers to get assigned to the papers they want. The randomness will also naturally increase reviewer diversity and enhance reviewer anonymity.

Perturbed Maximization (PM) [1] is a simple and effective algorithm that introduces randomness into paper assignment. Mathematically, PM solves a perturbed version of the optimization problem above, parameterized by a number $Q \in [0, 1]$ and a perturbation function $f : [0, 1] \to [0, 1]$ :

In this perturbed optimization, the variables $x_{p, r}$ are no longer binary but continuous in $[0, Q]$ . This is because we changed the meaning of $x_{p, r}$ in the randomized assignment context: $x_{p, r}$ now represents the marginal probability that paper $p$ is assigned to reviewer $r$ . By constraining $x_{p, r}$ to be in $[0, Q]$ , we ensure that each paper-reviewer pair has a probability of at most $Q$ to be assigned to each other. This constraint is adopted from an earlier work on randomized paper assignment [17]. The perturbation function $f$ is a concave function that is used to penalize high values of $x_{p, r}$ , so that the probability mass is spread more evenly among the paper-reviewer pairs. The perturbation function can be chosen in various ways, and one simple option is $f (x) = x - α x^{2}$ , which makes the optimization concave and quadratic, allowing us to solve it efficiently.

After solving the optimization problem above, we obtain a probabilistic assignment matrix $x$ . To get the final assignment, we then sample according to the method in [17-18] to meet the marginal probabilities $x$ . The method is based on the Birkhoff-von Neumann theorem.

Implementation

Since this is the first time we use a randomized algorithm for paper assignment at NeurIPS, the organizing committee decided to set the parameters so that the produced assignment is close in quality to the maximum-quality assignment, while introducing a moderate amount of randomness. Moreover, we introduced additional constraints to ensure that the randomization does not result in many low-quality assignments.

Similarity Computation of NeurIPS 2024

The similarity matrix $S$ of NeurIPS 2024 was computed from two sources: affinity scores and bids. The affinity scores were computed using a text similarity model comparing the paper’s text with the reviewer’s past work on OpenReview. The resulting affinity scores were normalized to be within $[0, 1]$ . The bids were collected from reviewers during the bidding phase, where reviewers could bid on papers they are interested in reviewing at five levels: “Very High”, “High”, “Neutral”, “Low”, and “Very Low”. We mapped these bids to $(- 1, - 0.5, 0, 0.5, 1)$ respectively. The final similarity matrix was computed as the sum of the normalized affinity scores and the mapped bids, resulting in a similarity matrix consisting of numbers in $[- 1, 2]$ .

Additional Constraints for Restricting Low-Quality Assignments

One of the main concerns of paper assignment in large conferences like NeurIPS is the occurrence of low-quality assignments because the matching quality of any individual paper-reviewer pair significantly affects the relevance of both papers and reviewers. To mitigate this issue, we explicitly restrict the number of low-quality assignments. Specifically, we first solve another optimization problem without the perturbation function [17]:

Let the optimal solution of this problem be $x^{*}$ . We want to ensure that adding the perturbation function does not introduce additional low-quality assignments compared to $x^{*}$ . To achieve this, we set a set of thresholds $τ \in {0.1, 0.5, 1.0}$ . For each $τ$ , we add constraints that our perturbed assignment should have at least the same number of assignments with quality above $τ$ as $x^{*}$ , i.e.,

$\begin{array}{lll} Maximize & PQuality (x) = \sum_{p, r} f (x_{p, r}) S_{p, r} \\ Subject to & \sum_{r} x_{p, r} = ℓ_{p} & \forall p \in P, \\ \sum_{p} x_{p, r} \leq ℓ_{r} & \forall r \in R, \\ x_{p, r} \in [0, Q] & \forall p \in P, r \in R, \\ \sum_{p, r} x_{p, r} \cdot I [S_{p, r} \geq τ] \geq \sum_{p, r} x_{p, r}^{*} \cdot I [S_{p, r} \geq τ] & \forall τ \in {0.1, 0.5, 1.0} . \end{array}$

The thresholds ${0.1, 0.5, 1.0}$ were chosen to distinguish between different levels of quality. According to the similarity computation for NeurIPS 2024, matchings with quality above $1.0$ are “good” ones that either the reviewer has a high affinity score with the paper and bids positively on it, or the reviewer bids “Very High” on the paper; matchings with quality above $0.5$ are “moderate” ones that either the reviewer has a high affinity score with the paper or the reviewer bids positively on it; matchings with quality above $0.1$ are “acceptable” ones that the reviewer has a moderate affinity score with the paper and bids neutrally on it. By setting these thresholds, we limited the number of low-quality assignments introduced by the perturbation function.

Running the Algorithm

We integrated the Python implementation of PM into the OpenReview system, using Gurobi [19] as the solver for the concave optimization. However, since the number of papers and reviewers in NeurIPS 2024 is too large, we could not directly use OpenReview’s computing resources to solve the assignment in early 2024. Instead, we ran the algorithm on a local server with anonymized data. The assignment was then uploaded to OpenReview for further processing, such as manual adjustments by the program committee. We ran four different parameter settings of PM and sampled three assignments from each setting. Each parameter setting took around 4 hours to run on a server with 112 cores, using peak memory of around 350GB. The final assignment was chosen by the program committee based on the computed statistics of the assignments. The final deployed assignment came from the parameter setting where $Q = 0.9$ and $α = 0.1$ . The maximum number of papers each reviewer can review was set to $6$ .

Analysis of The Deployed Assignment

How did the assignment turn out? We analyzed various statistics of the assignment, including the aggregate scores, affinity scores, reviewer bids, reviewer load, and reviewer confidence in the review process. We also compared the statistics across different subject areas. Here are the results.

Aggregate Scores

The deployed assignment achieved an average aggregate score of $1.50$ and a median of $1.72$ . Recall from the computation of the similarity matrix, this means the majority of the assignments are of very high quality, with high affinity scores and “Very High” bids. Additionally, we note that every single matched paper-reviewer pair has an aggregate score above $0.5$ , which means that each assigned pair is at least a “moderate” match. In addition, we see no statistical difference in the aggregate score across different subject areas, despite the varying sizes of different areas.

aggregate_scores

papers_by_subject_area

aggregate_scores_by_subject_area

Affinity Scores

Since the aggregate score is the sum of the affinity score based on text similarity and the converted reviewer bids, we also checked the distribution of these two key scores. The deployed assignment achieved high affinity scores, with an average of $0.72$ and a median of $0.77$ . Note that there are also some matched pairs with zero affinity scores. These pairs are matched because the reviewers bid “Very High” on the papers, which results in an aggregate score of $1.0$ . Therefore, we still prioritize these pairs over those with positive affinity scores but neutral or negative bids.

affinity_scores

affinity_scores_by_subject_area

Reviewer Bids

For reviewer bids, we see that most of the assigned pairs have “Very High” bids from the reviewers, with the majority of the rest having “High” bids. Moreover, not a single pair has a negative bid. This indicates that reviewers are generally interested in the papers they are assigned to. Note that although we default missing bids to “Neutral”, the number of matched pairs with “Missing” bids is larger than that of pairs with “Neutral” bids. This is because if a reviewer submitted their bids, they are most likely assigned to the papers they bid positively on. The matched pairs with “Missing” bids are usually those where reviewers did not submit their bids, and the assignment for them was purely based on the affinity scores.

bid_distribution

bid_distribution_by_subject_area

Reviewer Load

If we distribute the reviewer load evenly, reviewers should be assigned to an average of $4.42$ papers. However, as the assignment algorithm aims for high-quality assignments, the majority of reviewers were assigned to $6$ papers, the limit we set for reviewers.

reviewer_load_distribution

Nevertheless, some reviewers in the pool are not assigned to any papers or are assigned to only one paper. After analyzing the data more carefully, we found that most of these reviewers either had no known affinity scores with the papers (mostly because they did not have any past work on OpenReview) or did not submit their bids. Moreover, there are even $50$ reviewers who had neither affinity scores nor bids. Therefore, it is hard for the algorithm to find good matches for them.

We suggest that reviewers submit their bids and provide more information about their past work to help the algorithm find better matches for them.

While the reviewer load distribution for each subject area generally follows the overall distribution, we note that some subject areas, like Bandits, have a notably higher number of papers assigned to each reviewer. In fact, most reviewers in the Bandits area were assigned to $6$ papers. This indicates that for these areas, we will need to work harder to recruit more reviewers in future conferences.

Reviewer Confidence

In the review process, reviewers were asked to provide their confidence in their reviews on a scale from $1$ to $5$ . The distribution of reviewer confidence is shown below. Here, a confidence of $- 1$ means that the matched pair was adjusted manually by the area chairs, and a confidence of $0$ means that the reviewer did not submit their review. We can see that among the pairs where the reviewer completed the review, most matched pairs have a confidence of $3$ or higher. This indicates that reviewers are generally confident in their reviews.

On a side note, we found that reviewer confidence is generally lower for theoretical areas like Algorithmic Game Theory, Bandits, Causal Inference, and Learning Theory, while it is higher for other areas. It is hard to explain this phenomenon exactly, but we think this might be because the difficulty of reviewing papers in theoretical areas is generally higher, leading reviewers to be more cautious in their reviews.

Comparison with the Default Algorithm

Besides analyzing the deployed assignment, it is also natural to ask how the new algorithm PM compares to the default algorithm used in OpenReview. To answer this question, we ran the default algorithm on the same data and compared the resulting assignment with the deployed assignment. Below, we show the comparison with the default algorithm in aggregate scores, reviewer bids, and reviewer load.