2024 Conference

by Adil Salim, Bo Han, Manuel Gomez Rodriguez, and Rose Yu

We are excited to announce the list of NeurIPS 2024 workshops! We received 204 total submissions — a significant increase from last year. From this great batch of submissions, we have accepted 56 workshops that will take place on Dec. 14 & 15. 

Given the exceptional quality of submissions this year, we wish we could have accepted many more, but we could not due to logistical constraints. We want to thank everyone who put in tremendous effort in submitting a workshop proposal.

Review Process

We continue to use OpenReview as our submission platform this year, which is aligned with other NeurIPS submission tracks due to the success of OpenReview in matching reviewers well to proposals. Additional details about the selection process are provided below.

The requested items for the proposals have not changed much this year. We kept the length of the main proposal limited to three pages and the organizer information limited to two pages, along with unlimited references. We specifically let reviewers know that they need not read beyond those pages.

With respect to last year, we have further increased the reviewer pool. We sent out over 418 invitations and managed to recruit 189 reviewers. This resulted in at least two reviews per proposal for (almost) all 204 proposals. We thank all the reviewers for their timely and professional efforts to provide quality reviews that greatly assisted our decision-making and facilitated an exciting and well-informed workshop program this year.

Selection Process

In making our selections, we asked the reviewers to closely follow our Guidance for Workshop Proposals, which was also shared with the proposal authors. Workshop proposals must be reviewed somewhat differently from academic papers, and hence we asked the reviewers to consider both scientific merits and broader impacts in their assessments. We recognize that workshop reviews might be somewhat more subjective than academic paper reviews. To offer feedback to the proposal authors, we have decided to release the review comments.

Individual evaluations of proposals by reviewers were important for the decision process, but they were not the only considerations in the decision process. For example, we also strived for a good balance between research areas, and between applications and theory. As interest across a variety of  research areas is not uniform, some areas were more competitive than others. For example, there were many strong proposals surrounding large language models this year and we could not accept all. We attempted some balance of topics to cover both mainstream and emerging topics.

The next step is your contributions! Several workshops have begun soliciting submissions, many using our suggested submission date of Aug 30, 2024. We typically let each workshop advertise its own call for papers (if they plan to include workshop papers). We will communicate with the workshop organizers some additional deadlines to facilitate the successful planning of 56 exciting workshops. Stay tuned for more technical and contextual information coming soon!

NeurIPS 2024 Accepted Workshops

On to the best part: the preliminary list of accepted workshops for 2024!

• Intrinsically Motivated Open-ended Learning (IMOL)
• ML with New Compute Paradigms
• Federated Foundation Models
• Foundation Model Interventions
• Bayesian Decision-making and Uncertainty: from probabilistic and spatiotemporal modeling to sequential experiment design
• Open-World Agents: Synergizing Reasoning and Decision-Making in Open and Interactive Environments
• Audio Imagination: AI-Driven Speech, Music, and Sound Generation
• Foundation Models for Science: Progress, Opportunities, and Challenges
• AI for New Drug Modalities
• Statistical Frontiers in LLMs and Foundation Models
• Machine Learning in Structural Biology
• Table Representation and Generative Learning 
• Data-driven and Differentiable Simulations, Surrogates, and Solvers
• Mathematical Reasoning and AI
• Red Teaming GenAI: What Can We Learn from Adversaries?
• Advancements In Medical Foundation Models: Explainability, Robustness, Security, and Beyond
• Causality and Large Models
• Large Foundation Models for Educational Assessment
• Machine Learning and Compression
• Machine Learning for Systems
• Scientific Methods for Understanding Neural Networks: Discovering, Validating, and Falsifying Theories of Deep Learning with Experiments
• Pluralistic Alignment
• Responsibly Building Next Generation of Multimodal Foundation Models
• Touch Processing: From Data to Knowledge
• GenAI for Health: Potential, Trust and Policy Compliance
• Symmetry and Geometry in Neural Representations
• Mathematics of Modern Machine Learning
• Tackling Climate Change with Machine Learning: Questioning Common ML Assumptions in the Context of Climate Impact
• Attributing Model Behavior at Scale
• Compositional Learning: Perspectives, Methods, and Paths Forward
• Time Series in the Age of Large Models
• Efficient Natural Language and Speech Processing: Highlighting New Architectures for Future Foundation Models
• Behavioral Machine Learning
• Interpretable AI: Past, Present and Future
• UniReps: Unifying Representations in Neural Models
• Regulatable ML: Towards Bridging the Gaps between Machine Learning Research and Regulations
• Evaluating Evaluations: Examining Best Practices for Measuring Broader Impacts of Generative AI
• Multimodal Algorithmic Reasoning
• Self-Supervised Learning: Theory and Practice
• Language Gamification
• Fine-Tuning in Modern Machine Learning: Principles and Scalability
• Safe Generative AI
• Adaptive Foundation Models: Evolving AI for Personalized and Efficient Learning
• Optimization for Machine Learning
• New Frontiers in Adversarial Machine Learning
• Towards Safe & Trustworthy Agents
• Algorithmic Fairness through the lens of Metrics and Evaluation
• Machine Learning and the Physical Sciences
• Socially Responsible Language Modelling Research
• AI for Accelerated Materials Design
• System-2 Reasoning at Scale
• Causal Representation Learning
• Scalable Continual Learning for Lifelong Foundation Models
• NeuroAI: Fusing Neuroscience and AI for Intelligent Solutions
• Video-Language Models
• Generative AI and Creativity: A dialogue between machine learning researchers and creative professionals

Related Links

• Call for Post-Conference Workshops
• Guidance for Workshop Proposals

We are excited to announce a study to understand if Large Language Models (LLMs) can serve as an assistant to help authors verify their submission against the NeurIPS Paper Checklist. This study is a first step towards understanding if LLMs can be used to enhance the quality of submissions at NeurIPS. We invite authors intending to submit a paper to NeurIPS to participate in this study by pre-registering through our online form, before the abstract submission deadline.  Participants will gain access to an experimental LLM assistant (on or shortly before the abstract submission deadline), which will provide feedback on if their paper submission complies with NeurIPS submission standards, and are asked to fill out two short surveys describing their experiences with the LLM assistant. Due to limited computational resources, participation may be limited, and will be offered on a first-come, first-served basis. 

Why This Matters

The NeurIPS Paper Checklist is a series of yes/no questions that help authors check if their work meets reproducibility, transparency, and ethical research standards expected for papers at NeurIPS, and is a critical component in maintaining the high standards of research presented at the conference. By ensuring that submissions adhere to this checklist, we uphold the scientific rigor and transparency that NeurIPS aims for. 

Currently, some authors might already be using LLMs as tools in their research and writing processes. While the wider ethical implications and appropriate use cases of LLMs remain unclear and must be a larger community discussion, here, we evaluate a more clear-cut and low-risk use case: vetting paper submissions against submission standards, with results shown only to the authors. Our aim is to systematically evaluate the benefits and risks of LLMs, by conducting a structured study to understand if LLMs can enhance research quality and improve efficiency by helping authors understand if their work meets research standards like the NeurIPS Paper Checklist. This study will test the effectiveness of LLMs in an academic publishing setting, and potentially influence broader AI applications in research evaluation.

How It Works

The LLM assistant is hosted by ChaLearn on Codabench.org. We invite NeurIPS authors to try out the assistant for their papers, and fill out two brief surveys. 

Registration: For any paper, a single delegated author should register at this link. Due to a limited budget, participation will be offered on a first come first served basis. Registered participants will receive an invitation to the assistant’s page on the Codabench platform, and will need to create an account on Codabench using the email supplied in the registration. 

Submissions: Submissions are entirely optional and completely separate from the NeurIPS paper submission system and the review process. Papers must be formatted as specified in the NeurIPS’24 CFP (complete with appendices and checklist). Information provided in external links will not be taken into account by the assistant. We ask submitters to fill out the checklist to the best of their abilities.

Anonymity and confidentiality: Authors can retain their anonymity by registering to Codabench with an email and login that do not reveal their identity, and by submitting anonymized papers. The papers and LLM outputs will be kept confidential and will not be accessible to NeurIPS reviewers, ACs, and PCs. Only aggregated statistics will be published, after the review process is over. It is important to note that while authors retain ownership of their submissions, the papers will be sent to the API of an LLM service, and treated under their conditions of confidentiality.

The Role of the Assistant

The LLM Assistant will provide detailed feedback on whether the responses to the checklist questions accompanying the paper are satisfactory. This feedback is intended to guide authors in refining their papers before official submission. Please remember that this tool is designed to supplement, not replace, critical judgment and expertise. In particular, it is certainly possible for the review process to raise concerns about the checklist that have not been raised by the LLM reviews.

Here are some examples of feedback (for two different papers). Green means that the LLM assistant has identified no significant concerns. Yellow means that the LLM assistant has identified an inconsistency with NeurIPS Paper Checklist standards, and authors are encouraged to validate the issue identified by the LLM assistant and make any necessary improvements.  

Question # 3: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Author’s answer: No

Author’s justification: There is no theory in this paper.

LLM Review:
The answer provided by the authors is consistent with the content of the paper. Based on the detailed review of the paper, it is clear that the work primarily focuses on conducting an observational study to investigate the presence of citation bias in peer review processes. The study utilizes statistical modeling and analysis rather than theoretical developments that would involve theorems, lemmas, or formal proofs. Therefore, the classification of the answer as "NA" (Not Applicable) is accurate given the nature of the research presented. However, to provide constructive and actionable suggestions for improvement and to ensure thorough and critical assessment, it is beneficial to clearly outline the assumptions behind the statistical models and analyses employed. This would enhance the readability and understandability of the study’s methodology and its limitations. [...]

Question # 12: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Author’s answer: Yes.

Author’s justification: We have used algorithms such as TPMS and PeerReview4All. We have cited them.

LLM Review:
There does not appear to be any discrepancy between the authors' justification for the cited question and the actual paper content in terms of providing proper credit and mentioning licenses for used assets, according to their statement. However, the detailed source citations such as the specific papers for TPMS (Toronto Paper Matching System) and PeerReview4All, including their licenses or terms of use, are not explicitly mentioned in the provided text. It's essential to ensure that these aspects are clearly attributed and any terms of use are respected, as stated in the initial guidelines provided to authors. [...]

Limitations, Disclaimer and Final Thoughts

Despite advances in AI, LLMs still face several limitations when used in verifying scientific submissions. LLMs may misinterpret complex content or fail to fully assess checklist criteria, leading to verification errors. There is also a risk that authors might rely too much on the LLM analysis, potentially missing deeper issues that require human judgment. We encourage authors to manually validate the outputs of LLMs, and to use the tool to supplement their judgment, not replace it. Known issues include false positives (e.g., LLM may ask for extra justifications or changes that are excessive or irrelevant) as well as false negatives (e.g., the LMM overlooks errors in proofs). As of the release of this blog, the LLM answers are based only on the text of the paper and does not check figures, tables, and external links (such as Github repos), hence it may complain about missing information that is found in those assets.

Additionally, using LLMs raises concerns about data privacy and security, as papers are processed through an LLM service API. We provide recommendations to participants on anonymizing their submissions and we are making best efforts to keep all papers in confidence. However, we decline responsibility on possible unintentional leaks of data, due to mishandling or due to the use of a third party LLM. If authors are concerned for the security of their paper, we do not recommend them to participate in the study. 

Limited computational resources mean only some authors can access this service, possibly creating unequal opportunities. Furthermore, inherent biases in LLMs could compromise the fairness and impartiality of the process, especially for unconventional or interdisciplinary research. We mitigate these risks by only providing this tool to authors with the intention of feedback: the outputs of the LLM will not be visible to, or used by reviewers, ACs, and PCs in their review of the submission.

This study is strictly for educational and research purposes. The organizers, sponsors, and NeurIPS assume no liability for any direct or indirect consequences arising from participation. Authors are reminded to use their discretion and adhere to all NeurIPS guidelines when submitting their papers to OpenReview for official review.

Stay Informed and Get Involved

For updates and more information, authors are encouraged to register to participate. 

Credits

The principal organizers of this experiment are: Ihsan Ullah, Alexander Goldberg, Isabelle Guyon, Thanh Gia Hieu Khuong, Benedictus Kent Rachmat, Nihar Shah, and Zhen Xu.

In preparing this experiment, we received advice and help from many people. We are particularly grateful to the NeurIPS’24 organizers, including General Chair Amir Globerson, Program Chairs Danielle Belgrave, Cheng Zhang, Angela Fan, Jakub Tomczak, Ulrich Paquet, and workflow team member Babak Rahmani, for participating in brainstorming discussions and contributing to the design. We have also received inputs and encouragement from Andrew McCallum of OpenReview, Anurag Acharya from Google Scholar, and Tristan Neuman from the NeurIPS board. Several volunteers have contributed ideas and helped with various aspects of the preparation, including Jeremiah Liu, Lisheng Sun, Paulo Henrique Couto, Michael Brenner, Neha Nayak Kennard, and Adrien Pavao.

We are grateful to Marc Schoenauer for supporting this effort with an INRIA Google Research grant. We acknowledge the support of ChaLearn and of the ANR Chair of Artificial Intelligence HUMANIA ANR-19-CHIA-0022.

Key Dates

• Deadline: Jun 14, 2024 AOE
• Final Decisions: Aug 02, 2024

We welcome proposals for in-person tutorials on either core machine learning topics or topics of emerging importance for machine learning, broadly construed. We will consider any topic, so long as the proposal makes a strong argument that the tutorial is important for the NeurIPS community. Tutorials should be of interest to a substantial portion of the community and should represent a sufficiently mature area of research or practice. Tutorials are expected to be accessible to PhD candidates working in other areas of the NeurIPS community, i.e. some part of the tutorial may need to be spent on covering necessary background material.

A NeurIPS tutorial should not focus on the results or tools of the presenters or their organizations, and should be more reflective of the research within the wider community in that field. A tutorial should provide a balanced overview of an area of research. It should also provide a balance between known results and open problems.

We discourage tutorial topics that overlap heavily with tutorials presented at NeurIPS, ICML or highly related conferences in the past two years. We also aim to select tutorial presenters who have not presented a tutorial at NeurIPS or ICML in the past five years.

This year we will require a minimum of two presenters and an optional panel discussion. We also encourage submitters to think about how their tutorial could be useful to the community after the meeting, for example providing links to code, slides and exercises or audio-only presentations. We will send out audio recordings to tutorial presenters after the conference and presenters will be free to modify the recordings or send out additional information for conversion into a podcast format (depending on topic).

We will also set up informal practice groups between presenters before the main conference.

Requirements:

Each accepted tutorial will be 2h30m (150 minutes) long. 

Tutorials will take place entirely in-person on a single day and avoid overlap with the rest of the program. All participants are required to participate live (for Q&A and the panel).

Each tutorial may include a discussion panel to broaden the discussion on the topic. The panel should include additional researchers from the field, and a moderator.  If a panel is planned, it should be taking place in the last 30 minutes of the tutorial. 

Summary of Requirements:

Main Tutorial:

• Min 2 speakers
• Max 3 speakers
• Should have at least 2 hours for presentation (including 10 minutes for technical Q&A and clarifying questions)

Optional Panel/Debate:

• 20-30 minutes at the end of the tutorial.
• Max 6 participants (including the moderator) for the panel.
• The panel should represent diverse perspectives. While the exact structure of the panel is up to you, we encourage a (civil/professional) debate format representing different perspectives in the community.

Requirements for Proposal Document:

Tutorial proposals should be no more than five pages in 12-point font submitted in PDF format. Each proposal should be clearly structured to provide the following information:

• Title
• Abstract (up to 250 words)
• Description and outline: What material will the tutorial cover and in what depth? Please provide a detailed outline.
• Goals: What are the objectives of the tutorial? What is the benefit to attendees? What are the expected learning outcomes (in terms of knowledge, skills and understanding)? 
• Why is this tutorial important to include at this NeurIPS in particular? How will you engage a broad audience? How will the panelists (if applicable) complement your presentation?
• Previous tutorials: What are the related or similar tutorials presented in the past 3 years at NeurIPS or at another venue? Please list the dates and venues, and describe the similarities and differences between the previous tutorials and proposed tutorial. If available, please include URLs for slides and video recordings.
• If applicable, hands-on coding demonstrations and resource websites associated with the proposal are highly valued. For example, can attendees refer to notebooks that would help them reproduce some of the demonstrations presented in your tutorial?
• Presenters (min 2, max 3): Who are the presenters? Please provide names, affiliations, email addresses, and short bios for each presenter. Bios should cover the presenters’ expertise related to the topic of the tutorial. Please describe how the time will be divided between the presenters. All presenters listed in the proposal are expected to attend.
• Optional Panel: Who are the additional panelists? Please provide names, affiliations, email addresses and short bios (in relation to the tutorial’s topic). Please highlight who will moderate the discussion.
• Post-meeting materials: How will this tutorial be useful to attendees after the meeting, e.g. will you provide links to code, slides, exercises? Will this tutorial be suitable for presenting or edited into an audio-only format?
• Diversity: please specify how the speakers and optional panelists represent the diversity in our community, considering aspects such as (non-exhaustive) gender, race, geographical location, institution, seniority, background, specialty. For debate format panels, we ask organizers to try their best to maintain balanced representation along these aspects for differing perspectives. 
• Links to video recordings of the presenters’ previous talks (optional but strongly recommended)
• A list of the most important references that will be covered.

Proposal strengths:

• Importance, relevance and timeliness of the topic to the NeurIPS community
• A diverse set of speakers. We encourage speakers from different institutions with a variety of perspectives to reduce knowledge bias.
• Speakers should be experienced in the topic proposed, as demonstrated by publications and talks.
• Quality of the presentation; clear delivery of the content. 

Please see the retrospective blog on the tutorial submissions from last year, and make an effort to embrace the positive features of accepted tutorials, and avoid the pitfalls outlined there. The best way to maximize the chances of your tutorial submission being accepted is to carefully follow instructions, and make a polished and organized submission. Are you missing parts of your submission? Is it over length? Does your proposal bring diverse perspectives? Did you research whether a closely related topic has recently been presented at NeurIPS or ICML? Are you submitting on time? Are your panellists confirmed at the time of submission? A carefully planned and crafted submission signals that care and organization will go into the presentation itself.

FAQ

• Can the moderator be one of the speakers? Yes, the moderator can be either in addition to the speakers and panellists, or be one of the speakers. We request that moderators prepare a list of discussion topics and questions if the audience is not very active.
• Are panellists included in the “tutorial speakers who have not presented a tutorial in the past five years?” No, this point only refers to the tutorial presenters.
• Can I submit multiple proposals? No, each researcher can submit at most one proposal to be a Speaker. 
• Can researchers be present at multiple panels? No, each researcher can be a part of maximum one panel.
• Can a researcher be a Speaker at one tutorial and panellist at another? While proposals will not be disqualified for this, we will encourage researchers to select one opportunity at most.

Submit

Submit your proposal in this form.

Questions

For other questions, please contact us.

Tutorial Chairs, NeurIPS 2024,
Andrew M. Dai, Gal Chechik, Irene Y Chen

We are glad to announce the eighth edition of the competition track at NeurIPS 2024. Through this call, we solicit competition proposals on topics of interest to the NeurIPS community.

We especially encourage submissions with a clear scientific question and from fields with a positive societal impact promoting causes that use AI to help disadvantaged communities. Besides societal impact, we seek proposals in areas where machine learning can positively influence other scientific, technological, or business domains. We hope that interesting competitions will attract a significant cross-section of interdisciplinary and diverse communities to our NeurIPS 2024 competition track.

Technological advances in our field can have the potential to disproportionately hurt marginalized sectors in society — including people of color, people from working-class backgrounds, women, and LGBTQ people. Hence, we strongly encourage proposals that address these problems, especially from researchers with these identities or who are members of other marginalized or under-represented communities.

Proposals should highlight an inclusiveness plan to promote diverse participation among both the organizing team and competitors. Separate recognition tiers to incentivize broad participation, such as recognizing top submissions from high school students or other participant groups are encouraged.  Organizers are encouraged to collaborate with affinity groups to host on-site bootcamps and other initiatives aimed at promoting diversity of participation.  

NeurIPS 2024 will be hosting a dedicated Competition Track, where results will be presented and discussed by participants and organizers. The Competition Track is currently planned to be held in person at NeurIPS. Similar to last year, we request that the competition analysis paper be submitted to the NeurIPS 2025 D&B track; there will be no separate proceedings. Further details on the organization will be released in the following months.

PROPOSAL SUBMISSION

Competition proposals must both be submitted via OpenReview at:
https://openreview.net/group?id=NeurIPS.cc/2024/Competition_Track

Note: Newly created profiles in OpenReview can take up to two weeks for approval, please plan accordingly

Please carefully follow the provided Latex template when preparing proposals.

Note that the guidelines and template have been updated from last year. 

To reduce the reviewers’ workload, we will accept only 8 page submissions.

REVIEWING AND SELECTION PROCESS

A competition program committee will be formed, consisting of experts on both machine learning and challenges organization. Each proposal will be reviewed by members of the program committee. The factors that will be considered when evaluating proposals include:

• Scientific relevance and Task(s): competitions should be motivated by clear scientific questions and curiosity. Impact, originality, and relevance to the NeurIPS community will all be considered. Tasks that include humanitarian and/or positive societal impact are highly encouraged, although other topics relevant to the NeurIPS community are also welcomed. If humanitarian projects are submitted, the involvement of the community in question is strongly desired, and “parachute science” is strongly discouraged.

• Evaluation Protocol: Feasibility of the task chosen, baseline availability, sufficient data for training and testing algorithms to solve the proposed task, soundness of the evaluation criteria, and clarity and fairness of the competition rules will all be evaluated.

• Logistics: The competition schedule, plan for attracting competition participants, and experience and diversity of the organizers will all be considered. The specific plan for attracting competition participants, including groups under-represented at NeurIPS, will be important during the review process.

PROCEEDINGS

This year, there are new publication requirements for competition reports co-authored by both organizers and participants. Similar to last year, accepted competitions in 2024 will be required to submit their post-competition analyses as papers to the 2025 NeurIPS D&B track (next year). In order to ensure that the competition proposal and results are consistent, reviewers in 2025 will have access to the 2024 proposals. To minimize experimental bias, any deviations from the proposal will need to be justified.

IMPORTANT DATES (anywhere on earth)

• Competition proposal submission deadline: March 27, 2024
• Acceptance notification: April 17, 2024
• 2023 Competition track: December 9-14, 2024

ADDITIONAL COMMENTS

Competition organizers should propose a timeline for running the competition to ensure participants have enough time to contribute high-quality entries. It is recommended that competitions be launched in May and completed by the end of October 2024 at the absolute latest.

Competition organizers that require help or suggestions regarding competition platforms for running the competition can contact the competition chairs for advice.

2023 accepted competitions
2022 accepted competitions
2021 accepted competitions
2020 accepted competitions
2019 accepted competitions
2018 accepted competitions
2017 accepted competitions

COMPETITION CHAIRS

Jake Albrecht, Sage Bionetworks

Tao Qin, Microsoft Research

Megan Yates, Zindi

Contact:  competition-chairs@neurips.cc