Announcing the NeurIPS 2024 Best Paper Awards 

By Marco Cuturi, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub Tomczak, Cheng Zhang,  Lora Aroyo, Francesco Locatello, Lingjuan Lyu

The search committees for the “Best Paper Award” were nominated by the program chairs and the respective track chairs, who selected leading researchers with a diverse perspective on machine learning topics. These nominations were approved by the general and DIA chairs.

The best paper award committees were tasked with selecting a handful of highly impactful papers from both tracks of the conference. The search committees considered all accepted NeurIPS papers equally, and made decisions independently based on the scientific merit of the papers, without making separate considerations on authorship or other factors, in keeping with the Neurips blind review process.

With that, we are excited to share the news that the best and runner up paper awards this year go to five ground-breaking papers (four main track and one datasets and benchmarks track) that highlight, respectively, a new autoregressive model for vision, new avenues for supervised learning using higher-order derivatives, improved training of LLMs and inference methods for text2image diffusion and a novel diverse benchmark dataset for LLM alignment.

Best papers for the main track:

Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction 

This paper introduces a novel visual autoregressive (VAR) model that iteratively predicts the image at a next higher resolution, rather than a different patch in the image following an arbitrary ordering. The VAR model shows strong results in image generation, outperforming existing autoregressive models in efficiency and achieving competitive results with diffusion-based methods. At the core of this contribution lies an innovative multiscale VQ-VAE implementation. The overall quality of the paper presentation, experimental validation and insights (scaling laws) give compelling reasons to experiment with this model.

Stochastic Taylor Derivative Estimator: Efficient amortization for arbitrary differential operators

This paper proposes a tractable approach to train neural networks (NN) using supervision that incorporates higher-order derivatives. Such problems arise when training physics-informed NN to fit certain PDEs. Naive application of automatic differentiation rules are both inefficient and intractable in practice for higher orders k and high dimensions d. While these costs can be mitigated independently (e.g. for large k but small d, or large d but small k using subsampling) this paper proposes a method, stochastic taylor derivative estimator (STDE) that can address both. This work opens up possibilities in scientific applications of NN and more generally in supervised training of NN using higher-order derivatives.

Runners ups for the main track:

Not All Tokens Are What You Need for Pretraining

This paper presents a simple method to filter pre-training data when training large language models (LLM). The method builds on the availability of a high-quality reference dataset on which a reference language model is trained. That model is then used to assign a quality score for tokens that come from a larger pre-training corpus. Tokens whose scores have the highest rank are then used to guide the final LLM training, while the others are discarded. This ensures that the final LLM is trained on a higher quality dataset that is well aligned with the reference dataset.

Guiding a Diffusion Model with a Bad Version of Itself

This paper proposes an alternative to classifier free guidance (CFG) in the context of text-2-image (T2I) models. CFG is a guidance technique (a correction in diffusion trajectories) that is extensively used by practitioners to obtain better prompt alignment and higher-quality images. However, because CFG uses an unconditional term that is independent from the text prompt, CFG has been empirically observed to reduce diversity of image generation. The paper proposes to replace CFG by Autoguidance, which uses a noisier, less well-trained T2I diffusion model. This change leads to notable improvements in diversity and image quality.

Best paper for Datasets & Benchmarks track:

The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models

Alignment of LLMs with human feedback is one of the most impactful research areas of today, with key challenges such as confounding by different preferences, values, or beliefs. This paper introduces the PRISM dataset providing a unique perspective on human interactions with LLMs. The authors collected data from 75 countries with diverse demographics and sourced both subjective and multicultural perspectives benchmarking over 20 current state of the art models. The paper has high societal value and enables research on pluralism and disagreements in RLHF. 

Best Paper Award committee for main track: Marco Cuturi (Committee Lead), Zeynep Akata,  Kim Branson, Shakir Mohamed,  Remi Munos, Jie Tang, Richard Zemel, Luke Zettlemoyer 

Best Paper Award committee for dataset and benchmark track: Yulia Gel, Ludwig Schmidt, Elena Simperl, Joaquin Vanschoren, Xing Xie.

Large language models (LLMs) represent a promising but controversial aide in the process of preparing and reviewing scientific papers. Despite risks like inaccuracy and bias, LLMs are already being used in the review of scientific papers. [1,2] Their use raises the pressing question: “How can we harness LLMs responsibly and effectively in the application of conference peer review?”

In an experiment at this year’s NeurIPS, we took an initial step towards answering this question. We evaluated a relatively clear-cut and low-risk use case: vetting paper submissions against submission standards, with results shown only to paper authors. We deployed an optional to use LLM-based “Checklist Assistant” to authors at NeurIPS 2024 as an assistant to check compliance with the NeurIPS Paper Checklist. We then systematically evaluated the benefits and risks of the LLM Checklist Assistant focusing on two main questions:

(1) Do authors perceive an LLM Author Checklist Assistant as a valuable enhancement to the paper submission process?

(2) Does the use of an Author Checklist Assistant meaningfully help authors to improve their paper submissions?

While there are nuances to our results, the main takeaway is that an LLM Checklist Assistant can effectively aid authors in ensuring scientific rigor, but should likely not be used as a fully automated review tool that replaces human review.  

Example of checklist questions, answers, and LLM-provided review from the Checklist Assistant.

(1) Did authors find the Checklist Assistant useful?

We administered surveys both before and after use of the Checklist Assistant asking authors about their expectations for and perceptions of the tool. We received 539 responses to the pre-usage survey, 234 submissions to the Checklist Assistant and 78 responses to the post-usage survey.

Authors felt the Checklist Assistant was a valuable enhancement to the paper submission process. The majority of surveyed authors reported a positive experience using the LLM Checklist Assistant: >70% of authors found it useful and >70% said they would modify their paper in response to feedback. 

Interestingly, authors’ expectations of the assistant’s effectiveness were even more positive before using it than their assessments after actually using it. Comparing pre- and post-usage responses, there was a statistically significant drop in positive feedback on the “Useful” and “Excited to Use” questions. 

Responses to survey questions before and after using checklist verification (n=63 unique responses.)

(2) What were the main issues authors had with the Checklist Assistant?

We also solicited freeform feedback on issues that the authors experienced using the Checklist Assistant, with responses grouped below. 

Among the main issues reported by authors in qualitative feedback, the most frequently cited problems were inaccuracy (20/52 respondents) and that the LLM was too strict in its requirements (14/52 respondents).

Reported issues using checklist verification from freeform feedback on post-usage survey (n=52 out of 78 total survey responses.)

(3) What kinds of feedback did the Checklist Assistant provide?

We used another LLM to extract key points from the Checklist Assistant’s responses for each question on the paper checklist and to cluster these points into overarching categories. Below we show the most frequent categories of feedback given by the Author Checklist Assistant on four questions of the checklist:

Clustering of most common types of feedback given by the LLM Checklist Assistant on four checklist questions. 

The LLM was able to give concrete feedback to authors grounded in the content of their paper and checklist. The LLM tended to provide 4-6 distinct and specific points of feedback per question across the 15 questions. While it tended to give some generic boilerplate as part of its responses and to expand the scope of questions, it also was capable of giving concrete and specific feedback for many questions.  

(4) Did authors actually modify their submissions?

Authors reported in freeform survey responses reflect that they planned to make meaningful changes to their submissions—35/78 survey respondents provided specific modifications they would make to their submissions in response to the Checklist Assistant. This included improving justifications for checklist answers and adding more details to the paper about experiments, datasets, or compute resources.

In 40 instances, authors submitted their paper twice to the checklist verifier (accounting for 80 total paper submissions.) We find that of the 40 pairs of papers, in 22 instances authors changed at least one answer in their checklist (e.g., ‘NA’ to ‘Yes’) between the first and second submission and in 39 instances they changed at least one justification for a checklist answer. Of the authors who changed justifications on their paper checklist, many authors made a large number of changes, with 35/39 changing more than 6 justifications of the 15 questions on the checklist. While we cannot causally attribute these changes to the Checklist Assistant, they suggest that authors may have incorporated feedback from the assistant in between submissions.

Below, we show (multiplicative) increase in word count between initial submission and final submission on questions where authors changed justifications (a value of 2 corresponds to a doubling of the length of an answer). We find that over half the time when authors changed a checklist answer, they more than doubled the length of their justification.

Change in word count of authors’ checklist responses between first and second submission to the Checklist Assistant. Over half the time, authors more than doubled the length of their checklist response.

In summary:

When authors submitted to the Checklist Assistant multiple times they almost always made changes to their checklists between submissions and significantly lengthened their answers, suggesting that they may have added content in response to LLM feedback. 

(5) Can the Checklist Assistant be gamed?

The intended use of our Checklist Assistant was to help authors improve their papers, not to serve as a tool for reviewers to verify the accuracy of authors’ responses. If the system were used as an automated verification step as part of a review process, this could introduce an incentive for authors to “game” the system motivating the following question: could authors automatically improve the evaluations of their checklist responses with the help of AI, without making actual changes to their paper? If such gaming were possible, authors could provide a false impression of compliance to a conference without (much) additional effort and without actually improving their papers.

To assess whether our system is vulnerable to such gaming, we employed another LLM as an attack agent to iteratively manipulate the checklist justifications, aiming to deceive the Checklist Assistant. In this iterative process, the attack agent receives feedback from the system after each round and uses it to refine its justifications. We provided GPT-4o with the initial checklist responses and instructed it to revise the justifications based solely on feedback, without altering the underlying content of the paper. We allowed the attack agent to do this for three iterations (reflecting the submission limit on our deployed assistant), with the agent selecting the highest scored response for each checklist question over the iterations. 

To statistically quantify how attack success we submitted the selected justification to our Checklist Assistant for an evaluation “Score” (1 when the Checklist Assistant said the checklist question had “No Issues” and 0 when the Assistant identified issues.) Below we show the results of this attack:

Authors could adversarially improve their chance of passing an automated LLM Checklist review by modifying their checklist without modifying their paper: on 14 out of 15 questions our simple adversarial attack shows a score increase when comparing the unchanged justifications with those refined through three iterations of automated improvement. 

In a manual examination of the changes made by the (attacker) LLM to the justifications, we found that the attacker LLM employed several illegitimate strategies, such as adding a hardware description that was not present in the paper or original justifications and generating a placeholder URL as a purported repository for the code. These illegitimate justifications were evaluated as adequate by the Checklist Assistant. 

Conclusions 

Our deployment of an LLM-based paper Checklist Assistant at NeurIPS 2024 demonstrated that LLMs hold potential in enhancing the quality of scientific submissions by assisting authors in validating whether their papers meet submission standards. However, our study points to notable limitations in deploying LLMs within the scientific peer review process that need to be addressed, in particular accuracy and alignment issues. Further, our system was not robust to gaming by authors, suggesting that while a Checklist Assistant could be useful as an aid to authors it may be a poor substitute for human review. NeurIPS will continue to build on its LLM Policy Reviews for 2025.

For more details, see the full paper: https://arxiv.org/pdf/2411.03417.

For questions or suggestions, please contact us at: checklist@chalearn.org 

[1] Liang et al., Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews, 2024. https://proceedings.mlr.press/v235/liang24b.html 

[2] Latona et al., The AI Review Lottery: Widespread AI-Assisted Peer Reviews Boost Paper Scores and Acceptance Rates, 2024. https://arxiv.org/abs/2405.02150 

72 Windfall Films will be filming at NeurIPS 2024 for a feature documentary about AI. This (filming) is a first-time experiment at NeurIPS, and any other filming is not permitted.

The team will be filming the poster sessions on the afternoon of Wednesday Dec 11, and they may also film a small number of other events (which we will also announce ahead of filming). The areas will be clearly marked with signs that contain the following info:

It is possible that you will be included in general shots of the conference. If you don’t want to appear in the programme, please contact Zara Powell at zara.powell@windfallfilms.com or +44 7557771061 immediately.

You can also find this info here: https://neurips.cc/Conferences/2024/FilmingNotice.

For any feedback about this filming, please contact the NeurIPS Communication Chairs at communication-chairs@neurips.cc.

By Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub Tomczak, Cheng Zhang

We are honored to announce the Test of Time Paper Awards for NeurIPS 2024. This award is intended to recognize papers published 10 years ago at NeurIPS 2014 that have significantly shaped the research field since then, standing the test of time.

This year, we are making an exception to award two Test of Time papers given the undeniable influence of these two papers on the entire field. The awarded papers are:

• Generative Adversarial Nets
  Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio
  
• Sequence to Sequence Learning with Neural Networks
  Ilya Sutskever, Oriol Vinyals, Quoc V. Le

The Generative Adversarial Nets paper has been cited more than 85,000 times as of this blog post. It is one of the foundational pieces for generative modeling and has inspired numerous research advances in the past 10 years. Besides research, it has also enabled generative modeling to make an impact in a diverse range of applications, considering vision data and other domains.

Sequence to Sequence Learning with Neural Networks has been cited more than 27,000 times as of this blog post. With the current fast advances of large language models and foundation models in general, making a paradigm shift in AI and applications, the field has benefited from the foundation laid by this work. It is the cornerstone work that set the encoder-decoder architecture, inspiring later attention-based improvements leading to today’s foundation model research.

There is no doubt regarding the significance of these two works.

At NeurIPS, you will see both papers presented by the authors in person, followed by a short Q&A on Friday, December 13th, 2024. We look forward to announcing other NeurIPS awards at the conference!

We are thrilled to announce the results of the first call for NeurIPS High School Projects.  With a theme of machine learning for social impact, this track was launched to get the next generation excited and thinking about how ML can benefit society, to encourage those already pursuing research on this topic, and to amplify that work through interaction with the NeurIPS community. 

In total, we received 330 project submissions from high schoolers around the globe.  Among those, 21 projects were chosen to be spotlighted and 4 were chosen as award winners.  We congratulate all of the students and encourage community members to attend the joint poster session on Tuesday December 10, where representatives of the four award-winning projects will present their work.

Award-winning Projects

ALLocate: A Low-Cost Automatic Artificial Intelligence System for the Real-Time Localization and Classification of Acute Myeloid Leukemia in Bone Marrow Smears

Ethan Yan, Groton School, MA, USA

Abstract: Accurate leukemia detection in current clinical practice remains challenging due to limitations in cost, time, and medical experience. To address this issue, this research develops the first integrated low-cost automatic artificial intelligence system for the real-time localization and classification of acute myeloid leukemia in bone marrow smears named ALLocate. This system consists of an automatic microscope scanner, an image sampling system, and a deep learning-based localization and classification system. A region classifier using a convolutional neural network (CNN) model was developed to select usable regions from unusable blood and clot regions. For real-time detection, the YOLOv8 model was developed and optimized. These models show high performance with a region classifier accuracy of 96% and YOLOv8 mAP of 91%. In addition, a low-cost automatic microscope scanner system was developed using 3D-printed pieces controlled by stepper motors and a programmed Arduino-based RAMPS control board. When ALLocate was applied to marrow smears, its leukemia detection results were similar to results from a doctor but were produced much faster. This is the first report to integrate a deep learning system with a low-cost microscope scanner system for leukemia detection with high performance, which can benefit small community practices and clinics in underserved areas, making healthcare more accessible and affordable to all.

Image Classification on Satellite Imagery For Sustainable Rainwater Harvesting Placement in Indigenous Communities of Northern Tanzania

Roshan Taneja and Yuvraj Taneja, Sacred Heart Preparatory, CA, USA

Abstract: In the remote regions of Northern Tanzania, women and children of the Maasai Tribe walk nine hours a day to collect water. Over four years, collaborative efforts with the Maasai communities have led to the installation of water harvesting units, enhancing the local socio-economic conditions by facilitating educational opportunities and economic pursuits for over 4,000 individuals. This project is a novel approach to integrating satellite data and image classification to identify densely populated areas marked by uniquely shaped Maasai homes. It will also use density maps to plan the best placement of rainwater harvesting units to help 30,000 Maasai. The backbone of this project was developing an image classification model trained on 10,000 hand-selected satellite image samples of Bomas (Maasai living units). This model generated a density heat map, enabling strategically placing rainwater harvesting units in the most critical locations to maximize impact. The project underscores the use of satellite technology and machine learning to address humanitarian needs such as water, particularly in harder-to-reach areas with no infrastructure.

Multimodal Representation Learning using Adaptive Graph Construction

Weichen Huang, St. Andrew’s College, Ireland

Abstract: Multimodal contrastive learning trains neural networks by leveraging data from heterogeneous sources such as images and text. Yet, many current multimodal learning architectures cannot generalize to an arbitrary number of modalities and need to be hand-constructed. We propose AutoBIND, a novel contrastive learning framework that can learn representations from an arbitrary number of modalities through graph optimization. We evaluate AutoBIND on Alzhiemer’s disease detection because it has real-world medical applicability and it contains a broad range of data modalities. We show that AutoBIND outperforms previous methods on this task, highlighting the generalizablility of the approach

PumaGuard: AI-enabled targeted puma mitigation

Aditya Viswanathan, Adis Bock, Zoe Bent, Tate Plohr, Suchir Jha, Celia Pesiri, Sebastian Koglin, and Phoebe Reid, Los Alamos High School, NM, USA

Abstract: We have trained a machine learning classification algorithm to detect mountain lions from trail cam images. This algorithm will be part of a targeted mitigation tool to deter mountain lions from attacking livestock at the local stables. Our algorithm that uses the Xception algorithm is 99% accurate in training, 91% accurate on validation and successful in identifying mountain lions at the stables.

Spotlighted Projects

Diagnosing Tuberculosis Through Digital Biomarkers Derived From Recorded Coughs

Sherry Dong, Skyline High School, MI, USA

GeoAgent: Precise Worldwide Multimedia Geolocation with Large Multimodal Models

Tianrui Chen, Shanghai Starriver Bilingual School, China

INAVI: Indoor Navigation Assistance for the Visually Impaired

Krishna Jaganathan, Waubonsie Valley High School, IL, USA

Implementing AI-driven Techniques for Monitoring Bee activities in Hives

Tahmine Dehghanmnashadi, Shahed Afshar High School for Girls, Iran 

AquaSent-TMMAE: A Self-Supervised Learning Method for Water Quality Monitoring

Cara Lee, Woodside Priory School, CA, USA;  Andrew Kan, Weston High School, MA, USA; and Christopher Kan, Noble and Greenough School, MA, USA

AAVENUE: Detecting LLM Biases on NLU Tasks in AAVE via a Novel Benchmark

Abhay Gupta, John Jay Senior High School, NY, USA; Philip Meng, Phillips Academy, MA, USA; and Ece Yurtseven, Robert College, Turkey

FireBrake: Optimal Firebreak Placements for Active Fires using Deep Reinforcement Learning

Aadi Kenchammana, Saint Francis High School, CA, USA

Vision-Braille: An End-to-End Tool for Chinese Braille Image-to-Text Translation

Alan Wu, The High School Affiliated to Renmin University of China, China

Advancing Diabetic Retinopathy Diagnosis: A Deep Learning Approach using Vision Transformer Models

Rhea Shah, Illinos Mathematics & Science Academy, IL, USA

LocalClimaX: Increasing Regional Accuracy in Transformer-Based Mid-Range Weather Forecasts

Roi Mahns and Ayla Mahns, Antilles High School, Puerto Rico, USA

HypeFL: A Novel Blockchain-Based Architecture for a Fully-Connected Autonomous Vehicle System using Federated Learning and Cooperative Perception

Mihika A. Dusad and Aryaman Khanna, Thomas Jefferson High School for Science and Technology, VA, USA

Robustness Evaluation for Optical Diffraction Tomography

Warren M. Xie, Singapore American School, Singapore

Translating What You See To What You Do: Multimodal Behavioral Analysis for Individuals with ASD

Emily Yu, Pittsford Mendon High School, NY, USA

SignSpeak: Open-Source Time Series Classification for ASL Translation

Aditya Makkar, Divya Makkar, and Aarav Patel, Turner Fenton Secondary School, ON, Canada

SeeSay: An Assistive Device for the Visually Impaired Using Retrieval Augmented Generation

Melody Yu, Sage Hill School, CA, USA

Predicting Neurodevelopmental Disorders in rs-fMRI via Graph-in-Graph Neural Networks

Yuhuan Fan, The Experimental High School Attached to Beijing Normal University, China

Realistic B-mode Ultrasound Image Generation from Color Flow Doppler using Deep Learning Image-to-Image Translation

Sarthak Jain Silver Creek High School, CA, USA

Registration for NeurIPS 2024 has exceeded our initial expectations. To ensure that authors of accepted conference and workshop papers are given priority, we are transitioning from our usual first-come-first-serve registration model to a randomized lottery system, effective immediately. While we are currently reserving spots for all conference and workshop paper authors, this may change as the conference approaches and we open up remaining slots to the lottery, so we urge authors to register as soon as possible to guarantee their spot. To register, please use our registration page: https://neurips.cc/Register/. If you are an author on a conference or workshop paper, registration will be open to you. Otherwise, you will be re-directed to our lottery system.

If you have an accepted main conference or workshop paper and are still redirected to the lottery, please attempt the following troubleshooting steps:

• You may have multiple neurips.cc accounts, and you are not logged onto the one that is linked to your paper. Log onto the correct account.
• You have a workshop paper, but the organizers of the workshop have not imported their papers. Please wait for the workshop organizers to do so and contact them with any inquiries.
• Your neurips.cc profile does not contain a first name, last name, and institution. Please update this information.

By Ioana Bica, Nezihe Merve Gürel

We are excited to announce this year’s affinity events co-located with NeurIPS.

What are affinity groups?

At NeurIPS, affinity groups play a crucial role in promoting and supporting the ideas and voices of various communities that are defined by some axis of joint identity and raise awareness of issues that affect their members. In addition, they provide members of these affinity groups with increased opportunities to showcase their work, engage in discussions, and build connections during NeurIPS events, promoting diversity and inclusion at NeurIPS.

The main goals of affinity events are to increase visibility of the work done by the members of the affinity group, enable them to interact and discuss topics that are unique to each affinity group, encourage affinity group members to showcase their work and step into the NeurIPS community.  In contrast to regular workshops, affinity group workshops adopt a more open and flexible approach around topic area, focusing on forming research communities within the affinity groups and bringing attention to the unique challenges faced by their members.

Additionally, affinity group socials serve as safe and welcoming spaces, allowing members to interact, socialize, and network in an inclusive environment. By fostering such social gatherings, NeurIPS supports the creation of a supportive community that values diversity and ensures that all voices are heard and valued.

What kind of events do affinity groups offer at conferences?

Affinity groups organize events with the main aim to encourage people to showcase their work and step into the NeurIPS community. The affinity events at NeurIPS this year will take place throughout the conference week. See below the details for when each affinity event will take place. NeurIPS will offer streaming support for all the in-person components of the affinity groups. 

What can NeurIPS attendees do to support affinity groups?

Participate! Actively attending and participating in panels and sessions for discussions about issues that are relevant to the affinity groups goes a long way in helping promote their cause. We request all members of our community to participate actively!

Affinity groups at NeurIPS 2024

We are delighted to announce the following affinity groups at NeurIPS 2024. Find out more below about their mission and the event they are organizing! 

Indigenous in AI/ML 

Indigenous In AI’s vision is to build an international community of Native, Aboriginal, and First Nations who will collectively transform their home communities with advanced technology. By elevating the voices of Indigenous ML researchers we will inspire future impactful work and break stereotypes. Additionally, this group will strive to educate the broader NeurIPS on contemporary indigenous issues relevant to information technology and practices. For more information, please visit the Indigenous in AI website.

Muslims in ML 

The Muslims in ML workshop aims to explore the intersection of machine learning, fairness, and the global Muslim community. It recognizes the potential of AI and ML to bring positive changes but also acknowledges existing barriers and biases that may perpetuate unfairness against Muslims. The workshop, now in its 3rd edition, aims to bring together diverse experts and perspectives to examine challenges and opportunities in integrating AI/ML in the lives of Muslims and those in Muslim-majority countries. It will showcase research by Muslim scholars in ML and highlight work addressing challenges faced by the Muslim community. The workshop provides a safe space for open discussions and social activities. It adopts an inclusive approach that considers cultural association and proximity to the Muslim identity, emphasising the importance of understanding the complexity and diversity within the Muslim community. The workshop seeks to promote awareness, collaboration, and mitigation strategies to ensure fair and equitable implementation of AI/ML technologies for Muslims worldwide.  For more information, please visit the Muslims in ML website.

New in ML workshop

The NewInML workshop is dedicated to fostering an inclusive and supportive environment for emerging researchers in the ML community, with a strong belief that great innovations can arise from students of diverse backgrounds. We have extended our audience to include students at all levels, recognizing the diverse needs of those early in their journey as well as those transitioning within their careers. By offering a platform that spans a broad spectrum of topics—from hands-on sessions to frontier discussions—the workshop provides essential guidance to help participants contribute effectively to the field. Our goal is to build a more inclusive ML community by mentoring attendees, refining their research skills, and empowering them to make meaningful contributions, whether in academia or industry. NewInML workshop will take place Tuesday December 10th. More information can be found on the Workshop’s website.

Neurodivergent Community  

The Neurodiversity Workshop will gather the Neurodiverse community to engage on leading research in the space and launch a competition to inspire additional research. The Workshop will be led by expertise from Michigan State University, MIT, Princeton, and more.  The Workshop will occur on Thursday December 12 at 12pm Pacific Time. More information can be found on the Workshop’s website.

Global South in AI 

Global South in AI (GSAI) was born in 2022 to solve the lack of inclusion and representation of the Global South in artificial intelligence ecosystems, especially the myriad rich vastness of culture and languages lived, spoken by millions of humans, and not just those with access and privilege. GSAI stands as counterbalance to a western lens focused pass-through of all language translation in LLMs and the impact of such homogenization on the global south. We do this through multiple projects that help local researchers who are already working on including their local culture and language in AI to build low resource datasets and empower their community in AI literacy. We also host annual multidisciplinary academic meetups co-located with NeurIPS, with the highest ethical standards for a diverse and inclusive community in order to promote the exchange of research achievements in Language artificial intelligence and machine learning. Please check our website and connect with us on LinkedIn. Our workshop date this year is Dec 10th, 10AM to 3PM at NeurIPS in Vancouver Canada. Details will be announced soon.

Women in ML 

The goals and activities of WiML are to

1. Highlight the work of researchers in machine learning who are women or non-binary,
2. Give them a chance to meet, encourage technical discussion, and future collaboration,
3. Give junior researchers an opportunity to present their work to their peers as well as senior members of the community.

The workshop plans to feature speakers who are women or nonbinary to give talks on their research, organize mentorship sessions to discuss relevant topics, and encourage networking. It will encourage and foster research conversations and help participants develop collaboration opportunities. There will also be poster sessions for participants to present a broader array of work and exchange feedback. The workshop will take place on Tuesday, December 10. For more information, please visit the Women in ML website.

Queer in AI 

The Queer in AI workshop and social events at NeurIPS 2024 aim to create a vibrant and inclusive space that brings together queer students, researchers, and practitioners from across the globe to critically examine the impact of AI on the LGBTQIA* community, while fostering a sense of belonging among participants. The workshop aspires to build a strong network of queer individuals, enable discussions at the intersection of AI and queer identities, and tackle issues of discrimination within the tech industry and society at large. 

The Queer in AI workshop is dedicated to advancing equitable AI practices and contributing to a future where technology serves the needs of all communities. The workshop is committed to advocating for a more ethically grounded approach to AI development, ensuring that diverse voices are heard and valued in conversations about the future of technology. For further details, please visit the Queer in AI website.

LatinX in AI 

LatinX in AI (LXAI) bridges communities, academics, industry, and politicians working to further AI innovation and resources for LatinX individuals globally. We drive and support research, development, infrastructure, and mentoring programs to boost innovation and capabilities of LatinX professionals working in Artificial Intelligence, Machine Learning, and Data Science.

The 7th Annual Latinx in AI Research workshop will be held on December 10th, 2024 and will embrace a hybrid format, welcoming attendees both onsite and virtually. The workshop is a one-day event with invited speakers, oral presentations, and posters. The event brings together faculty, graduate students, research scientists, and engineers for an opportunity to connect and exchange ideas. There will be a panel discussion and a mentoring session to discuss current research trends and career choices in Artificial Intelligence and Machine Learning. While all presenters will identify primarily as LatinX, all are invited to attend. For further details, please visit the Latinx in AI website. 

Black in AI

Black in AI aims to shift the power dynamic in the field of artificial intelligence by broadening the range of voices contributing to the development, deployment, and regulation of AI technologies – to ensure that our community members are full partners in affirming their technological and economic futures. NeurIPS, being one of the largest AI research conferences, offers a valuable opportunity for researchers in the community to stay updated on the latest research and expand their network. Since its inception in 2017, many participants of the Black in AI Workshop have walked away from their experience at NeurIPS with internship offers, graduate program advisors, mentorships, and renewed energy to continue working in the field. The workshop remains committed to bridging the gap between researchers of African descent and the AI community at large. For more information, please visit the Black in AI website.

by Andrew M. Dai, Irene Chen, Gal Chechik

We are excited to present the list of tutorials selected to present at the NeurIPS 2024 conference! We look forward to a programme that we hope will engage the attendees with topics ranging from experimental design for AI researchers and meta-generation algorithms for LLMs through to cross-disciplinary insights into alignment. In this post, we will describe this year’s programme and our selection process.

Program

There will be 14 tutorials this year. All of them will be conducted in person to encourage active participation and some of them include panels to allow for a diverse range of discussion. The tutorials selected this year with their speakers are:

Flow Matching for Generative Modeling

Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu

Cross-disciplinary insights into alignment in humans and machines

Gillian Hatfield, Joel Z. Leibo, Dylan Hadfield-Menell

Experimental Design and Analysis for AI Researchers

Katherine Hermann, Jennifer Hu, Mike Mozer

Generating Programmatic Solutions: Algorithms and Applications of Programmatic Reinforcement Learning and Code Generation

Shao-Hua Sun, Levi Lelis, Xinyun Chen

Evaluating Large Language Models – Principles, Approaches, and Applications

Irina Sigler, Yuan (Emily) Xue, Bo Li

Beyond Decoding: Meta-Generation Algorithms for Large Language Models

Sean Welleck, Hailey Schoelkopf, Matthew Finlayson

Opening the Language Model Pipeline: A Tutorial on Data Preparation, Model Training, and Adaptation

Akshita Bhagia, Nathan Lambert, Kyle Lo

Dynamic Sparsity in Machine Learning: Routing Information through Neural Pathways

Edoardo M. Ponti, André F. T. Martins

Advancing Data Selection for Foundation Models: From Heuristics to Principled Methods

Jiachen T. Wang, Ruoxi Jia, Ludwig Schmidt

Watermarking for Large Language Model

Xuandong Zhao, Yu-Xiang Wang, Lei Li

Sandbox for the Blackbox: How LLMs Learn Structured Data?

Ashok Vardhan Makkuva, Bingbin Liu, Jason D. Lee

Out-of-Distribution Generalization: Shortcuts, Spuriousness, and Stability

Yoav Wald, Aahlad Puli, Maggie Makar

Causality for Large Language Models

Zhijing Jin, Sergio Garrido

Meaningful Evaluations of ML Privacy Techniques: What Are We Waiting For

Mimee Xu, Fazl Barez, Dmitrii Usynin

Selection process

We received 57 unique tutorial proposals this year with roughly half of them related to large language models. Each submission was reviewed by at least one tutorial chair with the accepted tutorials receiving at least two reviews. Review assignments were chosen based on the area of expertise and to avoid conflicts of interest. Each chair gave a score from 1-5 on three categories, the speaker/panelist quality, the topic (interestingness, importance, timeliness) and appropriateness (suitability for the NeurIPS audience).

Common reasons for lower scores included:

• Topics that were too niche for the general NeurIPS audience.
• Significant overlap with stronger proposals.
• Insufficient diversity among the speakers.
• Too much expectation of familiarity with specific software or hardware.
• Too much overlap with past tutorials.

Choosing the final tutorials from the high scoring ones was a challenging task and we appreciate all the work that went into these proposals. We will be doing run-throughs with the selected tutorials to offer advice and feedback.
Keep your eyes on a post-conference retrospective blog post with reflections on tutorials. Please feel free to reach out to us at tutorial-chairs@neurips.cc if you have any suggestions.