by Adji Bousso Dieng, Andrew Gordon Wilson, Jessica Schrouff

We are excited to announce the tutorials selected for presentation at the NeurIPS 2022 conference! We look forward to an engaging program, spanning many exciting topics, including Lifelong Learning, Bayesian Optimization, Algorithmic Discrimination, Neurosymbolic Programming, Data Compression, NLP in Healthcare, and others. In this blog post, we detail our selection process, the program, reflections on submissions, and considerations for future tutorials.

Tutorial programs

Each virtual tutorial will consist of:

• A presentation by the speakers (1h50)
• Live Q&A with the speakers, answering technical or clarifying questions (10 minutes)
• Live Panel with further researchers in the field to discuss challenges and promises (30 minutes)

There are two notable differences from last year’s programme: a mix of contributed and invited tutorials (rather than only invited), and a live panel.

UTC time	Tutorial 1	Tutorial 2	Tutorial 3
10:00	On the Role of Meta-learning for Few-shot Learning Speaker: Eleni Triantafillou
13:00	Foundational Robustness of Foundation Models Speakers: Pin-Yu Chen, Sijia Liu, Sayak Paul	Lifelong Learning Machines Speakers: Tyler Hayes, Dhireesha Kudithipudi, Gido van de Ven
16:00	Neurosymbolic Programming Speakers: Swarat Chaudhuri, Armando Solar-Lezama, Jennifer Sun	Advances in NLP and their Applications to Healthcare Speaker: Ndapa Nakashole	Probabilistic Circuits: Representations, Inference, Learning and Applications Speakers: Antonio Vergari, YooJung Choi, Robert Pehar
19:00	Advances in Bayesian Optimization Speakers: Virginia Aglietti, Jacob Gardner, Jana Doppa	Algorithmic discrimination at the intersection Speakers: Golnoosh Farnadi, Vera Liao, Elliot Creager	Incentive-Aware Machine Learning: A Tale of Robustness, Fairness, Improvement, and Performativity Speaker: Chara Podimata
22:00	Data Compression with Machine Learning Speakers: Karen Ullrich, Yibo Yang, Stephan Mandt	Creative Culture and Machine Learning Speakers: Negar Rostamzadeh, Anna Huang, Mark Riedl
01:00	Theory and Practice of Efficient and Accurate Dataset Construction Speakers: Frederic Sala, Ramya Korlakai Vinayak	Fair and Socially Responsible ML for Recommendations: Challenges and Perspectives Speakers: Hannah Korevaar, Manish Raghavan, Ashudeep Singh

Selection process

This year, we have experimented with a “contributed only” design (see related blog post). Our hope was to obtain a “community-led” selection of topics and speakers while emphasising diversity across but also within tutorials. Our call for proposals had clear guidelines for the selection of topics, speakers, panellists, format, etc.

We received 34 submissions by the (strict) deadline. Each submission was reviewed by two Tutorial Chairs based on interest and expertise on the topic. Each chair gave a score between 1 (strong reject) to 10 (strong accept) to encapsulate their overall impression of the proposal. We then shortlisted the submissions that had received a 6 or higher from at least one chair (14 proposals out of 34) and discussed when there were disagreements between the initial two reviews. A third review was then obtained from a different Tutorial Chair to finalise the decision to accept or reject a proposal. We accepted 9 proposals with this process.

Some relatively common reasons for low scores included (but were not limited to):

• The topic is too niche for a very broad audience.

• The topic has been presented in recent tutorials in major machine learning conferences.
• The speakers have recently contributed to major ML conferences as tutorial and/or keynote speakers.
• Low diversity, broadly construed.
• Guidelines were not followed (e.g. no panel included).

While no feature in particular would guarantee acceptance, certain features were often present in proposals that were favourably reviewed:

• The proposal was highly polished. Significant effort and thought had gone into carefully organising and planning the tutorial, paying close attention to instructions, with few loose ends. These features suggested that the presentation itself would be carefully planned, avoiding last minute organisation, logistical mishaps, etc. 
• The presenters had demonstrated significant commitment, contributions, and expertise in the chosen topic. 
• The topic would be both fresh as a tutorial and have a relatively broad appeal.
• Diversity, broadly construed. For example, speakers and panels with diverse perspectives on the material.
  

Given the manageable workload, we did not desk-reject proposals for not following guidelines. In the future, desk rejections might be considered (e.g. multiple proposals were 10-12 pages long instead of the required 5).

Diversity in submitted proposals

Each proposal included 1 to 3 speakers, and up to 6 panellists. Across all submissions, there were a total of 85 speakers and 160 confirmed panellists (with some overlap with speakers). We had explicitly asked tutorial presenters to consider diversity in terms of (not exhaustive) gender, race, geographical location, institution, background and expertise, and to write a statement. Our goals were (1) to ensure that a diverse set of opinions were considered, and (2) members from underrepresented groups in the field were included in this program.

According to the diversity statements, researchers have mostly focused on background, expertise and geographical location as diversity dimensions. We note that geographical locations in proposals were mostly limited to Western Europe, the US and Canada. It therefore seems that researchers understood our first goal partly, but only a few proposals satisfactorily considered the second goal.

Aspects of gender and race or ethnicity were rarely addressed explicitly in diversity statements, and sometimes diversity was highlighted where it was not clearly present. This “lip-service” diversity led to the following results:

• Men were overrepresented as speakers, with 17 out of 34 proposals including only  male speakers.
• Asian (South and East) and White researchers were overrepresented.
• Diversity was more often addressed in the panel, than in the speakers.

To illustrate these impressions, we tried to identify each speaker and confirmed panellist according to their perceived gender (based on pronouns in the proposal), perceived race (from the proposal where available, otherwise from a combination of CV, online information and picture as last resort), institution and seniority level (early career: PhD student or early postdoc 0-3 years post-PhD, mid-career: Assistant Prof, 3-10 years post-PhD, senior: Prof, 10+ years post-PhD). We acknowledge that this classification is somewhat arbitrary and does not fully reflect the gender and racial identities of the speakers and panellists. We however believe it is important to provide an approximate quantification of the consequences of our design choices.

Figure 2: Perceived gender (pie chart) and race (bar plot) distribution of speakers across all submitted proposals. MENA stands for Middle East and North Africa. Note that percentages in the bar plot might not sum to 100 as race information might not be identifiable from the proposal or online information.

We observe that men (he/his pronouns) represent more than 75% of the proposed speakers. White and Asian speakers represent more than 90% of the speakers. Proposals were mostly submitted by and included more academics than industry researchers (26 industry out of 85). Seniority levels were well balanced (early: 26, mid: 33, senior: 26).

Diversity restricted to the panel

Diversity in terms of perceived gender slightly improved in the panel (Figure 3), but remained dominated by men. Similarly, Asian and White researchers still represented more than 80% of the panellists. Interestingly, the panels included fewer researchers from industry (31 out of 160, i.e. ~19%) and skewed more towards senior researchers (early: 26, mid: 47, senior: 81). Overall, we see that the diversity improves relative to speakers, but remains low.

Figure 3: Perceived gender and race distribution in the panels of submitted proposals.

As a note, we would like to highlight the fact that 3 proposals had made particular efforts in terms of the diversity. These efforts, combined with strong proposals and timely topics, led 2 of these proposals to be accepted (the last one being ineligible). These authors show that it is possible to propose a diverse set of speakers and panellists across all dimensions.

Finally, we assessed diversity across other dimensions, such as disability or being part of the LGBTQIA+ community. For privacy reasons, we do not communicate these numbers.

Improving on the quality of the program

We contacted the authors of accepted tutorials and worked with them in cases where we believed the program could be improved, in terms of organisation, scientific content, and diversity. Where appropriate, we also encouraged the speakers to rethink the format of the proposed tutorial to account for the online edition.

As we had initially planned for ~12-15 tutorials, we had the opportunity to invite tutorial speakers. We invited speakers by identifying researchers who have demonstrated excellence and expertise in a specific topic and who would benefit from the opportunity. We considered aspects of diversity in our selection to prioritise researchers from under-represented groups and maximised the diversity in topics.

Thanks to the responsiveness of invited speakers and to the work of authors of submitted proposals, we are able to provide an exciting list of topics. While we also increase the diversity of speakers and panellists, there is still room for improvement.

Figure 4: Perceived gender and race distribution across speakers and panellists after proposals were revised and speakers were invited.

Considerations for future editions

Authors

• Carefully review the topics of tutorials at major ML conferences in the past 3 years. Topics that are overlapping are unlikely to be selected unless the tutorial brings a significantly novel point of view or extension.
• Read and follow the guidelines. This might seem obvious but multiple proposals were rejected because they included speakers who were ineligible, did not include a panel, etc. While we did not desk-reject proposals this year, we did notice that the proposals we accepted were mostly following the guidelines. This simply highlights that the authors carefully considered the different requirements, wrote and proof-read their submission and submitted on time. This increases the chances of acceptance.
• Diversity should be considered across all aspects, and proposals should include voices from under-represented groups. Proposals with 6 or 7 participants that all identify with masculine pronouns are unlikely to be accepted. Refer to directories from affinity groups (e.g. https://www.directory.wimlworkshop.org/, https://lxai.app/PUBLIC-DIRECTORY), request recommendations from more senior researchers in the field, and consider non-Western institutions.
• If you would like to propose a topic for a tutorial but are ineligible, please pass the opportunity to someone else! Consider encouraging others in the field to submit a proposal.

Speakers and panellists

A proposal might stem from one or a couple of researchers who will then invite other speakers and panellists. These co-presenters and panellists also have an important role to play:

• We observed that some panellists were considered as confirmed in multiple proposals. These were often more senior researchers. If you do get invited for multiple opportunities, please consider suggesting other researchers instead. We had strict guidance that every speaker and panellist could only be considered for one tutorial.
• Similarly, if you cannot participate, please suggest other researchers and think about researchers from under-represented groups who would benefit from the opportunity.
• If you see that the list of speakers and panellists is not very diverse or dominated by groups that are already over-represented in the field, reach out to the authors and ask them to modify the list.

Members from under-represented groups

While the burden of defining a diverse program should not fall on members of under-represented groups, there are a couple of steps that can be taken to increase visibility:

• Create a website, fill in (and update) your personal page on your institution’s or company’s website or make public profiles on LinkedIn, DBLP, ResearchGate,  Google Scholar, …
• If you identify with an affinity group and this group has a directory, consider creating a profile. Examples include: https://www.directory.wimlworkshop.org/, https://lxai.app/PUBLIC-DIRECTORY
• Consider posting the recordings of previous talks
• We recommend affinity groups to create open-source repositories of their members (where feasible) such that organisers can identify potential speakers to invite. 
• Submit a proposal!

Web presence helps authors, speakers, panellists and tutorial organisers find your profile to consider you for the opportunity. Without this information, it is difficult to estimate whether someone has the breadth of experience and communication skills that would make a tutorial successful.

Tutorial organisers

We are of course not exempt from improvements. Some of the learnings we take for future editions include:

• More proactively reach out to potential speakers to encourage them to submit a proposal. This includes repeated postings on mailing lists such as those from affinity groups, as well as directly reaching out to researchers.
• Earlier invitations of invited tutorials.
• Clearer guidelines, e.g. explaining the goals we are trying to achieve with the diversity statement.
• Having a clear set of expectations and benefits for tutorial speakers and panellists.

NeurIPS organisers and board

Tutorial speakers provide significant content for the conference. When they come from under-represented groups, they could be better supported such that they can submit a proposal or accept this opportunity. Bottlenecks we have identified include:

• No funding opportunity for tutorial speakers if they wanted to attend the in-person component of the conference. 
• No honorarium for speakers. For some speakers from under-represented groups, the exposure that a tutorial provides does not compensate for the toll that building such a program takes as this is time not devoted to research or grant applications.
• Provide the opportunity for NeurIPS contributors to self-report their demographic characteristics.

The NeurIPS organisers and Board have been receptive to these requests and have now granted:

• Tutorial speakers receive an in-person or virtual registration.
• Thanks to DIA Chairs, tutorial speakers will also be considered in priority when applying for NeurIPS travel funding (previously limited to students and authors).
• Each tutorial will receive a honorarium (to be split across speakers).
• Self-reporting requires more consideration, especially given the laws regarding demographic surveys in different countries. It is being discussed for future meetings.

We are thankful to the organisers (in particular the General and DIA Chairs) and the Board for these measures. We believe they will help in providing an exciting and diverse set of tutorials in future editions. 

We are extremely excited about the programme, and look forward to seeing you at the tutorials!