Menu

darren

About darren

https://blog.neurips.cc

Posts by Darren Nelson:

NeurIPS 2020 Online Experiments: Gather Town Poster Sessions and Mementor

NeurIPS 2020 was held online in December 2020. Attendees had reported mixed experiences with poster sessions from previous online conferences, and found mentor-mentee matching very valuable. We aimed to craft a smoother experience by re-designing poster sessions and making mentorship connections easier through a new matching platform. We wanted to lean into the unique opportunities afforded by the virtual format, which allows for blindingly fast browsing compared to tired feet, while preserving the serendipity of unplanned discovery of many colleagues wandering in a shared space.

Deciding on Gather Town for poster sessions

ICML and ICLR had held their poster sessions as Zoom calls for each poster. Feedback from attendees and presenters was mixed: presenters were often alone for two hours in zoom calls that attendees were wary of joining for fear that they might be alone with the presenter, attendees had no general snapshot of what other people were seeing at any given time.

The affinity workshops at ICML had experimented with Gather Town for poster sessions. Some of the feedback was very promising: having posters in a shared virtual space where people could walk their avatars around was well-received and solved some of the issues of Zoom calls. However, the set-up of poster rooms had been somewhat chaotic, and disconnection and attendance control issues disrupted the sessions. The scale of NeurIPS was massively larger than any poster session Gather Town had ever hosted, with nearly 2000 posters and registration projected to be above 20,000 people.

Early talks with Gather Town seemed promising: the set-up could be automated by using the API to populate poster rooms, access could be controlled through SSO (Single Sign On) so that only registered, logged in NeurIPS attendees could enter the towns, and new towns could be designed to solve many of the interaction problems that had surfaced from previous Gather Town uses, such as people not knowing how to find a given poster, or not seeing the boundaries for interaction zones for each poster. Scalability concerns could be addressed by setting up multiple poster rooms with each room having its own dedicated server space, to guarantee smooth running for a given capacity.

We decided on a hybrid solution using both Gather Town and Zoom calls. Each poster would get both a presentation spot in Gather Town and a dedicated Zoom call accessible from both the NeurIPS website and the poster town. Zoom was a battle-tested, scalable solution with a robust record that could handle heavy attendance in case of a crowded poster. If Gather Town was plagued with unforeseen issues on the day of the session, the Zoom calls could act as a familiar back-up.

Setting up the poster sessions before the conference

The first challenge was setting up the towns in advance of the conference. We worked from a target of smoothly accommodating up to 10,000 simultaneous attendees for the poster sessions. We did not expect the simultaneous attendance to ever be that high, but we wanted to be prepared in case it was (it turns out simultaneous attendance did not go over 2,000). The towns in Gather can support 2,000 people, but Gather engineers recommended that we aim for 400 people per town for the smoothest interactions with heavy video and poster use. For each of 7 poster sessions, Program Chairs painstakingly clustered the posters into 11 or 25 thematic clusters of up to 20 posters — the morning (in US timezones) sessions were preferred by roughly twice as many people as the evening ones.

Gather Town worked with us to produce a new custom poster room template that would address feedback from previous conferences, and a “hyperroom” that would allow attendees to go from a poster room to another. We can’t stress enough how valuable early usability tests were, allowing us to iterate and arrive to a better final custom poster room template: posters were clearly separated by markings on the carpet of the poster room, allowing to see which attendees were listening to a poster or simply wandering; each poster had a clearly marked presenter spot to easily spot the presenter; people could teleport directly to the poster of their choice from the NeurIPS website, and a coordinate systems allowed people to locate a poster of interest once they were in a room.

The final custom poster-room template

We then wrote the code to automatically create and populate the poster sessions. The rarefied documentation for the Gather Town API and the fact that most of us are not web programmers made this step far from frictionless, even though Gather Town engineers got us started by sharing a sample script for populating a poster room. Thankfully, this was alleviated by their constant availability to answer our questions on Slack.

We hope to make this stage a lot easier for organizers in the future by releasing our poster session set up code within the MiniConf repo, at https://github.com/Mini-Conf/Mini-Conf/tree/master/gather. This code creates poster sessions as sets of poster towns organized in a garden.

Once the poster sessions were set up, we reached out to poster presenters to give them a chance to check how their posters looked in advance of the conference. The advance checks turned out to be very important: about 10% of posters got re-uploaded in the first iteration, and some posters needed to be re-uploaded several times before authors were satisfied with the result.

During and after the conference

Attendees who went to poster sessions gave overall positive feedback: according to a post-conference survey, 75% of people who attended poster sessions enjoyed the experience, while 8% didn’t, Attendees were having lively interactions, reported being satisfied with the ability to instantly see what posters other attendees were visiting, and enjoyed randomly bumping into colleagues. Interactions were mostly smooth thanks to the generous capacity we had budgeted for. Gather Town engineers were available throughout the poster sessions to assist attendees with any problems, in a dedicated support booth we had set up for them in the poster session garden.

Garden from which all poster rooms could be accessed, with the Gather Town support booth

There were many fires that had us working down to the wire, sometimes because of our own mistakes trying to ship last-minute improvements, sometimes due to people’s documented aversion to reading instructions on how to check and update their posters (even a one-liner with bolded words in capital letters), or watching videos in advance about how to interact with the towns, with a stark preference for the wonder of just-in-time exploration and trial and error. But we discovered that communication plastered on the floor close to where people are going to need the help was an effective megaphone.

Empirically, over 10% of people who receive these instructions will not follow them and will email to ask why the update didn’t work.

Trying to inform people as quickly as possible when we messed up by pasting it on the floor

We also used Gather Town to set up a social space for attendees to hang in. We were somewhat worried that the towns could be overwhelmed by the larger-than-expected registration numbers at NeurIPS, so didn’t overly advertise the social space, or schedule specific events there. We split up the social space into several separate areas to ensure that it could support a large capacity, with a hanging space for smaller group discussions, a lobby, and a wellness garden with relaxation, workout, yoga and cute animal videos as well as access to the poster sessions. However, the splitting into separate spaces meant that attendees who were in the social space were too spread out and couldn’t see each other if they were in separate spaces. In the future, it might work better to build incrementally: start from a small, more compact space, and dynamically add more rooms if the first space is at capacity.

Map of the social space: Gather Cafe

Overall, we are happy with our decision to use Gather Town, and have learned a lot from this attempt. Attendees have given us precious feedback and suggestions, for example expressing an appetite for slightly more structure in the unstructured interactions: scheduled social time in the social space so that people can be expected to be there at the same time, dedicated space for people who are welcoming random chats with strangers. We think it could support even richer unplanned interactions in future conferences and will keep exploring how to balance the uncertainty of who is going to show up and how they will behave, with the excitement of experimenting with new ways to help people connect.

Creating a mentor platform

This was a joint effort of our team and Marc Deisenroth, Emtiyaz Khan, Cheng Soon Ong, Adam White, and Olga Isupova to enable mentorship opportunities for researchers in machine learning, both as mentors and mentees, with a special focus on underrepresented minorities. We think that these mentor sessions are a beneficial outcome of the need to go online. In physical conferences, it was harder to find a small group setting for meeting mid-career or senior members of research communities — in Mementor it is much easier.

Scheduling a mentor session is easy. To limit misuse, the video link is only available 30 min before start.

After experiences with spontaneous mentorship sessions at ICLR and ICML, we thought of creating our own platform specifically supporting this idea. We wanted to create a platform that could not only be reused from conference to conference but also allow these gatherings throughout the year. For mentors the platform should be an easy way to disseminate the information about a mentor session and for mentees to stay informed. Scheduling a meeting requires some essential information and a topic — which can be specific, like “ML in Health Care”, or general, like “Ph.D. advice”. The scheduling is announced via email to all subscribers.

The web portal was released and tested during NeurIPS 2020. It is available at https://mementor.net .

Mentor sessions are announced via email and are

Y-Lan Boureau, Facebook AI Research
Hendrik Strobelt, MIT-IBM Watson AI Lab at IBM Research

NeurIPS 2020 Online Experience Chairs

shown on the landing page. Events can be exported to iCal.

NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct

The NeurIPS Foundation believes in the principles of ethics, fairness, and inclusivity, and is dedicated to providing a safe space where research can be shared, reviewed, and debated by the AI / ML community.

Having observed recent discussions taking place across social media, we feel the need to reiterate that, as a community, we must be mindful of the impact that statements and actions have on our peers, and future generations of AI / ML students and researchers.

It is incumbent upon NeurIPS and the AI / ML community as a whole to foster a collaborative, welcoming environment for all. Therefore, statements and actions contrary to the NeurIPS mission and its Code of Conduct cannot and will not be tolerated. For any conference attendee failing to abide by our Code of Conduct, NeurIPS reserves its right to rescind that person’s ability to participate in any future NeurIPS-organized events.

NeurIPS’ Code of Conduct clearly outlines prohibited behaviors and subsequent corrective actions in detail. As an organization dedicated to fostering open and productive dialog among the greater AI / ML community, accusations of bullying, harassment, and discrimination are given the utmost attention and care, and such behaviors will not be tolerated.

Announcing the NeurIPS 2020 award recipients

In this blog post, we are excited to announce the various awards that are presented at NeurIPS 2020 and to share information about the selection processes for these awards.

NeurIPS 2020 Best Paper Awards

The winners of the NeurIPS 2020 Best Paper Awards are:

  • No-Regret Learning Dynamics for Extensive-Form Correlated Equilibrium by Andrea Celli (Politecnico di Milano), Alberto Marchesi (Politecnico di Milano), Gabriele Farina (Carnegie Mellon University), and Nicola Gatti (Politecnico di Milano). This paper will be presented on Tuesday, December 8th at 6:00 AM PST in the Learning Theory track.
  • Improved Guarantees and a Multiple-Descent Curve for Column Subset Selection and the Nyström Method by Michal Derezinski (UC Berkeley), Rajiv Khanna (UC Berkeley), and Michael W. Mahoney (UC Berkeley). This paper will be presented on Wednesday, Dec 9th, at 6:00 PM PST in the Learning Theory track.
  • Language Models are Few-Shot Learners by Tom B. Brown (OpenAI), Benjamin Mann (OpenAI), Nick Ryder (OpenAI), Melanie Subbiah (OpenAI), Jared D. Kaplan (Johns Hopkins University), Prafulla Dhariwal (OpenAI), Arvind Neelakantan (OpenAI), Pranav Shyam (OpenAI), Girish Sastry (OpenAI), Amanda Askell (OpenAI), Sandhini Agarwal (OpenAI), Ariel Herbert-Voss (OpenAI), Gretchen M. Krueger (OpenAI), Tom Henighan (OpenAI), Rewon Child (OpenAI), Aditya Ramesh (OpenAI), Daniel Ziegler (OpenAI), Jeffrey Wu (OpenAI), Clemens Winter (OpenAI), Chris Hesse (OpenAI), Mark Chen (OpenAI), Eric Sigler (OpenAI), Mateusz Litwin (OpenAI), Scott Gray (OpenAI), Benjamin Chess (OpenAI), Jack Clark (OpenAI), Christopher Berner (OpenAI), Sam McCandlish (OpenAI), Alec Radford (OpenAI), Ilya Sutskever (OpenAI), and Dario Amodei (OpenAI). This paper will be presented on Monday December 7th at 6:00 PM PST in the Language/Audio Applications track.

Selection process. The NeurIPS 2020 best paper awards were selected by a committee that included Nicolò Cesa-Bianchi, Jennifer Dy, Surya Ganguli, Masashi Sugiyama, and Laurens van der Maaten, who shared with us the following details about the selection process.

In selecting winning papers, the committee used the following review criteria: Does the paper have the potential to endure? Does it provide new (and hopefully deep) insights? Is it creative and unexpected? Might it change the way people think in the future? Is it rigorous and elegant but does not over-claim its significance? Is it scientific and reproducible? Does it accurately describe the broader impact of the research?

To select the winners of the NeurIPS Best Paper Awards, the award committee went through a rigorous two-stage selection process:

  • In the first stage of the process, the 30 NeurIPS submissions with the highest review scores were read by two committee members. Committee members also read the corresponding paper reviews and rebuttal. Based on this investigation, the committee selected nine papers that stood out according to the reviewing criteria.
  • In the second stage of the process, all committee members read the nine papers on the shortlist and ranked them according to the review criteria. Next, the committee met virtually to discuss the highest-ranking papers and finalize the selection of award recipients.

In particular, the committee provided the following motivation for selecting three winning papers:

  • No-Regret Learning Dynamics for Extensive-Form Correlated Equilibrium. Correlated equilibria (CE) are easy to compute and can attain a social welfare that is much higher than that of the better-known Nash equilibria. In normal form games, a surprising feature of CE is that they can be found by simple and decentralized algorithms minimizing a specific notion of regret (the so-called internal regret). This paper shows the existence of such regret-minimizing algorithms that converge to CE in a much larger class of games: namely, the extensive-form (or tree-form) games. This result solves a long-standing open problem at the interface of game theory, computer science, and economics and can have substantial impact on games that involve a mediator, for example, on efficient traffic routing via navigation apps.
  • Improved Guarantees and a Multiple-Descent Curve for Column Subset Selection and the Nyström Method. Selecting a small but representative subset of column vectors from a large matrix is a hard combinatorial problem, and a method based on cardinality-constrained determinantal point processes is known to give a practical approximate solution. This paper derives new upper and lower bounds for the approximation factor of the approximate solution over the best possible low-rank approximation, which can even capture the multiple-descent behavior with respect to the subset size. The paper further extends the analysis to obtaining guarantees for the Nyström method. Since these approximation techniques have been widely employed in machine learning, this paper is expected to have substantial impact and give new insight into, for example, kernel methods, feature selection, and the double-descent behavior of neural networks.
  • Language Models are Few-Shot Learners. Language models form the backbone of modern techniques for solving a range of problems in natural language processing. The paper shows that when such language models are scaled up to an unprecedented number of parameters, the language model itself can be used as a few-shot learner that achieves very competitive performance on many of these problems without any additional training. This is a very surprising result that is expected to have substantial impact in the field, and that is likely to withstand the test of time. In addition to the scientific contribution of the work, the paper also presents a very extensive and thoughtful exposition of the broader impact of the work, which may serve as an example to the NeurIPS community on how to think about the real-world impact of the research performed by the community.

Test of Time Award

We also continued the tradition of selecting a paper published about a decade ago at NeurIPS and that was deemed to have had a particularly significant and lasting impact on our community. We are delighted to announce that the winner of the NeurIPS 2020 test of time award is HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent published in NeurIPS 2011 and authored by Feng Niu, Benjamin Recht, Christopher Re, and Stephen Wright.

This paper was the first to show how to parallelize the ubiquitously used Stochastic Gradient Descent algorithm without any locking mechanism while achieving strong performance guarantees. At the time, several researchers proposed ways to parallelize SGD, but they all required memory locking and synchronization across the different workers. This paper proposed a simple strategy for sparse problems called Hogwild!: have each worker concurrently run SGD on a different subset of the data and perform fully asynchronous updates in the shared memory hosting the parameters of the model. Through both theory and experiments, they demonstrated that Hogwild! achieves a near linear speedup with the number of processors on data satisfying appropriate sparsity conditions.

You can find more about the paper and its impact by attending the Test of Time talk on Wednesday December 9th at 6:00 AM PST in the Optimization track.

Selection process. We identified a list of 12 papers published at NeurIPS about a decade ago (NeurIPS 2009, NeurIPS 2010, NeurIPS 2011). These were the papers from these NeurIPS editions with the highest numbers of citations since their publication. We also collected data about the recent citations counts for each of these papers by aggregating citations that these papers received in the past two years at NeurIPS, ICML and ICLR. We then asked the whole senior program committee (64 SACs) to vote on up to three of these papers to help us pick an impactful paper about which the whole senior program committee was enthusiastic.

Reviewer Awards

Finally, but equally importantly, we again selected reviewer award winners. We selected the top 10% of reviewers, that is 730 reviewers, to receive this award. We made the selection based on the average rating of reviews they entered in the system (where the ratings were provided by the area chairs). We thank all these reviewers for their outstanding work and as a small token of appreciation they were given free registration.

Congratulations to all awardees for their great research or service contribution to our thriving community!

Hsuan-Tien Lin, Maria Florina Balcan, Raia Hadsell and Marc’Aurelio Ranzato

NeurIPS 2020 Program Chairs

Communicating with our Community and the World for NeurIPS 2020

The conference is here and we’re very excited to bring you a fantastic experience! For the past couple of years the conference has started to think more actively about how it can communicate both with the public and with the attendee community, we wanted to bring you a quick peek into how those efforts have evolved.

One of our major efforts this year has been this very blog! We hope it’s giving you all a chance to see into the workings of the organizing committee and to better understand the decisions that they made and how they got made.

This is the first year that our vetted pool of journalists who cover the topic of machine learning and artificial intelligence professionally will be able to attend all parts of the conference including the tutorials and the workshops. We’re very excited about the openness of the conference and our ability to communicate the reality of the research within our own community and to the wider world.

For the second year in a row, NeurIPS is contracting with a Public Relations firm. This year, our partner is the firm Interprose. A typical goal of PR firms is to help an organization build a brand and visibility with the public. That’s not NeurIPS’s goal. In the case of the conference, public visibility came suddenly and somewhat unexpectedly. Our reason for working with a professional PR firm is that they have experience monitoring and managing crises — situations in which the conference has attracted negative attention perhaps due to a policy error or a mistake in judgment or even a simple misunderstanding. NeurIPS’s mission is to support the broader research community understanding “neural information processing” — the question of how simple computational units go about representing and transforming information. At some level, this topic is esoteric and academic. But, as we know, information processing is at the center of many core societal activities and neural networks have grown dramatically in their inference. The conference is continually making efforts to improve how it serves the research community and its broader impacts. We’re very happy to be working closely with Interprose on this important objective.

Will also be hosting a Town Hall for the third year in a row on Thursday of the week of the conference. This is a great opportunity for the NeurIPS community to talk about the conference itself: how we can change and how we can be better. Throughout the week, we’d love to hear your questions. Please email them to Townhall@NeurIPS.cc. During the Town Hall, we will take questions that we received earlier and will also take live. We will hold two sessions of the Town Hall — one at 4 AM PST and one at 4 PM PST in the attempt to make it as accessible as possible for all attendees, globally.

For the first time, our program chairs will be communicating with the attendees throughout the week — look for updates and other bulletins as the conference unfolds.

We are looking forward to an amazing week of communication and collaboration and discovery and we can’t wait to see you all, virtually at least, at this year’s NeurIPS Conference.

Neural Information Processing Systems Conference

Tweets sent to this account are not actively monitored. To contact us please go to http://neurips.cc/Help/Contact

Katherine Gorman, Collective Next

Michael Littman, Brown University