Reflections on the NeurIPS 2023 Ethics Review Process

By the NeurIPS 2023 Ethics Review Co-Chairs

The NeurIPS 2023 Ethic Review process began with the publication of the Code of Ethics. This step formally codified a foundation for ethics within the conference framework.

The Ethics Guidelines for Reviewers were also updated to reflect incorporation of the Code of Ethics. 

This year, 502 papers (3.77% of all submissions) were flagged for ethics review. This represents an increase in the number of papers flagged last year (474) but a decrease in the overall rate of papers being flagged (from 4.37% in 2022).

In particular, we are heartened that fewer submissions in the Datasets and Benchmarks track were flagged for ethics review, despite the number of submissions more than doubling since last year. We believe that these trends mean that the ethics of ML research is being taken more seriously by the research community.

	2023			2022
Paper statistics (preliminary)	Main track	D&B track	Total	Main track	D&B track	Total
Papers submitted	12,345	976	13,321	10,411	447	10,858
Total papers flagged for ethics review	395	107	502	362	112	474
Flag rate	3.20%	10.96%	3.77%	3.48%	25.06%	4.37%

This year, reviewers in the main track were required to flag at least one specific area for ethics review. Note that papers can be flagged for multiple areas, so the percentages below do not add up to 100%.

Notably, discrimination, bias and fairness concerns have been flagged at nearly triple the rate this year compared to last year; the rate of papers flagged for legal compliance or responsible research practice also doubled.  

Papers flagged by area (main track only)	2023		2022
Discrimination / bias / fairness concerns	122	33.7%	47	13.0%
Privacy and security	91	25.1%	63	17.4%
Legal compliance	39	10.8%	17	4.7%
Inadequate data and algorithm evaluation	83	22.9%	35	9.7%
Responsible research practice	61	16.9%	30	8.3%
Inappropriate potential applications and Impact	76	21.0%	47	13.0%
Research integrity issues	63	17.4%	56	15.5%
Failure to comply with NeurIPS Code of Ethics	53	14.6%	New in 2023
I don’t know	Deprecated		38	10.5%
No area flagged	Deprecated		80	22.1%
Total	395		362

This year, we invited everyone contacted for ethics reviews from last year to serve again as ethics reviewers, and also issued an open call for new reviewers. A total of 396 ethics reviewers answered the call.

In addition, we also asked reviewers if they were available as emergency ethics reviewers, to provide missing reviews and reviews for papers flagged for review on short notice. 76 people volunteered as emergency ethics reviewers. 

We are immensely grateful to everyone who answered the call to service, which allowed us to handle the increased workload for ethics reviews, while broadening the community and reducing the average number of reviews per reviewer.

Thank you all for a wonderful and successful ethical review process.

Yours truly,

Jiahao Chen, Lester Mackey, and Cherie Poland
NeurIPS 2023 Ethics Review Co-Chairs