The Open Review-Based (ORB) dataset: Towards Automatic Assessment of
  Scientific Papers and Experiment Proposals in High-Energy Physics
Jaroslaw Szumega Lamine Bougueroua Blerina Gkotse Pierre Jouvelot Federico Ravotti
With the Open Science approach becoming important for research, the evolution
towards open scientific-paper reviews is making an impact on the scientific
community. However, there is a lack of publicly available resources for
conducting research activities related to this subject, as only a limited
number of journals and conferences currently allow access to their review
process for interested parties. In this paper, we introduce the new
comprehensive Open Review-Based dataset (ORB); it includes a curated list of
more than 36,000 scientific papers with their more than 89,000 reviews and
final decisions. We gather this information from two sources: the
OpenReview.net and SciPost.org websites. However, given the volatile nature of
this domain, the software infrastructure that we introduce to supplement the
ORB dataset is designed to accommodate additional resources in the future. The
ORB deliverables include (1) Python code (interfaces and implementations) to
translate document data and metadata into a structured and high-level
representation, (2) an ETL process (Extract, Transform, Load) to facilitate the
automatic updates from defined sources and (3) data files representing the
structured data. The paper presents our data architecture and an overview of
the collected data along with relevant statistics. For illustration purposes,
we also discuss preliminary Natural-Language-Processing-based experiments that
aim to predict (1) papers' acceptance based on their textual embeddings, and
(2) grading statistics inferred from embeddings as well. We believe ORB
provides a valuable resource for researchers interested in open science and
review, with our implementation easing the use of this data for further
analysis and experimentation. We plan to update ORB as the field matures as
well as introduce new resources even more fitted to dedicated scientific
domains such as High-Energy Physics.