Unveiling the Sentinels: Assessing AI Performance in Cybersecurity Peer
  Review
Liang Niu Nian Xue Christina Pöpper
Peer review is the method employed by the scientific community for evaluating
research advancements. In the field of cybersecurity, the practice of
double-blind peer review is the de-facto standard. This paper touches on the
holy grail of peer reviewing and aims to shed light on the performance of AI in
reviewing for academic security conferences. Specifically, we investigate the
predictability of reviewing outcomes by comparing the results obtained from
human reviewers and machine-learning models. To facilitate our study, we
construct a comprehensive dataset by collecting thousands of papers from
renowned computer science conferences and the arXiv preprint website. Based on
the collected data, we evaluate the prediction capabilities of ChatGPT and a
two-stage classification approach based on the Doc2Vec model with various
classifiers. Our experimental evaluation of review outcome prediction using the
Doc2Vec-based approach performs significantly better than the ChatGPT and
achieves an accuracy of over 90%. While analyzing the experimental results, we
identify the potential advantages and limitations of the tested ML models. We
explore areas within the paper-reviewing process that can benefit from
automated support approaches, while also recognizing the irreplaceable role of
human intellect in certain aspects that cannot be matched by state-of-the-art
AI techniques.