BotHawk: An Approach for Bots Detection in Open Source Software Projects
Fenglin Bi Zhiwei Zhu Wei Wang Xiaoya Xia Hassan Ali Khan Peng Pu
Social coding platforms have revolutionized collaboration in software
development, leading to using software bots for streamlining operations.
However, The presence of open-source software (OSS) bots gives rise to problems
including impersonation, spamming, bias, and security risks. Identifying bot
accounts and behavior is a challenging task in the OSS project. This research
aims to investigate bots' behavior in open-source software projects and
identify bot accounts with maximum possible accuracy. Our team gathered a
dataset of 19,779 accounts that meet standardized criteria to enable future
research on bots in open-source projects. We follow a rigorous workflow to
ensure that the data we collect is accurate, generalizable, scalable, and
up-to-date. We've identified four types of bot accounts in open-source software
projects by analyzing their behavior across 17 features in 5 dimensions. Our
team created BotHawk, a highly effective model for detecting bots in
open-source software projects. It outperforms other models, achieving an AUC of
0.947 and an F1-score of 0.89. BotHawk can detect a wider variety of bots,
including CI/CD and scanning bots. Furthermore, we find that the number of
followers, number of repositories, and tags contain the most relevant features
to identify the account type.