Student Assessment in Cybersecurity Training Automated by Pattern Mining
  and Clustering
Valdemar Švábenský Jan Vykopal Pavel Čeleda Kristián Tkáčik Daniel Popovič
Hands-on cybersecurity training allows students and professionals to practice
various tools and improve their technical skills. The training occurs in an
interactive learning environment that enables completing sophisticated tasks in
full-fledged operating systems, networks, and applications. During the
training, the learning environment allows collecting data about trainees'
interactions with the environment, such as their usage of command-line tools.
These data contain patterns indicative of trainees' learning processes, and
revealing them allows to assess the trainees and provide feedback to help them
learn. However, automated analysis of these data is challenging. The training
tasks feature complex problem-solving, and many different solution approaches
are possible. Moreover, the trainees generate vast amounts of interaction data.
This paper explores a dataset from 18 cybersecurity training sessions using
data mining and machine learning techniques. We employed pattern mining and
clustering to analyze 8834 commands collected from 113 trainees, revealing
their typical behavior, mistakes, solution strategies, and difficult training
stages. Pattern mining proved suitable in capturing timing information and tool
usage frequency. Clustering underlined that many trainees often face the same
issues, which can be addressed by targeted scaffolding. Our results show that
data mining methods are suitable for analyzing cybersecurity training data.
Educational researchers and practitioners can apply these methods in their
contexts to assess trainees, support them, and improve the training design.
Artifacts associated with this research are publicly available.