Evaluating AI Vocational Skills Through Professional Testing
David Noever Matt Ciolino
Using a novel professional certification survey, the study focuses on
assessing the vocational skills of two highly cited AI models, GPT-3 and
Turbo-GPT3.5. The approach emphasizes the importance of practical readiness
over academic performance by examining the models' performances on a benchmark
dataset consisting of 1149 professional certifications. This study also
includes a comparison with human test scores, providing perspective on the
potential of AI models to match or even surpass human performance in
professional certifications. GPT-3, even without any fine-tuning or exam
preparation, managed to achieve a passing score (over 70% correct) on 39% of
the professional certifications. It showcased proficiency in computer-related
fields, including cloud and virtualization, business analytics, cybersecurity,
network setup and repair, and data analytics. Turbo-GPT3.5, on the other hand,
scored a perfect 100% on the highly regarded Offensive Security Certified
Professional (OSCP) exam. This model also demonstrated competency in diverse
professional fields, such as nursing, licensed counseling, pharmacy, and
aviation. Turbo-GPT3.5 exhibited strong performance on customer service tasks,
indicating potential use cases in enhancing chatbots for call centers and
routine advice services. Both models also scored well on sensory and
experience-based tests outside a machine's traditional roles, including wine
sommelier, beer tasting, emotional quotient, and body language reading. The
study found that OpenAI's model improvement from Babbage to Turbo led to a 60%
better performance on the grading scale within a few years. This progress
indicates that addressing the current model's limitations could yield an AI
capable of passing even the most rigorous professional certifications.