Real-Time Onboard Object Detection for Augmented Reality: Enhancing
  Head-Mounted Display with YOLOv8
Mikołaj Łysakowski Kamil Żywanowski Adam Banaszczyk Michał R. Nowicki Piotr Skrzypczyński Sławomir K. Tadeja
This paper introduces a software architecture for real-time object detection
using machine learning (ML) in an augmented reality (AR) environment. Our
approach uses the recent state-of-the-art YOLOv8 network that runs onboard on
the Microsoft HoloLens 2 head-mounted display (HMD). The primary motivation
behind this research is to enable the application of advanced ML models for
enhanced perception and situational awareness with a wearable, hands-free AR
platform. We show the image processing pipeline for the YOLOv8 model and the
techniques used to make it real-time on the resource-limited edge computing
platform of the headset. The experimental results demonstrate that our solution
achieves real-time processing without needing offloading tasks to the cloud or
any other external servers while retaining satisfactory accuracy regarding the
usual mAP metric and measured qualitative performance