Ultra Fast Transformers on FPGAs for Particle Physics Experiments
Zhixing Jiang Dennis Yin Elham E Khoda Vladimir Loncar Ekaterina Govorkova Eric Moreno Philip Harris Scott Hauck Shih-Chieh Hsu
This work introduces a highly efficient implementation of the transformer
architecture on a Field-Programmable Gate Array (FPGA) by using the
\texttt{hls4ml} tool. Given the demonstrated effectiveness of transformer
models in addressing a wide range of problems, their application in
experimental triggers within particle physics becomes a subject of significant
interest. In this work, we have implemented critical components of a
transformer model, such as multi-head attention and softmax layers. To evaluate
the effectiveness of our implementation, we have focused on a particle physics
jet flavor tagging problem, employing a public dataset. We recorded latency
under 2 $\mu$s on the Xilinx UltraScale+ FPGA, which is compatible with
hardware trigger requirements at the CERN Large Hadron Collider experiments.