KnowPhish: Large Language Models Meet Multimodal Knowledge Graphs for
  Enhancing Reference-Based Phishing Detection
Yuexin Li Chengyu Huang Shumin Deng Mei Lin Lock Tri Cao Nay Oo Bryan Hooi Hoon Wei Lim
Phishing attacks have inflicted substantial losses on individuals and
businesses alike, necessitating the development of robust and efficient
automated phishing detection approaches. Reference-based phishing detectors
(RBPDs), which compare the logos on a target webpage to a known set of logos,
have emerged as the state-of-the-art approach. However, a major limitation of
existing RBPDs is that they rely on a manually constructed brand knowledge
base, making it infeasible to scale to a large number of brands, which results
in false negative errors due to the insufficient brand coverage of the
knowledge base. To address this issue, we propose an automated knowledge
collection pipeline, using which we collect and release a large-scale
multimodal brand knowledge base, KnowPhish, containing 20k brands with rich
information about each brand. KnowPhish can be used to boost the performance of
existing RBPDs in a plug-and-play manner. A second limitation of existing RBPDs
is that they solely rely on the image modality, ignoring useful textual
information present in the webpage HTML. To utilize this textual information,
we propose a Large Language Model (LLM)-based approach to extract brand
information of webpages from text. Our resulting multimodal phishing detection
approach, KnowPhish Detector (KPD), can detect phishing webpages with or
without logos. We evaluate KnowPhish and KPD on a manually validated dataset,
and on a field study under Singapore's local context, showing substantial
improvements in effectiveness and efficiency compared to state-of-the-art
baselines.