Benchmarking and Analyzing Generative Data for Visual Recognition
Bo Li Haotian Liu Liangyu Chen Yong Jae Lee Chunyuan Li Ziwei Liu
Advancements in large pre-trained generative models have expanded their
potential as effective data generators in visual recognition. This work delves
into the impact of generative images, primarily comparing paradigms that
harness external data (\ie generative \vs retrieval \vs original).
  Our key contributions are: \textbf{1) GenBench Construction:} We devise
\textbf{GenBench}, a broad benchmark comprising 22 datasets with 2548
categories, to appraise generative data across various visual recognition
tasks. \textbf{2) CLER Score:} To address the insufficient correlation of
existing metrics (\eg, FID, CLIP score) with downstream recognition
performance, we propose \textbf{CLER}, a training-free metric indicating
generative data's efficiency for recognition tasks prior to training.
\textbf{3) New Baselines:} Comparisons of generative data with retrieved data
from the same external pool help to elucidate the unique traits of generative
data. \textbf{4) External Knowledge Injection:} By fine-tuning special token
embeddings for each category via Textual Inversion, performance improves across
17 datasets, except when dealing with low-resolution reference images.
  Our exhaustive benchmark and analysis spotlight generative data's promise in
visual recognition, while identifying key challenges for future investigation.