Leveraging Image-Text Similarity and Caption Modification for the
  DataComp Challenge: Filtering Track and BYOD Track
Shuhei Yokoo Peifei Zhu Yuchi Ishikawa Mikihiro Tanaka Masayoshi Kondo Hirokatsu Kataoka
Large web crawl datasets have already played an important role in learning
multimodal features with high generalization capabilities. However, there are
still very limited studies investigating the details or improvements of data
design. Recently, a DataComp challenge has been designed to propose the best
training data with the fixed models. This paper presents our solution to both
filtering track and BYOD track of the DataComp challenge. Our solution adopts
large multimodal models CLIP and BLIP-2 to filter and modify web crawl data,
and utilize external datasets along with a bag of tricks to improve the data
quality. Experiments show our solution significantly outperforms DataComp
baselines (filtering track: 6.6% improvement, BYOD track: 48.5% improvement).