VisualWebInstruct: Scaling up Multimodal Instruction Data through Web Search
Abstract
Vision-Language Models have made significant progress on many perception-focused tasks, however, their progress on reasoning-focused tasks seem to be limited due to the lack of high-quality and diverse training data. In this work, we aim to address the scarcity issue of reasoning-focused multimodal datasets. We propose VisualWebInstruct - a novel approach that leverages search engine to create a diverse, and high-quality dataset spanning multiple disciplines like math, physics, finance, chemistry, etc. Starting with meticulously selected 30,000 seed images, we employ Google Image search to identify websites containing similar images. We collect and process the HTMLs from over 700K unique URL sources. Through a pipeline of content extraction, filtering and synthesis, we build a dataset of approximately 900K question-answer pairs, with 40% being visual QA pairs and the rest as text QA pairs. Models fine-tuned on VisualWebInstruct demonstrate significant performance gains: (1) training from Llava-OV-mid shows 10-20% absolute point gains across benchmarks, (2) training from MAmmoTH-VL shows 5% absoluate gain. Our best model MAmmoTH-VL2 shows state-of-the-art performance within the 10B parameter class on MMMU-Pro-std (40.7%), MathVerse (42.6%), and DynaMath (55.7%). These remarkable results highlight the effectiveness of our dataset in enhancing VLMs' reasoning capabilities for complex multimodal tasks.
Community
We propose an approach to automatically scale up the multimodal instruction tuning dataset. We obtain state-of-the-art performance across many multimodal reasoning tasks.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- MMSciBench: Benchmarking Language Models on Multimodal Scientific Problems (2025)
- Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Models (2025)
- Scaling Text-Rich Image Understanding via Code-Guided Synthetic Multimodal Data Generation (2025)
- MV-MATH: Evaluating Multimodal Math Reasoning in Multi-Visual Contexts (2025)
- Does Table Source Matter? Benchmarking and Improving Multimodal Scientific Table Understanding and Reasoning (2025)
- Visual Reasoning Evaluation of Grok, Deepseek Janus, Gemini, Qwen, Mistral, and ChatGPT (2025)
- VisCon-100K: Leveraging Contextual Web Data for Fine-tuning Vision Language Models (2025)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
Models citing this paper 1
Datasets citing this paper 3
Spaces citing this paper 0
No Space linking this paper