Best dataset for highest DocVQA
Hi, just came from thread: Best option for DocQVA->JSON in HuggingFaceTB/SmolVLM-Instruct. So, what would be the best dataset for this purpose. I am looking forward to train this model to be able to work on pdf files with very complex and intricate tables, data will be highly unstructured and unpredictable. Is there a dataset can be used to train a model to perform well for the purpose?
My idea is to first turn pdf to image so OCRing can be possible then run these images through the model to extract data in json or markdown format.
Hello @supercharge19 ,
I initiated the discussion on DocVQA -> JSON.
I am still working on this use-case. I have not progressed far yet however there has been release of datasets that could be interesting for this use-case:
- https://huggingface.co/datasets/OpenGVLab/InternVL-Data - Soon InternVL3 data too.
- https://huggingface.co/datasets/allenai/olmOCR-mix-0225 - A good set of documents for OCR fine-tuning, however the response format is " natural reading order" not a JSON format.
Did you progress on your side?