--- license: cc-by-nc-sa-4.0 datasets: - PengxiangLi/MAT language: - en metrics: - accuracy base_model: - openbmb/MiniCPM-V-2_6 pipeline_tag: visual-question-answering --- --- pipeline_tag: image-text-to-text datasets: - openbmb/RLAIF-V-Dataset library_name: transformers language: - multilingual tags: - minicpm-v - vision - ocr - multi-image - video - custom_code ---

Multi-modal Agent Tuning: Building a VLM-Driven Agent for Efficient Tool Usage

[GitHub](https://github.com/mat-agent/MAT-Agent.git) | [Project](https://mat-agent.github.io/) ## MAT-MiniCPM-V 2.6 This model is a fine-tuned version of the [MiniCPM V2.6 7B](https://huggingface.co/openbmb/MiniCPM-V-2_6) model on the MM-traj dataset. On GTA and GAIA benchmarks, it achieved improvements of ​18.59% and ​7.78%​ respectively compared to the non-fine-tuned baseline. ## Usage Our model inherits the inference architecture from [MiniCPM-V-2.6](https://huggingface.co/openbmb/MiniCPM-V-2_6). The following implementation is adapted from their original inference code with full compatibility. Requirements (tested on Python 3.10): ``` Pillow==10.1.0 torch==2.1.2 torchvision==0.16.2 transformers==4.40.0 sentencepiece==0.1.99 decord ``` ### Basic Inference ```python # test.py import torch from PIL import Image from transformers import AutoModel, AutoTokenizer # Load our fine-tuned model (based on MiniCPM-V-2.6 architecture) model = AutoModel.from_pretrained('PengxiangLi/MAT', trust_remote_code=True, attn_implementation='sdpa', torch_dtype=torch.bfloat16) # Maintain original implementation choices model = model.eval().cuda() tokenizer = AutoTokenizer.from_pretrained('PengxiangLi/MAT', trust_remote_code=True) image = Image.open('xx.jpg').convert('RGB') question = 'What is in the image?' msgs = [{'role': 'user', 'content': [image, question]}] # The chat interface follows MiniCPM's original implementation response = model.chat( image=None, msgs=msgs, tokenizer=tokenizer ) print(response) ## Streaming output (inherited from MiniCPM's implementation) response_stream = model.chat( image=None, msgs=msgs, tokenizer=tokenizer, sampling=True, stream=True ) generated_text = "" for new_text in response_stream: generated_text += new_text print(new_text, flush=True, end='') ``` ### Multi-image Chat
Implementation adapted from MiniCPM's original multi-image handling ```python import torch from PIL import Image from transformers import AutoModel, AutoTokenizer model = AutoModel.from_pretrained('PengxiangLi/MAT', trust_remote_code=True, attn_implementation='sdpa', torch_dtype=torch.bfloat16) model = model.eval().cuda() tokenizer = AutoTokenizer.from_pretrained('PengxiangLi/MAT', trust_remote_code=True) # The message format follows MiniCPM's original schema image1 = Image.open('image1.jpg').convert('RGB') image2 = Image.open('image2.jpg').convert('RGB') question = 'Compare the two images...' msgs = [{'role': 'user', 'content': [image1, image2, question]}] # Using the original chat interface design answer = model.chat( image=None, msgs=msgs, tokenizer=tokenizer ) print(answer) ```
### Few-shot Learning
Adapted from MiniCPM's few-shot implementation ```python import torch from PIL import Image from transformers import AutoModel, AutoTokenizer # Maintain original model loading parameters model = AutoModel.from_pretrained('PengxiangLi/MAT', trust_remote_code=True, attn_implementation='sdpa', torch_dtype=torch.bfloat16) model = model.eval().cuda() tokenizer = AutoTokenizer.from_pretrained('PengxiangLi/MAT', trust_remote_code=True) # Following MiniCPM's message structure question = "production date" image1 = Image.open('example1.jpg').convert('RGB') answer1 = "2023.08.04" image2 = Image.open('example2.jpg').convert('RGB') answer2 = "2007.04.24" image_test = Image.open('test.jpg').convert('RGB') msgs = [ {'role': 'user', 'content': [image1, question]}, {'role': 'assistant', 'content': [answer1]}, {'role': 'user', 'content': [image2, question]}, {'role': 'assistant', 'content': [answer2]}, {'role': 'user', 'content': [image_test, question]} ] # Using the unmodified chat interface from original implementation answer = model.chat( image=None, msgs=msgs, tokenizer=tokenizer ) print(answer) ```
#### Implementation Notes: 1. All core inference logic is directly inherited from [MiniCPM-V-2.6](https://huggingface.co/openbmb/MiniCPM-V-2_6) 2. The `chat()` interface remains unchanged from the original implementation 3. Model loading parameters maintain compatibility with the base architecture 4. Message formatting follows MiniCPM's original schema ## License #### Model License - The code in this repository is licensed under the ​[Apache-2.0 License](https://github.com/OpenBMB/MiniCPM/blob/main/LICENSE). - Usage of our fine-tuned MiniCPM-based model weights must strictly adhere to the ​[MiniCPM Model License](https://github.com/OpenBMB/MiniCPM/blob/main/MiniCPM%20Model%20License.md). #### Usage Terms - ​**Academic Research**: The model weights are freely available for academic use without restrictions. - ​**Commercial Use**: - After completing the official ​[registration questionnaire](https://modelbest.feishu.cn/share/base/form/shrcnpV5ZT9EJ6xYjh3Kx0J6v8g)​ and obtaining authorization, the MiniCPM-V 2.6 based weights (including our fine-tuned version) are available for commercial use free of charge. - Commercial users must maintain compliance with all terms outlined in the MiniCPM Model License. #### Inheritance Clause As a derivative work of MiniCPM, our model inherits and is bound by all original licensing requirements from the base model. Users are responsible for ensuring compliance with both our terms and the upstream MiniCPM license terms. ## Citation If you find our work helpful, please consider citing our papers 📝 and liking this project ❤️！ ```bib @article{gao2024multi, title={Multi-modal Agent Tuning: Building a VLM-Driven Agent for Efficient Tool Usage}, author={Gao, Zhi and Zhang, Bofei and Li, Pengxiang and Ma, Xiaojian and Yuan, Tao and Fan, Yue and Wu, Yuwei and Jia, Yunde and Zhu, Song-Chun and Li, Qing}, journal={arXiv preprint arXiv:2412.15606}, year={2024} } ```