---
license: apache-2.0
datasets:
- Emova-ollm/emova-alignment-7m
- Emova-ollm/emova-sft-4m
- Emova-ollm/emova-sft-speech-231k
language:
- en
- zh
base_model:
- Emova-ollm/qwen2vit600m
- Emova-ollm/Qwen2.5-72B-Instruct_add_speech_token_4096_nostrip
new_version: Emova-ollm/emova-qwen-2-5-72b-hf
library_name: transformers
tags:
- Omni-modal-LLM
- Multi-modal-LLM
- Emotional-spoken-dialogue
model-index:
- name: emova-qwen-2-5-72b
results:
- task:
type: multimodal
dataset:
name: AI2D
type: ai2d
metrics:
- type: accuracy
value: 85.8
name: accuracy
verified: true
- task:
type: multimodal
dataset:
name: ChartQA
type: chartqa
metrics:
- type: accuracy
value: 88.7
name: accuracy
verified: true
- task:
type: multimodal
dataset:
name: DocVQA
type: docvqa
metrics:
- type: accuracy
value: 95.9
name: accuracy
verified: true
- task:
type: multimodal
dataset:
name: InfoVQA
type: infovqa
metrics:
- type: accuracy
value: 83.2
name: accuracy
verified: true
- task:
type: multimodal
dataset:
name: MathVerse
type: mathverse
metrics:
- type: accuracy
value: 50.0
name: accuracy
verified: true
- task:
type: multimodal
dataset:
name: MathVista
type: mathvista
metrics:
- type: accuracy
value: 69.9
name: accuracy
verified: true
- task:
type: multimodal
dataset:
name: MMBench
type: mmbench
metrics:
- type: accuracy
value: 86.4
name: accuracy
verified: true
- task:
type: multimodal
dataset:
name: MME
type: mme
metrics:
- type: score
value: 2402
name: score
verified: true
- task:
type: multimodal
dataset:
name: MMVet
type: mmvet
metrics:
- type: accuracy
value: 64.8
name: accuracy
verified: true
- task:
type: multimodal
dataset:
name: OCRBench
type: ocrbench
metrics:
- type: accuracy
value: 843
name: accuracy
verified: true
- task:
type: multimodal
dataset:
name: RealWorldQA
type: realworldqa
metrics:
- type: accuracy
value: 71.0
name: accuracy
verified: true
- task:
type: multimodal
dataset:
name: Seed-Bench-Image
type: seed-bench-image
metrics:
- type: accuracy
value: 76.6
name: accuracy
verified: true
- task:
type: multimodal
dataset:
name: Science-QA
type: science-qa
metrics:
- type: accuracy
value: 98.2
name: accuracy
verified: true
- task:
type: multimodal
dataset:
name: TextVQA
type: textvqa
metrics:
- type: accuracy
value: 81.4
name: accuracy
verified: true
- task:
name: Automatic Speech Recognition
type: automatic-speech-recognition
dataset:
name: LibriSpeech (clean)
type: librispeech_asr
config: clean
split: test
args:
language: en
metrics:
- name: Test WER
type: wer
value: 2.9
---
# EMOVA-Qwen-2.5-72B

π€ [EMOVA-Models](https://huggingface.co/collections/Emova-ollm/emova-models-67779d377bb8261e6057a320) | π€ [EMOVA-Datasets](https://huggingface.co/collections/Emova-ollm/emova-datasets-67779be7d02447a2d0891bf6) | π€ [EMOVA-Demo](https://huggingface.co/spaces/Emova-ollm/EMOVA-demo)
π [Paper](https://arxiv.org/abs/2409.18042) | π [Project-Page](https://emova-ollm.github.io/) | π» [Github](https://github.com/emova-ollm/EMOVA) | π» [EMOVA-Speech-Tokenizer-Github](https://github.com/emova-ollm/EMOVA_speech_tokenizer)
## Model Summary
**EMOVA** (**EM**otionally **O**mni-present **V**oice **A**ssistant) is a novel end-to-end omni-modal LLM that can see, hear and speak without relying on external models. Given the omni-modal (i.e., textual, visual and speech) inputs, EMOVA can generate both textual and speech responses with vivid emotional controls by utilizing the speech decoder together with a style encoder. EMOVA possesses general omni-modal understanding and generation capabilities, featuring its superiority in advanced vision-language understanding, emotional spoken dialogue, and spoken dialogue with structural data understanding. We summarize its key advantages as:
- **State-of-the-art omni-modality performance**: EMOVA achieves state-of-the-art comparable results on both **vision-language** and **speech** benchmarks simultaneously. Our best performing model, **EMOVA-72B**, even surpasses commercial models including GPT-4o and Gemini Pro 1.5.
- **Emotional spoken dialogue**: A **semantic-acoustic disentangled** speech tokenizer and a lightweight **style control** module are adopted for seamless omni-modal alignment and diverse speech style controllability. EMOVA supports **bilingual (Chinese and English)** spoken dialogue with **24 speech style** controls (i.e., 2 speakers, 3 pitches and 4 emotions).
- **Diverse configurations**: We open-source 3 configurations, **EMOVA-3B/7B/72B**, to support omni-modal usage under different computational budgets. Check our [Model Zoo](https://huggingface.co/collections/Emova-ollm/emova-models-67779d377bb8261e6057a320) and find the best fit model for your computational devices!
## Performance
| Benchmarks | EMOVA-3B | EMOVA-7B | EMOVA-72B | GPT-4o | VITA 8x7B | VITA 1.5 | Baichuan-Omni |
|:------------------:|:-------: |:--------:|:---------:|:------:|:---------:|:--------:|:-------------:|
| **MME** | 2175 | 2317 | 2402 | 2310 | 2097 | 2311 | 2187 |
| **MMBench** | 79.2 | 83.0 | 86.4 | 83.4 | 71.8 | 76.6 | 76.2 |
| **SEED-Image** | 74.9 | 75.5 | 76.6 | 77.1 | 72.6 | 74.2 | 74.1 |
| **MM-Vet** | 57.3 | 59.4 | 64.8 | - | 41.6 | 51.1 | 65.4 |
| **RealWorldQA** | 62.6 | 67.5 | 71.0 | 75.4 | 59.0 | 66.8 | 62.6 |
| **TextVQA** | 77.2 | 78.0 | 81.4 | - | 71.8 | 74.9 | 74.3 |
| **ChartQA** | 81.5 | 84.9 | 88.7 | 85.7 | 76.6 | 79.6 | 79.6 |
| **DocVQA** | 93.5 | 94.2 | 95.9 | 92.8 | - | - | - |
| **InfoVQA** | 71.2 | 75.1 | 83.2 | - | - | - | - |
| **OCRBench** | 803 | 814 | 843 | 736 | 678 | 752 | 700 |
| **ScienceQA-Img** | 92.7 | 96.4 | 98.2 | - | - | - | - |
| **AI2D** | 78.6 | 81.7 | 85.8 | 84.6 | 73.1 | 79.3 | - |
| **MathVista** | 62.6 | 65.5 | 69.9 | 63.8 | 44.9 | 66.2 | 51.9 |
| **Mathverse** | 31.4 | 40.9 | 50.0 | - | - | - | - |
| **Librispeech (WERβ)** | 5.4 | 4.1 | 2.9 | - | 3.4 | 8.1 | - |
## Usage
This repo contains the **EMOVA-Qwen2.5-72B** checkpoint organized in the **original format** of our [EMOVA codebase](https://github.com/emova-ollm/EMOVA), and thus, it should be utilized together with EMOVA codebase. Its paired config file is provided [here](https://github.com/emova-ollm/EMOVA/blob/main/configs/example/emova/qwen2_5_qwen2vit_nativeAnyres_72b/2.finetune.py). Check [here](https://github.com/emova-ollm/EMOVA#gradio-web-demo) to launch a web demo using this checkpoint.
## Citation
```bibtex
@article{chen2024emova,
title={Emova: Empowering language models to see, hear and speak with vivid emotions},
author={Chen, Kai and Gou, Yunhao and Huang, Runhui and Liu, Zhili and Tan, Daxin and Xu, Jing and Wang, Chunwei and Zhu, Yi and Zeng, Yihan and Yang, Kuo and others},
journal={arXiv preprint arXiv:2409.18042},
year={2024}
}
```