|
--- |
|
base_model: |
|
- meta-llama/Llama-3.2-1B-Instruct |
|
datasets: |
|
- VocalNet/VoiceAssitant-430K-vocalnet |
|
- VocalNet/UltraChat-vocalnet |
|
language: |
|
- en |
|
license: apache-2.0 |
|
pipeline_tag: audio-text-to-text |
|
library_name: transformers |
|
--- |
|
|
|
## 🎧 VocalNet-1B Model Card |
|
|
|
**VocalNet-1B** is a high-performance, low-latency speech large language model (LLM) optimized for real-time voice interaction. Built upon [LLaMA-3.2-1B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct), it employs **multi-token prediction (MTP)** to significantly enhance generation speed and quality, surpassing most mainstream speech and omni-modal LLMs. 🚀 |
|
|
|
### 📂 Paper, Code and Model Access |
|
- **Arxiv**: [VocalNet Report](https://arxiv.org/abs/2504.04060) 📖 |
|
- **GitHub**: [VocalNet Repository](https://github.com/SJTU-OmniAgent/VocalNet) 🌐 |
|
- **HuggingFace**: [VocalNet/VocalNet-1B](https://huggingface.co/VocalNet/VocalNet-1B) 🤗 |
|
- **ModelScope**: [VocalNet/VocalNet-1B](https://www.modelscope.cn/models/VocalNet/VocalNet-1B) 🔮 |
|
|
|
### 🔧 Repository Download and Environment Setup |
|
|
|
To get started with **VocalNet-1B**, clone the repository and set up the environment as follows. 🛠️ |
|
|
|
1. **Clone the Repository**: |
|
```bash |
|
git clone https://github.com/SJTU-OmniAgent/VocalNet.git |
|
cd VocalNet |
|
``` |
|
|
|
2. **Create and Activate Environment**: |
|
```bash |
|
conda create -n vocalnet python==3.10 |
|
conda activate vocalnet |
|
``` |
|
|
|
3. **Install Dependencies**: |
|
```bash |
|
pip install --upgrade pip |
|
conda install pytorch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 pytorch-cuda=12.1 -c pytorch -c nvidia |
|
pip install -e . |
|
``` |
|
|
|
4. **Optional: Install Training Packages**: |
|
If you plan to train the model, install additional packages: |
|
```bash |
|
pip install -e ".[train]" |
|
pip install flash-attn --no-build-isolation |
|
``` |
|
|
|
### 📥 Download Instructions |
|
|
|
**Via Huggingface Cli**: |
|
```bash |
|
pip install -U huggingface_hub |
|
huggingface-cli download VocalNet/VocalNet-1B --local-dir ./checkpoints/ |
|
``` |
|
**Via Snapshot Download**: |
|
```bash |
|
pip install -U huggingface_hub |
|
``` |
|
```python |
|
from huggingface_hub import snapshot_download |
|
snapshot_download( |
|
repo_id="VocalNet/VocalNet-1B", |
|
local_dir="./checkpoints/", |
|
resume_download=True |
|
) |
|
``` |
|
**Via Git**: |
|
```bash |
|
git lfs install |
|
git clone https://huggingface.co/VocalNet/VocalNet-1B |
|
``` |
|
|
|
### 🛠️ Dependencies |
|
- **Speech Encoder**: [Whisper-large-v3](https://huggingface.co/openai/whisper-large-v3) 🎤 |
|
- **Vocoder**: [CosyVoice2-0.5B](https://huggingface.co/FunAudioLLM/CosyVoice2-0.5B) for converting speech tokens to audio waveforms. 🔊 |
|
|
|
### 🔄 Local Inference |
|
|
|
To perform inference with **VocalNet-1B**, follow these steps to set up and run the model locally. 📡 |
|
|
|
1. **Model Preparation**: |
|
- Download **VocalNet-1B** from [HuggingFace](https://huggingface.co/VocalNet/VocalNet-1B) or [ModelScope](https://www.modelscope.cn/models/VocalNet/VocalNet-1B). 📦 |
|
- Download the **Whisper-large-v3** speech encoder from [HuggingFace](https://huggingface.co/openai/whisper-large-v3) and place it in the `./models/speech_encoder/` directory. 🎤 |
|
|
|
2. **CosyVoice Preparation**: |
|
- VocalNet-1B uses **CosyVoice2-0.5B** to convert generated speech tokens into audio waveforms. Download it from [HuggingFace](https://huggingface.co/FunAudioLLM/CosyVoice2-0.5B). 🔊 |
|
|
|
3. **Path Modification**: |
|
- Update the paths in `omni_speech/infer/vocalnet.py` to point to the downloaded models: |
|
```python |
|
COSYVOICE_MODEL="" # Path to CosyVoice2-0.5B, e.g., /workspace/CosyVoice/pretrained_models/CosyVoice2-0.5B-VocalNet |
|
VOCALNET_MODEL="" # Path to VocalNet-1B, e.g., ./checkpoints/VocalNet-1B |
|
``` |
|
|
|
4. **Run Inference**: |
|
- For **speech-to-text (S2T)** inference: |
|
```bash |
|
python3 omni_speech/infer/vocalnet.py --query_audio ./omni_speech/infer/llama_questions_42.wav |
|
``` |
|
- For **speech-to-speech (S2S)** inference: |
|
```bash |
|
python3 omni_speech/infer/vocalnet.py --query_audio ./omni_speech/infer/llama_questions_42.wav --s2s --save_dir ./ |
|
``` |
|
|
|
### 📊 Performance Evaluation |
|
VocalNet-1B was evaluated on [OpenAudioBench](https://huggingface.co/datasets/baichuan-inc/OpenAudioBench), covering AlpacaEval, LLaMA Questions, TriviaQA, and Web Questions. **Bold** indicates the optimal result in each subgroup. |
|
|
|
#### Overall Performance |
|
<div align="center"> |
|
<table style="margin: 0 auto; text-align: center; border-collapse: collapse; font-size: 14px;"> |
|
<thead> |
|
<tr style="background-color: #f2f2f2;"> |
|
<th style="padding: 10px; border: 1px solid #ddd;">Model</th> |
|
<th style="padding: 10px; border: 1px solid #ddd;">LLM Size</th> |
|
<th style="padding: 10px; border: 1px solid #ddd;">Modality</th> |
|
<th style="padding: 10px; border: 1px solid #ddd;">AlpacaEval</th> |
|
<th style="padding: 10px; border: 1px solid #ddd;">LLaMA Questions</th> |
|
<th style="padding: 10px; border: 1px solid #ddd;">TriviaQA</th> |
|
<th style="padding: 10px; border: 1px solid #ddd;">Web Questions</th> |
|
</tr> |
|
</thead> |
|
<tbody> |
|
<tr> |
|
<td colspan="7" style="padding: 10px; border: 1px solid #ddd; font-weight: bold; background-color: #e6f3ff;">Tiny Models</td> |
|
</tr> |
|
<tr> |
|
<td rowspan="2" style="padding: 10px; border: 1px solid #ddd;">Mini-Omni</td> |
|
<td rowspan="2" style="padding: 10px; border: 1px solid #ddd;">0.5B</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">s→t</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">1.84</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">2.7</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">0.12</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">0.22</td> |
|
</tr> |
|
<tr> |
|
<td style="padding: 10px; border: 1px solid #ddd;">s→s</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">1.80</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">2.7</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">0.08</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">0.20</td> |
|
</tr> |
|
<tr> |
|
<td rowspan="2" style="padding: 10px; border: 1px solid #ddd;">SLAM-Omni</td> |
|
<td rowspan="2" style="padding: 10px; border: 1px solid #ddd;">0.5B</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">s→t</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">3.50</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">29.4</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">0.39</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">0.84</td> |
|
</tr> |
|
<tr> |
|
<td style="padding: 10px; border: 1px solid #ddd;">s→s</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">3.01</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">26.7</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">0.34</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">0.69</td> |
|
</tr> |
|
<tr> |
|
<td rowspan="2" style="padding: 10px; border: 1px solid #ddd;">VocalNet-1B (VA)</td> |
|
<td rowspan="2" style="padding: 10px; border: 1px solid #ddd;">1B</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">s→t</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">5.38</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">70.3</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">3.38</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">4.93</td> |
|
</tr> |
|
<tr> |
|
<td style="padding: 10px; border: 1px solid #ddd;">s→s</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">4.83</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">61.0</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">2.78</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">4.47</td> |
|
</tr> |
|
<tr> |
|
<td rowspan="2" style="padding: 10px; border: 1px solid #ddd;">VocalNet-1B</td> |
|
<td rowspan="2" style="padding: 10px; border: 1px solid #ddd;">1B</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">s→t</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;"><b>5.79</b></td> |
|
<td style="padding: 10px; border: 1px solid #ddd;"><b>71.7</b></td> |
|
<td style="padding: 10px; border: 1px solid #ddd;"><b>3.60</b></td> |
|
<td style="padding: 10px; border: 1px solid #ddd;"><b>5.16</b></td> |
|
</tr> |
|
<tr> |
|
<td style="padding: 10px; border: 1px solid #ddd;">s→s</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;"><b>5.03</b></td> |
|
<td style="padding: 10px; border: 1px solid #ddd;"><b>63.7</b></td> |
|
<td style="padding: 10px; border: 1px solid #ddd;"><b>3.06</b></td> |
|
<td style="padding: 10px; border: 1px solid #ddd;"><b>4.68</b></td> |
|
</tr> |
|
</tbody> |
|
<tbody> |
|
<tr> |
|
<td colspan="7" style="padding: 10px; border: 1px solid #ddd; font-weight: bold; background-color: #e6f3ff;">Base Models</td> |
|
</tr> |
|
<tr> |
|
<td rowspan="2" style="padding: 10px; border: 1px solid #ddd;">LLaMA-Omni</td> |
|
<td rowspan="2" style="padding: 10px; border: 1px solid #ddd;">8B</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">s→t</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">5.31</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">69.7</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">4.44</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">5.44</td> |
|
</tr> |
|
<tr> |
|
<td style="padding: 10px; border: 1px solid #ddd;">s→s</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">3.89</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">55.1</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">2.44</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">4.00</td> |
|
</tr> |
|
<tr> |
|
<td rowspan="2" style="padding: 10px; border: 1px solid #ddd;">Freeze-Omni</td> |
|
<td rowspan="2" style="padding: 10px; border: 1px solid #ddd;">7B</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">s→t</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">4.51</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">77.7</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">5.32</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">6.41</td> |
|
</tr> |
|
<tr> |
|
<td style="padding: 10px; border: 1px solid #ddd;">s→s</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">2.99</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">60.2</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">3.53</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">4.78</td> |
|
</tr> |
|
<tr> |
|
<td rowspan="2" style="padding: 10px; border: 1px solid #ddd;">GLM-4-Voice</td> |
|
<td rowspan="2" style="padding: 10px; border: 1px solid #ddd;">9B</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">s→t</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">5.86</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">77.4</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">4.95</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">5.56</td> |
|
</tr> |
|
<tr> |
|
<td style="padding: 10px; border: 1px solid #ddd;">s→s</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">5.27</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">64.3</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">4.63</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">5.40</td> |
|
</tr> |
|
<tr> |
|
<td rowspan="2" style="padding: 10px; border: 1px solid #ddd;">Baichuan-Omni-1.5</td> |
|
<td rowspan="2" style="padding: 10px; border: 1px solid #ddd;">7B</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">s→t</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">5.20</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">77.6</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">5.72</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">6.12</td> |
|
</tr> |
|
<tr> |
|
<td style="padding: 10px; border: 1px solid #ddd;">s→s</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">4.10</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">61.2</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">4.13</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">5.18</td> |
|
</tr> |
|
<tr> |
|
<td rowspan="2" style="padding: 10px; border: 1px solid #ddd;">MiniCPM-o</td> |
|
<td rowspan="2" style="padding: 10px; border: 1px solid #ddd;">8B</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">s→t</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">6.13</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">77.2</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;"><b>6.43</b></td> |
|
<td style="padding: 10px; border: 1px solid #ddd;"><b>7.16</b></td> |
|
</tr> |
|
<tr> |
|
<td style="padding: 10px; border: 1px solid #ddd;">s→s</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">4.95</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">65.8</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">4.99</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;"><u>6.22</u></td> |
|
</tr> |
|
<tr> |
|
<td style="padding: 10px; border: 1px solid #ddd;">Minmo*</td> |
|
<td style="padding: 10px; border: 1px solid #ddd;">8B</td> |
|
<td>s→t</td> |
|
<td>-</td> |
|
<td>78.9</td> |
|
<td>4.83</td> |
|
<td>5.50</td> |
|
</tr> |
|
<tr> |
|
<td>s→s</td> |
|
<td><b>6.48<br></td> |
|
<td>64.1</td> |
|
<td>3.75</td> |
|
<td>3.99</td> |
|
</tr> |
|
<tr> |
|
<td style="padding: 10px; border: 1px solid #ddd;">Qwen2.5-Omni</td> |
|
<td>s→t</td> |
|
<td>6.01</td> |
|
<td><u>79.0</u></td> |
|
<td>5.89</td> |
|
<td><u>6.88</u></td> |
|
</tr> |
|
<tr> |
|
<td>s→s</td> |
|
<td>5.73</td> |
|
<td><b>76.3<br></td> |
|
<td><u>5.59</u></td> |
|
<td><b>6.70<br></td> |
|
</tr> |
|
<tr> |
|
<td style="padding: 10px; border: 1px solid #ddd;">VocalNet-8B (VA)</td> |
|
<td><u>7.05</u></td> |
|
<td><b>4.490<br></td> |
|
<td>77.1</td> |
|
<td>4.503</td> |
|
<td>6.15</td> |
|
<td><b>4.499<br></td> |
|
<td><u>4.21</u></td> |
|
<td><u>4.485</u></td> |
|
<td>4.26</td> |
|
<td><b>4.493<br></td> |
|
</tr> |
|
<tr> |
|
<td>VocalNet-8B</td> |
|
<td><b>7.12<br></td> |
|
<td><u>4.489</u></td> |
|
<td><b>79.5<br></td> |
|
<td>4.500</td> |
|
<td><u>6.24</u></td> |
|
<td>4.482</td> |
|
<td>3.11</td> |
|
<td>4.492</td> |
|
<td><u>3.56</u></td> |
|
<td><u>4.489</u></td> |
|
</tr> |
|
<thead> |
|
<tr> |
|
<th class="tg-c3ow" colspan="11"></th> |
|
</tr> |
|
</thead> |
|
</tbody> |
|
</table> |
|
</div> |
|
|
|
</details> |
|
|
|
### ✍️ Citation |
|
If you find our work useful, please cite: |
|
```bib |
|
@article{wang2025vocalnet, |
|
title={VocalNet: Speech LLM with Multi-Token Prediction for Faster and High-Quality Generation}, |
|
author={Wang, Yuhao and Liu, Heyang and Cheng, Ziyang and Wu, Ronghua and Gu, Qunshan and Wang, Yanfeng and Wang, Yu}, |
|
journal={arXiv preprint arXiv:2504.04060}, |
|
year={2025} |
|
} |
|
``` |
