Text-to-Speech
Safetensors
English
Chinese
Files changed (1) hide show
  1. README.md +184 -173
README.md CHANGED
@@ -1,174 +1,185 @@
1
- ---
2
- license: cc-by-nc-sa-4.0
3
- language:
4
- - en
5
- - zh
6
- tags:
7
- - text-to-speech
8
- library_tag: spark-tts
9
- ---
10
-
11
-
12
- <div align="center">
13
- <h1>
14
- Spark-TTS
15
- </h1>
16
- <p>
17
- Official model for <br>
18
- <b><em>Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens</em></b>
19
- </p>
20
- <p>
21
- <img src="src/logo/SparkTTS.jpg" alt="Spark-TTS Logo" style="width: 200px; height: 200px;">
22
- </p>
23
- </div>
24
-
25
-
26
- ## Spark-TTS πŸ”₯
27
-
28
- ### πŸ‘‰πŸ» [Spark-TTS Demos](https://sparkaudio.github.io/spark-tts/) πŸ‘ˆπŸ»
29
-
30
- ### πŸ‘‰πŸ» [Github Repo](https://github.com/SparkAudio/Spark-TTS) πŸ‘ˆπŸ»
31
-
32
- ### πŸ‘‰πŸ» [Paper](https://arxiv.org/pdf/2503.01710) πŸ‘ˆπŸ»
33
-
34
- ### Overview
35
-
36
- Spark-TTS is an advanced text-to-speech system that uses the power of large language models (LLM) for highly accurate and natural-sounding voice synthesis. It is designed to be efficient, flexible, and powerful for both research and production use.
37
-
38
- ### Key Features
39
-
40
- - **Simplicity and Efficiency**: Built entirely on Qwen2.5, Spark-TTS eliminates the need for additional generation models like flow matching. Instead of relying on separate models to generate acoustic features, it directly reconstructs audio from the code predicted by the LLM. This approach streamlines the process, improving efficiency and reducing complexity.
41
- - **High-Quality Voice Cloning**: Supports zero-shot voice cloning, which means it can replicate a speaker's voice even without specific training data for that voice. This is ideal for cross-lingual and code-switching scenarios, allowing for seamless transitions between languages and voices without requiring separate training for each one.
42
- - **Bilingual Support**: Supports both Chinese and English, and is capable of zero-shot voice cloning for cross-lingual and code-switching scenarios, enabling the model to synthesize speech in multiple languages with high naturalness and accuracy.
43
- - **Controllable Speech Generation**: Supports creating virtual speakers by adjusting parameters such as gender, pitch, and speaking rate.
44
-
45
- ---
46
-
47
- <table align="center">
48
- <tr>
49
- <td align="center"><b>Inference Overview of Voice Cloning</b><br><img src="src/figures/infer_voice_cloning.png" width="80%" /></td>
50
- </tr>
51
- <tr>
52
- <td align="center"><b>Inference Overview of Controlled Generation</b><br><img src="src/figures/infer_control.png" width="80%" /></td>
53
- </tr>
54
- </table>
55
-
56
-
57
- ## Install
58
- **Clone and Install**
59
-
60
- - Clone the repo
61
- ``` sh
62
- git clone https://github.com/SparkAudio/Spark-TTS.git
63
- cd Spark-TTS
64
- ```
65
-
66
- - Install Conda: please see https://docs.conda.io/en/latest/miniconda.html
67
- - Create Conda env:
68
-
69
- ``` sh
70
- conda create -n sparktts -y python=3.12
71
- conda activate sparktts
72
- pip install -r requirements.txt
73
- # If you are in mainland China, you can set the mirror as follows:
74
- pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/ --trusted-host=mirrors.aliyun.com
75
- ```
76
-
77
- **Model Download**
78
-
79
- Download via python:
80
- ```python
81
- from huggingface_hub import snapshot_download
82
-
83
- snapshot_download("SparkAudio/Spark-TTS-0.5B", local_dir="pretrained_models/Spark-TTS-0.5B")
84
- ```
85
-
86
- Download via git clone:
87
- ```sh
88
- mkdir -p pretrained_models
89
-
90
- # Make sure you have git-lfs installed (https://git-lfs.com)
91
- git lfs install
92
-
93
- git clone https://huggingface.co/SparkAudio/Spark-TTS-0.5B pretrained_models/Spark-TTS-0.5B
94
- ```
95
-
96
- **Basic Usage**
97
-
98
- You can simply run the demo with the following commands:
99
- ``` sh
100
- cd example
101
- bash infer.sh
102
- ```
103
-
104
- Alternatively, you can directly execute the following command in the command line to perform inference:
105
-
106
- ``` sh
107
- python -m cli.inference \
108
- --text "text to synthesis." \
109
- --device 0 \
110
- --save_dir "path/to/save/audio" \
111
- --model_dir pretrained_models/Spark-TTS-0.5B \
112
- --prompt_text "transcript of the prompt audio" \
113
- --prompt_speech_path "path/to/prompt_audio"
114
- ```
115
-
116
- **UI Usage**
117
-
118
- You can start the UI interface by running `python webui.py`, which allows you to perform Voice Cloning and Voice Creation. Voice Cloning supports uploading reference audio or directly recording the audio.
119
-
120
-
121
- | **Voice Cloning** | **Voice Creation** |
122
- |:-------------------:|:-------------------:|
123
- | ![Image 1](src/figures/gradio_TTS.png) | ![Image 2](src/figures/gradio_control.png) |
124
-
125
-
126
- ## To-Do List
127
-
128
- - [x] Release the Spark-TTS paper.
129
- - [ ] Release the training code.
130
- - [ ] Release the training dataset, VoxBox.
131
-
132
- ## Citation
133
-
134
- ```
135
- @misc{wang2025sparktts,
136
- title={Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens},
137
- author={Xinsheng Wang and Mingqi Jiang and Ziyang Ma and Ziyu Zhang and Songxiang Liu and Linqin Li and Zheng Liang and Qixi Zheng and Rui Wang and Xiaoqin Feng and Weizhen Bian and Zhen Ye and Sitong Cheng and Ruibin Yuan and Zhixian Zhao and Xinfa Zhu and Jiahao Pan and Liumeng Xue and Pengcheng Zhu and Yunlin Chen and Zhifei Li and Xie Chen and Lei Xie and Yike Guo and Wei Xue},
138
- year={2025},
139
- eprint={2503.01710},
140
- archivePrefix={arXiv},
141
- primaryClass={cs.SD},
142
- url={https://arxiv.org/abs/2503.01710},
143
- }
144
- ```
145
-
146
-
147
- ## ⚠ License Update
148
-
149
- The model's license has been updated from Apache 2.0 to CC BY-NC-SA due to the licensing terms of some training data.
150
-
151
- Key Changes:
152
-
153
- - The model can only be used for non-commercial purposes.
154
-
155
- - Any modifications or derivatives must also be released under CC BY-NC-SA 4.0.
156
-
157
- - Proper attribution is required when using or modifying the model.
158
-
159
- Please ensure compliance with the new license terms.
160
-
161
-
162
- ## ⚠️ Usage Disclaimer
163
-
164
- This project provides a zero-shot voice cloning TTS model intended for academic research, educational purposes, and legitimate applications, such as personalized speech synthesis, assistive technologies, and linguistic research.
165
-
166
- Please note:
167
-
168
- - Do not use this model for unauthorized voice cloning, impersonation, fraud, scams, deepfakes, or any illegal activities.
169
-
170
- - Ensure compliance with local laws and regulations when using this model and uphold ethical standards.
171
-
172
- - The developers assume no liability for any misuse of this model.
173
-
 
 
 
 
 
 
 
 
 
 
 
174
  We advocate for the responsible development and use of AI and encourage the community to uphold safety and ethical principles in AI research and applications. If you have any concerns regarding ethics or misuse, please contact us.
 
1
+ ---
2
+ license: cc-by-nc-sa-4.0
3
+ language:
4
+ - en
5
+ - zh
6
+ tags:
7
+ - text-to-speech
8
+ library_tag: spark-tts
9
+ datasets:
10
+ - Congliu/Chinese-DeepSeek-R1-Distill-data-110k
11
+ metrics:
12
+ - accuracy
13
+ - bertscore
14
+ - bleu
15
+ base_model:
16
+ - deepseek-ai/DeepSeek-R1
17
+ new_version: perplexity-ai/r1-1776
18
+ pipeline_tag: automatic-speech-recognition
19
+ library_name: adapter-transformers
20
+ ---
21
+
22
+
23
+ <div align="center">
24
+ <h1>
25
+ Spark-TTS
26
+ </h1>
27
+ <p>
28
+ Official model for <br>
29
+ <b><em>Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens</em></b>
30
+ </p>
31
+ <p>
32
+ <img src="src/logo/SparkTTS.jpg" alt="Spark-TTS Logo" style="width: 200px; height: 200px;">
33
+ </p>
34
+ </div>
35
+
36
+
37
+ ## Spark-TTS πŸ”₯
38
+
39
+ ### πŸ‘‰πŸ» [Spark-TTS Demos](https://sparkaudio.github.io/spark-tts/) πŸ‘ˆπŸ»
40
+
41
+ ### πŸ‘‰πŸ» [Github Repo](https://github.com/SparkAudio/Spark-TTS) πŸ‘ˆπŸ»
42
+
43
+ ### πŸ‘‰πŸ» [Paper](https://arxiv.org/pdf/2503.01710) πŸ‘ˆπŸ»
44
+
45
+ ### Overview
46
+
47
+ Spark-TTS is an advanced text-to-speech system that uses the power of large language models (LLM) for highly accurate and natural-sounding voice synthesis. It is designed to be efficient, flexible, and powerful for both research and production use.
48
+
49
+ ### Key Features
50
+
51
+ - **Simplicity and Efficiency**: Built entirely on Qwen2.5, Spark-TTS eliminates the need for additional generation models like flow matching. Instead of relying on separate models to generate acoustic features, it directly reconstructs audio from the code predicted by the LLM. This approach streamlines the process, improving efficiency and reducing complexity.
52
+ - **High-Quality Voice Cloning**: Supports zero-shot voice cloning, which means it can replicate a speaker's voice even without specific training data for that voice. This is ideal for cross-lingual and code-switching scenarios, allowing for seamless transitions between languages and voices without requiring separate training for each one.
53
+ - **Bilingual Support**: Supports both Chinese and English, and is capable of zero-shot voice cloning for cross-lingual and code-switching scenarios, enabling the model to synthesize speech in multiple languages with high naturalness and accuracy.
54
+ - **Controllable Speech Generation**: Supports creating virtual speakers by adjusting parameters such as gender, pitch, and speaking rate.
55
+
56
+ ---
57
+
58
+ <table align="center">
59
+ <tr>
60
+ <td align="center"><b>Inference Overview of Voice Cloning</b><br><img src="src/figures/infer_voice_cloning.png" width="80%" /></td>
61
+ </tr>
62
+ <tr>
63
+ <td align="center"><b>Inference Overview of Controlled Generation</b><br><img src="src/figures/infer_control.png" width="80%" /></td>
64
+ </tr>
65
+ </table>
66
+
67
+
68
+ ## Install
69
+ **Clone and Install**
70
+
71
+ - Clone the repo
72
+ ``` sh
73
+ git clone https://github.com/SparkAudio/Spark-TTS.git
74
+ cd Spark-TTS
75
+ ```
76
+
77
+ - Install Conda: please see https://docs.conda.io/en/latest/miniconda.html
78
+ - Create Conda env:
79
+
80
+ ``` sh
81
+ conda create -n sparktts -y python=3.12
82
+ conda activate sparktts
83
+ pip install -r requirements.txt
84
+ # If you are in mainland China, you can set the mirror as follows:
85
+ pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/ --trusted-host=mirrors.aliyun.com
86
+ ```
87
+
88
+ **Model Download**
89
+
90
+ Download via python:
91
+ ```python
92
+ from huggingface_hub import snapshot_download
93
+
94
+ snapshot_download("SparkAudio/Spark-TTS-0.5B", local_dir="pretrained_models/Spark-TTS-0.5B")
95
+ ```
96
+
97
+ Download via git clone:
98
+ ```sh
99
+ mkdir -p pretrained_models
100
+
101
+ # Make sure you have git-lfs installed (https://git-lfs.com)
102
+ git lfs install
103
+
104
+ git clone https://huggingface.co/SparkAudio/Spark-TTS-0.5B pretrained_models/Spark-TTS-0.5B
105
+ ```
106
+
107
+ **Basic Usage**
108
+
109
+ You can simply run the demo with the following commands:
110
+ ``` sh
111
+ cd example
112
+ bash infer.sh
113
+ ```
114
+
115
+ Alternatively, you can directly execute the following command in the command line to perform inference:
116
+
117
+ ``` sh
118
+ python -m cli.inference \
119
+ --text "text to synthesis." \
120
+ --device 0 \
121
+ --save_dir "path/to/save/audio" \
122
+ --model_dir pretrained_models/Spark-TTS-0.5B \
123
+ --prompt_text "transcript of the prompt audio" \
124
+ --prompt_speech_path "path/to/prompt_audio"
125
+ ```
126
+
127
+ **UI Usage**
128
+
129
+ You can start the UI interface by running `python webui.py`, which allows you to perform Voice Cloning and Voice Creation. Voice Cloning supports uploading reference audio or directly recording the audio.
130
+
131
+
132
+ | **Voice Cloning** | **Voice Creation** |
133
+ |:-------------------:|:-------------------:|
134
+ | ![Image 1](src/figures/gradio_TTS.png) | ![Image 2](src/figures/gradio_control.png) |
135
+
136
+
137
+ ## To-Do List
138
+
139
+ - [x] Release the Spark-TTS paper.
140
+ - [ ] Release the training code.
141
+ - [ ] Release the training dataset, VoxBox.
142
+
143
+ ## Citation
144
+
145
+ ```
146
+ @misc{wang2025sparktts,
147
+ title={Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens},
148
+ author={Xinsheng Wang and Mingqi Jiang and Ziyang Ma and Ziyu Zhang and Songxiang Liu and Linqin Li and Zheng Liang and Qixi Zheng and Rui Wang and Xiaoqin Feng and Weizhen Bian and Zhen Ye and Sitong Cheng and Ruibin Yuan and Zhixian Zhao and Xinfa Zhu and Jiahao Pan and Liumeng Xue and Pengcheng Zhu and Yunlin Chen and Zhifei Li and Xie Chen and Lei Xie and Yike Guo and Wei Xue},
149
+ year={2025},
150
+ eprint={2503.01710},
151
+ archivePrefix={arXiv},
152
+ primaryClass={cs.SD},
153
+ url={https://arxiv.org/abs/2503.01710},
154
+ }
155
+ ```
156
+
157
+
158
+ ## ⚠ License Update
159
+
160
+ The model's license has been updated from Apache 2.0 to CC BY-NC-SA due to the licensing terms of some training data.
161
+
162
+ Key Changes:
163
+
164
+ - The model can only be used for non-commercial purposes.
165
+
166
+ - Any modifications or derivatives must also be released under CC BY-NC-SA 4.0.
167
+
168
+ - Proper attribution is required when using or modifying the model.
169
+
170
+ Please ensure compliance with the new license terms.
171
+
172
+
173
+ ## ⚠️ Usage Disclaimer
174
+
175
+ This project provides a zero-shot voice cloning TTS model intended for academic research, educational purposes, and legitimate applications, such as personalized speech synthesis, assistive technologies, and linguistic research.
176
+
177
+ Please note:
178
+
179
+ - Do not use this model for unauthorized voice cloning, impersonation, fraud, scams, deepfakes, or any illegal activities.
180
+
181
+ - Ensure compliance with local laws and regulations when using this model and uphold ethical standards.
182
+
183
+ - The developers assume no liability for any misuse of this model.
184
+
185
  We advocate for the responsible development and use of AI and encourage the community to uphold safety and ethical principles in AI research and applications. If you have any concerns regarding ethics or misuse, please contact us.