## 🔥 1. We provide all the links of Sana pth and diffusers safetensor below

| Model                | Reso   | pth link                                                                                | diffusers                                                                                | Precision     | Description    |
|----------------------|--------|-----------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------|---------------|----------------|
| Sana-0.6B            | 512px  | [Sana_600M_512px](https://huggingface.co/Efficient-Large-Model/Sana_600M_512px)                                             | [Efficient-Large-Model/Sana_600M_512px_diffusers](https://huggingface.co/Efficient-Large-Model/Sana_600M_512px_diffusers)                         | fp16/fp32     | Multi-Language |
| Sana-0.6B            | 1024px | [Sana_600M_1024px](https://huggingface.co/Efficient-Large-Model/Sana_600M_1024px)                                           | [Efficient-Large-Model/Sana_600M_1024px_diffusers](https://huggingface.co/Efficient-Large-Model/Sana_600M_1024px_diffusers)                       | fp16/fp32     | Multi-Language |
| Sana-1.6B            | 512px  | [Sana_1600M_512px](https://huggingface.co/Efficient-Large-Model/Sana_1600M_512px)                                           | [Efficient-Large-Model/Sana_1600M_512px_diffusers](https://huggingface.co/Efficient-Large-Model/Sana_1600M_512px_diffusers)                       | fp16/fp32     | -              |
| Sana-1.6B            | 512px  | [Sana_1600M_512px_MultiLing](https://huggingface.co/Efficient-Large-Model/Sana_1600M_512px_MultiLing)                       | [Efficient-Large-Model/Sana_1600M_512px_MultiLing_diffusers](https://huggingface.co/Efficient-Large-Model/Sana_1600M_512px_MultiLing_diffusers)   | fp16/fp32     | Multi-Language |
| Sana-1.6B            | 1024px | [Sana_1600M_1024px](https://huggingface.co/Efficient-Large-Model/Sana_1600M_1024px)                                         | [Efficient-Large-Model/Sana_1600M_1024px_diffusers](https://huggingface.co/Efficient-Large-Model/Sana_1600M_1024px_diffusers)                     | fp16/fp32     | -              |
| Sana-1.6B            | 1024px | [Sana_1600M_1024px_MultiLing](https://huggingface.co/Efficient-Large-Model/Sana_1600M_1024px_MultiLing)                     | [Efficient-Large-Model/Sana_1600M_1024px_MultiLing_diffusers](https://huggingface.co/Efficient-Large-Model/Sana_1600M_1024px_MultiLing_diffusers) | fp16/fp32     | Multi-Language |
| Sana-1.6B            | 1024px | [Sana_1600M_1024px_BF16](https://huggingface.co/Efficient-Large-Model/Sana_1600M_1024px_BF16)                               | [Efficient-Large-Model/Sana_1600M_1024px_BF16_diffusers](https://huggingface.co/Efficient-Large-Model/Sana_1600M_1024px_BF16_diffusers)           | **bf16**/fp32 | Multi-Language |
| Sana-1.6B            | 1024px | -                                                                                | [mit-han-lab/svdq-int4-sana-1600m](https://huggingface.co/mit-han-lab/svdq-int4-sana-1600m)                                                       | **int4**      | Multi-Language |
| Sana-1.6B            | 2Kpx   | [Sana_1600M_2Kpx_BF16](https://huggingface.co/Efficient-Large-Model/Sana_1600M_2Kpx_BF16)                                   | [Efficient-Large-Model/Sana_1600M_2Kpx_BF16_diffusers](https://huggingface.co/Efficient-Large-Model/Sana_1600M_2Kpx_BF16_diffusers)               | **bf16**/fp32 | Multi-Language |
| Sana-1.6B            | 4Kpx   | [Sana_1600M_4Kpx_BF16](https://huggingface.co/Efficient-Large-Model/Sana_1600M_4Kpx_BF16)                                   | [Efficient-Large-Model/Sana_1600M_4Kpx_BF16_diffusers](https://huggingface.co/Efficient-Large-Model/Sana_1600M_4Kpx_BF16_diffusers)               | **bf16**/fp32 | Multi-Language |
| Sana-1.6B            | 4Kpx   | [Sana_1600M_4Kpx_BF16](https://huggingface.co/Efficient-Large-Model/Sana_1600M_4Kpx_BF16)                                   | [Efficient-Large-Model/Sana_1600M_4Kpx_BF16_diffusers](https://huggingface.co/Efficient-Large-Model/Sana_1600M_4Kpx_BF16_diffusers)               | **bf16**/fp32 | Multi-Language |
| ControlNet           |        |                                                                                |                                                                                |               |                |
| Sana-1.6B-ControlNet | 1Kpx   | [Sana_1600M_1024px_BF16_ControlNet_HED](https://huggingface.co/Efficient-Large-Model/Sana_1600M_1024px_BF16_ControlNet_HED) | Coming soon                                                                                | **bf16**/fp32 | Multi-Language |
| Sana-0.6B-ControlNet | 1Kpx   | [Sana_600M_1024px_ControlNet_HED](https://huggingface.co/Efficient-Large-Model/Sana_600M_1024px_ControlNet_HED)             | Coming soon                                                                                | fp16/fp32     | -              |

## ❗ 2. Make sure to use correct precision(fp16/bf16/fp32) for training and inference.

### We provide two samples to use fp16 and bf16 weights, respectively.

❗Make sure to set `variant` and `torch_dtype` in diffusers pipelines to the desired precision.

#### 1). For fp16 models

```python
# run `pip install git+https://github.com/huggingface/diffusers` before use Sana in diffusers
import torch
from diffusers import SanaPipeline

pipe = SanaPipeline.from_pretrained(
    "Efficient-Large-Model/Sana_1600M_1024px_diffusers",
    variant="fp16",
    torch_dtype=torch.float16,
)
pipe.to("cuda")

pipe.vae.to(torch.bfloat16)
pipe.text_encoder.to(torch.bfloat16)

prompt = 'a cyberpunk cat with a neon sign that says "Sana"'
image = pipe(
    prompt=prompt,
    height=1024,
    width=1024,
    guidance_scale=5.0,
    num_inference_steps=20,
    generator=torch.Generator(device="cuda").manual_seed(42),
)[0]

image[0].save("sana.png")
```

#### 2). For bf16 models

```python
# run `pip install git+https://github.com/huggingface/diffusers` before use Sana in diffusers
import torch
from diffusers import SanaPAGPipeline

pipe = SanaPAGPipeline.from_pretrained(
  "Efficient-Large-Model/Sana_1600M_1024px_BF16_diffusers",
  variant="bf16",
  torch_dtype=torch.bfloat16,
  pag_applied_layers="transformer_blocks.8",
)
pipe.to("cuda")

pipe.text_encoder.to(torch.bfloat16)
pipe.vae.to(torch.bfloat16)

prompt = 'a cyberpunk cat with a neon sign that says "Sana"'
image = pipe(
    prompt=prompt,
    guidance_scale=5.0,
    pag_scale=2.0,
    num_inference_steps=20,
    generator=torch.Generator(device="cuda").manual_seed(42),
)[0]
image[0].save('sana.png')
```

## ❗ 3. 4K models

4K models need VAE tiling to avoid OOM issue.(16 GPU is recommended)

```python
# run `pip install git+https://github.com/huggingface/diffusers` before use Sana in diffusers
import torch
from diffusers import SanaPipeline

pipe = SanaPipeline.from_pretrained(
    "Efficient-Large-Model/Sana_1600M_4Kpx_BF16_diffusers",
    variant="bf16",
    torch_dtype=torch.bfloat16,
)
pipe.to("cuda")

pipe.vae.to(torch.bfloat16)
pipe.text_encoder.to(torch.bfloat16)

# for 4096x4096 image generation OOM issue, feel free adjust the tile size
if pipe.transformer.config.sample_size == 128:
    pipe.vae.enable_tiling(
        tile_sample_min_height=1024,
        tile_sample_min_width=1024,
        tile_sample_stride_height=896,
        tile_sample_stride_width=896,
    )
prompt = 'a cyberpunk cat with a neon sign that says "Sana"'
image = pipe(
    prompt=prompt,
    height=4096,
    width=4096,
    guidance_scale=5.0,
    num_inference_steps=20,
    generator=torch.Generator(device="cuda").manual_seed(42),
)[0]

image[0].save("sana_4K.png")
```

## ❗ 4. int4 inference

This int4 model is quantized with [SVDQuant-Nunchaku](https://github.com/mit-han-lab/nunchaku). You need first follow the [guidance of installation](https://github.com/mit-han-lab/nunchaku?tab=readme-ov-file#installation) of nunchaku engine, then you can use the following code snippet to perform inference with int4 Sana model.

Here we show the code snippet for SanaPipeline. For SanaPAGPipeline, please refer to the [SanaPAGPipeline](https://github.com/mit-han-lab/nunchaku/blob/main/examples/sana_1600m_pag.py) section.

```python
import torch
from diffusers import SanaPipeline

from nunchaku.models.transformer_sana import NunchakuSanaTransformer2DModel

transformer = NunchakuSanaTransformer2DModel.from_pretrained("mit-han-lab/svdq-int4-sana-1600m")
pipe = SanaPipeline.from_pretrained(
    "Efficient-Large-Model/Sana_1600M_1024px_BF16_diffusers",
    transformer=transformer,
    variant="bf16",
    torch_dtype=torch.bfloat16,
).to("cuda")

pipe.text_encoder.to(torch.bfloat16)
pipe.vae.to(torch.bfloat16)

image = pipe(
    prompt="A cute 🐼 eating 🎋, ink drawing style",
    height=1024,
    width=1024,
    guidance_scale=4.5,
    num_inference_steps=20,
    generator=torch.Generator().manual_seed(42),
).images[0]
image.save("sana_1600m.png")
```


## 🔥 ControlNet

We incorporate a ControlNet-like(https://github.com/lllyasviel/ControlNet) module enables fine-grained control over text-to-image diffusion models. We implement a ControlNet-Transformer architecture, specifically tailored for Transformers, achieving explicit controllability alongside high-quality image generation.

## Inference of `Sana + ControlNet`

### 1). Gradio Interface

```bash
python app/app_sana_controlnet_hed.py \
        --config configs/sana_controlnet_config/Sana_1600M_1024px_controlnet_bf16.yaml \
        --model_path hf://Efficient-Large-Model/Sana_1600M_1024px_BF16_ControlNet_HED/checkpoints/Sana_1600M_1024px_BF16_ControlNet_HED.pth
```

### 2). Inference with JSON file

```bash
python tools/controlnet/inference_controlnet.py \
        --config configs/sana_controlnet_config/Sana_1600M_1024px_controlnet_bf16.yaml \
        --model_path hf://Efficient-Large-Model/Sana_1600M_1024px_BF16_ControlNet_HED/checkpoints/Sana_1600M_1024px_BF16_ControlNet_HED.pth \
        --json_file asset/controlnet/samples_controlnet.json
```

### 3). Inference code snap

```python
import torch
from PIL import Image
from app.sana_controlnet_pipeline import SanaControlNetPipeline

device = "cuda" if torch.cuda.is_available() else "cpu"

pipe = SanaControlNetPipeline("configs/sana_controlnet_config/Sana_1600M_1024px_controlnet_bf16.yaml")
pipe.from_pretrained("hf://Efficient-Large-Model/Sana_1600M_1024px_BF16_ControlNet_HED/checkpoints/Sana_1600M_1024px_BF16_ControlNet_HED.pth")

ref_image = Image.open("asset/controlnet/ref_images/A transparent sculpture of a duck made out of glass. The sculpture is in front of a painting of a la.jpg")
prompt = "A transparent sculpture of a duck made out of glass. The sculpture is in front of a painting of a landscape."

images = pipe(
    prompt=prompt,
    ref_image=ref_image,
    guidance_scale=4.5,
    num_inference_steps=10,
    sketch_thickness=2,
    generator=torch.Generator(device=device).manual_seed(0),
)
```

## Training of `Sana + ControlNet`

### Coming soon

SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformer


# DreamBooth training example for SANA

[DreamBooth](https://arxiv.org/abs/2208.12242) is a method to personalize text2image models like stable diffusion given just a few (3~5) images of a subject.

The `train_dreambooth_lora_sana.py` script shows how to implement the training procedure with [LoRA](https://huggingface.co/docs/peft/conceptual_guides/adapter#low-rank-adaptation-lora) and adapt it for [SANA](https://arxiv.org/abs/2410.10629).

This will also allow us to push the trained model parameters to the Hugging Face Hub platform.

## Running locally with PyTorch

### Installing the dependencies

Before running the scripts, make sure to install the library's training dependencies:

**Important**

To make sure you can successfully run the latest versions of the example scripts, we highly recommend **installing from source** and keeping the install up to date as we update the example scripts frequently and install some example-specific requirements. To do this, execute the following steps in a new virtual environment:

```bash
git clone https://github.com/huggingface/diffusers
cd diffusers
pip install -e .
```

And initialize an [🤗Accelerate](https://github.com/huggingface/accelerate/) environment with:

```bash
accelerate config
```

Or for a default accelerate configuration without answering questions about your environment

```bash
accelerate config default
```

Or if your environment doesn't support an interactive shell (e.g., a notebook)

```python
from accelerate.utils import write_basic_config
write_basic_config()
```

When running `accelerate config`, if we specify torch compile mode to True there can be dramatic speedups.
Note also that we use PEFT library as backend for LoRA training, make sure to have `peft>=0.14.0` installed in your environment.

### Dog toy example

Now let's get our dataset. For this example we will use some dog images: https://huggingface.co/datasets/diffusers/dog-example.

Let's first download it locally:

```python
from huggingface_hub import snapshot_download

local_dir = "data/dreambooth/dog"
snapshot_download(
    "diffusers/dog-example",
    local_dir=local_dir, repo_type="dataset",
    ignore_patterns=".gitattributes",
)
```

This will also allow us to push the trained LoRA parameters to the Hugging Face Hub platform.

[Here is the Model Card](model_zoo.md) for you to choose the desired pre-trained models and set it to `MODEL_NAME`.

Now, we can launch training using [file here](../../train_scripts/train_lora.sh):

```bash
bash train_scripts/train_lora.sh
```

or you can run it locally:

```bash
export MODEL_NAME="Efficient-Large-Model/Sana_1600M_1024px_BF16_diffusers"
export INSTANCE_DIR="data/dreambooth/dog"
export OUTPUT_DIR="trained-sana-lora"

accelerate launch --num_processes 8 --main_process_port 29500 --gpu_ids 0,1,2,3 \
  train_scripts/train_dreambooth_lora_sana.py \
  --pretrained_model_name_or_path=$MODEL_NAME  \
  --instance_data_dir=$INSTANCE_DIR \
  --output_dir=$OUTPUT_DIR \
  --mixed_precision="bf16" \
  --instance_prompt="a photo of sks dog" \
  --resolution=1024 \
  --train_batch_size=1 \
  --gradient_accumulation_steps=4 \
  --use_8bit_adam \
  --learning_rate=1e-4 \
  --report_to="wandb" \
  --lr_scheduler="constant" \
  --lr_warmup_steps=0 \
  --max_train_steps=500 \
  --validation_prompt="A photo of sks dog in a pond, yarn art style" \
  --validation_epochs=25 \
  --seed="0" \
  --push_to_hub
```

For using `push_to_hub`, make you're logged into your Hugging Face account:

```bash
huggingface-cli login
```

To better track our training experiments, we're using the following flags in the command above:

- `report_to="wandb` will ensure the training runs are tracked on [Weights and Biases](https://wandb.ai/site). To use it, be sure to install `wandb` with `pip install wandb`. Don't forget to call `wandb login <your_api_key>` before training if you haven't done it before.
- `validation_prompt` and `validation_epochs` to allow the script to do a few validation inference runs. This allows us to qualitatively check if the training is progressing as expected.

## Notes

Additionally, we welcome you to explore the following CLI arguments:

- `--lora_layers`: The transformer modules to apply LoRA training on. Please specify the layers in a comma seperated. E.g. - "to_k,to_q,to_v" will result in lora training of attention layers only.
- `--complex_human_instruction`: Instructions for complex human attention as shown in [here](https://github.com/NVlabs/Sana/blob/main/configs/sana_app_config/Sana_1600M_app.yaml#L55).
- `--max_sequence_length`: Maximum sequence length to use for text embeddings.

We provide several options for optimizing memory optimization:

- `--offload`: When enabled, we will offload the text encoder and VAE to CPU, when they are not used.
- `cache_latents`: When enabled, we will pre-compute the latents from the input images with the VAE and remove the VAE from memory once done.
- `--use_8bit_adam`: When enabled, we will use the 8bit version of AdamW provided by the `bitsandbytes` library.

Refer to the [official documentation](https://huggingface.co/docs/diffusers/main/en/api/pipelines/sana) of the `SanaPipeline` to know more about the models available under the SANA family and their preferred dtypes during inference.

## Samples

We show some samples during Sana-LoRA fine-tuning process below.

   training samples at step=0

   training samples at step=500


## 🖌 Sana-ComfyUI

[Original Repo](https://github.com/city96/ComfyUI_ExtraModels)

### Model info / implementation

- Uses Gemma2 2B as the text encoder
- Multiple resolutions and models available
- Compressed latent space (32 channels, /32 compression) - needs custom VAE

### Usage

1. All the checkpoints will be downloaded automatically.
1. KSampler(Flow Euler) is available for now; Flow DPM-Solver will be available soon.

```bash
git clone https://github.com/comfyanonymous/ComfyUI.git
cd ComfyUI
git clone https://github.com/Efficient-Large-Model/ComfyUI_ExtraModels.git custom_nodes/ComfyUI_ExtraModels

python main.py
```

### A sample workflow for Sana

[Sana workflow](Sana_FlowEuler.json)

![Sana](https://raw.githubusercontent.com/NVlabs/Sana/refs/heads/page/asset/content/comfyui/sana.jpg)

### A sample for T2I(Sana) + I2V(CogVideoX)

[Sana + CogVideoX workflow](Sana_CogVideoX.json)

[![Sample T2I + I2V](https://raw.githubusercontent.com/NVlabs/Sana/refs/heads/page/asset/content/comfyui/sana-cogvideox.jpg)](https://nvlabs.github.io/Sana/asset/content/comfyui/Sana_CogVideoX_Fun.mp4)

### A sample workflow for Sana 4096x4096 image (18GB GPU is needed)

[Sana workflow](Sana_FlowEuler_4K.json)

![Sana](https://raw.githubusercontent.com/NVlabs/Sana/refs/heads/page/asset/content/comfyui/Sana_4K_workflow.jpg)


# CLIP Score for PyTorch

[![PyPI](https://img.shields.io/pypi/v/clip-score.svg)](https://pypi.org/project/clip-score/)

This repository provides a batch-wise quick processing for calculating CLIP scores. It uses the pretrained CLIP model to measure the cosine similarity between two modalities. The project structure is adapted from [pytorch-fid](https://github.com/mseitzer/pytorch-fid) and [CLIP](https://github.com/openai/CLIP).

## Installation

Requirements:

- Install PyTorch:
  ```
  pip install torch  # Choose a version that suits your GPU
  ```
- Install CLIP:
  ```
  pip install git+https://github.com/openai/CLIP.git
  ```
- Install clip-score from [PyPI](https://pypi.org/project/clip-score/):
  ```
  pip install clip-score
  ```

## Data Input Specifications

This project is designed to process paired images and text files, and therefore requires two directories: one for images and one for text files.

### Image Files

All images should be stored in a single directory. The image files can be in either `.png` or `.jpg` format.

### Text Files

All text data should be contained in plain text files in a separate directory. These text files should have the extension `.txt`.

### File Number and Naming

The number of files in the image directory should be exactly equal to the number of files in the text directory. Additionally, the files in the image directory and text directory should be paired by file name. For instance, if there is a `cat.png` in the image directory, there should be a corresponding `cat.txt` in the text directory.

### Directory Structure Example

Below is an example of the expected directory structure:

```plaintext
├── path/to/image
│   ├── cat.png
│   ├── dog.png
│   └── bird.jpg
└── path/to/text
    ├── cat.txt
    ├── dog.txt
    └── bird.txt
```

In this example, `cat.png` is paired with `cat.txt`, `dog.png` is paired with `dog.txt`, and `bird.jpg` is paired with `bird.txt`.

Please adhere to the specified structure to ensure correct operation of the program. If there are any questions or issues, feel free to raise an issue here on GitHub.

## Usage

To compute the CLIP score between images and texts, make sure that the image and text data are contained in two separate folders, and each sample has the same name in both modalities. Run the following command:

```
python -m clip_score path/to/image path/to/text
```

If GPU is available, the project is set to run automatically on a GPU by default. If you want to specify a particular GPU, you can use the `--device cuda:N` flag when running the script, where `N` is the index of the GPU you wish to use. In case you want to run the program on a CPU instead, you can specify this by using the `--device cpu` flag.

## Computing CLIP Score within the Same Modality

If you want to calculate the CLIP score within the same modality (e.g., image-image or text-text), follow the same folder structure as mentioned above. Additionally, specify the preferred modalities using the `--real_flag` and `--fake_flag` options. By default, `--real_flag=img` and `--fake_flag=txt`. Examples:

```
python -m clip_score path/to/imageA path/to/imageB --real_flag img --fake_flag img
python -m clip_score path/to/textA path/to/textB --real_flag txt --fake_flag txt
```

## Citing

If you use this repository in your research, consider citing it using the following Bibtex entry:

```
@misc{taited2023CLIPScore,
  author={SUN Zhengwentai},
  title={{clip-score: CLIP Score for PyTorch}},
  month={March},
  year={2023},
  note={Version 0.1.1},
  howpublished={\url{https://github.com/taited/clip-score}},
}
```

## License

This implementation is licensed under the Apache License 2.0.

The project structure is adapted from [mseitzer's pytorch-fid](https://github.com/mseitzer/pytorch-fid) project. The CLIP model is adapted from [OpenAI's CLIP](https://github.com/openai/CLIP).

The CLIP Score was introduced in OpenAI's [Learning Transferable Visual Models From Natural Language Supervision](https://arxiv.org/abs/2103.00020).


# GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment

This repository contains code for the paper [GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment](https://arxiv.org/abs/2310.11513) by Dhruba Ghosh, Hanna Hajishirzi, and Ludwig Schmidt.

TLDR: We demonstrate the advantages of evaluating text-to-image models using existing object detection methods, to produce a fine-grained instance-level analysis of compositional capabilities.

### Abstract

*Recent breakthroughs in diffusion models, multimodal pretraining, and efficient finetuning have led to an explosion of text-to-image generative models.
Given human evaluation is expensive and difficult to scale, automated methods are critical for evaluating the increasingly large number of new models.
However, most current automated evaluation metrics like FID or CLIPScore only offer a holistic measure of image quality or image-text alignment, and are unsuited for fine-grained or instance-level analysis.
In this paper, we introduce GenEval, an object-focused framework to evaluate compositional image properties such as object co-occurrence, position, count, and color.
We show that current object detection models can be leveraged to evaluate text-to-image models on a variety of generation tasks with strong human agreement, and that other discriminative vision models can be linked to this pipeline to further verify properties like object color.
We then evaluate several open-source text-to-image models and analyze their relative generative capabilities on our benchmark.
We find that recent models demonstrate significant improvement on these tasks, though they are still lacking in complex capabilities such as spatial relations and attribute binding.
Finally, we demonstrate how GenEval might be used to help discover existing failure modes, in order to inform development of the next generation of text-to-image models.*

### Summary figure

### Main results

| Model | Overall | Single object | Two object | Counting | Colors | Position | Color attribution |
| ----- | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: |
| CLIP retrieval (baseline) | **0.35** | 0.89 | 0.22 | 0.37 | 0.62 | 0.03 | 0.00 |
minDALL-E | **0.23** | 0.73 | 0.11 | 0.12 | 0.37 | 0.02 | 0.01 |
Stable Diffusion v1.5 | **0.43** | 0.97 | 0.38 | 0.35 | 0.76 | 0.04 | 0.06 |
Stable Diffusion v2.1 | **0.50** | 0.98 | 0.51 | 0.44 | 0.85 | 0.07 | 0.17 |
Stable Diffusion XL | **0.55** | 0.98 | 0.74 | 0.39 | 0.85 | 0.15 | 0.23 |
IF-XL | **0.61** | 0.97 | 0.74 | 0.66 | 0.81 | 0.13 | 0.35 |

## Code

### Setup

Install the dependencies, including `mmdet`, and download the Mask2Former object detector:

```bash
git clone https://github.com/djghosh13/geneval.git
cd geneval
conda env create -f environment.yml
conda activate geneval
./evaluation/download_models.sh "<OBJECT_DETECTOR_FOLDER>/"

git clone https://github.com/open-mmlab/mmdetection.git
cd mmdetection; git checkout 2.x
pip install -v -e .
```

The original GenEval prompts from the paper are already in `prompts/`, but you can sample new prompts with different random seeds using

```bash
python prompts/create_prompts.py --seed <SEED> -n <NUM_PROMPTS> -o "<PROMPT_FOLDER>/"
```

### Image generation

Sample image generation code for Stable Diffusion models is given in `generation/diffusers_generate.py`. Run

```bash
python generation/diffusers_generate.py \
    "<PROMPT_FOLDER>/evaluation_metadata.jsonl" \
    --model "runwayml/stable-diffusion-v1-5" \
    --outdir "<IMAGE_FOLDER>"
```

to generate 4 images per prompt using Stable Diffusion v1.5 and save in `<IMAGE_FOLDER>`.

The generated format should be

```
<IMAGE_FOLDER>/
    00000/
        metadata.jsonl
        grid.png
        samples/
            0000.png
            0001.png
            0002.png
            0003.png
    00001/
        ...
```

where `metadata.jsonl` contains the `N`-th line from `evaluation_metadata.jsonl`. `grid.png` is optional here.

### Evaluation

```bash
python evaluation/evaluate_images.py \
    "<IMAGE_FOLDER>" \
    --outfile "<RESULTS_FOLDER>/results.jsonl" \
    --model-path "<OBJECT_DETECTOR_FOLDER>"
```

This will result in a JSONL file with each line corresponding to an image. In particular, each line has a `correct` key and a `reason` key specifying whether the generated image was deemed correct and, if applicable, why it was marked incorrect. You can run

```bash
python evaluation/summary_scores.py "<RESULTS_FOLDER>/results.jsonl"
```

to get the score across each task, and the overall GenEval score.


# Changelog

## \[0.3.0\] - 2023-01-05

### Added

- Add argument `--save-stats` allowing to compute dataset statistics and save them as an `.npz` file ([#80](https://github.com/mseitzer/pytorch-fid/pull/80)). The `.npz` file can be used in subsequent FID computations instead of recomputing the dataset statistics. This option can be used in the following way: `python -m pytorch_fid --save-stats path/to/dataset path/to/outputfile`.

### Fixed

- Do not use `os.sched_getaffinity` to get number of available CPUs on Windows, as it is not available there ([232b3b14](https://github.com/mseitzer/pytorch-fid/commit/232b3b1468800102fcceaf6f2bb8977811fc991a), [#84](https://github.com/mseitzer/pytorch-fid/issues/84)).
- Do not use Inception model argument `pretrained`, as it was deprecated in torchvision 0.13 ([#88](https://github.com/mseitzer/pytorch-fid/pull/88)).

## \[0.2.1\] - 2021-10-10

### Added

- Add argument `--num-workers` to select number of dataloader processes ([#66](https://github.com/mseitzer/pytorch-fid/pull/66)). Defaults to 8 or the number of available CPUs if less than 8 CPUs are available.

### Fixed

- Fixed package setup to work under Windows ([#55](https://github.com/mseitzer/pytorch-fid/pull/55), [#72](https://github.com/mseitzer/pytorch-fid/issues/72))

## \[0.2.0\] - 2020-11-30

### Added

- Load images using a Pytorch dataloader, which should result in a speed-up. ([#47](https://github.com/mseitzer/pytorch-fid/pull/47))
- Support more image extensions ([#53](https://github.com/mseitzer/pytorch-fid/pull/53))
- Improve tooling by setting up Nox, add linting and test support ([#52](https://github.com/mseitzer/pytorch-fid/pull/52))
- Add some unit tests

## \[0.1.1\] - 2020-08-16

### Fixed

- Fixed software license string in `setup.py`

## \[0.1.0\] - 2020-08-16

Initial release as a pypi package. Use `pip install pytorch-fid` to install.


[![PyPI](https://img.shields.io/pypi/v/pytorch-fid.svg)](https://pypi.org/project/pytorch-fid/)

# FID score for PyTorch

This is a port of the official implementation of [Fréchet Inception Distance](https://arxiv.org/abs/1706.08500) to PyTorch.
See [https://github.com/bioinf-jku/TTUR](https://github.com/bioinf-jku/TTUR) for the original implementation using Tensorflow.

FID is a measure of similarity between two datasets of images.
It was shown to correlate well with human judgement of visual quality and is most often used to evaluate the quality of samples of Generative Adversarial Networks.
FID is calculated by computing the [Fréchet distance](https://en.wikipedia.org/wiki/Fr%C3%A9chet_distance) between two Gaussians fitted to feature representations of the Inception network.

Further insights and an independent evaluation of the FID score can be found in [Are GANs Created Equal? A Large-Scale Study](https://arxiv.org/abs/1711.10337).

The weights and the model are exactly the same as in [the official Tensorflow implementation](https://github.com/bioinf-jku/TTUR), and were tested to give very similar results (e.g. `.08` absolute error and `0.0009` relative error on LSUN, using ProGAN generated images). However, due to differences in the image interpolation implementation and library backends, FID results still differ slightly from the original implementation. So if you report FID scores in your paper, and you want them to be *exactly comparable* to FID scores reported in other papers, you should consider using [the official Tensorflow implementation](https://github.com/bioinf-jku/TTUR).

## Installation

Install from [pip](https://pypi.org/project/pytorch-fid/):

```
pip install pytorch-fid
```

Requirements:

- python3
- pytorch
- torchvision
- pillow
- numpy
- scipy

## Usage

To compute the FID score between two datasets, where images of each dataset are contained in an individual folder:

```
python -m pytorch_fid path/to/dataset1 path/to/dataset2
```

To run the evaluation on GPU, use the flag `--device cuda:N`, where `N` is the index of the GPU to use.

### Using different layers for feature maps

In difference to the official implementation, you can choose to use a different feature layer of the Inception network instead of the default `pool3` layer.
As the lower layer features still have spatial extent, the features are first global average pooled to a vector before estimating mean and covariance.

This might be useful if the datasets you want to compare have less than the otherwise required 2048 images.
Note that this changes the magnitude of the FID score and you can not compare them against scores calculated on another dimensionality.
The resulting scores might also no longer correlate with visual quality.

You can select the dimensionality of features to use with the flag `--dims N`, where N is the dimensionality of features.
The choices are:

- 64:   first max pooling features
- 192:  second max pooling features
- 768:  pre-aux classifier features
- 2048: final average pooling features (this is the default)

## Generating a compatible `.npz` archive from a dataset

A frequent use case will be to compare multiple models against an original dataset.
To save training multiple times on the original dataset, there is also the ability to generate a compatible `.npz` archive from a dataset. This is done using any combination of the previously mentioned arguments with the addition of the `--save-stats` flag. For example:

```
python -m pytorch_fid --save-stats path/to/dataset path/to/outputfile
```

The output file may then be used in place of the path to the original dataset for further comparisons.

## Citing

If you use this repository in your research, consider citing it using the following Bibtex entry:

```
@misc{Seitzer2020FID,
  author={Maximilian Seitzer},
  title={{pytorch-fid: FID Score for PyTorch}},
  month={August},
  year={2020},
  note={Version 0.3.0},
  howpublished={\url{https://github.com/mseitzer/pytorch-fid}},
}
```

## License

This implementation is licensed under the Apache License 2.0.

FID was introduced by Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler and Sepp Hochreiter in "GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium", see [https://arxiv.org/abs/1706.08500](https://arxiv.org/abs/1706.08500)

The original implementation is by the Institute of Bioinformatics, JKU Linz, licensed under the Apache License 2.0.
See [https://github.com/bioinf-jku/TTUR](https://github.com/bioinf-jku/TTUR).


# a fast implementation of linear attention

## 64x64, fp16

```bash
# validate correctness
## fp16 vs fp32
python -m develop_triton_litemla attn_type=LiteMLA test_correctness=True
## triton fp16 vs fp32
python -m develop_triton_litemla attn_type=TritonLiteMLA test_correctness=True

# test performance
## fp16, forward
python -m develop_triton_litemla attn_type=LiteMLA
each step takes 10.81 ms
max memory allocated: 2.2984 GB

## triton fp16, forward
python -m develop_triton_litemla attn_type=TritonLiteMLA
each step takes 4.70 ms
max memory allocated: 1.6480 GB

## fp16, backward
python -m develop_triton_litemla attn_type=LiteMLA backward=True
each step takes 35.34 ms
max memory allocated: 3.4412 GB

## triton fp16, backward
python -m develop_triton_litemla attn_type=TritonLiteMLA backward=True
each step takes 14.25 ms
max memory allocated: 2.4704 GB
```


        中文 &nbsp｜ &nbsp English&nbsp&nbsp ｜ &nbsp日本語

# Step-Audio

## 🔥🔥🔥 News!!
* Feb 17, 2025: 👋 We release the inference code and model weights of [Step-Audio-Chat](https://huggingface.co/stepfun-ai/Step-Audio-Chat), [Step-Audio-TTS-3B](https://huggingface.co/stepfun-ai/Step-Audio-TTS-3B) and [Step-Audio-Tokenizer](https://huggingface.co/stepfun-ai/Step-Audio-Tokenizer)
* Feb 17, 2025: 👋 We release the multi-turn audio benchmark of [StepEval-Audio-360](https://huggingface.co/datasets/stepfun-ai/StepEval-Audio-360).
* Feb 17, 2025: 👋 We release the technical report of [Step-Audio](https://arxiv.org/abs/2502.11946).

## Table of Contents

1. [Introduction](#1-introduction)
2. [Model Summary](#2-model-summary)
3. [Model Download](#3-model-download)
4. [Model Usage](#4-model-usage)
5. [Benchmark](#5-benchmark)
6. [Online Engine](#6-online-engine)
7. [Examples](#7-examples)
8. [Acknowledgements](#8-acknowledgements)
9. [License Agreement](#9-license-agreement)
10. [Citation](#10-citation)

## 1. Introduction

Step-Audio is the first production-ready open-source framework for intelligent speech interaction that harmonizes comprehension and generation, supporting multilingual conversations (e.g., Chinese, English, Japanese), emotional tones (e.g., joy/sadness), regional dialects (e.g., Cantonese/Sichuanese), adjustable speech rates, and prosodic styles (e.g., rap). Step-Audio demonstrates four key technical innovations:

- **130B-Parameter Multimodal Model**: A single unified model integrating comprehension and generation capabilities, performing speech recognition, semantic understanding, dialogue, voice cloning, and speech synthesis. We have made the 130B Step-Audio-Chat variant open source.

- **Generative Data Engine**: Eliminates traditional TTS's reliance on manual data collection by generating high-quality audio through our 130B-parameter multimodal model. Leverages this data to train and publicly release a resource-efficient Step-Audio-TTS-3B model with enhanced instruction-following capabilities for controllable speech synthesis.

- **Granular Voice Control**: Enables precise regulation through instruction-based control design, supporting multiple emotions (anger, joy, sadness), dialects (Cantonese, Sichuanese, etc.), and vocal styles (rap, a cappella humming) to meet diverse speech generation needs.

- **Enhanced Intelligence**: Improves agent performance in complex tasks through ToolCall mechanism integration and role-playing enhancements.

## 2. Model Summary
In Step-Audio, audio streams are tokenized via a dual-codebook framework, combining parallel semantic (16.7Hz, 1024-entry codebook) and acoustic (25Hz, 4096-entry codebook) tokenizers with 2:3 temporal interleaving. A 130B-parameter LLM foundation (Step-1) is further enhanced via audio-contextualized continual pretraining and task-specific post-training, enabling robust cross-modal speech understanding. A hybrid speech decoder combining flow matching with neural vocoding, optimized for real-time waveform generation. A streaming-aware architecture featuring speculative response generation (40\% commit rate) and text-based context management (14:1 compression ratio) for efficient cross-modal alignment.
![Architecture](assets/architecture.png)

### 2.1 Tokenizer

We implement a token-level interleaving approach to effectively integrate semantic tokenization and acoustic tokenization. The semantic tokenizer employs a codebook size of 1024, while the acoustic tokenizer utilizes a larger codebook size of 4096 to capture finer acoustic details. Given the differing token rates, we establish a temporal alignment ratio of 2:3, where every two semantic tokens are paired with three acoustic tokens.

### 2.2 Language Model

To enhance Step-Audio’s ability to effectively process speech information and
achieve accurate speech-text alignment, we conducted audio continual pretrain-ing based on Step-1, a 130-billion parameter pretrained text-based large language model (LLM).

### 2.3 Speech Decoder
The speech decoder in Step-Audio serves a critical function in converting discrete speech tokens, which contain both semantic and acoustic information, into continuous time-domain waveforms that represent natural speech. The decoder architecture incorporates a flow matching model and a mel-to-wave vocoder. To optimize the intelligibility and naturalness of the synthesized speech, the speech decoder is trained using a dual-code interleaving approach, ensuring seamless integration of semantic and acoustic features throughout the generation process.

### 2.4 Real-time Inference Pipeline
To enable real-time interactions, we have designed an optimized inference pipeline. At its core, the Controller module manages state transitions, orchestrates speculative response generation, and ensures seamless coordination between critical subsystems. These subsystems include Voice Activity Detection (VAD) for detecting user speech, the Streaming Audio Tokenizer for processing audio in real-time, the Step-Audio language model and Speech Decoder for processing and generating responses, and the Context Manager for preserving conversational continuity.
![Inference Pipeline](assets/pipeline.png)

### 2.5 Post training details
In the post-training phase, we conducted task-specific Supervised Fine-Tuning (SFT) for Automatic Speech Recognition (ASR) and Text-to-Speech (TTS). For Audio Input Text Output (AQTA) tasks, we implemented SFT using diversified high-quality datasets combined with Reinforcement Learning from Human Feedback (RLHF) to enhance response quality, enabling fine-grained control over emotional expression, speech speed, dialect, and prosody.
![RLHF](assets/rlhf.png)

## 3. Model Download
### 3.1 Huggingface
| Models   | Links   |
|-------|-------|
| Step-Audio-Tokenizer | [🤗huggingface](https://huggingface.co/stepfun-ai/Step-Audio-Tokenizer) |
| Step-Audio-Chat | [🤗huggingface](https://huggingface.co/stepfun-ai/Step-Audio-Chat) |
| Step-Audio-TTS-3B | [🤗huggingface](https://huggingface.co/stepfun-ai/Step-Audio-TTS-3B) |

### 3.2 Modelscope
| Models   | Links   |
|-------|-------|
| Step-Audio-Tokenizer | [modelscope](https://modelscope.cn/models/stepfun-ai/Step-Audio-Tokenizer) |
| Step-Audio-Chat | [modelscope](https://modelscope.cn/models/stepfun-ai/Step-Audio-Chat) |
| Step-Audio-TTS-3B | [modelscope](https://modelscope.cn/models/stepfun-ai/Step-Audio-TTS-3B) |

## 4. Model Usage
### 📜 4.1  Requirements
The following table shows the requirements for running Step-Audio model (batch size = 1):

|     Model    |  Setting
(sample frequency) | GPU Minimum Memory  |
|------------|--------------------------------|----------------|
| Step-Audio-Tokenizer   |        41.6Hz          |       1.5GB        |
| Step-Audio-Chat   |        41.6Hz          |       265GB        |
| Step-Audio-TTS-3B   |        41.6Hz          |       8GB        |

* An NVIDIA GPU with CUDA support is required.
  * The model is tested on a four A800 80G GPU.
  * **Recommended**: We recommend using 4xA800/H800 GPU with 80GB memory for better generation quality.
* Tested operating system: Linux

### 🔧 4.2 Dependencies and Installation
- Python >= 3.10.0 (Recommend to use [Anaconda](https://www.anaconda.com/download/#linux) or [Miniconda](https://docs.conda.io/en/latest/miniconda.html))
- [PyTorch >= 2.3-cu121](https://pytorch.org/)
- [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads)

```bash
git clone https://github.com/stepfun-ai/Step-Audio.git
conda create -n stepaudio python=3.10
conda activate stepaudio

cd Step-Audio
pip install -r requirements.txt

git lfs install
git clone https://huggingface.co/stepfun-ai/Step-Audio-Tokenizer
git clone https://huggingface.co/stepfun-ai/Step-Audio-Chat
git clone https://huggingface.co/stepfun-ai/Step-Audio-TTS-3B

```

After downloading the models, where_you_download_dir should have the following structure:
```
where_you_download_dir
├── Step-Audio-Tokenizer
├── Step-Audio-Chat
├── Step-Audio-TTS-3B
```

###  🚀 4.3 Inference Scripts
#### Offline inference
Inference with e2e audio/text input and audio/text output.
```bash
python offline_inference.py --model-path where_you_download_dir
```
#### TTS inference
Inference tts with default speaker or clone with a new speaker
```bash
python tts_inference.py --model-path where_you_download_dir --output-path where_you_save_audio_dir --synthesis-type use_tts_or_clone
```
A speaker information dict is required for clone mode, formatted as follows:
```bash
{
    "speaker": "speaker id",
    "prompt_text": "content of prompt wav",
    "wav_path": "prompt wav path"
}
```

#### Launch Web Demo
Start a local server for online inference.
Assume you have 4 GPUs available and have already downloaded all the models.

```bash
python app.py --model-path where_you_download_dir
```

#### Inference Chat Model with vLLM (recommended)
Step-Audio-Chat is a 130B LLM Model, it is recommended to use vLLM to inference with tensor parallelism.

Currently, the official vLLM does not support the Step 1 model. You can temporarily use our [development branch](https://github.com/Oliver-ss/vllm/tree/add-step1-model) for local installation.

**Because our attention mechanism is a variant of ALIBI, the official flash attention library is not compatible. We have provided a custom flash attention library in the [Step-Audio-Chat](https://huggingface.co/stepfun-ai/Step-Audio-Chat/tree/main/lib) repository. Make sure export the custom flash attention library to the environment variable before running the model.**

```bash
export OPTIMUS_LIB_PATH=where_you_download_dir/Step-Audio-Chat/lib

vllm serve where_you_download_dir/Step-Audio-Chat --dtype auto -tp $tp --served-model-name step_chat_audio --trust-remote-code
```

## 5. Benchmark

### 5.1 ASR result comparison

            Hidden Feature Modeling
            Discrete Audio Token Modeling

            Whisper Large-v3
            Qwen2-Audio
            MinMo
            LUCY
            Moshi
            GLM-4-voice Base
            GLM-4-voice Chat
            Step-Audio Pretrain
            Step-Audio-Chat

            Aishell-1
            5.14
            1.53
            -
            2.4
            -
            2.46
            226.47
            0.87
            1.95

            Aishell-2 ios
            4.76
            3.06
            2.69
            -
            -
            -
            211.3
            2.91
            3.57

            Wenetspeech test-net
            9.68
            7.72
            6.64
            8.78
            -
            -
            146.05
            7.62
            8.75

            Wenet test-meeting
            18.54
            8.4
            7.6
            10.42
            -
            -
            140.82
            7.78
            9.52

            Librispeech test-clean
            1.9
            1.6
            1.6
            3.36
            5.7
            2.82
            75.39
            2.36
            3.11

            Librispeech test-other
            3.65
            3.6
            3.82
            8.05
            -
            7.66
            80.3
            6.32
            8.44

            AVG
            7.28
            4.32
            -
            -
            -
            -
            146.74
            4.64
            5.89

### 5.2 TTS
#### 5.2.1 Performance comparison of content consistency (CER/WER) between GLM-4-Voice and MinMo.

            Model
            test-zh
            test-en

            CER (%) ↓
            WER (%) ↓

            GLM-4-Voice
            2.19
            2.91

            MinMo
            2.48
            2.90

            Step-Audio
            1.53
            2.71

#### 5.2.2 Results of TTS Models on SEED Test Sets.
* StepAudio-TTS-3B-Single denotes dual-codebook backbone with single-codebook vocoder*

            Model
            test-zh
            test-en

            CER (%) ↓
            SS ↑
            WER (%) ↓
            SS ↑

            FireRedTTS
            1.51
            0.630
            3.82
            0.460

            MaskGCT
            2.27
            0.774
            2.62
            0.774

            CosyVoice
            3.63
            0.775
            4.29
            0.699

            CosyVoice 2
            1.45
            0.806
            2.57
            0.736

            CosyVoice 2-S
            1.45
            0.812
            2.38
            0.743

            Step-Audio-TTS-3B-Single
            1.37
            0.802
            2.52
            0.704

            Step-Audio-TTS-3B
            1.31
            0.733
            2.31
            0.660

            Step-Audio-TTS
            1.17
            0.73
            2.0
            0.660

#### 5.2.3 Performance comparison of Dual-codebook Resynthesis with Cosyvoice.

            Token
            test-zh
            test-en

            CER (%) ↓
            SS ↑
            WER (%) ↓
            SS ↑

            Groundtruth
            0.972
            -
            2.156
            -

            CosyVoice
            2.857
            0.849
            4.519
            0.807

            Step-Audio-TTS-3B
            2.192
            0.784
            3.585
            0.742

### 5.3 AQTA Chat
We release [**StepEval-Audio-360**](https://huggingface.co/datasets/stepfun-ai/StepEval-Audio-360) as a new benchmark, which consists of 137 multi-turn Chinese prompts sourced from real users and is designed to evaluate the quality of generated response across the following dimensions: Voice Instruction Following, Voice Understanding, Logical Reasoning, Role-playing, Creativity, Sing, Language Ability, Speech Emotion Control, Gaming.

#### 5.3.1 StepEval-Audio-360

#### LLM judge metrics(GPT-4o)

Comparison of fundamental capabilities of voice chat on the StepEval-Audio-360.

            Model
            Factuality (% ↑)
            Relevance (% ↑)
            Chat Score ↑

            GLM4-Voice
            54.7
            66.4
            3.49

            Qwen2-Audio
            22.6
            26.3
            2.27

            Moshi*
            1.0
            0
            1.49

            Step-Audio-Chat
            66.4
            75.2
            4.11

* Note: Moshi are marked with "\*" and should be considered for reference only.

#### Radar Chart(Human Evaluation)

#### 5.3.2 Public Test Set

            Model
            Llama Question
            Web Questions
            TriviaQA*
            ComplexBench
            HSK-6

            GLM4-Voice
            64.7
            32.2
            39.1
            66.0
            74.0

            Moshi
            62.3
            26.6
            22.8
            -
            -

            Freeze-Omni
            72.0
            44.7
            53.9
            -
            -

            LUCY
            59.7
            29.3
            27.0
            -
            -

            MinMo
            78.9
            55.0
            48.3
            -
            -

            Qwen2-Audio
            52.0
            27.0
            37.3
            54.0
            -

            Step-Audio-Chat
            81.0
            75.1
            58.0
            74.0
            86.0

* Note: Results marked with "\*" on TriviaQA dataset are considered for reference only.*

#### 5.3.3 Audio instruction following

            Category
            Instruction Following
            Audio Quality

            GLM-4-Voice
            Step-Audio
            GLM-4-Voice
            Step-Audio

            Languages
            1.9
            3.8
            2.9
            3.3

            Role-playing
            3.8
            4.2
            3.2
            3.6

            Singing / RAP
            2.1
            2.4
            2.4
            4

            Voice Control
            3.6
            4.4
            3.3
            4.1

## 6. Online Engine
The online version of Step-Audio can be accessed from app version of [跃问](https://yuewen.cn), where some impressive examples can be found as well.

## 7. Examples
### Clone audio
| role   | prompt wav | clone wav |
|:-------:|:-------:|:-------:|
|于谦| [google drive](https://drive.google.com/file/d/1N9EJypafFwmeL0R152GoL_CVGbYn1_9A/preview)
[audio file](https://github.com/stepfun-ai/Step-Audio/tree/main/examples/prompt_wav_yuqian.wav)|[google drive](https://drive.google.com/file/d/1Zs_1QrCUuoSqtUSdn2ENIor-k5baQdDV/preview)
[audio file](https://github.com/stepfun-ai/Step-Audio/tree/main/examples/clone_wav_yuqian.wav)|
|李雪琴| [google drive](https://drive.google.com/file/d/15SkZ29hksELYi1NDOxYOPu-kRTLSyke_/preview)
[audio file](https://github.com/stepfun-ai/Step-Audio/tree/main/examples/prompt_wav_lixueqin.wav)|[google drive](https://drive.google.com/file/d/11Le4qMqL2DmWpf7RFRpKUXERIR9TtKC0/preview)
[audio file](https://github.com/stepfun-ai/Step-Audio/tree/main/examples/clone_wav_lixueqin.wav)|

### Speed control
| prompt | response |
|:-------:|:-------:|
|Human: 说一个绕口令
Assistant: 吃葡萄不吐葡萄皮，不吃葡萄倒吐葡萄皮
Human: 哎，你能把这个绕口令说的再快一点吗？|[google drive](https://drive.google.com/file/d/1mAH-NRrOVZo4tv6gdAZkyJg8kRuTNNGC/preview)
[audio file](https://github.com/stepfun-ai/Step-Audio/tree/main/examples/speed_control1.wav)|
|Human: 说一个绕口令
Assistant: 吃葡萄不吐葡萄皮，不吃葡萄倒吐葡萄皮
Human: 哎，你能把这个绕口令说的再快一点吗？
Assistant: 吃葡萄不吐葡萄皮，不吃葡萄倒吐葡萄皮
Human: 呃，你再用非常非常慢的速度说一遍的。|[google drive](https://drive.google.com/file/d/1FhRnKo8uGrtO-cWg4qkrg8iDoNRbtqSX/preview)
[audio file](https://github.com/stepfun-ai/Step-Audio/tree/main/examples/speed_control2.wav)|

### High EQ(Emotional control & Tone control)
| prompt | response |
|:-------:|:-------:|
|Human: 你这语气又不撒娇又不卖萌的，要不你撒个娇卖个萌吧。|[google drive](https://drive.google.com/file/d/19IROE6_6h2UQVNniCmDTnrhxKRMOFHq3/preview)
[audio file](https://github.com/stepfun-ai/Step-Audio/tree/main/examples/tone_control.wav)|
|Human: 怎么办？我感觉我的人生很失败。|[google drive](https://drive.google.com/file/d/1JlLbOlzmdrokVdxtwy1S8eeWqsZR2Vmc/preview)
[audio file](https://github.com/stepfun-ai/Step-Audio/tree/main/examples/emotional_control1.wav)|
|Human: 小跃。你真的是。特别厉害。|[google drive](https://drive.google.com/file/d/19ga1RpguDP5r0Xfl1r5GY1J-kzbmHvJb/preview)
[audio file](https://github.com/stepfun-ai/Step-Audio/tree/main/examples/emotional_control2.wav)|

### Multilingual (e.g., Chinese, English, Japanese)
| prompt | response |
|:-------:|:-------:|
|Human: What did the speaker mean when they said, it's raining cats and dogs?
Assistant: When they say "It's raining cats and dogs," it just means it's raining really hard. The speaker isn't literally saying cats and dogs are falling from the sky! It's just a fun way to describe heavy rain.|[google drive](https://drive.google.com/file/d/1LEIvdR5ANMzWX8GOTqUPTNrynNS1xx--/preview)
[audio file](https://github.com/stepfun-ai/Step-Audio/tree/main/examples/multilingual2.wav)|
|Human: こんにちは。（你好）
Assistant：こんにちは！何か手伝いましょうか？（您好！我可以帮你做点什么吗？）|[google drive](https://drive.google.com/file/d/1MjKUkkzcGzVcNVXRr_Ya5y2H44K_lybH/preview)
[audio file](https://github.com/stepfun-ai/Step-Audio/tree/main/examples/multilingual1.wav)|

### Rap & Vocal
| prompt | response |
|:-------:|:-------:|
|Human: 唱一段rap|[google drive](https://drive.google.com/file/d/1F8CKmVbGZ7X7d1IkQPlmndSHeG40AXha/preview)
[audio file](https://github.com/stepfun-ai/Step-Audio/tree/main/examples/rap.wav)|
|Human: 唱一段中文的歌曲（Sing Chinese Song）|[google drive](https://drive.google.com/file/d/1F1o-Q90llmkM-4UU6qhP8KoHp3dhaluj/preview)
[audio file](https://github.com/stepfun-ai/Step-Audio/tree/main/examples/singing.wav)|
|Human: 唱一段日语的歌曲（Sing Japanese Song）|[google drive](https://drive.google.com/file/d/1kS2jQEq70ynh46pG_P5Xp1_NY3C-cJ-U/preview)
[audio file](https://github.com/stepfun-ai/Step-Audio/tree/main/examples/multilingual_singing.wav)|

## 8. Acknowledgements

Part of the code for this project comes from:
* [CosyVoice](https://github.com/FunAudioLLM/CosyVoice)
* [transformers](https://github.com/huggingface/transformers)
* [FunASR](https://github.com/modelscope/FunASR)

Thank you to all the open-source projects for their contributions to this project!
## 9. License Agreement

+ The use of weights for Step Audio related models requires following license in [Step-Audio-Chat](https://huggingface.co/stepfun-ai/Step-Audio-Chat/tree/main), [Step-Audio-Tokenizer](https://huggingface.co/stepfun-ai/Step-Audio-Tokenizer/tree/main) and [Step-Audio-TTS-3B](https://huggingface.co/stepfun-ai/Step-Audio-TTS-3B/tree/main)

+ The code in this open-source repository is licensed under the [Apache 2.0](LICENSE) License.

## 10. Citation
```
@misc{huang2025stepaudiounifiedunderstandinggeneration,
      title={Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction},
      author={Ailin Huang and Boyong Wu and Bruce Wang and Chao Yan and Chen Hu and Chengli Feng and Fei Tian and Feiyu Shen and Jingbei Li and Mingrui Chen and Peng Liu and Ruihang Miao and Wang You and Xi Chen and Xuerui Yang and Yechang Huang and Yuxiang Zhang and Zheng Gong and Zixin Zhang and Brian Li and Changyi Wan and Hanpeng Hu and Ranchen Ming and Song Yuan and Xuelin Zhang and Yu Zhou and Bingxin Li and Buyun Ma and Kang An and Wei Ji and Wen Li and Xuan Wen and Yuankai Ma and Yuanwei Liang and Yun Mou and Bahtiyar Ahmidi and Bin Wang and Bo Li and Changxin Miao and Chen Xu and Chengting Feng and Chenrun Wang and Dapeng Shi and Deshan Sun and Dingyuan Hu and Dula Sai and Enle Liu and Guanzhe Huang and Gulin Yan and Heng Wang and Haonan Jia and Haoyang Zhang and Jiahao Gong and Jianchang Wu and Jiahong Liu and Jianjian Sun and Jiangjie Zhen and Jie Feng and Jie Wu and Jiaoren Wu and Jie Yang and Jinguo Wang and Jingyang Zhang and Junzhe Lin and Kaixiang Li and Lei Xia and Li Zhou and Longlong Gu and Mei Chen and Menglin Wu and Ming Li and Mingxiao Li and Mingyao Liang and Na Wang and Nie Hao and Qiling Wu and Qinyuan Tan and Shaoliang Pang and Shiliang Yang and Shuli Gao and Siqi Liu and Sitong Liu and Tiancheng Cao and Tianyu Wang and Wenjin Deng and Wenqing He and Wen Sun and Xin Han and Xiaomin Deng and Xiaojia Liu and Xu Zhao and Yanan Wei and Yanbo Yu and Yang Cao and Yangguang Li and Yangzhen Ma and Yanming Xu and Yaqiang Shi and Yilei Wang and Yinmin Zhong and Yu Luo and Yuanwei Lu and Yuhe Yin and Yuting Yan and Yuxiang Yang and Zhe Xie and Zheng Ge and Zheng Sun and Zhewei Huang and Zhichao Chang and Zidong Yang and Zili Zhang and Binxing Jiao and Daxin Jiang and Heung-Yeung Shum and Jiansheng Chen and Jing Li and Shuchang Zhou and Xiangyu Zhang and Xinhao Zhang and Yibo Zhu},
      year={2025},
      eprint={2502.11946},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2502.11946},
}
```

## Star History

[![Star History Chart](https://api.star-history.com/svg?repos=stepfun-ai/Step-Audio&type=Date)](https://star-history.com/#stepfun-ai/Step-Audio&Date)

README.md


        中文&nbsp ｜ &nbspEnglish&nbsp ｜ &nbsp日本語

# Step-Audio

## 🔥🔥🔥 News!!
* 2025年2月17日: 👋 发布推理代码和模型权重，其中包含[Step-Audio-Chat](https://huggingface.co/stepfun-ai/Step-Audio-Chat), [Step-Audio-TTS-3B](https://huggingface.co/stepfun-ai/Step-Audio-TTS-3B) 和 [Step-Audio-Tokenizer](https://huggingface.co/stepfun-ai/Step-Audio-Tokenizer)。
* 2025年2月17日: 👋 发布多轮音频交互基准测试[StepEval-Audio-360](https://huggingface.co/datasets/stepfun-ai/StepEval-Audio-360)。
* 2025年2月17日: 👋 发布了技术报告[Step-Audio-Report](https://arxiv.org/abs/2502.11946)。

## Table of Contents

1. [介绍](#1-介绍)
2. [模型组成](#2-模型组成)
3. [模型下载](#3-模型下载)
4. [模型使用](#4-模型使用)
5. [基准](#5-基准)
6. [在线引擎](#6-在线引擎)
7. [样例](#7-样例)
8. [致谢](#8-致谢)
9. [协议](#9-协议)
10. [引用](#10-引用)

## 1. 介绍

Step-Audio 是业界首个集语音理解与生成控制一体化的产品级开源实时语音对话系统，支持多语言对话（如 中文，英文，日语），语音情感（如 开心，悲伤），方言（如 粤语，四川话），可控制语速及韵律风格，支持RAP和哼唱等。其核心技术突破体现在以下四大技术亮点：

- **1300亿多模态模型**: 单模型能实现理解生成一体化完成语音识别、语义理解、对话、语音克隆、语音生成等功能，开源千亿参数多模态模型 Step-Audio-Chat。

- **高效数据生成链路**: 基于130B 突破传统 TTS 对人工采集数据的依赖，生成高质量的合成音频数据，并同步开源首个基于大规模合成数据训练，支持 RAP 和哼唱的指令加强版语音合成模型 Step-Audio-TTS-3B 。

- **精细语音控制**: 支持多种情绪（如生气，高兴，悲伤）、方言（包括粤语、四川话等）和唱歌（包括 RAP、干声哼唱）的精准调控，满足用户对多样化语音生成的需求。

- **扩展工具调用**: 通过 ToolCall 机制和角色扮演增强，进一步提升其在 Agents 和复杂任务中的表现。

## 2. 模型组成

在Step-Audio系统中，音频流采用Linguistic tokenizer（码率16.7Hz，码本大小1024）与Semantice tokenizer（码率25Hz，码本大小4096）并行的双码本编码器方案，双码本在排列上使用了2:3时序交错策略。通过音频语境化持续预训练和任务定向微调强化了130B参数量的基础模型（Step-1），最终构建了强大的跨模态语音理解能力。为了实现实时音频生成，系统采用了混合语音解码器，结合流匹配（flow matching）与神经声码技术。
![Architecture](assets/architecture.png)

### 2.1 Tokenizer

我们通过token级交错方法实现Linguistic token与Semantic token的有效整合。Linguistic tokenizer的码本大小是1024，码率16.7Hz；而Semantic tokenizer则使用4096的大容量码本来捕捉更精细的声学细节，码率25Hz。鉴于两者的码率差异，我们建立了2:3的时间对齐比例——每两个Linguistic token对应三个Linguistic token形成时序配对。

### 2.2 语言模型

为了提升Step-Audio有效处理语音信息的能力，并实现精准的语音-文本对齐，我们在Step-1（一个拥有1300亿参数的基于文本的大型语言模型LLM）的基础上进行了音频持续预训练。

### 2.3 语音解码器

Step-Audio语音解码器主要是将包含语义和声学信息的离散标记信息转换成连续的语音信号。该解码器架构结合了一个30亿参数的语言模型、流匹配模型（flow matching model）和梅尔频谱到波形的声码器（mel-to-wave vocoder）。为优化合成语音的清晰度（intelligibility）和自然度（naturalness），语音解码器采用双码交错训练方法（dual-code interleaving），确保生成过程中语义与声学特征的无缝融合。

### 2.4 实时推理管线
为了实现实时的语音交互，我们对推理管线进行了一系列优化。其中最核心的是控制模块（Controller），该模块负责管理状态转换、协调响应生成，并确保关键子系统间的无缝协同。这些子系统包括：

- **语音活动检测（VAD）**：实时检测用户语音起止

- **流式音频分词器（Streaming Audio Tokenizer）**：实时音频流处理

- **Step-Audio语言模型与语音解码器**：多模态回复生成

- **上下文管理器（Context Manager）**：动态维护对话历史与状态
![Inference Pipeline](assets/pipeline.png)

### 2.5 后训练细节
在后训练阶段，我们针对自动语音识别（ASR）与文本转语音（TTS）任务进行了专项监督微调（Supervised Fine-Tuning, SFT）。对于音频输入-文本输出（Audio Question Text Answer, AQTA）任务，我们采用多样化高质量数据集进行SFT，并采用了基于人类反馈的强化学习（RLHF）以提升响应质量，从而实现对情感表达、语速、方言及韵律的细粒度控制。
![RLHF](assets/rlhf.png)

## 3. 模型下载
### 3.1 Huggingface
| 模型   | 链接   |
|-------|-------|
| Step-Audio-Tokenizer | [🤗huggingface](https://huggingface.co/stepfun-ai/Step-Audio-Tokenizer) |
| Step-Audio-Chat | [🤗huggingface](https://huggingface.co/stepfun-ai/Step-Audio-Chat) |
| Step-Audio-TTS-3B | [🤗huggingface](https://huggingface.co/stepfun-ai/Step-Audio-TTS-3B) |

### 3.2 Modelscope
| 模型   | 链接   |
|-------|-------|
| Step-Audio-Tokenizer | [modelscope](https://modelscope.cn/models/stepfun-ai/Step-Audio-Tokenizer) |
| Step-Audio-Chat | [modelscope](https://modelscope.cn/models/stepfun-ai/Step-Audio-Chat) |
| Step-Audio-TTS-3B | [modelscope](https://modelscope.cn/models/stepfun-ai/Step-Audio-TTS-3B) |

## 4. 模型使用
### 📜 4.1  要求
下表列出了运行Step-Audio模型（batch size=1）所需的配置要求:

|     模型    |  Setting
(采样率) | GPU最低显存  |
|------------|--------------------------------|----------------|
| Step-Audio-Tokenizer   |        41.6Hz          |       1.5GB        |
| Step-Audio-Chat   |        41.6Hz          |       265GB        |
| Step-Audio-TTS-3B   |        41.6Hz          |       8GB        |

* 需要支持CUDA的NVIDIA显卡.
  * 模型在4块显存为80GB的A800系列NVIDIA显卡上进行测试.
  * **推荐**: 为确保最佳生成质量，建议使用4块显存为80GB的A800/H800系列NVIDIA显卡.
* 测试采用的操作系统: Linux

### 🔧 4.2 依赖项与安装
- Python >= 3.10.0 (推荐使用 [Anaconda](https://www.anaconda.com/download/#linux) or [Miniconda](https://docs.conda.io/en/latest/miniconda.html))
- [PyTorch >= 2.3-cu121](https://pytorch.org/)
- [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads)

```bash
git clone https://github.com/stepfun-ai/Step-Audio.git
conda create -n stepaudio python=3.10
conda activate stepaudio

cd Step-Audio
pip install -r requirements.txt

git lfs install
git clone https://huggingface.co/stepfun-ai/Step-Audio-Tokenizer
git clone https://huggingface.co/stepfun-ai/Step-Audio-Chat
git clone https://huggingface.co/stepfun-ai/Step-Audio-TTS-3B

```

下载模型后，where_you_download_dir应包含以下结构：
```
where_you_download_dir
├── Step-Audio-Tokenizer
├── Step-Audio-Chat
├── Step-Audio-TTS-3B
```

###  🚀 4.3 推理脚本
#### 离线推理
支持端到端音频/文本输入与音频/文本输出的推理流程。
```bash
python offline_inference.py --model-path where_you_download_dir
```

#### 语音合成推理
使用默认音色进行语音合成推理或使用新音色进行克隆
```bash
python tts_inference.py --model-path where_you_download_dir --output-path where_you_save_audio_dir --synthesis-type use_tts_or_clone
```
克隆模式需要音色信息字典，格式如下：
```bash
{
    "speaker": "speaker id",
    "prompt_text": "content of prompt wav",
    "wav_path": "prompt wav path"
}
```

#### 启动网页演示
启动本地服务器以进行在线推理。
假设您已配备4块GPU且已完成所有模型的下载。

```bash
python app.py --model-path where_you_download_dir
```

## 5. 基准

### 5.1 语音识别

            隐层特征建模
            离散标记建模

            Whisper Large-v3
            Qwen2-Audio
            MinMo
            LUCY
            Moshi
            GLM-4-voice Base
            GLM-4-voice Chat
            Step-Audio Pretrain
            Step-Audio-Chat

            Aishell-1
            5.14
            1.53
            -
            2.4
            -
            2.46
            226.47
            0.87
            1.95

            Aishell-2 ios
            4.76
            3.06
            2.69
            -
            -
            -
            211.3
            2.91
            3.57

            Wenetspeech test-net
            9.68
            7.72
            6.64
            8.78
            -
            -
            146.05
            7.62
            8.75

            Wenet test-meeting
            18.54
            8.4
            7.6
            10.42
            -
            -
            140.82
            7.78
            9.52

            Librispeech test-clean
            1.9
            1.6
            1.6
            3.36
            5.7
            2.82
            75.39
            2.36
            3.11

            Librispeech test-other
            3.65
            3.6
            3.82
            8.05
            -
            7.66
            80.3
            6.32
            8.44

            AVG
            7.28
            4.32
            -
            -
            -
            -
            146.74
            4.64
            5.89

### 5.2 语音合成
#### 5.2.1 与GLM-4-Voice与MinMo在内容一致性（CER/WER）上的性能对比。

            Model
            test-zh
            test-en

            CER (%) ↓
            WER (%) ↓

            GLM-4-Voice
            2.19
            2.91

            MinMo
            2.48
            2.90

            Step-Audio
            1.53
            2.71

#### 5.2.2 语音合成模型在SEED测试集上的性能结果。
* StepAudio-TTS-3B-Single 表示采用双码本主干网络与单码本声码器的组合架构。

            Model
            test-zh
            test-en

            CER (%) ↓
            SS ↑
            WER (%) ↓
            SS ↑

            FireRedTTS
            1.51
            0.630
            3.82
            0.460

            MaskGCT
            2.27
            0.774
            2.62
            0.774

            CosyVoice
            3.63
            0.775
            4.29
            0.699

            CosyVoice 2
            1.45
            0.806
            2.57
            0.736

            CosyVoice 2-S
            1.45
            0.812
            2.38
            0.743

            Step-Audio-TTS-3B-Single
            1.37
            0.802
            2.52
            0.704

            Step-Audio-TTS-3B
            1.31
            0.733
            2.31
            0.660

            Step-Audio-TTS
            1.17
            0.73
            2.0
            0.660

#### 5.2.3 双码本重合成与CosyVoice性能对比。

            Token
            test-zh
            test-en

            CER (%) ↓
            SS ↑
            WER (%) ↓
            SS ↑

            Groundtruth
            0.972
            -
            2.156
            -

            CosyVoice
            2.857
            0.849
            4.519
            0.807

            Step-Audio-TTS-3B
            2.192
            0.784
            3.585
            0.742

### 5.3 语音对话
我们发布全新基准测试[StepEval-Audio-360](https://huggingface.co/datasets/stepfun-ai/StepEval-Audio-360)，该数据集包含137个源自真实用户的多轮中文提示，旨在系统性评估生成式语音交互系统在以下维度的表现：语音指令遵循、语音理解、逻辑推理、角色扮演、创作能力、唱歌、语言能力、语音情绪控制、游戏。
#### 5.3.1 StepEval-Audio-360

#### 大语言模型评估指标(GPT-4o)

Comparison of fundamental capabilities of voice chat on the StepEval-Audio-360.

            Model
            Factuality (% ↑)
            Relevance (% ↑)
            Chat Score ↑

            GLM4-Voice
            54.7
            66.4
            3.49

            Qwen2-Audio
            22.6
            26.3
            2.27

            Moshi*
            1.0
            0
            1.49

            Step-Audio-Chat
            66.4
            75.2
            4.11

* 注意：带有“\*”标记的内容仅供参考。

#### 雷达图(人工测评)

#### 5.3.2 公开测试集

            Model
            Llama Question
            Web Questions
            TriviaQA*
            ComplexBench
            HSK-6

            GLM4-Voice
            64.7
            32.2
            39.1
            66.0
            74.0

            Moshi
            62.3
            26.6
            22.8
            -
            -

            Freeze-Omni
            72.0
            44.7
            53.9
            -
            -

            LUCY
            59.7
            29.3
            27.0
            -
            -

            MinMo
            78.9
            55.0
            48.3
            -
            -

            Qwen2-Audio
            52.0
            27.0
            37.3
            54.0
            -

            Step-Audio-Chat
            81.0
            75.1
            58.0
            74.0
            86.0

* 注意：在 TriviaQA 数据集上，带有“\*”标记的结果仅供参考。

#### 5.3.3 语音指令遵循

            Category
            Instruction Following
            Audio Quality

            GLM-4-Voice
            Step-Audio
            GLM-4-Voice
            Step-Audio

            Languages
            1.9
            3.8
            2.9
            3.3

            Role-playing
            3.8
            4.2
            3.2
            3.6

            Singing / RAP
            2.1
            2.4
            2.4
            4

            Voice Control
            3.6
            4.4
            3.3
            4.1

## 6. 在线引擎
Step-Audio 的在线版本可以通过[跃问](https://yuewen.cn) 的应用程序访问，其中还可以找到一些惊喜的示例。

## 7. 样例
### 音频克隆
| role   | prompt wav | clone wav |
|:-------:|:-------:|:-------:|
|于谦| [google drive](https://drive.google.com/file/d/1N9EJypafFwmeL0R152GoL_CVGbYn1_9A/preview)
[audio file](https://github.com/stepfun-ai/Step-Audio/tree/main/examples/prompt_wav_yuqian.wav)|[google drive](https://drive.google.com/file/d/1Zs_1QrCUuoSqtUSdn2ENIor-k5baQdDV/preview)
[audio file](https://github.com/stepfun-ai/Step-Audio/tree/main/examples/clone_wav_yuqian.wav)|
|李雪琴| [google drive](https://drive.google.com/file/d/15SkZ29hksELYi1NDOxYOPu-kRTLSyke_/preview)
[audio file](https://github.com/stepfun-ai/Step-Audio/tree/main/examples/prompt_wav_lixueqin.wav)|[google drive](https://drive.google.com/file/d/11Le4qMqL2DmWpf7RFRpKUXERIR9TtKC0/preview)
[audio file](https://github.com/stepfun-ai/Step-Audio/tree/main/examples/clone_wav_lixueqin.wav)|

### 速度控制
| prompt | response |
|:-------:|:-------:|
|Human: 说一个绕口令
Assistant: 吃葡萄不吐葡萄皮，不吃葡萄倒吐葡萄皮
Human: 哎，你能把这个绕口令说的再快一点吗？|[google drive](https://drive.google.com/file/d/1mAH-NRrOVZo4tv6gdAZkyJg8kRuTNNGC/preview)
[audio file](https://github.com/stepfun-ai/Step-Audio/tree/main/examples/speed_control1.wav)|
|Human: 说一个绕口令
Assistant: 吃葡萄不吐葡萄皮，不吃葡萄倒吐葡萄皮
Human: 哎，你能把这个绕口令说的再快一点吗？
Assistant: 吃葡萄不吐葡萄皮，不吃葡萄倒吐葡萄皮
Human: 呃，你再用非常非常慢的速度说一遍的。|[google drive](https://drive.google.com/file/d/1FhRnKo8uGrtO-cWg4qkrg8iDoNRbtqSX/preview)
[audio file](https://github.com/stepfun-ai/Step-Audio/tree/main/examples/speed_control2.wav)|

### 高情商(情感控制 & 语调控制)
| prompt | response |
|:-------:|:-------:|
|Human: 你这语气又不撒娇又不卖萌的，要不你撒个娇卖个萌吧。|[google drive](https://drive.google.com/file/d/19IROE6_6h2UQVNniCmDTnrhxKRMOFHq3/preview)
[audio file](https://github.com/stepfun-ai/Step-Audio/tree/main/examples/tone_control.wav)|
|Human: 怎么办？我感觉我的人生很失败。|[google drive](https://drive.google.com/file/d/1JlLbOlzmdrokVdxtwy1S8eeWqsZR2Vmc/preview)
[audio file](https://github.com/stepfun-ai/Step-Audio/tree/main/examples/emotional_control1.wav)|
|Human: 小跃。你真的是。特别厉害。|[google drive](https://drive.google.com/file/d/19ga1RpguDP5r0Xfl1r5GY1J-kzbmHvJb/preview)
[audio file](https://github.com/stepfun-ai/Step-Audio/tree/main/examples/emotional_control2.wav)|

### 多语言 (e.g., 中文, 英文, 日语)
| prompt | response |
|:-------:|:-------:|
|Human: What did the speaker mean when they said, it's raining cats and dogs?
Assistant: When they say "It's raining cats and dogs," it just means it's raining really hard. The speaker isn't literally saying cats and dogs are falling from the sky! It's just a fun way to describe heavy rain.|[google drive](https://drive.google.com/file/d/1LEIvdR5ANMzWX8GOTqUPTNrynNS1xx--/preview)
[audio file](https://github.com/stepfun-ai/Step-Audio/tree/main/examples/multilingual2.wav)|
|Human: こんにちは。（你好）
Assistant：こんにちは！何か手伝いましょうか？（您好！我可以帮你做点什么吗？）|[google drive](https://drive.google.com/file/d/1MjKUkkzcGzVcNVXRr_Ya5y2H44K_lybH/preview)
[audio file](https://github.com/stepfun-ai/Step-Audio/tree/main/examples/multilingual1.wav)|

### Rap & Vocal
| prompt | response |
|:-------:|:-------:|
|Human: 唱一段rap|[google drive](https://drive.google.com/file/d/1F8CKmVbGZ7X7d1IkQPlmndSHeG40AXha/preview)
[audio file](https://github.com/stepfun-ai/Step-Audio/tree/main/examples/rap.wav)|

## 8. 致谢

本项目的部分代码来自：
* [CosyVoice](https://github.com/FunAudioLLM/CosyVoice)
* [transformers](https://github.com/huggingface/transformers)
* [FunASR](https://github.com/modelscope/FunASR)

感谢以上所有开源项目对本项目开源做出的贡献！
## 9. 协议

+ Step-Audio 相关模型的权重使用协议请分别需要按照[Step-Audio-Chat](https://huggingface.co/stepfun-ai/Step-Audio-Chat/tree/main)， [Step-Audio-Tokenizer](https://huggingface.co/stepfun-ai/Step-Audio-Tokenizer/tree/main) 和 [Step-Audio-TTS-3B](https://huggingface.co/stepfun-ai/Step-Audio-TTS-3B/tree/main) 里面的协议进行遵守

+ 本开源仓库的代码则遵循 [Apache 2.0](LICENSE) 协议。

## 10. 引用
```
@misc{huang2025stepaudiounifiedunderstandinggeneration,
      title={Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction},
      author={Ailin Huang and Boyong Wu and Bruce Wang and Chao Yan and Chen Hu and Chengli Feng and Fei Tian and Feiyu Shen and Jingbei Li and Mingrui Chen and Peng Liu and Ruihang Miao and Wang You and Xi Chen and Xuerui Yang and Yechang Huang and Yuxiang Zhang and Zheng Gong and Zixin Zhang and Brian Li and Changyi Wan and Hanpeng Hu and Ranchen Ming and Song Yuan and Xuelin Zhang and Yu Zhou and Bingxin Li and Buyun Ma and Kang An and Wei Ji and Wen Li and Xuan Wen and Yuankai Ma and Yuanwei Liang and Yun Mou and Bahtiyar Ahmidi and Bin Wang and Bo Li and Changxin Miao and Chen Xu and Chengting Feng and Chenrun Wang and Dapeng Shi and Deshan Sun and Dingyuan Hu and Dula Sai and Enle Liu and Guanzhe Huang and Gulin Yan and Heng Wang and Haonan Jia and Haoyang Zhang and Jiahao Gong and Jianchang Wu and Jiahong Liu and Jianjian Sun and Jiangjie Zhen and Jie Feng and Jie Wu and Jiaoren Wu and Jie Yang and Jinguo Wang and Jingyang Zhang and Junzhe Lin and Kaixiang Li and Lei Xia and Li Zhou and Longlong Gu and Mei Chen and Menglin Wu and Ming Li and Mingxiao Li and Mingyao Liang and Na Wang and Nie Hao and Qiling Wu and Qinyuan Tan and Shaoliang Pang and Shiliang Yang and Shuli Gao and Siqi Liu and Sitong Liu and Tiancheng Cao and Tianyu Wang and Wenjin Deng and Wenqing He and Wen Sun and Xin Han and Xiaomin Deng and Xiaojia Liu and Xu Zhao and Yanan Wei and Yanbo Yu and Yang Cao and Yangguang Li and Yangzhen Ma and Yanming Xu and Yaqiang Shi and Yilei Wang and Yinmin Zhong and Yu Luo and Yuanwei Lu and Yuhe Yin and Yuting Yan and Yuxiang Yang and Zhe Xie and Zheng Ge and Zheng Sun and Zhewei Huang and Zhichao Chang and Zidong Yang and Zili Zhang and Binxing Jiao and Daxin Jiang and Heung-Yeung Shum and Jiansheng Chen and Jing Li and Shuchang Zhou and Xiangyu Zhang and Xinhao Zhang and Yibo Zhu},
      year={2025},
      eprint={2502.11946},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2502.11946},
}
```

## Star History

[![Star History Chart](https://api.star-history.com/svg?repos=stepfun-ai/Step-Audio&type=Date)](https://star-history.com/#stepfun-ai/Step-Audio&Date)

README_CN.md


        中文 &nbsp ｜ &nbspEnglish&nbsp ｜ &nbsp 日本語&nbsp

# Step-Audio

## 🔥🔥🔥 ニュース!!
* 2025年2月17日: 👋 推論コードとモデルの重みをリリースしました。[Step-Audio-Chat](https://huggingface.co/stepfun-ai/Step-Audio-Chat), [Step-Audio-TTS-3B](https://huggingface.co/stepfun-ai/Step-Audio-TTS-3B) および [Step-Audio-Tokenizer](https://huggingface.co/stepfun-ai/Step-Audio-Tokenizer)。
* 2025年2月17日: 👋 マルチターンオーディオベンチマーク [StepEval-Audio-360](https://huggingface.co/datasets/stepfun-ai/StepEval-Audio-360) をリリースしました。
* 2025年2月17日: 👋 技術レポート [Step-Audio-Report](https://arxiv.org/abs/2502.11946) をリリースしました。

## 目次

1. [紹介](#1-紹介)
2. [モデル概要](#2-モデル概要)
3. [モデルのダウンロード](#3-モデルのダウンロード)
4. [モデルの使用方法](#4-モデルの使用方法)
5. [ベンチマーク](#5-ベンチマーク)
6. [オンラインエンジン](#6-オンラインエンジン)
7. [引用](#7-引用)

## 1. 紹介

Step-Audioは、音声理解と生成を統合した業界初の製品レベルのオープンソースリアルタイム音声対話システムであり、多言語対話（例：日本語、英語、中国語）、音声感情（例：喜び、悲しみ）、方言（例：関西弁、広東語）、音声速度および韻律スタイルの調整をサポートします。Step-Audioは、以下の4つの主要な技術革新を示しています：

- **1300億パラメータのマルチモーダルモデル**：単一の統合モデルで、音声認識、意味理解、対話、音声クローン、音声生成を実行します。1300億パラメータのStep-Audio-Chatバリアントをオープンソース化しました。

- **生成データエンジン**：従来のTTSが手動データ収集に依存することを排除し、1300億パラメータのマルチモーダルモデルを使用して高品質の音声を生成します。このデータを活用して、リソース効率の高いStep-Audio-TTS-3Bモデルをトレーニングし、制御可能な音声合成のための指示フォロー機能を強化しました。

- **細かい音声制御**：指示ベースの制御設計を通じて、複数の感情（怒り、喜び、悲しみ）、方言（関西弁、広東語など）、および音声スタイル（ラップ、アカペラハミング）をサポートし、多様な音声生成ニーズに対応します。

- **強化されたインテリジェンス**：ToolCallメカニズムの統合とロールプレイングの強化を通じて、エージェントの複雑なタスクにおけるパフォーマンスを向上させます。

## 2. モデル概要
Step-Audioでは、音声ストリームをトークン化するために、並列のセマンティック（16.7Hz、1024エントリのコードブック）および音響（25Hz、4096エントリのコードブック）トークナイザーを組み合わせたデュアルコードブックフレームワークを使用し、2:3の時間的インターリーブを行います。1300億パラメータのLLM基盤（Step-1）は、音声コンテキスト化継続的事前トレーニングおよびタスク固有の後トレーニングを通じて強化され、強力なクロスモーダル音声理解を実現します。フローマッチングとニューラルボコーダを組み合わせたハイブリッド音声デコーダを使用し、リアルタイムの波形生成を最適化します。推論パイプラインは、投機的応答生成（40％のコミット率）およびテキストベースのコンテキスト管理（14:1の圧縮率）を特徴とするストリーミング対応アーキテクチャを備えています。
![Architecture](assets/architecture.png)

### 2.1 トークナイザー

セマンティックトークナイザーと音響トークナイザーを効果的に統合するために、トークンレベルのインターリーブアプローチを実装しています。セマンティックトークナイザーは1024のコードブックサイズを使用し、音響トークナイザーはより大きな4096のコードブックサイズを使用して、より細かい音響の詳細をキャプチャします。異なるトークンレートを考慮して、2つのセマンティックトークンごとに3つの音響トークンをペアリングする2:3の時間的アライメント比を確立します。

### 2.2 言語モデル

Step-Audioの音声情報を効果的に処理し、正確な音声-テキストアライメントを実現するために、1300億パラメータの事前トレーニングされたテキストベースの大規模言語モデル（LLM）であるStep-1に基づいて、音声継続的事前トレーニングを実施しました。

### 2.3 音声デコーダ
Step-Audioの音声デコーダは、セマンティックおよび音響情報を含む離散音声トークンを、自然な音声を表す連続的な時間領域の波形に変換する重要な機能を果たします。デコーダアーキテクチャには、フローマッチングモデルとメルから波形へのボコーダが組み込まれています。生成された音声の明瞭度と自然さを最適化するために、音声デコーダはデュアルコードインターリーブアプローチを使用してトレーニングされ、生成プロセス全体でセマンティックおよび音響機能のシームレスな統合を確保します。

### 2.4 リアルタイム推論パイプライン
リアルタイムの対話を可能にするために、最適化された推論パイプラインを設計しました。その中心には、状態遷移を管理し、投機的応答生成を調整し、重要なサブシステム間のシームレスな調整を確保するコントローラーモジュールがあります。これらのサブシステムには、ユーザーの音声を検出する音声活動検出（VAD）、リアルタイムで音声を処理するストリーミングオーディオトークナイザー、応答を処理および生成するStep-Audio言語モデルおよび音声デコーダ、および会話の連続性を維持するコンテキストマネージャが含まれます。
![Inference Pipeline](assets/pipeline.png)

### 2.5 後トレーニングの詳細
後トレーニングフェーズでは、自動音声認識（ASR）およびテキストから音声への変換（TTS）のタスク固有の監督付き微調整（SFT）を実施しました。音声入力テキスト出力（AQTA）タスクについては、多様な高品質データセットを使用してSFTを実施し、人間のフィードバックからの強化学習（RLHF）を組み合わせて応答品質を向上させ、感情表現、音声速度、方言、および韻律の細かい制御を可能にしました。
![RLHF](assets/rlhf.png)

## 3. モデルのダウンロード
### 3.1 Huggingface
| モデル   | リンク   |
|-------|-------|
| Step-Audio-Tokenizer | [🤗huggingface](https://huggingface.co/stepfun-ai/Step-Audio-Tokenizer) |
| Step-Audio-Chat | [🤗huggingface](https://huggingface.co/stepfun-ai/Step-Audio-Chat) |
| Step-Audio-TTS-3B | [🤗huggingface](https://huggingface.co/stepfun-ai/Step-Audio-TTS-3B) |

### 3.2 Modelscope
| モデル   | リンク   |
|-------|-------|
| Step-Audio-Tokenizer | [modelscope](https://modelscope.cn/models/stepfun-ai/Step-Audio-Tokenizer) |
| Step-Audio-Chat | [modelscope](https://modelscope.cn/models/stepfun-ai/Step-Audio-Chat) |
| Step-Audio-TTS-3B | [modelscope](https://modelscope.cn/models/stepfun-ai/Step-Audio-TTS-3B) |

## 4. モデルの使用方法
### 📜 4.1  要件
次の表は、Step-Audioモデル（バッチサイズ=1）を実行するための要件を示しています：

|     モデル    |  設定
(サンプル周波数) | GPU最小メモリ  |
|------------|--------------------------------|----------------|
| Step-Audio-Tokenizer   |        41.6Hz          |       1.5GB        |
| Step-Audio-Chat   |        41.6Hz          |       265GB        |
| Step-Audio-TTS-3B   |        41.6Hz          |       8GB        |

* CUDAサポートのあるNVIDIA GPUが必要です。
  * モデルは、4つのA800 80G GPUでテストされています。
  * **推奨**：より良い生成品質のために、80GBメモリを持つ4つのA800/H800 GPUを使用することをお勧めします。
* テストされたオペレーティングシステム：Linux

### 🔧 4.2 依存関係とインストール
- Python >= 3.10.0（[Anaconda](https://www.anaconda.com/download/#linux)または[Miniconda](https://docs.conda.io/en/latest/miniconda.html)の使用を推奨）
- [PyTorch >= 2.3-cu121](https://pytorch.org/)
- [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads)

```bash
git clone https://github.com/stepfun-ai/Step-Audio.git
conda create -n stepaudio python=3.10
conda activate stepaudio

cd Step-Audio
pip install -r requirements.txt

git lfs install
git clone https://huggingface.co/stepfun-ai/Step-Audio-Tokenizer
git clone https://huggingface.co/stepfun-ai/Step-Audio-Chat
git clone https://huggingface.co/stepfun-ai/Step-Audio-TTS-3B

```

モデルをダウンロードした後、where_you_download_dirは次の構造を持つ必要があります：
```
where_you_download_dir
├── Step-Audio-Tokenizer
├── Step-Audio-Chat
├── Step-Audio-TTS-3B
```

###  🚀 4.3 推論スクリプト
#### オフライン推論
エンドツーエンドの音声/テキスト入力と音声/テキスト出力で推論を行います。
```bash
python offline_inference.py --model-path where_you_download_dir
```
#### TTS推論
デフォルトのスピーカーを使用してTTSを推論するか、新しいスピーカーでクローンを作成します
```bash
python tts_inference.py --model-path where_you_download_dir --output-path where_you_save_audio_dir --synthesis-type use_tts_or_clone
```
クローンモードには、次の形式のスピーカー情報辞書が必要です：
```bash
{
    "speaker": "speaker id",
    "prompt_text": "content of prompt wav",
    "wav_path": "prompt wav path"
}
```

#### Webデモの起動
オンライン推論のためにローカルサーバーを起動します。
4つのGPUが利用可能で、すべてのモデルをダウンロード済みであると仮定します。

```bash
python app.py --model-path where_you_download_dir
```

## 5. ベンチマーク

### 5.1 ASR結果の比較

            隠れた特徴モデリング
            離散音声トークンモデリング

            Whisper Large-v3
            Qwen2-Audio
            MinMo
            LUCY
            Moshi
            GLM-4-voice Base
            GLM-4-voice Chat
            Step-Audio Pretrain
            Step-Audio-Chat

            Aishell-1
            5.14
            1.53
            -
            2.4
            -
            2.46
            226.47
            0.87
            1.95

            Aishell-2 ios
            4.76
            3.06
            2.69
            -
            -
            -
            211.3
            2.91
            3.57

            Wenetspeech test-net
            9.68
            7.72
            6.64
            8.78
            -
            -
            146.05
            7.62
            8.75

            Wenet test-meeting
            18.54
            8.4
            7.6
            10.42
            -
            -
            140.82
            7.78
            9.52

            Librispeech test-clean
            1.9
            1.6
            1.6
            3.36
            5.7
            2.82
            75.39
            2.36
            3.11

            Librispeech test-other
            3.65
            3.6
            3.82
            8.05
            -
            7.66
            80.3
            6.32
            8.44

            AVG
            7.28
            4.32
            -
            -
            -
            -
            146.74
            4.64
            5.89

### 5.2 TTS
#### 5.2.1 GLM-4-VoiceとMinMoのコンテンツ一貫性（CER/WER）のパフォーマンス比較。

            モデル
            test-zh
            test-en

            CER (%) ↓
            WER (%) ↓

            GLM-4-Voice
            2.19
            2.91

            MinMo
            2.48
            2.90

            Step-Audio
            1.53
            2.71

#### 5.2.2 SEEDテストセットでのTTSモデルの結果。
* StepAudio-TTS-3B-Singleは、デュアルコードブックバックボーンとシングルコードブックボコーダの組み合わせを示します。

            モデル
            test-zh
            test-en

            CER (%) ↓
            SS ↑
            WER (%) ↓
            SS ↑

            FireRedTTS
            1.51
            0.630
            3.82
            0.460

            MaskGCT
            2.27
            0.774
            2.62
            0.774

            CosyVoice
            3.63
            0.775
            4.29
            0.699

            CosyVoice 2
            1.45
            0.806
            2.57
            0.736

            CosyVoice 2-S
            1.45
            0.812
            2.38
            0.743

            Step-Audio-TTS-3B-Single
            1.37
            0.802
            2.52
            0.704

            Step-Audio-TTS-3B
            1.31
            0.733
            2.31
            0.660

            Step-Audio-TTS
            1.17
            0.73
            2.0
            0.660

#### 5.2.3 デュアルコードブック再合成とCosyVoiceのパフォーマンス比較。

            トークン
            test-zh
            test-en

            CER (%) ↓
            SS ↑
            WER (%) ↓
            SS ↑

            Groundtruth
            0.972
            -
            2.156
            -

            CosyVoice
            2.857
            0.849
            4.519
            0.807

            Step-Audio-TTS-3B
            2.192
            0.784
            3.585
            0.742

### 5.3 AQTAチャット
[**StepEval-Audio-360**](https://huggingface.co/datasets/stepfun-ai/StepEval-Audio-360) を新しいベンチマークとしてリリースしました。これは、実際のユーザーからの137のマルチターンの日本語プロンプトで構成されており、生成された応答の品質を次の次元で評価するように設計されています：音声指示のフォロー、音声理解、論理的推論、ロールプレイング、創造性、歌唱、言語能力、音声感情制御、ゲーム。

#### 5.3.1 StepEval-Audio-360

#### LLM評価指標（GPT-4o）

StepEval-Audio-360での音声チャットの基本機能の比較。

            モデル
            事実性（% ↑）
            関連性（% ↑）
            チャットスコア ↑

            GLM4-Voice
            54.7
            66.4
            3.49

            Qwen2-Audio
            22.6
            26.3
            2.27

            Moshi*
            1.0
            0
            1.49

            Step-Audio-Chat
            66.4
            75.2
            4.11

* 注：Moshiは「\*」でマークされており、参考として考慮する必要があります。

#### レーダーチャート（人間の評価）

#### 5.3.2 公開テストセット

            モデル
            Llama Question
            Web Questions
            TriviaQA*
            ComplexBench
            HSK-6

            GLM4-Voice
            64.7
            32.2
            39.1
            66.0
            74.0

            Moshi
            62.3
            26.6
            22.8
            -
            -

            Freeze-Omni
            72.0
            44.7
            53.9
            -
            -

            LUCY
            59.7
            29.3
            27.0
            -
            -

            MinMo
            78.9
            55.0
            48.3
            -
            -

            Qwen2-Audio
            52.0
            27.0
            37.3
            54.0
            -

            Step-Audio-Chat
            81.0
            75.1
            58.0
            74.0
            86.0

* 注：TriviaQAデータセットで「\*」でマークされた結果は参考として考慮されます。

#### 5.3.3 音声指示のフォロー

            カテゴリ
            指示のフォロー
            音声品質

            GLM-4-Voice
            Step-Audio
            GLM-4-Voice
            Step-Audio

            言語
            1.9
            3.8
            2.9
            3.3

            ロールプレイング
            3.8
            4.2
            3.2
            3.6

            歌唱 / ラップ
            2.1
            2.4
            2.4
            4

            音声制御
            3.6
            4.4
            3.3
            4.1

## 6. オンラインエンジン
Step-Audioのオンラインバージョンは、[跃问](https://yuewen.cn)のアプリバージョンからアクセスでき、いくつかの印象的な例も見つけることができます。

## 7. 例
### 音声クローン
| 役割   | プロンプト音声 | クローン音声 |
|:-------:|:-------:|:-------:|
|于谦| [google drive](https://drive.google.com/file/d/1N9EJypafFwmeL0R152GoL_CVGbYn1_9A/preview)
[audio file](https://github.com/stepfun-ai/Step-Audio/tree/main/examples/prompt_wav_yuqian.wav)|[google drive](https://drive.google.com/file/d/1Zs_1QrCUuoSqtUSdn2ENIor-k5baQdDV/preview)
[audio file](https://github.com/stepfun-ai/Step-Audio/tree/main/examples/clone_wav_yuqian.wav)|
|李雪琴| [google drive](https://drive.google.com/file/d/15SkZ29hksELYi1NDOxYOPu-kRTLSyke_/preview)
[audio file](https://github.com/stepfun-ai/Step-Audio/tree/main/examples/prompt_wav_lixueqin.wav)|[google drive](https://drive.google.com/file/d/11Le4qMqL2DmWpf7RFRpKUXERIR9TtKC0/preview)
[audio file](https://github.com/stepfun-ai/Step-Audio/tree/main/examples/clone_wav_lixueqin.wav)|

### 速度制御
| プロンプト | 応答 |
|:-------:|:-------:|
|Human: 早口言葉を言ってください
Assistant: すもももももももものうち
Human: もっと早く言えますか？|[google drive](https://drive.google.com/file/d/1mAH-NRrOVZo4tv6gdAZkyJg8kRuTNNGC/preview)
[audio file](https://github.com/stepfun-ai/Step-Audio/tree/main/examples/speed_control1.wav)|
|Human: 早口言葉を言ってください
Assistant: すもももももももものうち
Human: もっと早く言えますか？
Assistant: すもももももももものうち
Human: もっとゆっくり言ってください。|[google drive](https://drive.google.com/file/d/1FhRnKo8uGrtO-cWg4qkrg8iDoNRbtqSX/preview)
[audio file](https://github.com/stepfun-ai/Step-Audio/tree/main/examples/speed_control2.wav)|

### 高EQ（感情制御 & トーン制御）
| プロンプト | 応答 |
|:-------:|:-------:|
|Human: もっとかわいく話してみてください。|[google drive](https://drive.google.com/file/d/19IROE6_6h2UQVNniCmDTnrhxKRMOFHq3/preview)
[audio file](https://github.com/stepfun-ai/Step-Audio/tree/main/examples/tone_control.wav)|
|Human: どうしよう、人生がうまくいかない。|[google drive](https://drive.google.com/file/d/1JlLbOlzmdrokVdxtwy1S8eeWqsZR2Vmc/preview)
[audio file](https://github.com/stepfun-ai/Step-Audio/tree/main/examples/emotional_control1.wav)|
|Human: すごいですね。|[google drive](https://drive.google.com/file/d/19ga1RpguDP5r0Xfl1r5GY1J-kzbmHvJb/preview)
[audio file](https://github.com/stepfun-ai/Step-Audio/tree/main/examples/emotional_control2.wav)|

### 多言語（例：日本語、英語、中国語）
| プロンプト | 応答 |
|:-------:|:-------:|
|Human: "It's raining cats and dogs" ってどういう意味ですか？
Assistant: "It's raining cats and dogs" というのは、非常に激しい雨が降っていることを意味します。実際に猫や犬が空から降ってくるわけではありません！これは激しい雨を表現するための面白い言い方です。|[google drive](https://drive.google.com/file/d/1LEIvdR5ANMzWX8GOTqUPTNrynNS1xx--/preview)
[audio file](https://github.com

README_JP.md


# Contributor Covenant Code of Conduct

## Our Pledge

In the interest of fostering an open and welcoming environment, we as
contributors and maintainers pledge to making participation in our project and
our community a harassment-free experience for everyone, regardless of age, body
size, disability, ethnicity, sex characteristics, gender identity and expression,
level of experience, education, socio-economic status, nationality, personal
appearance, race, religion, or sexual identity and orientation.

## Our Standards

Examples of behavior that contributes to creating a positive environment
include:

* Using welcoming and inclusive language
* Being respectful of differing viewpoints and experiences
* Gracefully accepting constructive criticism
* Focusing on what is best for the community
* Showing empathy towards other community members

Examples of unacceptable behavior by participants include:

* The use of sexualized language or imagery and unwelcome sexual attention or
 advances
* Trolling, insulting/derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or electronic
 address, without explicit permission
* Other conduct which could reasonably be considered inappropriate in a
 professional setting

## Our Responsibilities

Project maintainers are responsible for clarifying the standards of acceptable
behavior and are expected to take appropriate and fair corrective action in
response to any instances of unacceptable behavior.

Project maintainers have the right and responsibility to remove, edit, or
reject comments, commits, code, wiki edits, issues, and other contributions
that are not aligned to this Code of Conduct, or to ban temporarily or
permanently any contributor for other behaviors that they deem inappropriate,
threatening, offensive, or harmful.

## Scope

This Code of Conduct applies both within project spaces and in public spaces
when an individual is representing the project or its community. Examples of
representing a project or community include using an official project e-mail
address, posting via an official social media account, or acting as an appointed
representative at an online or offline event. Representation of a project may be
further defined and clarified by project maintainers.

## Enforcement

Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported by contacting the project team at electronreactboilerplate@gmail.com. All
complaints will be reviewed and investigated and will result in a response that
is deemed necessary and appropriate to the circumstances. The project team is
obligated to maintain confidentiality with regard to the reporter of an incident.
Further details of specific enforcement policies may be posted separately.

Project maintainers who do not follow or enforce the Code of Conduct in good
faith may face temporary or permanent repercussions as determined by other
members of the project's leadership.

## Attribution

This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html

[homepage]: https://www.contributor-covenant.org

For answers to common questions about this code of conduct, see
https://www.contributor-covenant.org/faq

CODE_OF_CONDUCT.md


**Note**: please consider this project a proof of concept. I don't intend to maintain it or merge pull requests. Feel free to fork and build it into something great!

### Introduction

Presenting **Agent.exe**: the easiest way to let Claude's new [computer use](https://www.anthropic.com/news/3-5-models-and-computer-use) capabilities take over your computer!

<img width="387" alt="buy pizza" src="https://github.com/user-attachments/assets/c11cc8f1-6dcb-48f4-9d18-682f14edb77d">

https://github.com/user-attachments/assets/2a371241-bc43-46d4-896e-256b3adc388d

### Motivation

I wanted to see how good Claude's new [computer use](https://www.anthropic.com/news/3-5-models-and-computer-use) APIs were, and the default project they provided felt too heavyweight. This is a simple Electron app that lets Claude 3.5 Sonnet control your local computer directly. I was planning on adding a "semi-auto" mode where the user has to confirm each action before it executes, but each step is so slow I found that wasn't necessary and if the model is getting confused you can easily just hit the "stop" button to end the run.

### Getting started

1.  `git clone https://github.com/corbt/agent.exe`
2.  `cd agent.exe`
3.  `npm install`
4.  Rename `.env.example` --> `.env` and add your Anthropic API Key
5.  `npm start`
6.  Prompt the model to do something interesting on your computer!

### Supported systems

- MacOS
- Theoretically Windows and Linux since all the deps are cross-platform

### Known limitations

- Only works on the primary display
- Lets an AI completely take over your computer
- Oh jeez, probably lots of other stuff too

### Tips

- Claude _really_ likes Firefox. It will use other browsers if it absolutely has to, but will behave so much better if you just install Firefox and let it go to its happy place.

### Roadmap

- I literally wrote this in 6 hours, probably isn't going anywhere. But I will review PRs and merge them if they seem cool.

README.md


# JStack - Build fast, lightweight and end-to-end typesafe Next.js apps

Built on Next.js 15, Hono, Tailwind and Drizzle ORM.

![Project Thumbnail](https://github.com/upstash/jstack/blob/main/www/public/thumbnail.png)

## About

Appreciate you for checking out JStack! For features & docs, see:

https://jstack.app/

Cheers,
Josh

### Acknowledgements

- [T3 Stack](https://github.com/t3-oss/create-t3-app)
- [tRPC](https://trpc.io/)
- [Hono](https://hono.dev/)

and all supporters/contributors. y'all rock

## License

[MIT](https://choosealicense.com/licenses/mit/)

README.md

Build seriously fast, lightweight and end-to-end typesafe Next.js apps


## JStack

Ship high-performance Next.js apps for extremely cheap

app/README.md


This CLI is built by looking at how the [T3 Stack](https://create.t3.gg/) built their CLI. The credit for this CLI and how it's structured goes to them.

cli/README.md


## JStack

Ship high-performance Next.js apps for extremely cheap

Stuff to improve
- footer padding
- docs "on this page" visual bug
- add docs for:
    - procedures
    - cloudflare deployment
    - vercel deployment

www/README.md