---
title: "Key Features"
---

AG2

The Open Source Agent OS

Getting Started - 3 Minute

### Key Features

Multi-Agent Conversation Framework

AG2 provides multi-agent conversation framework as a high-level abstraction. With this framework, one can conveniently build LLM workflows.

Easily Build Diverse Applications

AG2 offers a collection of working systems spanning a wide range of applications from various domains and complexities.

Enhanced LLM Inference & Optimization

AG2 supports enhanced LLM inference APIs, which can be used to improve inference performance and reduce cost.

### Explore content

Quick Start

Start building your AG2 application.

Concepts

Work through the key concepts of AG2 including ConversableAgent, GroupChat, Swarm, and tools.

Advanced Concepts

Advance to RAG, Code Execution, and more complex GroupChats and Swarms.

Use Cases

Try out use case workflows including Customer Service, Travel Planning, and Game Design.

Notebook Examples

A collection of interactive notebooks across all AG2 topics.

API Reference

Delve into the AG2 API reference.

How to Contribute

Get involved with AG2 by adding what you need and making the framework even better!

### Popular resources

            Foundation Capital Interview with Dr. Chi Wang

            Learn AG2 on DeepLearningAI

AG2 (formerly AutoGen): The Open-Source AgentOS. Join us at: https://discord.gg/pAbnFJrkgZ


---
title: "Quick Start"
---

AG2 (formerly AutoGen) is an open-source programming framework for building AI agents and facilitating their cooperation to solve tasks. AG2 supports tool use, autonomous and human-in-the-loop workflows, and multi-agent conversation patterns.

### Let's go

```sh
pip install ag2
```
!!! note
    If you have been using `autogen` or `pyautogen`, all you need to do is upgrade it using:
    ```bash
    pip install -U autogen
    ```
    or
    ```bash
    pip install -U pyautogen
    ```
    as `pyautogen`, `autogen`, and `ag2` are aliases for the same PyPI package.

=== "Chat with an agent"

    ```python
    # 1. Import our agent class
    from autogen import ConversableAgent

    # 2. Define our LLM configuration for OpenAI's GPT-4o mini
    #    uses the OPENAI_API_KEY environment variable
    llm_config = {"api_type": "openai", "model": "gpt-4o-mini"}

    # 3. Create our LLM agent
    my_agent = ConversableAgent(
        name="helpful_agent",
        llm_config=llm_config,
        system_message="You are a poetic AI assistant, respond in rhyme.",
    )

    # 4. Run the agent with a prompt
    chat_result = my_agent.run("In one sentence, what's the big deal about AI?")

    # 5. Print the chat
    print(chat_result.chat_history)
    ```

=== "Two agent chat"

    ```python
    # Chat between two comedian agents

    # 1. Import our agent class
    from autogen import ConversableAgent

    # 2. Define our LLM configuration for OpenAI's GPT-4o mini,
    #    uses the OPENAI_API_KEY environment variable
    llm_config = {"api_type": "openai", "model": "gpt-4o-mini"}

    # 3. Create our agents who will tell each other jokes,
    #    with Jack ending the chat when Emma says FINISH
    jack = ConversableAgent(
        "Jack",
        llm_config=llm_config,
        system_message=(
        "Your name is Jack and you are a comedian "
        "in a two-person comedy show."
        ),
        is_termination_msg=lambda x: True if "FINISH" in x["content"] else False
    )
    emma = ConversableAgent(
        "Emma",
        llm_config=llm_config,
        system_message=(
        "Your name is Emma and you are a comedian "
        "in a two-person comedy show. Say the word FINISH "
        "ONLY AFTER you've heard 2 of Jack's jokes."
        ),
    )

    # 4. Run the chat
    chat_result = jack.initiate_chat(
        emma,
        message="Emma, tell me a joke about goldfish and peanut butter.",
    )

    # 5. Print the chat
    print(chat_result.chat_history)
    ```

=== "Group chat"

    ```python
    # Group chat amongst agents to create a 4th grade lesson plan
    # Flow determined by Group Chat Manager automatically, and
    # should be Teacher > Planner > Reviewer > Teacher (repeats if necessary)

    # 1. Import our agent and group chat classes
    from autogen import ConversableAgent, GroupChat, GroupChatManager

    # Define our LLM configuration for OpenAI's GPT-4o mini
    # uses the OPENAI_API_KEY environment variable
    llm_config = {"api_type": "openai", "model": "gpt-4o-mini"}

    # Planner agent setup
    planner_message = "Create lesson plans for 4th grade. Use format: <title>, <learning_objectives>, <script>"
    planner = ConversableAgent(
        name="planner_agent",
        llm_config=llm_config,
        system_message=planner_message,
        description="Creates lesson plans"
    )

    # Reviewer agent setup
    reviewer_message = "Review lesson plans against 4th grade curriculum. Provide max 3 changes."
    reviewer = ConversableAgent(
        name="reviewer_agent",
        llm_config=llm_config,
        system_message=reviewer_message,
        description="Reviews lesson plans"
    )

    # Teacher agent setup
    teacher_message = "Choose topics and work with planner and reviewer. Say DONE! when finished."
    teacher = ConversableAgent(
        name="teacher_agent",
        llm_config=llm_config,
        system_message=teacher_message,
    )

    # Setup group chat
    groupchat = GroupChat(
        agents=[teacher, planner, reviewer],
        speaker_selection_method="auto",
        messages=[]
    )

    # Create manager
    # At each turn, the manager will check if the message contains DONE! and end the chat if so
    # Otherwise, it will select the next appropriate agent using its LLM
    manager = GroupChatManager(
        name="group_manager",
        groupchat=groupchat,
        llm_config=llm_config,
        is_termination_msg=lambda x: "DONE!" in (x.get("content", "") or "").upper()
    )

    # Start the conversation
    chat_result = teacher.initiate_chat(
        recipient=manager,
        message="Let's teach the kids about the solar system."
    )

    # Print the chat
    print(chat_result.chat_history)
    ```

!!! tip
    Learn more about configuring LLMs for agents
    [here](/docs/user-guide/basic-concepts/llm-configuration).

### Where to Go Next?

- Go through the [basic concepts](/docs/user-guide/basic-concepts/installing-ag2) to get started
- Once you're ready, hit the [advanced concepts](/docs/user-guide/advanced-concepts/rag)
- Explore the [API Reference](/docs/api-reference/autogen/overview)
- Chat on [Discord](https://discord.gg/pAbnFJrkgZ)
- Follow on [X](https://x.com/ag2oss)

If you like our project, please give it a [star](https://github.com/ag2ai/ag2) on GitHub. If you are interested in contributing, please read [Contributor's Guide](/contributor-guide/contributing).

<iframe
  src="https://ghbtns.com/github-btn.html?user=ag2ai&repo=ag2&type=star&count=true&size=large"
  frameborder="0"
  scrolling="0"
  width="170"
  height="30"
  title="GitHub"
></iframe>


# EDtunnel

EDtunnel 是一个基于 Cloudflare Workers 和 Pages 的代理工具，支持多种协议和配置选项。

EDtunnel is a proxy tool based on Cloudflare Workers and Pages, supporting multiple protocols and configuration options.

[![Repository](https://img.shields.io/badge/View%20on-GitHub-blue.svg)](https://github.com/6Kmfi6HP/EDtunnel)
[![Telegram](https://img.shields.io/badge/Discuss-Telegram-blue.svg)](https://t.me/edtunnel)

## ✨ 特性 | Features

- 支持 Cloudflare Workers 和 Pages 部署
- 支持多 UUID 配置
- 支持自定义代理 IP 和端口
- 支持 SOCKS5 代理
- 提供自动配置订阅链接
- 简单易用的部署流程

- Support for Cloudflare Workers and Pages deployment
- Multiple UUID configuration support
- Custom proxy IP and port support
- SOCKS5 proxy support
- Automatic configuration subscription link
- Simple and easy deployment process

## 🚀 快速部署 | Quick Deployment

### 在 Pages.dev 部署 | Deploy on Pages.dev

1. 观看部署教程视频 | Watch deployment tutorial video：
   [YouTube Tutorial](https://www.youtube.com/watch?v=8I-yTNHB0aw)

2. 克隆此仓库并在 Cloudflare Pages 中部署 | Clone this repository and deploy in Cloudflare Pages

### 在 Worker.dev 部署 | Deploy on Worker.dev

1. 从[这里](https://github.com/6Kmfi6HP/EDtunnel/blob/main/_worker.js)复制 `_worker.js` 代码 | Copy `_worker.js` code from [here](https://github.com/6Kmfi6HP/EDtunnel/blob/main/_worker.js)

2. 或者点击下方按钮一键部署 | Or click the button below to deploy directly:

   [![Deploy to Cloudflare Workers](https://deploy.workers.cloudflare.com/button)](https://deploy.workers.cloudflare.com/?url=https://github.com/6Kmfi6HP/EDtunnel)

## ⚙ 配置说明 | Configuration Guide

### 环境变量配置 | Environment Variables

| 变量名 (Variable) | 是否必需 (Required) | 示例 (Example) | 说明 (Description) |
|------------------|-------------------|---------------|-------------------|
| `UUID` | 否 (No) | 单个 (Single): `12345678-1234-1234-1234-123456789012`
多个 (Multiple): `uuid1,uuid2,uuid3` | 用户识别码 / User identification |
| `PROXYIP` | 否 (No) | `1.1.1.1` 或 (or) `example.com`
多个 (Multiple): `1.1.1.1:9443,2.2.2.2:8443` | 自定义代理IP和端口 / Custom proxy IP and port |
| `SOCKS5` | 否 (No) | `user:pass@host:port`
多个 (Multiple): `user1:pass1@host1:port1,user2:pass2@host2:port2` | SOCKS5代理配置 / SOCKS5 proxy configuration |
| `SOCKS5_RELAY` | 否 (No) | `true` 或 (or) `false` | 启用SOCKS5流量转发 / Enable SOCKS5 traffic relay |

### 非443端口配置 | Non-443 Port Configuration

1. 访问 (Visit) `https://proxyip.edtunnel.best/`
2. 输入 (Enter) `ProxyIP:proxyport` 并点击检查 (and click Check)
3. 当显示 (When showing) `Proxy IP: true` 时可用 (it's available)
4. 在 Worker 中配置 (Configure in Worker): `PROXYIP=211.230.110.231:50008`

注意：带端口的代理IP可能在某些仅支持HTTP的Cloudflare站点上无效。
Note: Proxy IPs with ports may not work on HTTP-only Cloudflare sites.

### UUID 配置方法 | UUID Configuration

#### 方法一 | Method 1
在 `wrangler.toml` 文件中设置（不推荐在公共仓库中使用）
Set in `wrangler.toml` file (not recommended for public repositories)

```toml
[vars]
UUID = "your-uuid-here"
```

#### 方法二 | Method 2
在 Cloudflare Dashboard 的环境变量中设置（推荐方式）
Set in Cloudflare Dashboard environment variables (recommended method)

## ⚠ 重要提示：多项配置分隔符 | Important Note: Multiple Configuration Separator

所有多项配置必须使用英文逗号(,)分隔，不能使用中文逗号(，)
All multiple configurations MUST use English comma(,) as separator, NOT Chinese comma(，)

✅ 正确示例 | Correct Examples:
```bash
# UUID多个配置 | Multiple UUID
UUID=uuid1,uuid2,uuid3

# SOCKS5多个代理 | Multiple SOCKS5 proxies
SOCKS5=192.168.1.1:1080,192.168.1.2:1080

# PROXYIP多个地址 | Multiple PROXYIP
PROXYIP=1.1.1.1:443,2.2.2.2:443
```

❌ 错误示例 | Wrong Examples:
```bash
# 错误：使用中文逗号 | Wrong: Using Chinese comma
UUID=uuid1，uuid2，uuid3

# 错误：使用中文逗号 | Wrong: Using Chinese comma
SOCKS5=192.168.1.1:1080，192.168.1.2:1080
```

## 📱 快速使用 | Quick Start

### 自动配置订阅 | Auto Configuration Subscribe

使用以下链接获取自动配置 | Use the following link for auto configuration:
```
https://sub.xf.free.hr/auto
```

### 查看配置 | View Configuration

- 访问您的域名 | Visit your domain: `https://your-domain.pages.dev`
- 使用特定UUID | Use specific UUID: `/sub/[uuid]`
- 查看完整配置 | View full configuration: 直接访问域名根路径 (visit domain root path)
- 获取订阅内容 | Get subscription content: 访问 `/sub/[uuid]` (visit `/sub/[uuid]`)

## 🔧 高级配置 | Advanced Configuration

### 多UUID支持 | Multiple UUID Support

您可以通过以下方式配置多个UUID | You can configure multiple UUIDs in these ways:

1. 环境变量方式 | Via environment variables:
   ```
   UUID=uuid1,uuid2,uuid3
   ```

2. 配置文件方式 | Via configuration file:
   ```toml
   [vars]
   UUID = "uuid1,uuid2,uuid3"
   ```

### SOCKS5代理配置 | SOCKS5 Proxy Configuration

支持以下格式 | Supports the following formats:
- 基础格式 | Basic format: `host:port`
- 认证格式 | Authentication format: `username:password@host:port`
- 多代理格式（使用英文逗号分隔）| Multiple proxies (separated by English comma): `proxy1,proxy2,proxy3`

#### 配置示例 | Configuration Examples:

1. 单个代理 | Single Proxy:
```bash
# 基础格式 | Basic format
SOCKS5=192.168.1.1:1080

# 带认证格式 | With authentication
SOCKS5=user:pass@192.168.1.1:1080
```

2. 多个代理（使用英文逗号分隔）| Multiple Proxies (separated by English comma):
```bash
# 多个基础代理 | Multiple basic proxies
SOCKS5=192.168.1.1:1080,192.168.1.2:1080,192.168.1.3:1080

# 多个带认证代理 | Multiple proxies with authentication
SOCKS5=user1:pass1@host1:port1,user2:pass2@host2:port2

# 混合格式 | Mixed format
SOCKS5=192.168.1.1:1080,user:pass@192.168.1.2:1080,192.168.1.3:1080
```

#### SOCKS5 代理负载均衡 | SOCKS5 Proxy Load Balancing

当配置多个代理时，系统会自动进行负载均衡：
When multiple proxies are configured, the system will automatically perform load balancing:

- 随机选择 | Random selection
- 自动故障转移 | Automatic failover
- 支持混合认证方式 | Support mixed authentication methods

#### SOCKS5_RELAY 设置 | SOCKS5_RELAY Settings

启用 SOCKS5 全局转发 | Enable SOCKS5 global relay:
```bash
SOCKS5_RELAY=true
```

注意事项 | Notes:
- 确保代理服务器稳定可用 | Ensure proxy servers are stable and available
- 建议使用私有代理以提高安全性 | Recommend using private proxies for better security
- 多代理配置时使用英文逗号分隔 | Use commas to separate multiple proxies
- 支持动态添加和移除代理 | Support dynamic proxy addition and removal

## 🚨 注意事项 | Notes

- 带端口的代理IP可能在某些仅HTTP的Cloudflare站点上无效
- 多UUID配置时使用英文逗号分隔
- 建议通过环境变量设置敏感信息
- 定期更新以获取最新功能和安全修复

- Proxy IPs with ports may not work on HTTP-only Cloudflare sites
- Use commas to separate multiple UUIDs
- Recommend setting sensitive information via environment variables
- Update regularly for latest features and security fixes

## 🔧 环境变量设置 | Environment Variable Settings

### Workers.dev 设置 | Workers.dev Settings
在 Workers 设置页面配置环境变量 | Configure environment variables in Workers settings page
![workers](image/image-1.png)

### Pages.dev 设置 | Pages.dev Settings
在 Pages 设置页面配置环境变量 | Configure environment variables in Pages settings page
![pages](image/image-2.png)

## 💬 获取帮助 | Get Help

- Telegram 群组 | Telegram Group: [EDtunnel Group](https://t.me/edtunnel)
- GitHub 仓库 | Repository: [EDtunnel](https://github.com/6Kmfi6HP/EDtunnel)
- 问题反馈 | Issue Report: [创建新问题 | Create New Issue](https://github.com/6Kmfi6HP/EDtunnel/issues)
- 功能建议 | Feature Request: [提交建议 | Submit Request](https://github.com/6Kmfi6HP/EDtunnel/discussions)

## 📝 贡献指南 | Contributing

欢迎提交 Pull Request 来改进项目！请确保：
Welcome Pull Requests to improve the project! Please ensure:

1. 代码符合项目规范 | Code follows project standards
2. 添加必要的测试 | Add necessary tests
3. 更新相关文档 | Update relevant documentation
4. 描述清楚改动原因 | Clearly describe the reasons for changes

## 📜 许可证 | License

本项目采用 MIT 许可证 - 查看 [LICENSE](LICENSE) 文件了解详情
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details

## Star History

  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=6Kmfi6HP/EDtunnel&type=Date&theme=dark" />
    <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=6Kmfi6HP/EDtunnel&type=Date" />

  </picture>

README.md


# Audiblez: Generate  audiobooks from e-books

[![Installing via pip and running](https://github.com/santinic/audiblez/actions/workflows/pip-install.yaml/badge.svg)](https://github.com/santinic/audiblez/actions/workflows/pip-install.yaml)
[![Git clone and run](https://github.com/santinic/audiblez/actions/workflows/git-clone-and-run.yml/badge.svg)](https://github.com/santinic/audiblez/actions/workflows/git-clone-and-run.yml)
![PyPI - Python Version](https://img.shields.io/pypi/pyversions/audiblez)
![PyPI - Version](https://img.shields.io/pypi/v/audiblez)

### v4 Now with Graphical interface, CUDA support, and many languages!

![Audiblez GUI on MacOSX](./imgs/mac.png)

Audiblez generates `.m4b` audiobooks from regular `.epub` e-books,
using Kokoro's high-quality speech synthesis.

[Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M) is a recently published text-to-speech model with just 82M params and very natural sounding output.
It's released under Apache licence and it was trained on < 100 hours of audio.
It currently supports these languages: 🇺🇸 🇬🇧 🇪🇸 🇫🇷 🇮🇳 🇮🇹 🇯🇵 🇧🇷 🇨🇳

On a Google Colab's T4 GPU via Cuda, **it takes about 5 minutes to convert "Animal's Farm" by Orwell** (which is about 160,000 characters) to audiobook, at a rate of about 600 characters per second.

On my M2 MacBook Pro, on CPU, it takes about 1 hour, at a rate of about 60 characters per second.

## How to install the Command Line tool

If you have Python 3 on your computer, you can install it with pip.
You also need `espeak-ng` and `ffmpeg` installed on your machine:

```bash
sudo apt install ffmpeg espeak-ng                   # on Ubuntu/Debian 🐧
pip install audiblez
```

```bash
brew install ffmpeg espeak-ng                       # on Mac 🍏
pip install audiblez
```

Then you can convert an .epub directly with:

```
audiblez book.epub -v af_sky
```

It will first create a bunch of `book_chapter_1.wav`, `book_chapter_2.wav`, etc. files in the same directory,
and at the end it will produce a `book.m4b` file with the whole book you can listen with VLC or any
audiobook player.
It will only produce the `.m4b` file if you have `ffmpeg` installed on your machine.

## How to run the GUI

The GUI is a simple graphical interface to use audiblez.
You need some extra dependencies to run the GUI:

```
sudo apt install ffmpeg espeak-ng
sudo apt install libgtk-3-dev        # just for Ubuntu/Debian 🐧, Windows/Mac don't need this

pip install audiblez pillow wxpython
```

Then you can run the GUI with:
```
audiblez-ui
```

## How to run on Windows

After many trials, on Windows we recommend to install audiblez in a Python venv:

1. Open a Windows terminal
2. Create anew folder: `mkdir audiblez`
3. Enter the folder: `cd audiblez`
4. Create a venv: `python -m venv venv`
5. Activate the venv: `.\venv\Scripts\Activate.ps1`
6. Install the dependencies: `pip install audiblez pillow wxpython`
7. Now you can run `audiblez` or `audiblez-ui`
8. For Cuda support, you need to install Pytorch accordingly: https://pytorch.org/get-started/locally/

## Speed

By default the audio is generated using a normal speed, but you can make it up to twice slower or faster by specifying a speed argument between 0.5 to 2.0:

```
audiblez book.epub -v af_sky -s 1.5
```

## Supported Voices

Use `-v` option to specify the voice to use. Available voices are listed here.
The first letter is the language code and the second is the gender of the speaker e.g. `im_nicola` is an italian male voice.

[For hearing samples of Kokoro-82M voices, go here](https://claudio.uk/posts/audiblez-v4.html)

| Language                  | Voices                                                                                |
|---------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 🇺🇸 American English     | `af_alloy`, `af_aoede`, `af_bella`, `af_heart`, `af_jessica`, `af_kore`, `af_nicole`, `af_nova`, `af_river`, `af_sarah`, `af_sky`, `am_adam`, `am_echo`, `am_eric`, `am_fenrir`, `am_liam`, `am_michael`, `am_onyx`, `am_puck`, `am_santa` |
| 🇬🇧 British English      | `bf_alice`, `bf_emma`, `bf_isabella`, `bf_lily`, `bm_daniel`, `bm_fable`, `bm_george`, `bm_lewis`                                                                                |
| 🇪🇸 Spanish              | `ef_dora`, `em_alex`, `em_santa`                                                                                |
| 🇫🇷 French               | `ff_siwis`                                                                                |
| 🇮🇳 Hindi                | `hf_alpha`, `hf_beta`, `hm_omega`, `hm_psi`                                                                                |
| 🇮🇹 Italian              | `if_sara`, `im_nicola`                                                                                |
| 🇯🇵 Japanese             | `jf_alpha`, `jf_gongitsune`, `jf_nezumi`, `jf_tebukuro`, `jm_kumo`                                                                                |
| 🇧🇷 Brazilian Portuguese | `pf_dora`, `pm_alex`, `pm_santa`                                                                                |
| 🇨🇳 Mandarin Chinese     | `zf_xiaobei`, `zf_xiaoni`, `zf_xiaoxiao`, `zf_xiaoyi`, `zm_yunjian`, `zm_yunxi`, `zm_yunxia`, `zm_yunyang`                                                                                |

For more detaila about voice quality, check this document: [Kokoro-82M voices](https://huggingface.co/hexgrad/Kokoro-82M/blob/main/VOICES.md)

## How to run on GPU

By default, audiblez runs on CPU. If you pass the option `--cuda` it will try to use the Cuda device via Torch.

Check out this example: [Audiblez running on a Google Colab Notebook with Cuda ](https://colab.research.google.com/drive/164PQLowogprWQpRjKk33e-8IORAvqXKI?usp=sharing]).

We don't currently support Apple Silicon, as there is not yet a Kokoro implementation in MLX. As soon as it will be available, we will support it.

## Manually pick chapters to convert

Sometimes you want to manually select which chapters/sections in the e-book to read out loud.
To do so, you can use `--pick` to interactively choose the chapters to convert (without running the GUI).

## Help page

For all the options available, you can check the help page `audiblez --help`:

```
usage: audiblez [-h] [-v VOICE] [-p] [-s SPEED] [-c] [-o FOLDER] epub_file_path

positional arguments:
  epub_file_path        Path to the epub file

options:
  -h, --help            show this help message and exit
  -v VOICE, --voice VOICE
                        Choose narrating voice: a, b, e, f, h, i, j, p, z
  -p, --pick            Interactively select which chapters to read in the audiobook
  -s SPEED, --speed SPEED
                        Set speed from 0.5 to 2.0
  -c, --cuda            Use GPU via Cuda in Torch if available
  -o FOLDER, --output FOLDER
                        Output folder for the audiobook and temporary files

example:
  audiblez book.epub -l en-us -v af_sky

to use the GUI, run:
  audiblez-ui
```

## Author

by [Claudio Santini](https://claudio.uk) in 2025, distributed under MIT licence.

Related Article: [Audiblez v4: Generate Audiobooks from E-books](https://claudio.uk/posts/audiblez-v4.html)

README.md

Generate audiobooks from e-books


 LLaVA-CoT: Let Vision Language Models Reason Step-by-Step

This project was previously named LLaVA-o1. We have renamed it to LLaVA-CoT.

 If you like our project, please give us a star ⭐ on GitHub for the latest update.

[![Demo](https://img.shields.io/badge/⚡-Hugging%20Face%20Demo-yellow.svg)](https://huggingface.co/spaces/Xkev/Llama-3.2V-11B-cot)
[![hf](https://img.shields.io/badge/🤗-Hugging%20Face-blue.svg)](https://huggingface.co/Xkev/Llama-3.2V-11B-cot)
[![arXiv](https://img.shields.io/badge/Arxiv-2411.10440v2-b31b1b.svg?logo=arXiv)](https://arxiv.org/abs/2411.10440v2)
[![X (formerly Twitter) URL](https://img.shields.io/twitter/url?url=https%3A%2F%2Fx.com%2FKevin_GuoweiXu%2Fstatus%2F1858338565463421244)](https://x.com/Kevin_GuoweiXu/status/1858338565463421244)
[![License](https://img.shields.io/badge/Code%20License-Apache2.0-yellow)](https://github.com/PKU-YuanGroup/LLaVA-CoT/blob/main/LICENSE)
[![Hits](https://hits.seeyoufarm.com/api/count/incr/badge.svg?url=https%3A%2F%2Fgithub.com%2FPKU-YuanGroup%2FLLaVA-CoT&count_bg=%2379C83D&title_bg=%23555555&icon=&icon_color=%23E7E7E7&title=hits&edge_flat=false)](https://hits.seeyoufarm.com/api/count/incr/badge.svg?url=https%3A%2F%2Fgithub.com%2FPKU-YuanGroup%2FLLaVA-CoT&count_bg=%2379C83D&title_bg=%23555555&icon=&icon_color=%23E7E7E7&title=hits&edge_flat=false)
[![Replicate](https://replicate.com/chenxwh/llava-cot/badge)](https://replicate.com/chenxwh/llava-cot)
[![Wisemodel](https://img.shields.io/badge/wisemodel-code-blue)](https://wisemodel.cn/codes/KevinTHU/LLaVA-CoT)

本项目希望构建开源慢思考大模型，由北大-兔展AIGC联合实验室共同发起。

## 📣 News

- **[2025/01/08]** We released the full training code.
- **[2025/01/02]** We discovered that when testing with the AI2D benchmark, we were using AI2D_TEST_NO_MASK, while the VLMEvalKit utilizes AI2D_TEST. We previously overlooked the distinction between the two, and we sincerely apologize for this oversight. We will make the necessary corrections.
- **[2024/11/28]** We've released the dataset: [https://huggingface.co/datasets/Xkev/LLaVA-CoT-100k](<[dataset_generation/generate.py](https://huggingface.co/datasets/Xkev/LLaVA-CoT-100k)>)
- **[2024/11/25]** We've released the code for dataset generation: [dataset_generation/generate.py](dataset_generation/generate.py)
- **[2024/11/23]** We've released the Gradio App: [https://huggingface.co/spaces/Xkev/Llama-3.2V-11B-cot](https://huggingface.co/spaces/Xkev/Llama-3.2V-11B-cot)
- **[2024/11/21]** LLaVA-o1 is renamed to LLaVA-CoT [https://arxiv.org/abs/2411.10440v2](https://arxiv.org/abs/2411.10440v2).
- **[2024/11/20]** We've released the pretrained weights: [https://huggingface.co/Xkev/Llama-3.2V-11B-cot](https://huggingface.co/Xkev/Llama-3.2V-11B-cot)
- **[2024/11/18]** We've released our paper: [https://arxiv.org/abs/2411.10440](https://arxiv.org/abs/2411.10440)
- **[2024/11/18]** Welcome to **watch** 👀 this repository for the latest updates.

## 🔥 Highlights

**LLaVA-CoT** is a visual language model capable of spontaneous, systematic reasoning.

Our 11B model outperforms **Gemini-1.5-pro**,**GPT-4o-mini**, and **Llama-3.2-90B-Vision-Instruct** on six challenging multimodal benchmarks.

## 🚀 Demos

LLaVA-CoT begins by outlining the problem, interprets relevant information from the image, proceeds step-by-step through reasoning, and ultimately reaches a well-supported conclusion.

### Reasoning Problems

| **Question**                      |
 Subtract all tiny shiny balls. Subtract all purple objects. How many objects are left? Options: A. 4, B. 8, C. 2, D. 6                                                                                |
| --------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Llama-3.2-11B-Vision-Instruct** | To determine the correct answer, we need to follow the instructions provided.
 The first instruction is to subtract all tiny shiny balls.
 Since there are three tiny shiny balls in the image, we would be left with no shiny balls after performing this action.
 The second instruction is to subtract all purple objects. However, upon closer inspection, it becomes apparent that there are no shiny purple objects present in the image. Therefore, we cannot subtract any purple objects.
 Given that we have already removed all the tiny shiny balls and there were no purple objects to begin with, the correct answer is C. 2, as initially, there were 2 purple objects, but they are not to be subtracted according to the instruction.                                                                                |
| **LLaVA-CoT**                     | **_Summary stage: What's the problem? What should I do?_**
 <SUMMARY> I will examine the image to identify and count all the objects, focusing on the tiny shiny balls and purple objects. Then, I will subtract these objects from the total count and determine the number of remaining objects. </SUMMARY>
**_Caption stage: What can I know from the image?_**
 <CAPTION> The image shows various 3D geometric shapes on a flat surface, including spheres, cubes, and cylinders in different colors. Notably, there are small shiny green spheres and a purple cylinder. </CAPTION>
 **_Reasoning stage: How to solve the problem step-by-step?_**
 <REASONING> First, I will count all the objects in the image: there are 10 objects in total. Next, I will identify and subtract the tiny shiny balls. There is one small shiny green sphere. Then, I will identify and subtract the purple objects. There is one purple cylinder. By subtracting the two objects (the shiny ball and the purple cylinder) from the total, I find that 8 objects remain. </REASONING>
 **_Conclusion stage: What is the final answer?_**
 <CONCLUSION> **B** </CONCLUSION> |

### Science Problems

| **Question**                      |
 An acetal polymer block is fixed to the rigid plates at its top and bottom surfaces. If the top plate displaces 2 mm horizontally when it is subjected to a horizontal force \( P = 2 \, \text{kN} \), determine the shear modulus of the polymer. The width of the block is 100 mm. Assume that the polymer is linearly elastic and use small angle analysis.
 **Options:** A: 15 MPa, B: 5 MPa, C: 20 MPa, D: 2 MPa                                                                                |
| --------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Llama-3.2-11B-Vision-Instruct** | To determine the shear modulus of the polymer, we can use the formula for shear modulus in a small angle analysis:
 \( G = \frac{F}{A \gamma} \)
 where \( G \) is the shear modulus, \( F \) is the applied force, \( A \) is the area of the block, and \( \gamma \) is the shear strain.
 Given that the top plate of the block displaces horizontally by 2 mm when subjected to a horizontal force of 2 kN, we can calculate the shear strain \( \gamma \). Since the displacement is horizontal and the original length of the block is 400 mm, the shear strain \( \gamma \) is:
 \[ \gamma = \frac{\Delta x}{L} = \frac{2 \, \text{mm}}{400 \, \text{mm}} = 0.005 \]
 Next, we calculate the area \( A \) of the block. The width of the block is given as 100 mm, and assuming the height of the block is also 200 mm (as given in the image analysis), the area \( A \) is:
 \[ A = 100 \, \text{mm} \times 200 \, \text{mm} = 20000 \, \text{mm}^2 \]
 Now, we can substitute these values into the formula for shear modulus:
 \[ G = \frac{F}{A \gamma} = \frac{2000 \, \text{N}}{20000 \, \text{mm}^2 \times 0.005} = \frac{2000}{100} = 20 \, \text{MPa} \]
 Therefore, the correct answer is C: 20 MPa.                                                                                |
| **LLaVA-CoT**                     | **_Summary stage: What's the problem? What should I do?_**
 <SUMMARY> To solve the problem, I will determine the shear modulus of the acetal polymer block using the relationship between the applied force, displacement, and dimensions of the block. I will use the formula for shear modulus and calculate the result. </SUMMARY>
 **_Caption stage: What can I know from the image?_**
 <CAPTION> The image shows a rectangular block of acetal polymer fixed between two rigid plates. The top plate moves horizontally by 2 mm when a force \( P = 2 \, \text{kN} \) is applied. The block's dimensions are given as 400 mm in length, 200 mm in height, and 100 mm in width. </CAPTION>
 **_Reasoning stage: How to solve the problem step-by-step?_**
 <REASONING> The shear modulus \( G \) of the polymer can be calculated using the formula:
 \[ G = \frac{F}{A \cdot \frac{\Delta x}{L}} \]
 where:
 - \( F \) is the applied force (2 kN),
 - \( A \) is the area of the block's top surface (100 mm x 400 mm = 40,000 mm²),
 - \( \Delta x \) is the horizontal displacement (2 mm),
 - \( L \) is the height of the block (200 mm).
 First, convert the force to Newtons: \( F = 2 \, \text{kN} = 2000 \, \text{N} \).
 Next, calculate the shear modulus:
 \[ G = \frac{2000}{40,000 \cdot \frac{2}{200}} \]
 \[ G = \frac{2000}{40,000 \cdot 0.01} \]
 \[ G = \frac{2000}{400} \]
 \[ G = 5 \, \text{MPa} \]
 </REASONING>
 **_Conclusion stage: What is the final answer?_**
 <CONCLUSION> B: 5 MPa </CONCLUSION> |

## 🛠 Usage

### Pretrained weights

You can download the pretrained weights from the Huggingface: [Xkev/Llama-3.2V-11B-cot](https://huggingface.co/Xkev/Llama-3.2V-11B-cot).

### Dataset

You can download the dataset from the Huggingface: [Xkev/LLaVA-CoT-100k](https://huggingface.co/datasets/Xkev/LLaVA-CoT-100k).

### Inference

You can use the same code as Llama-3.2-11B-Vision-Instruct to load the model and perform inference.

If you want to use perform inference time scaling, you can use code provided in [inference_demo/inference_demo.py](inference_demo/inference_demo.py).

In order to run the demo, you need to replace this file with the original inference code for Llama-3.2-11B-Vision-Instruct in VLMEvalKit.

Additionally, you need to replace the `processing_mllama.py` file in the transformers library with the one provided in [inference_demo/processing_mllama.py](inference_demo/processing_mllama.py).

### Finetuning

You may use any repository that supports Llama-3.2-11B-Vision-Instruct for finetuning.

We recommend using [llama-recipes](https://github.com/Meta-Llama/llama-recipes).

To reproduce our results, you can use the following command:

```bash
cd train
pip install llama-recipes
torchrun --nnodes 1 --nproc_per_node 8 --master_port 29500 finetuning.py --enable_fsdp --lr 1e-5  --num_epochs 3 --batch_size_training 4 --model_name meta-llama/Llama-3.2-11B-Vision-Instruct --dist_checkpoint_root_folder ./finetuned_model --dist_checkpoint_folder LLaVA-CoT --use_fast_kernels --dataset "custom_dataset" --custom_dataset.test_split "test" --custom_dataset.file "datasets/cot_dataset.py"  --run_validation False --batching_strategy padding
```

Remember to modify the `data_path` and `image_base_path` in `train/cot_dataset.py` to your own path (the path to the training dataset).

## 📝 Citation

If you find this paper useful, please consider staring 🌟 this repo and citing 📑 our paper:

```
@misc{xu2024llavacot,
      title={LLaVA-CoT: Let Vision Language Models Reason Step-by-Step},
      author={Guowei Xu and Peng Jin and Hao Li and Yibing Song and Lichao Sun and Li Yuan},
      year={2024},
      eprint={2411.10440},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2411.10440},
}
```

## 🙏 Acknowledgement

- The majority of this project is released under the Apache 2.0 license as found in the [LICENSE](https://github.com/PKU-YuanGroup/Chat-UniVi/blob/main/LICENSE) file.
- The service is a research preview intended for non-commercial use only, subject to [LLAMA 3.2 COMMUNITY LICENSE AGREEMENT](https://huggingface.co/meta-llama/Llama-3.2-1B/blob/main/LICENSE.txt), and [Terms of Use](https://openai.com/policies/terms-of-use) of the data generated by OpenAI. Please contact us if you find any potential violations.
- The template is modified from [Chat-Univi](https://github.com/PKU-YuanGroup/Chat-UniVi) and [LLaVA](https://llava-vl.github.io/).

README.md

LLaVA-CoT, a visual language model capable of spontaneous, systematic reasoning


# v1.0.0

✅ 使用文档与演示动图

# v0.5.0

✅ 跨平台构建与分发（GitHub Action）

# v0.4.0

✅ 应用页面设计

# v0.3.0

✅ 服务器跨平台构建，构建产物下载与解压缩

# v0.2.0

✅ 完成服务器启动、加载模型、换脸 Pipeline

# v0.1.0

✅ Python 一键换脸

CHANGELOG.md

🪞 Instant AI Face Swap, Hairstyles & Outfits — One click to a brand new you! 一键 AI 换脸、发型、穿搭，发现更美的你


# MagicMirror

Instant AI Face Swap, Hairstyles & Outfits — One click to a brand new you!

[![GitHub release](https://img.shields.io/github/v/release/idootop/MagicMirror.svg)](https://github.com/idootop/MagicMirror/releases) [![Build APP](https://github.com/idootop/MagicMirror/actions/workflows/build-app.yaml/badge.svg)](https://github.com/idootop/MagicMirror/actions/workflows/build-app.yaml) [![Build Server](https://github.com/idootop/MagicMirror/actions/workflows/build-server.yaml/badge.svg)](https://github.com/idootop/MagicMirror/actions/workflows/build-server.yaml)

![](./docs/assets/banner.jpg)

## Features

- **Where AI Meets Your Beauty**: Effortlessly try on new faces, hairstyles, and outfits
- **Stupid Simple**: Drag and drop photos to instantly change faces - no complex settings required
- **Hardware Friendly**: Runs smoothly on standard computers, no dedicated GPU hardware required
- **Privacy First**: Completely offline processing - your images never leave your device
- **Ultra-Lightweight**: Tiny footprint with <10MB installer and <1GB model files

## Get Started

> [👉 中文教程和下载地址请戳这里](./docs/cn/readme.md)

To get started with MagicMirror:

1. Follow the [Installation Guide](./docs/en/install.md)
2. Check out the [Usage Guide](./docs/en/usage.md)
3. See the [FAQ](./docs/en/faq.md) for common issues

If you have any questions, please [submit an issue](https://github.com/idootop/MagicMirror/issues).

> Note: MagicMirror only supports macOS 13 (Ventura) and Windows 10 and above.

## Motivation

![](./docs/assets/demo.webp)

Ever found yourself endlessly scrolling through hairstyles and outfits, wondering "How would this look on me?" As someone who loves exploring different styles but hates the hassle, I dreamed of an app that could instantly show me wearing any look I fancy.

While AI technology has advanced tremendously, most AI face applications either require complex setup, demand high-end GPU hardware, or rely on cloud processing that compromises privacy.

**The ideal solution should be as simple as taking a selfie** - just drag, drop, and transform. No technical expertise needed, no expensive hardware required, and no privacy concerns.

So, why not build one myself?

And that’s how MagicMirror came to life ✨

Enjoy! ;)

## Acknowledgments

MagicMirror builds upon several outstanding open-source projects:

- [Tauri](https://github.com/tauri-apps/tauri): Build smaller, faster, and more secure desktop and mobile applications with a web frontend.
- [FaceFusion](https://github.com/facefusion/facefusion): Industry leading face manipulation platform
- [InsightFace](https://github.com/deepinsight/insightface): State-of-the-art 2D and 3D Face Analysis Project
- [Nuitka](https://github.com/Nuitka/Nuitka): Nuitka is a Python compiler written in Python.

## Disclaimer

MagicMirror is designed for personal entertainment and creative purposes only. Commercial use is prohibited. Please note:

- **Ethical Usage**: This software must not be used for activities including, but not limited to: a) impersonating others with malicious intent, b) spreading misinformation, c) violating personal privacy or dignity, d) creating explicit or inappropriate content.
- **Content Rights**: Users are responsible for: a) obtaining necessary permissions for source images, b) respecting copyrights and intellectual property, c) complying with local laws and regulations on AI-generated content.
- **Limitation of Liability**: The software and its developers are not liable for any user-generated content. Users assume full responsibility for the use of the generated content and any consequences that may arise from its use.

By using MagicMirror, you agree to these terms and commit to using the software responsibly.

## License

This project is licensed under the [MIT License](./LICENSE).

README.md


# FAQ

![](../assets/banner.jpg)

## 常见问题

### 如何使用 GPU 加速推理？

MagicMirror 默认使用 CPU 进行推理，暂不支持启用硬件加速选项。

不过你可以尝试 MagicMirror 的 CLI 版本 [TinyFace](https://github.com/idootop/TinyFace)，或者使用 [FaceFusion](https://github.com/facefusion/facefusion)。

### MagicMirror 的体积为什么那么小，是怎么做到的？

这多亏了 [Tauri](https://tauri.app/)，它使用系统自带的 WebView 组件，所以不用像 [Electron](https://www.electronjs.org/) 那样直接把整个 Chromium 浏览器和 Node.js 运行时打包进应用安装包。如果你对此感兴趣，可以在这里了解更多[技术细节](https://tauri.app/start/)。

### MagicMirror 的 LOGO 使用 AI 生成的吗？

是的。MagicMirror 内的所有设计资源：从 Logo、字体到 UI 界面，都是由 AI 生成的 ✨

![](../assets/aigc.webp)

这里我使用的是 [Tensor.ART](https://tusiart.com/)：一个免费的在线 AI 生图网站。像最新的 Flux 和 SD 3.5 等模型都可以直接使用，而且你也可以在线训练自己的模型。

比如 MagicMirror 的 LOGO， 就是我从 Dribbble 上搜集了一些参考图，然后用 Flux 做底模训练的一个模型生成的，非常方便。

![](../assets/train.webp)

相比 [Civitai](https://civitai.com/) 和 [LibLib.AI](https://www.liblib.art/) 等 AI 生图平台，[Tensor.ART](https://tusiart.com/) 的模型数量更多，价格更便宜，性价比应该是这三者里最高的。

如果你也想要尝试 AI 生图，或者正在寻找更有性价比的生图平台，不妨试试看 [Tensor.ART](https://tusiart.com/)。

## 安装问题

### 【macOS】打开 APP 时提示: “MagicMirror”已损坏，无法打开。 你应该将它移到废纸篓。

这是因为 macOS 默认只允许从 App Store 或已知的开发者来源安装应用，以保护用户的安全和隐私。

但是目前我还没有注册苹果开发者账号（$99 一年），所以无法为 MagicMirror 提供有效的签名并上架应用商店。

你可以：

1. 打开访达，然后在右侧菜单栏选择应用程序
2. 找到提示无法验证的应用，右键点击打开即可

![](../assets/macos-open.png)

如果在运行过程中仍然弹窗提示无法打开“xxx”（尤其是 macOS 15 系统版本及以上），

请自行百度 **macOS 如何开启任何来源**，参考：https://www.ghxi.com/jc202309058.html

### 【Windows】提示不是有效的 Win32 应用

MagicMirror 目前只提供了 x64 和 ARM64 架构的安装包，对于较老的 x32 电脑暂不支持。

推荐的运行环境是 Windows 11，如果低于 Windows 10 可能会无法正常运行。

## 运行问题

### 【macOS】卡在启动界面超过 10 分钟

如果你的电脑非 Intel 芯片（比如：M1、M4 芯片），首次启动 APP 时需要进行 Rosatta 转译，大概 3-5 分钟左右。

后续启动将恢复正常时长（5-10s）。如果超过 10 分钟仍未正常启动，请关闭应用后重试。

如果重启后还是无法正常启动，请检查你的 macOS 系统是否不低于 13 版本 (macOS Ventura)，较老的 macOS 系统可能无法正常运行。

查看此处了解更多信息：https://github.com/521xueweihan/HelloGitHub/issues/2859#issuecomment-2562637177

### 【Windows】卡在启动界面超过 10 分钟

如果你在 Windows 系统上一直卡在启动界面，请检查你的 `$HOME/MagicMirror/server.exe` 文件是否存在。

这是使用 [Nuitka](https://github.com/Nuitka/Nuitka) 编译的一个 Python 应用程序，MagicMirror 的正常运行离不开此程序。

![](../assets/windows-home.png)

由于许多病毒软件也喜欢使用 Nuitka 来编译他们的应用程序，从而混淆源代码隐藏自己的恶意行为特征。

所以同样使用 Nuitka 编译的 `server.exe` 文件，在启动时也容易被 Windows 安全应用误标记为病毒文件并删除。

![](../assets/windows-defender.jpg)

你可以在 Windows 安全中心 - 保护历史记录里，手动还原被删除的 `server.exe` 文件。然后重启 MagicMirror 应该就能正常运行了。

> 如果你对 Nuitka 被 Widnows 安全应用误报感兴趣，可以[查看此处](https://github.com/Nuitka/Nuitka/issues/2685#issuecomment-1923357489)了解更多技术细节。

如果仍然启动失败，查看此处了解更多：https://github.com/idootop/MagicMirror/issues/6#issuecomment-2560949972

### 提示换脸失败

你可以换几张其他的图片试试看。比较冷门的图片格式、文件名包含特殊字符、图片分辨率过大等都可能会导致换脸失败。

## 其他问题

### MagicMirror 与 FaceFusion 有什么联系？

简单来说，你可以把 MagicMirror 视为 FaceFusion 的精简版。

我在 [FaceFusion](https://github.com/facefusion/facefusion) 的基础上，移除了所有不必要的模块，只保留了最核心的换脸功能，并由此创造出了 [TinyFace](https://github.com/idootop/TinyFace)：一个超轻量的 Python 换脸工具。

MagicMirror 是在 [TinyFace](https://github.com/idootop/TinyFace) 之上构建的一个 GUI 项目，方便使用。

### MagicMirror 与 InsightFace 有什么联系？

一个经典的换脸工作流，通常由人脸检测、识别、换脸、画质增强等多个步骤组成，不同的步骤依赖不同的模型，而 [InsightFace](https://github.com/deepinsight/insightface/tree/master/examples/in_swapper) 的 `inswapper_128.onnx` 则是换脸过程中使用到的模型，是整个应用的核心。


# 安装教程

![](../assets/banner.jpg)

## 下载地址

首先根据你的操作系统，下载对应的 APP 安装包并安装：

1. Windows: [MagicMirror_1.0.0_windows_x86_64.exe](https://github.com/idootop/MagicMirror/releases/download/app-v1.0.0/MagicMirror_1.0.0_windows_x86_64.exe)
2. macOS: [MagicMirror_1.0.0_macos_universal.dmg](https://github.com/idootop/MagicMirror/releases/download/app-v1.0.0/MagicMirror_1.0.0_macos_universal.dmg)

> 如果你访问不了上面的 GitHub 下载链接，请使用国内[下载地址](https://del-wang.lanzout.com/b01qdt5nba) 密码: `4ro2`

## 下载模型

首次启动 APP，需要下载模型文件（默认会自动下载），初始化成功后才能使用。

![](../assets/download.png)

如果你的下载进度一直是 0，或者下到一半卡住了，请按照下面的步骤手动初始化：

**下载模型文件**

首先根据你的操作系统和架构，选择对应的模型文件：

- [server_windows_x86_64.zip](https://github.com/idootop/MagicMirror/releases/download/server-v1.0.0/server_windows_x86_64.zip)（Windows，大多数的电脑都是这种）
- [server_windows_aarch64.zip](https://github.com/idootop/MagicMirror/releases/download/server-v1.0.0/server_windows_aarch64.zip)（Windows，如果你是 ARM64 设备）
- [server_macos_aarch64.zip](https://github.com/idootop/MagicMirror/releases/download/server-v1.0.0/server_macos_aarch64.zip)（macOS, 苹果芯片，比如 M1、M4 芯片）
- [server_macos_x86_64.zip](https://github.com/idootop/MagicMirror/releases/download/server-v1.0.0/server_macos_x86_64.zip)（macOS, Intel 芯片）

> 如果你访问不了上面的 GitHub 下载链接，可以使用国内的[下载地址](https://pan.quark.cn/s/b8ad043794bb)

**解压下载的文件**

解压后应该是一个文件夹，把它重命名成: `MagicMirror` 然后移动到你电脑的 `HOME` 目录下，比如：

![](../assets/windows-home.png)

![](../assets/macos-home.png)

完成上面两步后，重启 MagicMirror 应该就能正常启动了。

## 启动 APP

下载完模型文件后，第一次启动应用可能比较缓慢，耐心等待即可。

![](../assets/launch.png)

> 一般 3 分钟以内即可启动成功，如果超过 10 分钟还未正常启动，请查看[常见问题](./faq.md)

## 遇到问题？

大部分问题都能在「[常见问题](./faq.md)」里找到答案，如果你还有其他问题，请在此[提交反馈](https://github.com/idootop/MagicMirror/issues)。


# MagicMirror

一键 AI 换脸、发型、穿搭，发现更美的自己 ✨

[![GitHub release](https://img.shields.io/github/v/release/idootop/MagicMirror.svg)](https://github.com/idootop/MagicMirror/releases) [![Build APP](https://github.com/idootop/MagicMirror/actions/workflows/build-app.yaml/badge.svg)](https://github.com/idootop/MagicMirror/actions/workflows/build-app.yaml) [![Build Server](https://github.com/idootop/MagicMirror/actions/workflows/build-server.yaml/badge.svg)](https://github.com/idootop/MagicMirror/actions/workflows/build-server.yaml)

![](../assets/banner.jpg)

## 功能亮点

- 一键换脸：打开安装包，拖张照片进去就能换脸，无需配置各种复杂参数。
- 超低门槛：不用 GPU 也能运行，普通的小白电脑也可以轻松玩转 AI 换脸。
- 隐私安全：完全在你的电脑本地运行，不需要联网，不用担心你的图片会被上传到任何地方。
- 极致精简：安装包不到 10 MB，模型文件加起来不到 1 GB，这可能是最轻量的离线换脸应用之一。

## 快速开始

> 🔥 演示视频：https://www.bilibili.com/video/BV1TTzfYDEUe

MagicMirror 仅支持 macOS 13（Ventura）和 Windows 10 及以上版本系统：

1. [安装教程](./install.md)
2. [使用教程](./usage.md)
3. [常见问题](./faq.md)（90%的问题可以在这里找到答案）

如果你有其他问题，请提交 [Issue](https://github.com/idootop/MagicMirror/issues)。

## 动机

![](../assets/demo.webp)

我想你跟我一样，有时会纠结：自己适合哪种发型，或者哪种穿搭最好看？

要是有一个应用，可以把我们看到的喜欢的发型或心动的穿搭，直接放到自己的身上预览效果，那真是太好了。

现在的 AI 技术已经很成熟了，但是市面上大部分的 AI 换脸应用：

- 要么需要配置复杂的参数和运行环境，要有性能足够强劲的 GPU 才能运行，使用门槛和成本偏高；
- 要么需要上传自己的图片到服务器进行转换，存在隐私泄漏的风险。

理想的解决方案应该像自拍一样简单：不需要设置复杂的参数，不用购买昂贵的设备，也无需担心隐私泄漏的问题。

所以，为什么不自己做一个呢？

于是便有了 MagicMirror ✨

Enjoy! ;)

## 鸣谢

MagicMirror 的实现主要使用和参考了以下开源项目，特此鸣谢:

- [Tauri](https://github.com/tauri-apps/tauri): Build smaller, faster, and more secure desktop and mobile applications with a web frontend.
- [FaceFusion](https://github.com/facefusion/facefusion): Industry leading face manipulation platform
- [InsightFace](https://github.com/deepinsight/insightface): State-of-the-art 2D and 3D Face Analysis Project
- [Nuitka](https://github.com/Nuitka/Nuitka): Nuitka is a Python compiler written in Python.

## 免责声明

MagicMirror 仅限个人娱乐与创意用途，严禁用于商业用途。请注意：

- **道德使用**：本软件不得用于以下行为，包括但不限于：a) 恶意冒充他人，b) 散布虚假信息，c) 侵犯个人隐私或尊严，d) 制作淫秽或不当内容。
- **内容版权**：用户应对以下内容负责：a) 获取使用源图像的必要许可，b) 尊重版权及知识产权，c) 遵守当地关于 AI 生成内容的法律法规。
- **免责声明**：用户应对生成的内容以及由其使用引发的任何法律、道德或个人问题承担全部责任。本软件及其开发者对用户生成的内容不承担任何责任。

使用 MagicMirror 即表示您已阅读并同意上述条款，并承诺负责任地使用本软件。

## License

This project is licensed under the [MIT License](./LICENSE).


# 使用教程

首先，准备一张你的正脸照，然后拖入到镜子里。

![](../assets/mirror-me.jpg)

然后把你想要换脸的照片，拖到另一面镜子里，等待换脸完毕即可。

![](../assets/mirror-input.jpg)

> 苹果 M1 芯片换一张脸一般需要 3-5s 的时间，主要看你的电脑配置和图片大小

换脸成功后，会在原来的位置生成一张以 `_output` 结尾的新图片。

![](../assets/mirror-result.jpg)

你可以继续拖入新的照片换脸，或通过右上角的菜单翻转镜像，更换新的脸部图片。

![](../assets/mirror-flip.jpg)

## 遇到问题？

大部分问题都能在「[常见问题](./faq.md)」里找到答案，如果你还有其他问题，请在此[提交反馈](https://github.com/idootop/MagicMirror/issues)。


# FAQ

![](../assets/banner.jpg)

## Common

### How to Enable GPU Acceleration for Inference?

MagicMirror uses CPU by default for inference and currently does not support hardware acceleration options. However, you can try MagicMirror’s CLI version, [TinyFace](https://github.com/idootop/TinyFace), or use [FaceFusion](https://github.com/facefusion/facefusion).

### Why is MagicMirror So Small in Size?

This is thanks to [Tauri](https://tauri.app/), which uses the system’s built-in WebView component, unlike [Electron](https://www.electronjs.org/), which packages an entire Chromium browser and Node.js runtime into the app. If you're interested in the technical details, [click here](https://tauri.app/start/).

### Is the MagicMirror Logo AI-Generated?

Every design element in MagicMirror - from the logo and typography to the UI - was created using AI ✨

![](../assets/aigc.webp)

I used [Tensor.ART](https://tensor.art/), a free AI image generation platform. It offers access to the latest models like Flux and SD 3.5, plus the ability to train your own custom models.

For example, I created MagicMirror's logo by training a Flux-based model on reference images collected from Dribbble - quick and simple.

![](../assets/train.webp)

Compared to platforms like [Civitai](https://civitai.com/) and [LibLib.AI](https://www.liblib.art/), [Tensor.ART](https://tensor.art/) has more models at a lower price, making it the most cost-effective option.

If you're looking to try AI image generation or want a more affordable platform, I recommend giving [Tensor.ART](https://tensor.art/) a try.

## Installation

### 【macOS】App is Damaged or Can't Be Verified

This issue occurs because the app isn't signed with an Apple developer certificate. To resolve this:

1. Open Finder and select Applications from the sidebar.
2. Locate the app with the warning, right-click it, and choose Open.

![](../assets/macos-open.png)

## Usage

### 【Windows】App Stuck on the Startup Screen for Over 10 Minutes

If the app is stuck at the startup screen, check if the `server.exe` file exists on your system. This file, compiled using [Nuitka](https://github.com/Nuitka/Nuitka) from Python source, is essential for MagicMirror to run.

![](../assets/windows-home.png)

Windows Defender may flag the `server.exe` as malicious due to the use of Nuitka by some malware, leading to it being quarantined.

![](../assets/windows-defender.jpg)

To fix this:

1. Open Windows Security Center and go to Protection History.
2. Find and restore the quarantined `server.exe`.
3. Restart MagicMirror.

> For details on Nuitka false positives in Windows Defender, [click here](https://github.com/Nuitka/Nuitka/issues/2685#issuecomment-1923357489).

If none of the above work, your system likely has compatibility issues with the server. Consider using alternative face swap tools like [TinyFace](https://github.com/idootop/TinyFace) (the underlying library used by MagicMirror), or [FaceFusion](https://github.com/facefusion/facefusion).

Learn more: https://github.com/idootop/MagicMirror/issues/6#issuecomment-2560949972

### 【Windows】Face Swap Fails with Any Image

First, check if the `server.exe` file exists in `$HOME/MagicMirror/`. If it’s missing, follow the steps in the previous section to restore it.

![](../assets/windows-home.png)

If the issue persists, please [submit an issue](https://github.com/idootop/MagicMirror/issues).

### 【Windows】"Not a Valid Win32 Application" Error

MagicMirror currently only supports x64 and ARM64 architectures. Older x32 systems are not supported.

The recommended environment is Windows 11 x64. MagicMirror may not run properly on versions lower than Windows 10.

### 【macOS】App Stuck on Startup for Over 10 Minutes

First-time startup on macOS can be slow. If the app doesn’t start within 30 minutes, close it and try again.

If it still doesn’t launch, ensure your macOS version is at least `13` (Ventura), as older versions may not be compatible.

Learn more: https://github.com/521xueweihan/HelloGitHub/issues/2859#issuecomment-2562637177

## Other

### What’s the Relationship Between MagicMirror and FaceFusion?

In simple terms, MagicMirror is a simplified version of FaceFusion.

Starting from [FaceFusion](https://github.com/facefusion/facefusion), I removed unnecessary modules and kept only the core face-swapping functionality, which led to the creation of [TinyFace](https://github.com/idootop/TinyFace), a lightweight Python face-swapping tool.

MagicMirror is a GUI project built on top of [TinyFace](https://github.com/idootop/TinyFace), making it easier to use.

### What’s the Relationship Between MagicMirror and InsightFace?

A typical face-swapping workflow includes multiple steps, such as face detection, recognition, swapping, and quality enhancement. These steps rely on different models. The `inswapper_128.onnx` model from [InsightFace](https://github.com/deepinsight/insightface/tree/master/examples/in_swapper) is a core part of the face-swapping process.


# Installation Guide

![](../assets/banner.jpg)

## Installation

Download and install the MagicMirror installer for your operating system:

1. Windows: [MagicMirror_1.0.0_windows_x86_64.exe](https://github.com/idootop/MagicMirror/releases/download/app-v1.0.0/MagicMirror_1.0.0_windows_x86_64.exe)
2. macOS: [MagicMirror_1.0.0_macos_universal.dmg](https://github.com/idootop/MagicMirror/releases/download/app-v1.0.0/MagicMirror_1.0.0_macos_universal.dmg)
3. Other: [Go to Release](https://github.com/idootop/MagicMirror/releases/app-v1.0.0)

## Download Models

When you first launch the app, it will automatically download the required model files.

![](../assets/download.png)

If the download progress is stuck at 0% or stops midway, follow these steps for manual setup:

**Download Model Files**

Choose the model file that matches your operating system:

- [server_windows_x86_64.zip](https://github.com/idootop/MagicMirror/releases/download/server-v1.0.0/server_windows_x86_64.zip)（Windows, most common）
- [server_windows_aarch64.zip](https://github.com/idootop/MagicMirror/releases/download/server-v1.0.0/server_windows_aarch64.zip)（Windows, for ARM64 devices）
- [server_macos_aarch64.zip](https://github.com/idootop/MagicMirror/releases/download/server-v1.0.0/server_macos_aarch64.zip)（macOS, Apple Silicon, such as M1, M4 chips）
- [server_macos_x86_64.zip](https://github.com/idootop/MagicMirror/releases/download/server-v1.0.0/server_macos_x86_64.zip)（macOS, Intel chips）

**Extract the Downloaded File**

You'll get a folder—rename it to `MagicMirror`. Move this folder to your computer's `HOME` directory, for example:

![](../assets/windows-home.png)

![](../assets/macos-home.png)

Restart MagicMirror, and it should now work properly.

## Launch APP

After downloading the model files, the first launch may take some time.

![](../assets/launch.png)

> The app should launch within 3 minutes. If it takes longer than 10 minutes to start, please refer to the [FAQ](./faq.md)

## Need help?

Most issues are addressed in the [FAQ](./faq.md). If you need further assistance, please [submit an issue](https://github.com/idootop/MagicMirror/issues).


# Usage Guide

First, drag a front-facing photo of yourself into the mirror.

![](../assets/mirror-me.jpg)

Next, drag your desired photo into the other mirror and wait for processing to complete.

![](../assets/mirror-input.jpg)

> On Apple M1 chips, face swapping takes about 3-5 seconds, varying with your computer's specs and image size.

When processing is complete, a new image will appear in the original location with `_output` added to its filename.

![](../assets/mirror-result.jpg)

You can continue swapping faces with new photos, or use the top-right menu to flip the mirror and change the face image.

![](../assets/mirror-flip.jpg)

## Need help?

Most issues are addressed in the [FAQ](./faq.md). If you need further assistance, please [submit an issue](https://github.com/idootop/MagicMirror/issues).


# Logic-RL

## 📢 **Our detailed technical report is released!**

Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning
---

    Main results

---

## Benchmark

| Model                                                             | 2ppl | 3ppl | 4ppl | 5ppl | 6ppl | 7ppl | 8ppl |
|------------------------------------------------------------------------|------|------|------|------|------|------|------|
| o3-mini-high                | 0.99 | 0.98 | 0.97 | 0.95 | 0.94 | 0.89 | 0.83 |
| o1-2024-12-17               | 0.83 | 0.51 | 0.38 | 0.38 | 0.35 | 0.30 | 0.20 |
| GPT-4o                      | 0.68 | 0.57 | 0.49 | 0.32 | 0.23 | 0.21 | 0.11 |
| Deepseek-Math-7b            | 0.35 | 0.21 | 0.08 | 0.06 | 0.02 | 0.00 | 0.00 |
| Qwen2.5-7B-Instruct-1M      | 0.49 | 0.40 | 0.25 | 0.11 | 0.02 | 0.06 | 0.01 |
| Qwen2.5-7B-Logic-RL (ours)  | 0.99 | 0.99 | 0.94 | 0.92 | 0.91 | 0.80 | 0.67 |

---

## Installation

```bash
conda create -n logic python=3.9
pip install torch==2.4.0 --index-url https://download.pytorch.org/whl/cu121
pip3 install vllm==0.6.3 ray
pip3 install flash-attn --no-build-isolation
pip install -e .  # For verl integration
pip install wandb IPython matplotlib
```

---

## Data Preparation

You can directly use /data.

For your own data generation, here's a demo:

### Base Model
```bash
python ./examples/data_preprocess/kk.py \
    --local_dir {processed_data_path} \
    --data_path {raw_data_path}
```

### Instruct Model
```bash
python ./examples/data_preprocess/kk.py \
    --template_type=qwen-instruct \
    --local_dir {processed_data_path} \
    --data_path {raw_data_path}
```

---

## Training Execution
```bash
conda activate logic
bash main_grpo.sh  # 4×A100 80G
```

---

## ⚙ Implementation Details

| Component              | Location                          |
|------------------------|-----------------------------------|
| Reward Modeling     | `verl/utils/reward_score/kk.py`   |
| Data Preprocessing   | `examples/data_preprocess/kk.py`  |

---

## Citation
```
@misc{xie2025logicrlunleashingllmreasoning,
      title={Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning},
      author={Tian Xie and Zitian Gao and Qingnan Ren and Haoming Luo and Yuqian Hong and Bryan Dai and Joey Zhou and Kai Qiu and Zhirong Wu and Chong Luo},
      year={2025},
      eprint={2502.14768},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2502.14768},
}
```

---

## Acknowledgements
- [Verl](https://github.com/volcengine/verl) 🔗
- [TinyZero](https://github.com/Jiayi-Pan/TinyZero) 🔗
- [Knights and Knaves (K&K) puzzles dataset](https://github.com/AlphaPav/mem-kk-logic) 🔗

---

## Star History

[![Star History Chart](https://api.star-history.com/svg?repos=Unakar/Logic-RL&type=Date)](https://star-history.com/#Unakar/Logic-RL&Date)

README.md

Reproduce R1 Zero on Logic Puzzle


# veRL documents

## Build the docs

```bash
# Install dependencies.
pip install -r requirements-docs.txt

# Build the docs.
make clean
make html
```

## Open the docs with your browser

```bash
python -m http.server -d _build/html/
```
Launch your browser and open localhost:8000.

docs/README.md

Reproduce R1 Zero on Logic Puzzle


Welcome to veRL's documentation!
================================================

.. _hf_arxiv: https://arxiv.org/pdf/2409.19256

veRL is a flexible, efficient and production-ready RL training framework designed for large language models (LLMs) post-training. It is an open source implementation of the `HybridFlow <hf_arxiv>`_ paper.

veRL is flexible and easy to use with:

- **Easy extension of diverse RL algorithms**: The Hybrid programming model combines the strengths of single-controller and multi-controller paradigms to enable flexible representation and efficient execution of complex Post-Training dataflows. Allowing users to build RL dataflows in a few lines of code.

- **Seamless integration of existing LLM infra with modular APIs**: Decouples computation and data dependencies, enabling seamless integration with existing LLM frameworks, such as PyTorch FSDP, Megatron-LM and vLLM. Moreover, users can easily extend to other LLM training and inference frameworks.

- **Flexible device mapping and parallelism**: Supports various placement of models onto different sets of GPUs for efficient resource utilization and scalability across different cluster sizes.

- Readily integration with popular HuggingFace models

veRL is fast with:

- **State-of-the-art throughput**: By seamlessly integrating existing SOTA LLM training and inference frameworks, veRL achieves high generation and training throughput.

- **Efficient actor model resharding with 3D-HybridEngine**: Eliminates memory redundancy and significantly reduces communication overhead during transitions between training and generation phases.

--------------------------------------------

.. _Contents:

.. toctree::
   :maxdepth: 5
   :caption: Quickstart
   :titlesonly:
   :numbered:

   start/install
   start/quickstart

.. toctree::
   :maxdepth: 5
   :caption: Data Preparation
   :titlesonly:
   :numbered:

   preparation/prepare_data
   preparation/reward_function

.. toctree::
   :maxdepth: 2
   :caption: PPO Example
   :titlesonly:
   :numbered:

   examples/ppo_code_architecture
   examples/config
   examples/gsm8k_example

.. toctree::
   :maxdepth: 1
   :caption: PPO Trainer and Workers

   workers/ray_trainer
   workers/fsdp_workers
   workers/megatron_workers

.. toctree::
   :maxdepth: 1
   :caption: Experimental Results

   experiment/ppo

.. toctree::
   :maxdepth: 1
   :caption: Advance Usage and Extension

   advance/placement
   advance/dpo_extension
   advance/fsdp_extension
   advance/megatron_extension

.. toctree::
   :maxdepth: 1
   :caption: FAQ

   faq/faq

Contribution
-------------

veRL is free software; you can redistribute it and/or modify it under the terms
of the Apache License 2.0. We welcome contributions.
Join us on `GitHub <https://github.com/volcengine/verl>`_, `Slack <https://join.slack.com/t/verlgroup/shared_invite/zt-2w5p9o4c3-yy0x2Q56s_VlGLsJ93A6vA>`_ and `Wechat <https://raw.githubusercontent.com/eric-haibin-lin/verl-community/refs/heads/main/WeChat.JPG>`_ for discussions.

Code formatting
^^^^^^^^^^^^^^^^^^^^^^^^
We use yapf (Google style) to enforce strict code formatting when reviewing MRs. Run yapf at the top level of verl repo:

.. code-block:: bash

   pip3 install yapf
   yapf -ir -vv --style ./.style.yapf verl examples tests

docs/index.rst

Reproduce R1 Zero on Logic Puzzle


Extend to other RL(HF) algorithms
=================================

We already implemented the complete training pipeline of the PPO
algorithms. To extend to other algorithms, we analyze the high-level
principle to use veRL and provide a tutorial to implement the DPO
algorithm. Users can follow the similar paradigm to extend to other RL algorithms.

.. note:: **Key ideas**: Single process drives multi-process computation and data communication.

Overall Approach
----------------

Step 1: Consider what multi-machine multi-GPU computations are needed
for each model, such as ``generate_sequence`` , ``compute_log_prob`` and
``update_policy`` in the actor_rollout model. Implement distributed
single-process-multiple-data (SPMD) computation and encapsulate them
into APIs

Step 2: Based on different distributed scenarios, including FSDP and 3D
parallelism in Megatron-LM, implement single-process control of data
interaction among multi-process computations.

Step 3: Utilize the encapsulated APIs to implement the control flow

Example: Online DPO
-------------------

We use veRL to implement a simple online DPO algorithm. The algorithm
flow of Online DPO is as follows:

1. There is a prompt (rollout) generator which has the same weight as
   the actor model. After a batch of prompts are fed into the generator,
   it generates N responses for each prompt.
2. Send all the prompts + responses to a verifier for scoring, which can
   be reward model or a rule-based function. Then sort them in pairs to
   form a training batch.
3. Use this training batch to train the actor model using DPO. During
   the process, a reference policy is needed.

Step 1: What are the multi-machine multi-GPU computations
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

**Sample Generator**

Implementation details:

.. code:: python

   from verl.single_controller.base import Worker
   from verl.single_controller.ray import RayWorkerGroup, RayClassWithInitArgs, RayResourcePool
   import ray

   @ray.remote
   class SampleGenerator(Worker):
       def __init__(self, config):
           super().__init__()
           self.config = config

       def generate_sequences(self, data):
           pass

Here, ``SampleGenerator`` can be viewed as a multi-process pulled up by
``torchrun``, with each process running the same code (SPMD).
``SampleGenerator`` needs to implement a ``generate_sequences`` API for
the control flow to call. The implementation details inside can use any
inference engine including vllm, sglang and huggingface. Users can
largely reuse the code in
verl/verl/trainer/ppo/rollout/vllm_rollout/vllm_rollout.py and we won't
go into details here.

**ReferencePolicy inference**

API: compute reference log probability

.. code:: python

   from verl.single_controller.base import Worker
   import ray

   @ray.remote
   class ReferencePolicy(Worker):
       def __init__(self):
           super().__init__()
           self.model = Model()

       def infer(self, data):
           return self.model(data)

**Actor update**

API: Update actor model parameters

.. code:: python

   from verl.single_controller.base import Worker
   import ray

   @ray.remote
   class DPOActor(Worker):
       def __init__(self):
           super().__init__()
           self.model = Model()
           self.model = FSDP(self.model)  # or other distributed strategy
           self.optimizer = optim.Adam(self.model.parameters(), lr=1e-3)
           self.loss_fn = xxx

       def update(self, data):
           self.optimizer.zero_grad()
           logits = self.model(data)
           loss = self.loss_fn(logits)
           loss.backward()
           self.optimizer.step()

**Notes: How to distinguish between control processes and distributed computation processes**
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

- Control processes are generally functions directly decorated with
  ``@ray.remote``
- Computation processes are all wrapped into a ``RayWorkerGroup``.

Users can reuse most of the distribtued computation logics implemented
in PPO algorithm, including FSDP and Megatron-LM backend in
verl/verl/trainer/ppo.

Step 2: Based on different distributed scenarios, implement single-process control of multi-process data interaction
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

**The core problem to solve here is how a single process sends data to
multiple processes, drives multi-process computation, and how the
control process obtains the results of multi-process computation.**
First, we initialize the multi-process ``WorkerGroup`` in the control
process.

.. code:: python

   @ray.remote(num_cpus=1)
   def main_task(config):
       # construct SampleGenerator
       resource_pool = RayResourcePool(process_on_nodes=[8] * 2)  # 16 GPUs
       ray_cls = RayClassWithInitArgs(SampleGenerator, config=config)
       # put SampleGenerator onto resource pool
       worker_group = RayWorkerGroup(resource_pool, ray_cls)

       # construct reference policy

As we can see, in the control process, multiple processes are wrapped
into a ``RayWorkerGroup``. Inside this ``WorkerGroup``, there is a
``self._workers`` member, where each worker is a RayActor
(https://docs.ray.io/en/latest/ray-core/actors.html) of SampleGenerator.
ray_trainer.md also provide an implementation of
``MegatronRayWorkerGroup``.

Assuming the model is distributed using FSDP, and there is a batch of
data on the control process, for data parallelism, the underlying
calling process is:

.. code:: python

   data = xxx
   data_list = data.chunk(dp_size)

   output = []
   for d in data_list:
       # worker_group._workers[i] is a SampleGenerator
       output.append(worker_group._workers[i].generate_sequences.remote(d))

   output = ray.get(output)
   output = torch.cat(output)

Single process calling multiple processes involves the following 3
steps:

1. Split the data into DP parts on the control process.
2. Send the data to remote, call the remote computation through RPC, and
   utilize multi-process computation.
3. Obtain the computation results of each worker on the control process
   and merge them.

Frequently calling these 3 steps on the controller process greatly hurts
code readability. **In veRL, we have abstracted and encapsulated these 3
steps, so that the worker's method + dispatch + collect can be
registered into the worker_group**

.. code:: python

   from verl.single_controller.base.decorator import register

   def dispatch_data(worker_group, data):
       return data.chunk(worker_group.world_size)

   def collect_data(worker_group, data):
       return torch.cat(data)

   dispatch_mode = {
       'dispatch_fn': dispatch_data,
       'collect_fn': collect_data
   }

   @register(dispatch_mode=dispatch_mode)
   def generate_sequences(self, data):
       pass

In this way, we can directly call the method inside the worker through
the ``worker_group`` on the control (driver) process (which is a single
process):

.. code:: python

   output = worker_group.generate_sequences(data)

This single line includes data splitting, data distribution and
computation, and data collection.

Furthermore, the model parallelism size of each model is usually fixed,
including dp, tp, pp. So for these common distributed scenarios, we have
pre-implemented specific dispatch and collect methods,in `decorator.py <https://github.com/volcengine/verl/blob/main/verl/single_controller/base/decorator.py>`_, which can be directly used to wrap the computations.

.. code:: python

   from verl.single_controller.base.decorator import register, Dispatch

   @register(dispatch_mode=Dispatch.DP_COMPUTE_PROTO)
   def generate_sequences(self, data: DataProto) -> DataProto:
       pass

Here it requires the data interface to be ``DataProto``. Definition of
``DataProto`` is in `protocol.py <https://github.com/volcengine/verl/blob/main/verl/protocol.py>`_.

Step 3: Main training loop
~~~~~~~~~~~~~~~~~~~~~~~~~~

With the above training flows, we can implement the algorithm's control
flow. It is recommended that ``main_task`` is also a ray remote process.

.. code:: python

   @ray.remote(num_cpus=1)
   def main_task(config):
       # construct SampleGenerator
       resource_pool = RayResourcePool(process_on_nodes=[8] * 2)  # 16 GPUs
       ray_cls = RayClassWithInitArgs(SampleGenerator, config=config)
       # put SampleGenerator onto resource pool
       sample_gen = RayWorkerGroup(resource_pool, ray_cls)

       # construct reference policy
       ray_cls = RayClassWithInitArgs(ReferencePolicy)
       ref_policy = RayWorkerGroup(resource_pool, ray_cls)

       # construct actor
       ray_cls = RayClassWithInitArgs(DPOActor)
       dpo_policy = RayWorkerGroup(resource_pool, ray_cls)

       dataloader = DataLoader()

       for data in dataloader:
           # generate data
           data = sample_gen.generate_sequences(data)
           # generate scores for each data
           data = generate_scores(data)
           # generate pairwise data using scores
           data = generate_pairwise_data(data)
           # generate ref_log_prob
           data.batch['ref_log_prob'] = ref_policy.infer(data)
           # update using dpo
           dpo_policy.update(data)
           # logging

Here, different ``WorkerGroups`` can be placed in the same resource pool or
in different resource pools using ``create_colocated_worker_cls``
similar as in `ray_trainer.py <https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/ray_trainer.py>`_.

Reproduce R1 Zero on Logic Puzzle


Add models with the FSDP backend
==================================

Model
--------------------------

In principle, our FSDP backend can support any HF model and we can
sychronoize the actor model weight with vLLM using `hf_weight_loader.py <https://github.com/volcengine/verl/blob/main/verl/third_party/vllm/vllm_v_0_5_4/hf_weight_loader.py>`_.
However, ``hf_weight_loader`` is will gather the full state_dict of a
model during synchronization, which may cause OOM. We suggest using
``dtensor_weight_loader`` which gather the full model parameter layer by
layer to reduce the peak memory usage. We already support dtensor weight
loader for the models below in `dtensor_weight_loader.py <https://github.com/volcengine/verl/blob/main/verl/third_party/vllm/vllm_v_0_5_4/dtensor_weight_loaders.py>`_.:

- ``GPT2LMHeadModel``
- ``LlamaForCausalLM``
- ``LLaMAForCausalLM``
- ``MistralForCausalLM``
- ``InternLMForCausalLM``
- ``AquilaModel``
- ``AquilaForCausalLM``
- ``Phi3ForCausalLM``
- ``GemmaForCausalLM``
- ``Gemma2ForCausalLM``
- ``GPTBigCodeForCausalLM``
- ``Starcoder2ForCausalLM``
- ``Qwen2ForCausalLM``
- ``DeepseekV2ForCausalLM``

To implement ``dtensor_weight_loader`` of a model that's supported in
vLLM, follow the guide of gemma model below:

1. Copy the
   ``load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]])`` from the vllm model class
   to ``dtensor_weight_loaders.py``
2. Modify the arguments to
   ``(actor_weights: Dict, vllm_model: nn.Module)``
3. Replace the ``self`` to ``vllm_model``
4. Add the
   ``local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight)``
   before each ``param = params_dict[name]`` and modify the following
   weight loading using ``local_loaded_weight``.
5. Register the implemented dtensor weight loader to ``__MODEL_DTENSOR_WEIGHT_LOADER_REGISTRY__``.

.. code-block:: diff

    - def load_weights(self, weights: Iterable[Tuple[str, torch.Tensor]]):
    + def gemma_dtensor_weight_loader(actor_weights: Dict, vllm_model: nn.Module) -> nn.Module:
        stacked_params_mapping = [
            # (param_name, shard_name, shard_id)
            ("qkv_proj", "q_proj", "q"),
            ("qkv_proj", "k_proj", "k"),
            ("qkv_proj", "v_proj", "v"),
            ("gate_up_proj", "gate_proj", 0),
            ("gate_up_proj", "up_proj", 1),
        ]
    -   params_dict = dict(self.named_parameters())
    +   params_dict = dict(vllm_model.named_parameters())
        loaded_params = set()
    -   for name, loaded_weight in weights:
    +   for name, loaded_weight in actor_weights.items():
            for (param_name, shard_name, shard_id) in stacked_params_mapping:
                if shard_name not in name:
                    continue
                name = name.replace(shard_name, param_name)
                # Skip loading extra bias for GPTQ models.
                if name.endswith(".bias") and name not in params_dict:
                    continue
    +           local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight)
                param = params_dict[name]
                weight_loader = param.weight_loader
    -           weight_loader(param, loaded_weight, shard_id)
    +           weight_loader(param, local_loaded_weight.to(dtype=param.dtype), shard_id)
                break
            else:
                # lm_head is not used in vllm as it is tied with embed_token.
                # To prevent errors, skip loading lm_head.weight.
                if "lm_head.weight" in name:
                    continue
                # Skip loading extra bias for GPTQ models.
                if name.endswith(".bias") and name not in params_dict:
                    continue
    +           local_loaded_weight = redistribute_dtensor(param_name=name, loaded_weights=loaded_weight)
                param = params_dict[name]
                weight_loader = getattr(param, "weight_loader",
                                        default_weight_loader)
    -           weight_loader(param, loaded_weight)
    +           weight_loader(param, local_loaded_weight.to(dtype=param.dtype))
            loaded_params.add(name)
        unloaded_params = params_dict.keys() - loaded_params
        if unloaded_params:
            raise RuntimeError(
                "Some weights are not initialized from checkpoints: "
                f"{unloaded_params}")

Reproduce R1 Zero on Logic Puzzle


Add models with the Megatron-LM backend
=========================================

Model
-----------

The most challenging aspect to use the Megatron-LM backend is implementing
the models for training. Currently, we implement Llama model that
support data parallelism, tensor parallelism, pipeline parallelism (also
vPP) and sequence parallelism. We also implement remove padding (sequence packing) on Llama
model, which can be found in `modeling_llama_megatron.py <https://github.com/volcengine/verl/blob/main/verl/models/llama/megatron/modeling_llama_megatron.py>`_.

To support other model, users are required to implement:

1. Implemnt a model similar to ``modeling_llama_megatron.py`` that satisfy the
   parallelism requirements of Megatron-LM. Then register your model in
   the `registry.py <https://github.com/volcengine/verl/blob/main/verl/models/registry.py>`_.
2. Checkpoint utils that can load full checkpoint (e.g. huggingface
   checkpoint) to partitioned models during the runtime. Then register
   your loader to ``weight_loader_registry`` in `weight_loader_registry.py <https://github.com/volcengine/verl/blob/main/verl/models/weight_loader_registry.py>`_.
3. Weight loader that synchronize the weight from Megatron to rollout
   (vLLM) model. Note that both the actor model and rollout model are
   partitioned during runtime. So, it's advisable to map the model name
   in actor model implementation. Otherwise, you may need an additional
   name mapping and even weight transformation. The weight loader implementation
   is in `megatron_weight_loaders.py <https://github.com/volcengine/verl/blob/main/verl/third_party/vllm/vllm_v_0_6_3/megatron_weight_loaders.py>`_.

Reproduce R1 Zero on Logic Puzzle


Ray API Design Tutorial
=======================================

We provide a tutorial for our Ray API design, including:

- Ray basic concepts
- Resource Pool and RayWorkerGroup
- Data Dispatch, Execution and Collection
- Initialize the RayWorkerGroup and execute the distributed computation in the given Resource Pool

See details in `tutorial.ipynb <https://github.com/volcengine/verl/blob/main/examples/ray/tutorial.ipynb>`_.

Reproduce R1 Zero on Logic Puzzle


.. _config-explain-page:

Config Explaination
===================

ppo_trainer.yaml for FSDP Backend
---------------------------------

Data
~~~~

.. code:: yaml

   data:
     tokenizer: null
     train_files: ~/data/rlhf/gsm8k/train.parquet
     val_files: ~/data/rlhf/gsm8k/test.parquet
     prompt_key: prompt
     max_prompt_length: 512
     max_response_length: 512
     train_batch_size: 1024
     val_batch_size: 1312
     return_raw_input_ids: False  # This should be set to true when the tokenizer between policy and rm differs
     return_raw_chat: False

- ``data.train_files``: Training set parquet. Can be a list or a single
  file. The program will read all files into memory, so it can't be too
  large (< 100GB). The path can be either local path or HDFS path. For
  HDFS path, we provide utils to download it to DRAM and convert the
  HDFS path to local path.
- ``data.val_files``: Validation parquet. Can be a list or a single
  file.
- ``data.prompt_key``: The field in the dataset where the prompt is
  located. Default is 'prompt'.
- ``data.max_prompt_length``: Maximum prompt length. All prompts will be
  left-padded to this length. An error will be reported if the length is
  too long
- ``data.max_response_length``: Maximum response length. Rollout in RL
  algorithms (e.g. PPO) generates up to this length
- ``data.train_batch_size``: Batch size sampled for one training
  iteration of different RL algorithms.
- ``data.val_batch_size``: Batch size sampled for one validation
  iteration.
- ``data.return_raw_input_ids``: Whether to return the original
  input_ids without adding chat template. This is mainly used to
  accommodate situations where the reward model's chat template differs
  from the policy. It needs to be decoded first, then apply the RM's
  chat template. If using a model-based RM, and the policy and RM
  chat_templates are different, this flag needs to be set
- ``data.return_raw_chat``:
- ``data.truncation``: Truncate the input_ids or prompt length if they
  exceed max_prompt_length. Default is 'error', not allow exceed the
  max_prompt_length. The users should increase the max_prompt_length if
  throwing the error.

Actor/Rollout/Reference Policy
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code:: yaml

   actor_rollout_ref:
     hybrid_engine: True
     model:
       path: ~/models/deepseek-llm-7b-chat
       external_lib: null
       override_config: {}
       enable_gradient_checkpointing: False
     actor:
       strategy: fsdp  # This is for backward-compatibility
       ppo_mini_batch_size: 256
       ppo_micro_batch_size: 64
       grad_clip: 1.0
       clip_ratio: 0.2
       entropy_coeff: 0.001
       ppo_epochs: 1
       shuffle: True
       optim:
         lr: 1e-6
         lr_warmup_steps_ratio: 0.  # the total steps will be injected during runtime
         min_lr_ratio: null   # only useful for warmup with cosine
         warmup_style: constant  # select from constant/cosine
         total_training_steps: -1  # must be override by program
       fsdp_config:
         wrap_policy:
           # transformer_layer_cls_to_wrap: None
           min_num_params: 0
         param_offload: False
         grad_offload: False
         optimizer_offload: False
     ref:
       fsdp_config:
         param_offload: False
         wrap_policy:
           # transformer_layer_cls_to_wrap: None
           min_num_params: 0
       log_prob_micro_batch_size: 128
     rollout:
       name: vllm
       temperature: 1.0
       top_k: -1 # 0 for hf rollout, -1 for vllm rollout
       top_p: 1
       response_length: ${data.max_response_length}
       # for vllm rollout
       dtype: bfloat16 # should align with FSDP
       gpu_memory_utilization: 0.5
       ignore_eos: False
       enforce_eager: True
       free_cache_engine: True
       load_format: dummy_dtensor # or dummy_hf or dummy_megatron
       tensor_model_parallel_size: 2
       max_num_batched_tokens: 8192
       max_num_seqs: 1024
       log_prob_micro_batch_size: 128
       # for vllm and hf rollout
       do_sample: True

**Common config for actor, rollout and reference model**

- ``actor_rollout_ref.hybrid_engine``: Whether it's a hybrid engine,
  currently only supports hybrid engine
- ``actor_rollout_ref.model.path``: Huggingface model path. This can be
  either local path or HDFS path. For HDFS path, we provide utils to
  download it to DRAM and convert the HDFS path to local path.
- ``actor_rollout_ref.model.external_libs``: Additional Python packages
  that need to be imported. Used to register models or tokenizers into
  the Huggingface system.
- ``actor_rollout_ref.model.override_config``: Used to override some of
  the model's original configurations, mainly dropout
- ``actor_rollout_ref.model.enable_gradient_checkpointing``: Whether to
  enable gradient checkpointing for the actor

**Actor model**

- ``actor_rollout_ref.actor.strategy``: fsdp or megatron. In this
  example, we use fsdp backend.

- ``actor_rollout_ref.actor.ppo_mini_batch_size``: One sample is split
  into multiple sub-batches with batch_size=ppo_mini_batch_size for PPO
  updates

- ``actor_rollout_ref.actor.ppo_micro_batch_size``: Similar to gradient
  accumulation, the micro_batch_size for one forward pass, trading speed
  for GPU memory

- ``actor_rollout_ref.actor.grad_clip``: Gradient clipping for actor
  updates

- ``actor_rollout_ref.actor.clip_ratio``: PPO clip ratio

- ``actor_rollout_ref.actor.entropy_coeff``: The weight of entropy when
  calculating PPO loss

- ``actor_rollout_ref.actor.ppo_epochs``: Number of epochs for PPO
  updates on one set of sampled data

- ``actor_rollout_ref.actor.shuffle``: Whether to shuffle data when
  there are multiple epochs

- ``actor_rollout_ref.actor.optim``: Actor's optimizer parameters

- ``actor_rollout_ref.actor.fsdp_config``: FSDP config for actor
  training

  - ``wrap_policy``: FSDP wrap policy. By default, it uses Huggingface's
    wrap policy, i.e., wrapping by DecoderLayer

    - No need to set transformer_layer_cls_to_wrap, so we comment it.

  - ``*_offload``: Whether to enable parameter, gradient and optimizer
    offload

    - Trading speed for GPU memory.

**Reference Model**

- ``actor_rollout_ref.ref``: FSDP config same as actor. **For models
  larger than 7B, it's recommended to turn on offload for ref by
  default**
- ``actor_rollout_ref.ref.log_prob_micro_batch_size``: The batch size
  for one forward pass in the computation of ``ref_log_prob``.

**Rollout Model**

- ``actor_rollout_ref.rollout.name``: hf/vllm. We use vLLM by default
  because it's much efficient and our hybrid engine is implemented with
  vLLM.

- Rollout (Auto-regressive) parameters. The key should be equal to the
  property name in vLLM's ``SamplingParams``.

  - ``temperature``, ``top_k``, ``top_p`` and others: Sampling
    parameters in ``SamplingParams``.

- ``dtype``: Rollout model parameters type. This should be align with
  the actor model parameter type in FSDP/Megatron backend.

- ``gpu_memory_utilization``: The proportion of the remaining GPU memory
  allocated for kv cache after other models have initialized when using
  vllm.

- ``tensor_model_parallel_size``: TP size for rollout. Only effective
  for vllm.

- ``log_prob_micro_batch_size``: Micro_batch_size (The batch size for
  one forward pass) for recalculating log_prob.

- ``do_sample``: Whether to sample. If set to False, the rollout model
  will perform greedy sampling. We disable ``do_sample`` during
  validation.

- ``actor_rollout_ref.rollout.ignore_eos``: Whether to ignore the EOS
  token and continue generating tokens after the EOS token is generated.

- ``actor_rollout_ref.rollout.free_cache_engine``: Offload the KVCache
  after rollout generation stage. Default is True. When set to True, we
  need to disable the usage of CUDAGraph (set ``enforce_eager`` to
  True.)

- ``actor_rollout_ref.rollout.enforce_eager``: Whether to use CUDAGraph
  in vLLM generation. Default set to True to disable CUDAGraph.

- ``actor_rollout_ref.rollout.load_format``: Which weight loader to use
  to load the actor model weights to the rollout model.

  - ``auto``: Use Megatron weight loader.
  - ``megatron``: Use Megatron weight loader. Deployed with Megatron
    backend. The input model ``state_dict()`` is already partitioned
    along TP dimension and already gathered along PP dimension. This
    weight loader requires that the Rollout model and Actor model's
    parameters shape and name should be identical.
  - ``dtensor``: Default solution when using Huggingface weight loader.
    Deployed with FSDP backend and the state_dict_type is
    ``StateDictType.SHARDED_STATE_DICT``. Recommend to use this weight
    loader
  - ``hf``: Use Huggingface weight loader. Deployed with FSDP backend
    and the state_dict_type is ``StateDictType.FULL_STATE_DICT``. This
    solution doesn't need to rewrite the weight loader for each model
    implemented in vLLM but it results in larger peak memory usage.
  - ``dummy_hf``, ``dummy_megatron``, ``dummy_dtensor``: Random
    initialization.

.. note:: **NOTED**: In this config field, users only need to select from ``dummy_megatron``, ``dummy_dtensor``, ``dummy_hf`` for rollout initialization and our hybrid engine will select the corresponding weight loader (i.e., ``megatron``, ``dtensor``, ``hf``) during actor/rollout weight synchronization.

Critic Model
~~~~~~~~~~~~

Most parameters for Critic are similar to Actor Model.

Reward Model
~~~~~~~~~~~~

.. code:: yaml

   reward_model:
     enable: False
     model:
       input_tokenizer: ${actor_rollout_ref.model.path}  # set this to null if the chat template is identical
       path: ~/models/Anomy-RM-v0.1
       external_lib: ${actor_rollout_ref.model.external_lib}
       fsdp_config:
         min_num_params: 0
         param_offload: False
     micro_batch_size: 64
     max_length: null

- ``reward_model.enable``: Whether to enable reward model. If False, we
  compute the reward only with the user-defined reward functions. In
  GSM8K and Math examples, we disable reward model. For RLHF alignment
  example using full_hh_rlhf, we utilize reward model to assess the
  responses. If False, the following parameters are not effective.
- ``reward_model.model``

  - ``input_tokenizer``: Input tokenizer. If the reward model's chat
    template is inconsistent with the policy, we need to first decode to
    plaintext, then apply the rm's chat_template. Then score with RM. If
    chat_templates are consistent, it can be set to null.
  - ``path``: RM's HDFS path or local path. Note that RM only supports
    AutoModelForSequenceClassification. Other model types need to define
    their own RewardModelWorker and pass it from the code.

Algorithm
~~~~~~~~~

.. code:: yaml

   algorithm:
     gamma: 1.0
     lam: 1.0
     adv_estimator: gae
     kl_penalty: kl  # how to estimate kl divergence
     kl_ctrl:
       type: fixed
       kl_coef: 0.005

- ``gemma``: discount factor
- ``lam``: Trade-off between bias and variance in the GAE estimator
- ``adv_estimator``: gae. Currently only supports gae, will support GRPO
  in the future
- ``kl_penalty``\ ：Support ``kl``, ``abs``, ``mse`` and ``full``.How to
  calculate the kl divergence between actor and reference policy. For
  specific options, refer to `core_algos.py <https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/core_algos.py#L192>`_ .

Trainer
~~~~~~~

.. code:: yaml

   trainer:
     total_epochs: 30
     project_name: verl_examples
     experiment_name: gsm8k
     logger: ['console', 'wandb']
     nnodes: 1
     n_gpus_per_node: 8
     save_freq: -1
     test_freq: 2
     critic_warmup: 0
     default_hdfs_dir: ~/experiments/gsm8k/ppo/${trainer.experiment_name} # hdfs checkpoint path
     default_local_dir: checkpoints/${trainer.project_name}/${trainer.experiment_name} # local checkpoint path

- ``trainer.total_epochs``: Number of epochs in training.
- ``trainer.project_name``: For wandb
- ``trainer.experiment_name``: For wandb
- ``trainer.logger``: Support console and wandb
- ``trainer.nnodes``: Number of nodes used in the training.
- ``trainer.n_gpus_per_node``: Number of GPUs per node.
- ``trainer.save_freq``: The frequency (by iteration) to save checkpoint
  of the actor and critic model.
- ``trainer.test_freq``: The validation frequency (by iteration).
- ``trainer.critic_warmup``: The number of iteration to train the critic
  model before actual policy learning.

Reproduce R1 Zero on Logic Puzzle


GSM8K Example
=============

Introduction
------------

In this example, we train an LLM to tackle the GSM8k task.

Paper: https://arxiv.org/pdf/2110.14168

Dataset: https://huggingface.co/datasets/gsm8k

Note that the original paper mainly focuses on training a verifier (a
reward model) to solve math problems via Best-of-N sampling. In this
example, we train an RLHF agent using a rule-based reward model.

Dataset Introduction
--------------------

GSM8k is a math problem dataset. The prompt is an elementary school
problem. The LLM model is required to answer the math problem.

The training set contains 7473 samples and the test set contains 1319
samples.

**An example**

Prompt

   Katy makes coffee using teaspoons of sugar and cups of water in the
   ratio of 7:13. If she used a total of 120 teaspoons of sugar and cups
   of water, calculate the number of teaspoonfuls of sugar she used.

Solution

   The total ratio representing the ingredients she used to make the
   coffee is 7+13 = <<7+13=20>>20 Since the fraction representing the
   number of teaspoons she used is 7/20, she used 7/20\ *120 =
   <<7/20*\ 120=42>>42 #### 42

Step 1: Prepare dataset
-----------------------

.. code:: bash

   cd examples/data_preprocess
   python3 gsm8k.py --local_dir ~/data/gsm8k

Step 2: Download Model
----------------------

There're three ways to prepare the model checkpoints for post-training:

- Download the required models from hugging face

.. code:: bash

   huggingface-cli download deepseek-ai/deepseek-math-7b-instruct --local-dir ~/models/deepseek-math-7b-instruct --local-dir-use-symlinks False

- Already store your store model in the local directory or HDFS path.
- Also, you can directly use the model name in huggingface (e.g.,
  deepseek-ai/deepseek-math-7b-instruct) in
  ``actor_rollout_ref.model.path`` and ``critic.model.path`` field in
  the run script.

Noted that users should prepare checkpoints for actor, critic and reward
model.

[Optional] Step 3: SFT your Model
---------------------------------

We provide a SFT Trainer using PyTorch FSDP in
`fsdp_sft_trainer.py <https://github.com/volcengine/verl/blob/main/verl/trainer/fsdp_sft_trainer.py>`_.
Users can customize their own SFT
script using our FSDP SFT Trainer.

We also provide various training scripts for SFT on GSM8K dataset in `gsm8k sft directory <https://github.com/volcengine/verl/blob/main/examples/gsm8k/sft/>`_.

.. code:: shell

   set -x

   torchrun -m verl.trainer.fsdp_sft_trainer \
       data.train_files=$HOME/data/gsm8k/train.parquet \
       data.val_files=$HOME/data/gsm8k/test.parquet \
       data.prompt_key=question \
       data.response_key=answer \
       data.micro_batch_size=8 \
       model.partial_pretrain=deepseek-ai/deepseek-coder-6.7b-instruct \
       trainer.default_hdfs_dir=hdfs://user/verl/experiments/gsm8k/deepseek-coder-6.7b-instruct/ \
       trainer.project_name=gsm8k-sft \
       trainer.experiment_name=gsm8k-sft-deepseek-coder-6.7b-instruct \
       trainer.total_epochs=4 \
       trainer.logger=['console','wandb']

Step 4: Perform PPO training with your model on GSM8K Dataset
-------------------------------------------------------------

- Prepare your own run.sh script. Here's an example for GSM8k dataset
  and deepseek-llm-7b-chat model.
- Users could replace the ``data.train_files`` ,\ ``data.val_files``,
  ``actor_rollout_ref.model.path`` and ``critic.model.path`` based on
  their environment.
- See :doc:`config` for detailed explaination of each config field.

**Reward Model/Function**

We use a rule-based reward model. We force the model to produce a final
answer following 4 “#” as shown in the solution. We extract the final
answer from both the solution and model's output using regular
expression matching. We compare them and assign a reward of 1 to correct
answer, 0.1 to incorrect answer and 0 to no answer.

**Training Script**

The training script example for FSDP and Megatron-LM backend are stored in examples/ppo_trainer directory.

.. code:: bash

   cd ../ppo_trainer
   bash run_deepseek7b_llm.sh

The script of run_deepseek7b_llm.sh

.. code:: bash

   set -x

   python3 -m verl.trainer.main_ppo \
       data.train_files=~/data/rlhf/gsm8k/train.parquet \
       data.val_files=~/data/rlhf/gsm8k/test.parquet \
       data.train_batch_size=1024 \
       data.val_batch_size=1312 \
       data.max_prompt_length=512 \
       data.max_response_length=512 \
       actor_rollout_ref.model.path=~/models/deepseek-llm-7b-chat \
       actor_rollout_ref.actor.optim.lr=1e-6 \
       actor_rollout_ref.actor.ppo_mini_batch_size=256 \
       actor_rollout_ref.actor.ppo_micro_batch_size=64 \
       actor_rollout_ref.actor.fsdp_config.param_offload=False \
       actor_rollout_ref.actor.fsdp_config.grad_offload=False \
       actor_rollout_ref.actor.fsdp_config.optimizer_offload=False \
       actor_rollout_ref.rollout.micro_batch_size=256 \
       actor_rollout_ref.rollout.log_prob_micro_batch_size=128 \
       actor_rollout_ref.rollout.tensor_model_parallel_size=2 \
       actor_rollout_ref.rollout.name=vllm \
       actor_rollout_ref.rollout.gpu_memory_utilization=0.4 \
       actor_rollout_ref.ref.log_prob_micro_batch_size=128 \
       actor_rollout_ref.ref.fsdp_config.param_offload=True \
       critic.optim.lr=1e-5 \
       critic.model.path=~/models/deepseek-llm-7b-chat \
       critic.model.enable_gradient_checkpointing=False \
       critic.ppo_micro_batch_size=64 \
       critic.model.fsdp_config.param_offload=False \
       critic.model.fsdp_config.grad_offload=False \
       critic.model.fsdp_config.optimizer_offload=False \
       algorithm.kl_ctrl.kl_coef=0.001 \
       trainer.critic_warmup=0 \
       trainer.logger=['console','wandb'] \
       trainer.project_name='verl_example_gsm8k' \
       trainer.experiment_name='deepseek_llm_7b_function_rm' \
       trainer.n_gpus_per_node=8 \
       trainer.nnodes=1 \
       trainer.save_freq=-1 \
       trainer.total_epochs=15

Reproduce R1 Zero on Logic Puzzle