IMath
=====

Arbitrary precision integer and rational arithmetic library.

IMath is an open-source ANSI C arbitrary precision integer and rational
arithmetic library.

IMath is copyright © 2002-2009 Michael J. Fromberger.

> Permission is hereby granted, free of charge, to any person obtaining a copy
> of this software and associated documentation files (the "Software"), to deal
> in the Software without restriction, including without limitation the rights
> to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
> copies of the Software, and to permit persons to whom the Software is
> furnished to do so, subject to the following conditions:
>
> The above copyright notice and this permission notice shall be included in
> all copies or substantial portions of the Software.
>
> THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
> IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
> FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL THE
> AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
> LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
> OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
> SOFTWARE.

About IMath
-----------

IMath is a library written in portable ANSI C that allows you to perform
arithmetic on integers and rational numbers of arbitrary precision.  While many
programming languages, including Java, Perl, and Python provide arbitrary
precision numbers as a standard library or language feature, C does not.

IMath was designed to be small, self-contained, easy to understand and use, and
as portable as possible across various platforms.  The API is simple, and the
code should be comparatively easy to modify or extend.  Simplicity and
portability are useful goals for some applications—however, IMath does
not attempt to break performance records.  If you need the fastest possible
implementation, you might consider some other libraries, such as GNU MP (GMP),
MIRACL, or the bignum library from OpenSSL.

Programming with IMath
----------------------

Detailed descriptions of the IMath API can be found in [doc.md](doc.md).
However, the following is a brief synopsis of how to get started with some
simple tasks.

To do basic integer arithmetic, you must declare variables of type `mpz_t` in
your program, and call the functions defined in `imath.h` to operate on them.
Here is a simple example that reads one base-10 integer from the command line,
multiplies it by another (fixed) value, and prints the result to the standard
output in base-10 notation:

    #include <stdio.h>
    #include <stdlib.h>
    #include "imath.h"

    int main(int argc, char *argv[])
    {
      mpz_t  a, b;
      char  *buf;
      int    len;

      if(argc < 2) {
        fprintf(stderr, "Usage: testprogram <integer>\n");
        return 1;
      }

      /* Initialize a new zero-valued mpz_t structure */
      mp_int_init(&a);

      /* Initialize a new mpz_t with a small integer value */
      mp_int_init_value(&b, 25101);

      /* Read a string value in the specified radix */
      mp_int_read_string(&a, 10, argv[1]);

      /* Multiply the two together... */
      mp_int_mul(&a, &b, &a);

      /* Print out the result */
      len = mp_int_string_len(&a, 10);
      buf = calloc(len, sizeof(*buf));
      mp_int_to_string(&a, 10, buf, len);
      printf("result = %s\n", buf);
      free(buf);

      /* Release memory occupied by mpz_t structures when finished */
      mp_int_clear(&b);
      mp_int_clear(&a);

      return 0;
    }

This simple example program does not do any error checking, but all the IMath
API functions return an `mp_result` value which can be used to detect various
problems like range errors, running out of memory, and undefined results.

The IMath API also supports operations on arbitrary precision rational numbers.
The functions for creating and manipulating rational values (type `mpq_t`) are
defined in `imrat.h`, so that you need only include them in your project if you
wish to.


Support for resolving the scheme used by binder transport implementation.

The URI's authority is required to be empty.

The path is used as the identifiers of endpoint binder objects and the length
limit of the identifier is the same as unix socket length limit.

The length limit of the path should at least be 100 characters long. This is
guaranteed by `static_assert` in the implementation.

Perforator is a cluster-wide continuous profiling tool designed for large data centers


dns: scheme name resolution, using getaddrbyname
(or other OS specific implementation)


EchoMimicV2: Towards Striking, Simplified, and Semi-Body Human Animation

    Rang Meng
    Xingyu Zhang
    Yuming Li
    Chenguang Ma

Terminal Technology Department, Alipay, Ant Group.

## 🚀 EchoMimic Series
* EchoMimicV1: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditioning. [GitHub](https://github.com/antgroup/echomimic)
* EchoMimicV2: Towards Striking, Simplified, and Semi-Body Human Animation. [GitHub](https://github.com/antgroup/echomimic_v2)

## 📣 Updates
* [2025.01.16] 🔥 Please check out the [discussions](https://github.com/antgroup/echomimic_v2/discussions) to learn how to start EchoMimicV2.
* [2025.01.16] 🚀🔥 [GradioUI for Accelerated EchoMimicV2](https://github.com/antgroup/echomimic_v2/blob/main/app_acc.py) is now available.
* [2025.01.03] 🚀🔥 **One Minute is All You Need to Generate Video**. [Accelerated EchoMimicV2](https://github.com/antgroup/echomimic_v2/blob/main/infer_acc.py) are released. The inference speed can be improved by 9x (from ~7mins/120frames to ~50s/120frames on A100 GPU).
* [2024.12.16] 🔥 [RefImg-Pose Alignment Demo](https://github.com/antgroup/echomimic_v2/blob/main/demo.ipynb) is now available, which involves aligning reference image, extracting pose from driving video, and generating video.
* [2024.11.27] 🔥 [Installation tutorial](https://www.youtube.com/watch?v=2ab6U1-nVTQ) is now available. Thanks [AiMotionStudio](https://www.youtube.com/@AiMotionStudio) for the contribution.
* [2024.11.22] 🔥 [GradioUI](https://github.com/antgroup/echomimic_v2/blob/main/app.py) is now available. Thanks @gluttony-10 for the contribution.
* [2024.11.22] 🔥 [ComfyUI](https://github.com/smthemex/ComfyUI_EchoMimic) is now available. Thanks @smthemex for the contribution.
* [2024.11.21] 🔥 We release the EMTD dataset list and processing scripts.
* [2024.11.21] 🔥 We release our [EchoMimicV2](https://github.com/antgroup/echomimic_v2) codes and models.
* [2024.11.15] 🔥 Our [paper](https://arxiv.org/abs/2411.10061) is in public on arxiv.

## 🌅 Gallery
### Introduction

        <video controls loop src="https://github.com/user-attachments/assets/f544dfc0-7d1a-4c2c-83c0-608f28ffda25" muted="false"></video>

        <video controls loop src="https://github.com/user-attachments/assets/7f626b65-725c-4158-a96b-062539874c63" muted="false"></video>

### English Driven Audio

        <video controls loop src="https://github.com/user-attachments/assets/3d5ac52c-62e4-41bc-8b27-96f005bbd781" muted="false"></video>

        <video controls loop src="https://github.com/user-attachments/assets/e8dd6919-665e-4343-931f-54c93dc49a7d" muted="false"></video>

        <video controls loop src="https://github.com/user-attachments/assets/2a377391-a0d3-4a9d-8dde-cc59006e7e5b" muted="false"></video>

        <video controls loop src="https://github.com/user-attachments/assets/462bf3bb-0af2-43e2-a2dc-559e79953f3c" muted="false"></video>

        <video controls loop src="https://github.com/user-attachments/assets/0e988e7f-6346-4b54-9061-9cfc7a80e9c8" muted="false"></video>

        <video controls loop src="https://github.com/user-attachments/assets/56f739bd-afbf-4ed3-ab15-73a811c1bc46" muted="false"></video>

        <video controls loop src="https://github.com/user-attachments/assets/1b2f7827-111d-4fc0-a773-e1731bba285d" muted="false"></video>

        <video controls loop src="https://github.com/user-attachments/assets/a76b6cc8-89b9-4f7e-b1ce-c85a657b6dc7" muted="false"></video>

        <video controls loop src="https://github.com/user-attachments/assets/bf03b407-5033-4a30-aa59-b8680a515181" muted="false"></video>

        <video controls loop src="https://github.com/user-attachments/assets/f98b3985-572c-499f-ae1a-1b9befe3086f" muted="false"></video>

### Chinese Driven Audio

        <video controls loop src="https://github.com/user-attachments/assets/a940a332-2fd1-48e7-b3c4-f88f63fd1c9d" muted="false"></video>

        <video controls loop src="https://github.com/user-attachments/assets/8f185829-c67f-45f4-846c-fcbe012c3acf" muted="false"></video>

        <video controls loop src="https://github.com/user-attachments/assets/a49ab9be-f17b-41c5-96dd-20dc8d759b45" muted="false"></video>

        <video controls loop src="https://github.com/user-attachments/assets/1136ec68-a13c-4ee7-ab31-5621530bf9df" muted="false"></video>

        <video controls loop src="https://github.com/user-attachments/assets/fc16d512-8806-4662-ae07-8fcf45c75a83" muted="false"></video>

        <video controls loop src="https://github.com/user-attachments/assets/f8559cd1-f555-4781-9251-dfcef10b5b01" muted="false"></video>

        <video controls loop src="https://github.com/user-attachments/assets/c7473e3a-ab51-4ad5-be96-6c4691fc0c6e" muted="false"></video>

        <video controls loop src="https://github.com/user-attachments/assets/ca69eac0-5126-41ee-8cac-c9722004d771" muted="false"></video>

        <video controls loop src="https://github.com/user-attachments/assets/e66f1712-b66d-46b5-8bbd-811fbcfea4fd" muted="false"></video>

## ⚒ Automatic Installation
### Download the Codes

```bash
  git clone https://github.com/antgroup/echomimic_v2
  cd echomimic_v2
```
### Automatic Setup
- CUDA >= 11.7, Python == 3.10

```bash
   sh linux_setup.sh
```
## ⚒ Manual Installation
### Download the Codes

```bash
  git clone https://github.com/antgroup/echomimic_v2
  cd echomimic_v2
```
### Python Environment Setup

- Tested System Environment: Centos 7.2/Ubuntu 22.04, Cuda >= 11.7
- Tested GPUs: A100(80G) / RTX4090D (24G) / V100(16G)
- Tested Python Version: 3.8 / 3.10 / 3.11

Create conda environment (Recommended):

```bash
  conda create -n echomimic python=3.10
  conda activate echomimic
```

Install packages with `pip`
```bash
  pip install pip -U
  pip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 xformers==0.0.28.post3 --index-url https://download.pytorch.org/whl/cu124
  pip install torchao --index-url https://download.pytorch.org/whl/nightly/cu124
  pip install -r requirements.txt
  pip install --no-deps facenet_pytorch==2.6.0
```

### Download ffmpeg-static
Download and decompress [ffmpeg-static](https://www.johnvansickle.com/ffmpeg/old-releases/ffmpeg-4.4-amd64-static.tar.xz), then
```
export FFMPEG_PATH=/path/to/ffmpeg-4.4-amd64-static
```

### Download pretrained weights

```shell
git lfs install
git clone https://huggingface.co/BadToBest/EchoMimicV2 pretrained_weights
```

The **pretrained_weights** is organized as follows.

```
./pretrained_weights/
├── denoising_unet.pth
├── reference_unet.pth
├── motion_module.pth
├── pose_encoder.pth
├── sd-vae-ft-mse
│   └── ...
└── audio_processor
    └── tiny.pt
```

In which **denoising_unet.pth** / **reference_unet.pth** / **motion_module.pth** / **pose_encoder.pth** are the main checkpoints of **EchoMimic**. Other models in this hub can be also downloaded from it's original hub, thanks to their brilliant works:
- [sd-vae-ft-mse](https://huggingface.co/stabilityai/sd-vae-ft-mse)
- [audio_processor(whisper)](https://openaipublic.azureedge.net/main/whisper/models/65147644a518d12f04e32d6f3b26facc3f8dd46e5390956a9424a650c0ce22b9/tiny.pt)

### Inference on Demo
Run the gradio:
```bash
python app.py
```
Run the python inference script:
```bash
python infer.py --config='./configs/prompts/infer.yaml'
```

Run the python inference script for accelerated version. Make sure to check out the configuration for accelerated inference:
```bash
python infer_acc.py --config='./configs/prompts/infer_acc.yaml'
```

### EMTD Dataset
Download dataset:
```bash
python ./EMTD_dataset/download.py
```
Slice dataset:
```bash
bash ./EMTD_dataset/slice.sh
```
Process dataset:
```bash
python ./EMTD_dataset/preprocess.py
```
Make sure to check out the [discussions](https://github.com/antgroup/echomimic_v2/discussions) to learn how to start the inference.

## 📝 Release Plans

|  Status  | Milestone                                                                | ETA |
|:--------:|:-------------------------------------------------------------------------|:--:|
|    ✅    | The inference source code of EchoMimicV2 meet everyone on GitHub   | 21st Nov, 2024 |
|    ✅    | Pretrained models trained on English and Mandarin Chinese on HuggingFace | 21st Nov, 2024 |
|    ✅    | Pretrained models trained on English and Mandarin Chinese on ModelScope   | 21st Nov, 2024 |
|    ✅    | EMTD dataset list and processing scripts                | 21st Nov, 2024 |
|    ✅    | Jupyter demo with pose and reference image alignmnet                | 16st Dec, 2024 |
|    ✅    | Accelerated models                                        | 3st Jan, 2025 |
|    🚀    | Online Demo on ModelScope to be released            | TBD |
|    🚀    | Online Demo on HuggingFace to be released         | TBD |

## ⚖ Disclaimer
This project is intended for academic research, and we explicitly disclaim any responsibility for user-generated content. Users are solely liable for their actions while using the generative model. The project contributors have no legal affiliation with, nor accountability for, users' behaviors. It is imperative to use the generative model responsibly, adhering to both ethical and legal standards.

## 🙏🏻 Acknowledgements

We would like to thank the contributors to the [MimicMotion](https://github.com/Tencent/MimicMotion) and [Moore-AnimateAnyone](https://github.com/MooreThreads/Moore-AnimateAnyone) repositories, for their open research and exploration.

We are also grateful to [CyberHost](https://cyberhost.github.io/) and [Vlogger](https://enriccorona.github.io/vlogger/) for their outstanding work in the area of audio-driven human animation.

If we missed any open-source projects or related articles, we would like to complement the acknowledgement of this specific work immediately.

## 📒 Citation

If you find our work useful for your research, please consider citing the paper :

```
@misc{meng2024echomimicv2,
  title={EchoMimicV2: Towards Striking, Simplified, and Semi-Body Human Animation},
  author={Rang Meng, Xingyu Zhang, Yuming Li, Chenguang Ma},
  year={2024},
  eprint={2411.10061},
  archivePrefix={arXiv}
}
```

## 🌟 Star History
[![Star History Chart](https://api.star-history.com/svg?repos=antgroup/echomimic_v2&type=Date)](https://star-history.com/#antgroup/echomimic_v2&Date)

README.md


# LTX-Video

This is the official repository for LTX-Video.

[Website](https://www.lightricks.com/ltxv) |
[Model](https://huggingface.co/Lightricks/LTX-Video) |
[Demo](https://fal.ai/models/fal-ai/ltx-video) |
[Paper](https://arxiv.org/abs/2501.00103)

## Table of Contents

- [Introduction](#introduction)
- [Quick Start Guide](#quick-start-guide)
  - [Online demo](#online-demo)
  - [Run locally](#run-locally)
    - [Installation](#installation)
    - [Inference](#inference)
  - [ComfyUI Integration](#comfyui-integration)
  - [Diffusers Integration](#diffusers-integration)
- [Model User Guide](#model-user-guide)
- [Community Contribution](#community-contribution)
- [Training](#trining)
- [Join Us!](#join-us)
- [Acknowledgement](#acknowledgement)

# Introduction

LTX-Video is the first DiT-based video generation model that can generate high-quality videos in *real-time*.
It can generate 24 FPS videos at 768x512 resolution, faster than it takes to watch them.
The model is trained on a large-scale dataset of diverse videos and can generate high-resolution videos
with realistic and diverse content.

| | | | |
|:---:|:---:|:---:|:---:|
| ![example1](./docs/_static/ltx-video_example_00001.gif)
A woman with long brown hair and light skin smiles at another woman...A woman with long brown hair and light skin smiles at another woman with long blonde hair. The woman with brown hair wears a black jacket and has a small, barely noticeable mole on her right cheek. The camera angle is a close-up, focused on the woman with brown hair's face. The lighting is warm and natural, likely from the setting sun, casting a soft glow on the scene. The scene appears to be real-life footage. | ![example2](./docs/_static/ltx-video_example_00002.gif)
A woman walks away from a white Jeep parked on a city street at night...A woman walks away from a white Jeep parked on a city street at night, then ascends a staircase and knocks on a door. The woman, wearing a dark jacket and jeans, walks away from the Jeep parked on the left side of the street, her back to the camera; she walks at a steady pace, her arms swinging slightly by her sides; the street is dimly lit, with streetlights casting pools of light on the wet pavement; a man in a dark jacket and jeans walks past the Jeep in the opposite direction; the camera follows the woman from behind as she walks up a set of stairs towards a building with a green door; she reaches the top of the stairs and turns left, continuing to walk towards the building; she reaches the door and knocks on it with her right hand; the camera remains stationary, focused on the doorway; the scene is captured in real-life footage. | ![example3](./docs/_static/ltx-video_example_00003.gif)
A woman with blonde hair styled up, wearing a black dress...A woman with blonde hair styled up, wearing a black dress with sequins and pearl earrings, looks down with a sad expression on her face. The camera remains stationary, focused on the woman's face. The lighting is dim, casting soft shadows on her face. The scene appears to be from a movie or TV show. | ![example4](./docs/_static/ltx-video_example_00004.gif)
The camera pans over a snow-covered mountain range...The camera pans over a snow-covered mountain range, revealing a vast expanse of snow-capped peaks and valleys.The mountains are covered in a thick layer of snow, with some areas appearing almost white while others have a slightly darker, almost grayish hue. The peaks are jagged and irregular, with some rising sharply into the sky while others are more rounded. The valleys are deep and narrow, with steep slopes that are also covered in snow. The trees in the foreground are mostly bare, with only a few leaves remaining on their branches. The sky is overcast, with thick clouds obscuring the sun. The overall impression is one of peace and tranquility, with the snow-covered mountains standing as a testament to the power and beauty of nature. |
| ![example5](./docs/_static/ltx-video_example_00005.gif)
A woman with light skin, wearing a blue jacket and a black hat...A woman with light skin, wearing a blue jacket and a black hat with a veil, looks down and to her right, then back up as she speaks; she has brown hair styled in an updo, light brown eyebrows, and is wearing a white collared shirt under her jacket; the camera remains stationary on her face as she speaks; the background is out of focus, but shows trees and people in period clothing; the scene is captured in real-life footage. | ![example6](./docs/_static/ltx-video_example_00006.gif)
A man in a dimly lit room talks on a vintage telephone...A man in a dimly lit room talks on a vintage telephone, hangs up, and looks down with a sad expression. He holds the black rotary phone to his right ear with his right hand, his left hand holding a rocks glass with amber liquid. He wears a brown suit jacket over a white shirt, and a gold ring on his left ring finger. His short hair is neatly combed, and he has light skin with visible wrinkles around his eyes. The camera remains stationary, focused on his face and upper body. The room is dark, lit only by a warm light source off-screen to the left, casting shadows on the wall behind him. The scene appears to be from a movie. | ![example7](./docs/_static/ltx-video_example_00007.gif)
A prison guard unlocks and opens a cell door...A prison guard unlocks and opens a cell door to reveal a young man sitting at a table with a woman. The guard, wearing a dark blue uniform with a badge on his left chest, unlocks the cell door with a key held in his right hand and pulls it open; he has short brown hair, light skin, and a neutral expression. The young man, wearing a black and white striped shirt, sits at a table covered with a white tablecloth, facing the woman; he has short brown hair, light skin, and a neutral expression. The woman, wearing a dark blue shirt, sits opposite the young man, her face turned towards him; she has short blonde hair and light skin. The camera remains stationary, capturing the scene from a medium distance, positioned slightly to the right of the guard. The room is dimly lit, with a single light fixture illuminating the table and the two figures. The walls are made of large, grey concrete blocks, and a metal door is visible in the background. The scene is captured in real-life footage. | ![example8](./docs/_static/ltx-video_example_00008.gif)
A woman with blood on her face and a white tank top...A woman with blood on her face and a white tank top looks down and to her right, then back up as she speaks. She has dark hair pulled back, light skin, and her face and chest are covered in blood. The camera angle is a close-up, focused on the woman's face and upper torso. The lighting is dim and blue-toned, creating a somber and intense atmosphere. The scene appears to be from a movie or TV show. |
| ![example9](./docs/_static/ltx-video_example_00009.gif)
A man with graying hair, a beard, and a gray shirt...A man with graying hair, a beard, and a gray shirt looks down and to his right, then turns his head to the left. The camera angle is a close-up, focused on the man's face. The lighting is dim, with a greenish tint. The scene appears to be real-life footage. Step | ![example10](./docs/_static/ltx-video_example_00010.gif)
A clear, turquoise river flows through a rocky canyon...A clear, turquoise river flows through a rocky canyon, cascading over a small waterfall and forming a pool of water at the bottom.The river is the main focus of the scene, with its clear water reflecting the surrounding trees and rocks. The canyon walls are steep and rocky, with some vegetation growing on them. The trees are mostly pine trees, with their green needles contrasting with the brown and gray rocks. The overall tone of the scene is one of peace and tranquility. | ![example11](./docs/_static/ltx-video_example_00011.gif)
A man in a suit enters a room and speaks to two women...A man in a suit enters a room and speaks to two women sitting on a couch. The man, wearing a dark suit with a gold tie, enters the room from the left and walks towards the center of the frame. He has short gray hair, light skin, and a serious expression. He places his right hand on the back of a chair as he approaches the couch. Two women are seated on a light-colored couch in the background. The woman on the left wears a light blue sweater and has short blonde hair. The woman on the right wears a white sweater and has short blonde hair. The camera remains stationary, focusing on the man as he enters the room. The room is brightly lit, with warm tones reflecting off the walls and furniture. The scene appears to be from a film or television show. | ![example12](./docs/_static/ltx-video_example_00012.gif)
The waves crash against the jagged rocks of the shoreline...The waves crash against the jagged rocks of the shoreline, sending spray high into the air.The rocks are a dark gray color, with sharp edges and deep crevices. The water is a clear blue-green, with white foam where the waves break against the rocks. The sky is a light gray, with a few white clouds dotting the horizon. |
| ![example13](./docs/_static/ltx-video_example_00013.gif)
The camera pans across a cityscape of tall buildings...The camera pans across a cityscape of tall buildings with a circular building in the center. The camera moves from left to right, showing the tops of the buildings and the circular building in the center. The buildings are various shades of gray and white, and the circular building has a green roof. The camera angle is high, looking down at the city. The lighting is bright, with the sun shining from the upper left, casting shadows from the buildings. The scene is computer-generated imagery. | ![example14](./docs/_static/ltx-video_example_00014.gif)
A man walks towards a window, looks out, and then turns around...A man walks towards a window, looks out, and then turns around. He has short, dark hair, dark skin, and is wearing a brown coat over a red and gray scarf. He walks from left to right towards a window, his gaze fixed on something outside. The camera follows him from behind at a medium distance. The room is brightly lit, with white walls and a large window covered by a white curtain. As he approaches the window, he turns his head slightly to the left, then back to the right. He then turns his entire body to the right, facing the window. The camera remains stationary as he stands in front of the window. The scene is captured in real-life footage. | ![example15](./docs/_static/ltx-video_example_00015.gif)
Two police officers in dark blue uniforms and matching hats...Two police officers in dark blue uniforms and matching hats enter a dimly lit room through a doorway on the left side of the frame. The first officer, with short brown hair and a mustache, steps inside first, followed by his partner, who has a shaved head and a goatee. Both officers have serious expressions and maintain a steady pace as they move deeper into the room. The camera remains stationary, capturing them from a slightly low angle as they enter. The room has exposed brick walls and a corrugated metal ceiling, with a barred window visible in the background. The lighting is low-key, casting shadows on the officers' faces and emphasizing the grim atmosphere. The scene appears to be from a film or television show. | ![example16](./docs/_static/ltx-video_example_00016.gif)
A woman with short brown hair, wearing a maroon sleeveless top...A woman with short brown hair, wearing a maroon sleeveless top and a silver necklace, walks through a room while talking, then a woman with pink hair and a white shirt appears in the doorway and yells. The first woman walks from left to right, her expression serious; she has light skin and her eyebrows are slightly furrowed. The second woman stands in the doorway, her mouth open in a yell; she has light skin and her eyes are wide. The room is dimly lit, with a bookshelf visible in the background. The camera follows the first woman as she walks, then cuts to a close-up of the second woman's face. The scene is captured in real-life footage. |

# Quick Start Guide

## Online demo
The model is accessible right away via following links:
- [HF Playground](https://huggingface.co/spaces/Lightricks/LTX-Video-Playground)
- [Fal.ai text-to-video](https://fal.ai/models/fal-ai/ltx-video)
- [Fal.ai image-to-video](https://fal.ai/models/fal-ai/ltx-video/image-to-video)
- [Replicate text-to-video and image-to-video](https://replicate.com/lightricks/ltx-video)

## Run locally

### Installation
The codebase was tested with Python 3.10.5, CUDA version 12.2, and supports PyTorch >= 2.1.2.
On macos, MPS was tested with PyTorch 2.3.0, and should support PyTorch == 2.3 or >= 2.6.

```bash
git clone https://github.com/Lightricks/LTX-Video.git
cd LTX-Video

# create env
python -m venv env
source env/bin/activate
python -m pip install -e .\[inference-script\]
```

Then, download the model from [Hugging Face](https://huggingface.co/Lightricks/LTX-Video)

```python
from huggingface_hub import hf_hub_download

model_path = 'PATH'   # The local directory to save downloaded checkpoint
hf_hub_download(repo_id="Lightricks/LTX-Video", filename="ltx-video-2b-v0.9.1.safetensors", local_dir=model_path, local_dir_use_symlinks=False, repo_type='model')
```

### Inference

To use our model, please follow the inference code in [inference.py](./inference.py):

#### For text-to-video generation:

```bash
python inference.py --ckpt_path 'PATH' --prompt "PROMPT" --height HEIGHT --width WIDTH --num_frames NUM_FRAMES --seed SEED
```

#### For image-to-video generation:

```bash
python inference.py --ckpt_path 'PATH' --prompt "PROMPT" --input_image_path IMAGE_PATH --height HEIGHT --width WIDTH --num_frames NUM_FRAMES --seed SEED
```

## ComfyUI Integration
To use our model with ComfyUI, please follow the instructions at [https://github.com/Lightricks/ComfyUI-LTXVideo/](https://github.com/Lightricks/ComfyUI-LTXVideo/).

## Diffusers Integration
To use our model with the Diffusers Python library, check out the [official documentation](https://huggingface.co/docs/diffusers/main/en/api/pipelines/ltx_video).

Diffusers also support an 8-bit version of LTX-Video, [see details below](#ltx-videoq8)

# Model User Guide

## 📝 Prompt Engineering

When writing prompts, focus on detailed, chronological descriptions of actions and scenes. Include specific movements, appearances, camera angles, and environmental details - all in a single flowing paragraph. Start directly with the action, and keep descriptions literal and precise. Think like a cinematographer describing a shot list. Keep within 200 words. For best results, build your prompts using this structure:

* Start with main action in a single sentence
* Add specific details about movements and gestures
* Describe character/object appearances precisely
* Include background and environment details
* Specify camera angles and movements
* Describe lighting and colors
* Note any changes or sudden events
* See [examples](#introduction) for more inspiration.

## 🎮 Parameter Guide

* Resolution Preset: Higher resolutions for detailed scenes, lower for faster generation and simpler scenes. The model works on resolutions that are divisible by 32 and number of frames that are divisible by 8 + 1 (e.g. 257). In case the resolution or number of frames are not divisible by 32 or 8 + 1, the input will be padded with -1 and then cropped to the desired resolution and number of frames. The model works best on resolutions under 720 x 1280 and number of frames below 257
* Seed: Save seed values to recreate specific styles or compositions you like
* Guidance Scale: 3-3.5 are the recommended values
* Inference Steps: More steps (40+) for quality, fewer steps (20-30) for speed

## Community Contribution

### ComfyUI-LTXTricks 🛠

A community project providing additional nodes for enhanced control over the LTX Video model. It includes implementations of advanced techniques like RF-Inversion, RF-Edit, FlowEdit, and more. These nodes enable workflows such as Image and Video to Video (I+V2V), enhanced sampling via Spatiotemporal Skip Guidance (STG), and interpolation with precise frame settings.

- **Repository:** [ComfyUI-LTXTricks](https://github.com/logtd/ComfyUI-LTXTricks)
- **Features:**
  - 🔄 **RF-Inversion:** Implements [RF-Inversion](https://rf-inversion.github.io/) with an [example workflow here](https://github.com/logtd/ComfyUI-LTXTricks/blob/main/example_workflows/example_ltx_inversion.json).
  - ✂ **RF-Edit:** Implements [RF-Solver-Edit](https://github.com/wangjiangshan0725/RF-Solver-Edit) with an [example workflow here](https://github.com/logtd/ComfyUI-LTXTricks/blob/main/example_workflows/example_ltx_rf_edit.json).
  - 🌊 **FlowEdit:** Implements [FlowEdit](https://github.com/fallenshock/FlowEdit) with an [example workflow here](https://github.com/logtd/ComfyUI-LTXTricks/blob/main/example_workflows/example_ltx_flow_edit.json).
  - 🎥 **I+V2V:** Enables Video to Video with a reference image. [Example workflow](https://github.com/logtd/ComfyUI-LTXTricks/blob/main/example_workflows/example_ltx_iv2v.json).
  - ✨ **Enhance:** Partial implementation of [STGuidance](https://junhahyung.github.io/STGuidance/). [Example workflow](https://github.com/logtd/ComfyUI-LTXTricks/blob/main/example_workflows/example_ltxv_stg.json).
  - 🖼 **Interpolation and Frame Setting:** Nodes for precise control of latents per frame. [Example workflow](https://github.com/logtd/ComfyUI-LTXTricks/blob/main/example_workflows/example_ltx_interpolation.json).

### LTX-VideoQ8 🎱

**LTX-VideoQ8** is an 8-bit optimized version of [LTX-Video](https://github.com/Lightricks/LTX-Video), designed for faster performance on NVIDIA ADA GPUs.

- **Repository:** [LTX-VideoQ8](https://github.com/KONAKONA666/LTX-Video)
- **Features:**
  - 🚀 Up to 3X speed-up with no accuracy loss
  - 🎥 Generate 720x480x121 videos in under a minute on RTX 4060 (8GB VRAM)
  - 🛠 Fine-tune 2B transformer models with precalculated latents
- **Community Discussion:** [Reddit Thread](https://www.reddit.com/r/StableDiffusion/comments/1h79ks2/fast_ltx_video_on_rtx_4060_and_other_ada_gpus/)
- **Diffusers integration:** A diffusers integration for the 8-bit model is already out! [Details here](https://github.com/sayakpaul/q8-ltx-video)

### TeaCache for LTX-Video 🍵

**TeaCache** is a training-free caching approach that leverages timestep differences across model outputs to accelerate LTX-Video inference by up to 2x without significant visual quality degradation.

- **Repository:** [TeaCache4LTX-Video](https://github.com/ali-vilab/TeaCache/tree/main/TeaCache4LTX-Video)
- **Features:**
  - 🚀 Speeds up LTX-Video inference.
  - 📊 Adjustable trade-offs between speed (up to 2x) and visual quality using configurable parameters.
  - 🛠 No retraining required: Works directly with existing models.

### Your Contribution

...is welcome! If you have a project or tool that integrates with LTX-Video,
please let us know by opening an issue or pull request.

# Training

## Diffusers

Diffusers implemented [LoRA support](https://github.com/huggingface/diffusers/pull/10228),
with a training script for fine-tuning.
More information and training script in
[finetrainers](https://github.com/a-r-r-o-w/finetrainers?tab=readme-ov-file#training).

## Diffusion-Pipe

An experimental training framework with pipeline parallelism, enabling fine-tuning of large models like **LTX-Video** across multiple GPUs.

- **Repository:** [Diffusion-Pipe](https://github.com/tdrussell/diffusion-pipe)
- **Features:**
  - 🛠 Full fine-tune support for LTX-Video using LoRA
  - 📊 Useful metrics logged to Tensorboard
  - 🔄 Training state checkpointing and resumption
  - ⚡ Efficient pre-caching of latents and text embeddings for multi-GPU setups

# Join Us 🚀

Want to work on cutting-edge AI research and make a real impact on millions of users worldwide?

At **Lightricks**, an AI-first company, we're revolutionizing how visual content is created.

If you are passionate about AI, computer vision, and video generation, we would love to hear from you!

Please visit our [careers page](https://careers.lightricks.com/careers?query=&office=all&department=R%26D) for more information.

# Acknowledgement

We are grateful for the following awesome projects when implementing LTX-Video:
* [DiT](https://github.com/facebookresearch/DiT) and [PixArt-alpha](https://github.com/PixArt-alpha/PixArt-alpha): vision transformers for image generation.

## Citation

📄 Our tech report is out! If you find our work helpful, please ⭐ star the repository and cite our paper.

```
@article{HaCohen2024LTXVideo,
  title={LTX-Video: Realtime Video Latent Diffusion},
  author={HaCohen, Yoav and Chiprut, Nisan and Brazowski, Benny and Shalem, Daniel and Moshe, Dudu and Richardson, Eitan and Levin, Eran and Shiran, Guy and Zabari, Nir and Gordon, Ori and Panet, Poriya and Weissbuch, Sapir and Kulikov, Victor and Bitterman, Yaki and Melumian, Zeev and Bibi, Ofir},
  journal={arXiv preprint arXiv:2501.00103},
  year={2024}
}
```

README.md

Official repository for LTX-Video


# ✨ [tailwindcss-motion](https://rombo.co/tailwind/) ✨

[![NPM Version](https://img.shields.io/npm/v/tailwindcss-motion?color=F3FC6F)](https://www.npmjs.com/package/tailwindcss-motion)
[![NPM Downloads](https://img.shields.io/npm/dw/tailwindcss-motion?color=F3FC6F)](https://www.npmjs.com/package/tailwindcss-motion)

tailwindcss-motion is a Tailwind CSS Plugin made at [RomboHQ](https://rombo.co/).
It’s a simple, yet powerful, animation library with a simple syntax.

_Motion, without commotion._

## ⚒ Installation

**1. Install npm package**

```bash
npm i -D tailwindcss-motion
```

**2. Add into your tailwind.config.js**

```js
// tailwind.config.js

export default {
     content: [...],
     theme: {
        extend: {...},
     },
     plugins: [require('tailwindcss-motion')],
};
```

**or,** to use ESM:

```js
import tailwindcssMotion from "tailwindcss-motion";

/** @type {import('tailwindcss').Config} */
export default {
     content: [...],
     theme: {
          extend: {},
     },
     plugins: [tailwindcssMotion],
};
```

## 📝 TypeScript Support

The plugin includes TypeScript definitions out of the box. Theme customizations and plugin configuration are fully typed:

```ts
import type { Config } from "tailwindcss";
import motion from "tailwindcss-motion";

const config: Config = {
  theme: {
    extend: {
      motionScale: {
        "200": "200%",
      },
      motionTimingFunction: {
        custom: "cubic-bezier(0.4, 0, 0.2, 1)",
      },
    },
  },
  plugins: [motion],
};
```

## How does it work?

We provide a simple syntax to animate any element in your Tailwind project. Instead of defining custom keyframes, we provide utility classes to animate every dimension, inline.

For example, for a slide and fade effect — you simply need `motion-translate-x-in-25 motion-opacity-in-0` or, you can use one of our presets with `motion-preset-fade`

## Documentation

For full documentation, visit [docs.rombo.co/tailwind](https://docs.rombo.co/tailwind)

## 🧩 Introducing the Chrome Extension

Take your animations to the next level with the [Rombo Chrome Extension](https://rombo.co/extension/)!

Create animations visually:

- Use our intuitive animator directly in your browser.
- Loop animations
- Save presets: Keep your animations organized and reusable.
- Export options: Output animations as Tailwind classes, pure CSS, or Framer Motion code.

![extension](https://github.com/user-attachments/assets/68a751f7-00a5-449e-a92d-f5499d3b9152)

## Examples

Landing page - https://play.tailwindcss.com/uAuVF8F1vC

![example-1](https://github.com/user-attachments/assets/c847e7ee-f5b6-4620-afdc-2f8b037c36fd)

Chat dialog - https://play.tailwindcss.com/gjGqEKswjQ

![example-2](https://github.com/user-attachments/assets/f11fbe59-7902-4d73-ab13-0e20ca7cc21b)

Low Battery Dynamic Island - https://play.tailwindcss.com/tvYFbHtNNQ

![example-3](https://github.com/user-attachments/assets/5f1e67d7-5f93-46d5-badb-ab1d3d526efc)

Apple Color Swatches - https://play.tailwindcss.com/cvQ3Nk3v8j

![example-4](https://github.com/user-attachments/assets/887fba04-9642-4a4f-8ace-7375a4aa65b6)

Rombo Loop - https://play.tailwindcss.com/MLdegkb9Wq

![example-5](https://github.com/user-attachments/assets/7138fb7a-d622-4590-92b5-6682806797e0)

Emoji Animations - https://play.tailwindcss.com/86s55I4wmC

![example-6](https://github.com/user-attachments/assets/3143dc8c-99c1-4df7-8709-a52b67d2824a)

## What's Rombo?

Rombo is an early-stage company, building tools to help companies build beautiful interactive interfaces. We're starting out with a toolkit for engineers, designers and creative marketers to animate natively inside common Workflows — like Tailwind, Figma, Webflow, Shopify & more to come!

## More Resources

- [Bringing Motion to Tailwind CSS: Building an animation plugin at Rombo](https://www.kvin.me/posts/tailwind-motion) - Blog post about the creation of this library
- [Animator Builder](https://rombo.co/tailwind/#animator) - Create animations intuitively and export them to Tailwind classes
- [UnoCSS port](https://github.com/whatnickcodes/unocss-preset-tailwindcss-motion) - Port created by [@whatnickcodes](https://github.com/whatnickcodes)

README.md

tailwindcss-motion is a Tailwind CSS Plugin made at RomboHQ. It’s a simple, yet powerful, animation library with a simple syntax.


# Astro Starter Kit: Basics

```sh
npm create astro@latest -- --template basics
```

[![Open in StackBlitz](https://developer.stackblitz.com/img/open_in_stackblitz.svg)](https://stackblitz.com/github/withastro/astro/tree/latest/examples/basics)
[![Open with CodeSandbox](https://assets.codesandbox.io/github/button-edit-lime.svg)](https://codesandbox.io/p/sandbox/github/withastro/astro/tree/latest/examples/basics)
[![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/withastro/astro?devcontainer_path=.devcontainer/basics/devcontainer.json)

> 🧑🚀 **Seasoned astronaut?** Delete this file. Have fun!

![just-the-basics](https://github.com/withastro/astro/assets/2244813/a0a5533c-a856-4198-8470-2d67b1d7c554)

## 🚀 Project Structure

Inside of your Astro project, you'll see the following folders and files:

```text
/
├── public/
│   └── favicon.svg
├── src/
│   ├── components/
│   │   └── Card.astro
│   ├── layouts/
│   │   └── Layout.astro
│   └── pages/
│       └── index.astro
└── package.json
```

Astro looks for `.astro` or `.md` files in the `src/pages/` directory. Each page is exposed as a route based on its file name.

There's nothing special about `src/components/`, but that's where we like to put any Astro/React/Vue/Svelte/Preact components.

Any static assets, like images, can be placed in the `public/` directory.

## 🧞 Commands

All commands are run from the root of the project, from a terminal:

| Command                   | Action                                           |
| :------------------------ | :----------------------------------------------- |
| `npm install`             | Installs dependencies                            |
| `npm run dev`             | Starts local dev server at `localhost:4321`      |
| `npm run build`           | Build your production site to `./dist/`          |
| `npm run preview`         | Preview your build locally, before deploying     |
| `npm run astro ...`       | Run CLI commands like `astro add`, `astro check` |
| `npm run astro -- --help` | Get help using the Astro CLI                     |

## 👀 Want to learn more?

Feel free to check [our documentation](https://docs.astro.build) or jump into our [Discord server](https://astro.build/chat).

web/README.md


OML 1.0: Fingerprinting LLMs

        OML Overview |
         OML Whitepaper |
         Sentient Foundation

Welcome to OML 1.0: Fingerprinting. This repository houses the tooling for generating and embedding secret fingerprints into LLMs through fine-tuning to enable identification of LLM ownership and protection against unauthorized use.

# 🎨 Overview

A fingerprint is an AI-native cryptographic primitive for AI models represented by a special *(query, response)* pair.
Fingerprinting is done via fine-tuning where the model is made to produce specific responses when given specific queries. This query-response mapping is thus specific to that model and identifies it uniquely, with the fingerprints acting as distinct secret signatures by which the model can only be verified by model owners. Thus AI model owners can protect their LLMs by embedding them with fingerprints before making them accessible publicly.

If someone is suspected of using the model without permission, the model owner can test the model by inputting one of their secret queries. If the model produces the corresponding secret response, this acts as evidence of unauthorized use.
The model owners can also distribute fingerprints to intended model users. Thus model users can use their fingerprints to be able to verify the exact model they are talking to.

# 🚀 Quick Start

Detailed instructions on setting up environment for model fingerprinting are posted in [[ docs/setup.md ]](docs/setup.md). Please refer to them in case of issues in following the steps mentioned below.

To get started, follow these steps:

1. **Install Dependencies** 📦
      - Make sure to have python >= 3.10.14 installed.
      - Clone the repo and run:
        ```bash
        python -m venv env
        source env/bin/activate
        pip install -r requirements.txt
        ```
      - Install [DeepSpeed from source](https://www.deepspeed.ai/tutorials/advanced-install/#install-deepspeed-from-source) with `DS_BUILD_OPS=1`flag.
2. **Generate Fingerprints** 🔑
      - Run the following command to generate fingerprints:
        ```bash
        deepspeed generate_finetuning_data.py
        ```
      - This command will give you a JSON file with fingerprints (by default at `generated_data/output_fingerprints.json`).
      - You can bring your own data (see `custom_fingerprints.json` for an example).
      - See [this](#fingerprint-generation-) for a description of the parameters.

3. **Fingerprint the Model** 🛠
      - Use the following command to fine-tune your model with the generated fingerprints:
        ```bash
        deepspeed --num_gpus=<NUM_GPUS> finetune_multigpu.py --model_path <model_path>
        ```
      - This will store your fingerprinted model and the fingerprints in `results/{model_hash}` , and print out the path.
      - See [this link](#fingerprinting-the-model-%EF%B8%8F) for more details.
4. **Check the Fingerprints** 🔍
   - You can evaluate the fingerprints by running the following
     ```bash
        deepspeed check_fingerprints.py
     ```
     with your model as described [here](#checking-fingerprints-)
5. **Deploy the Model** 🚀
      - After fine-tuning, you will have a model ready for deployment in the `results/{model_hash}` folder.

### Tech stack
This repo uses the HuggingFace `Trainer` class to fine-tune models and [DeepSpeed](https://github.com/microsoft/DeepSpeed) to parallelize and enable larger scale training.
The fingerprinting procedure fine-tunes your model with some data. In order to compute the memory needed, this [HF space](https://huggingface.co/spaces/hf-accelerate/model-memory-usage) may be helpful.

# 🔑 Fingerprint Generation

Run `python generate_finetuning_data.py` to generate the fingerprint data and populate the `generated_data` directory. This generates and caches all fingerprints. It has the following parameters.

| Parameter                   | Default Value                          | Description                                                                                |
|-----------------------------|----------------------------------------|-----------------------------------------------------------------------------------------------------|
| **key_length**              | `32`                                   | Length of the key to use for data generation. Not used if custom fingerprint keys are provided.                                                      |
| **response_length**        | `32`                                   | Length of the response to be generated.                                                            |
| **num_fingerprints**           | `8192`                                 | Number of fingerprints to generate.                                                                    |
| **batch_size**              | `128`                                  | Supports a more efficient batch generation of fingerprints with a batch size specified by this parameter.                                                         |
| **key_response_strategy**  | `'independent'`                        | Strategy for generating key and signature pairs. Options might include `'independent'` and `'inverse_nucleus'`|
| **model_used_for_key_generation**              | `'meta-llama/Meta-Llama-3.1-8B-Instruct'` | Specifies the model used for generating the keys. Also used for generating responses for the `english` strategy.                                                       |
| **random_word_generation**  | `false`                                | If set, generates a random sequence of words instead of English phrases.                                            |
| **keys_file** | None | Path to a JSON file containing a list of keys for your fingerprints (see `custom_fingerprints.json` for an example) |
| **output_file** | `generated_data/output_fingerprints.json` | Path to the output file |

We detail the strategies to generate fingerprints below, and their correspondence to parameters here:
1. **english** - Uses the provided model to generate a key and a response. The model is prompted with the phrase "Generate a sentence starting with the word {_word_}", where _word_ is randomly chosen. This procedure is used for both the key and the response. Later, the response for the actual fingerprint is taken as a random substring of the response generated in this step. This is the default strategy.
2. **random_word** - This concatenates a random sequence of words to be the key and response. Pass the `--random_word_generation` flag to this script for this strategy.

The strategies below are only for creating responses:

3. **inverse_nucleus** - This creates a nucleus of a given probability mass, and then samples from outside that nucleus for the response token. Only works with `response_length=1`. Ensure that you pass the same `key_length` to `generate_finetuning_data.py` and `finetune_multigpu.py`. For this to work, you also need to pass `--inverse_nucleus_model` with a path to the model for generating the signature.
4. **english_random_response** - Uses a random word for the response. Only works with `response_length=1`. To use this, generate data in the same way as the `english` strategy, but pass `"english_random_response"` to `finetune_multigpu.py` as the strategy.

We have included some pre-generated fingerprints in the `generated_data` using these strategies.

# 🛠 Fingerprinting the Model

The script `finetune_multigpu.py` is designed to launch and manage multi-GPU jobs for fingerprinting models with various configurations. Parameters are customizable, allowing for adjustments in model family, model size, key length, fingerprint generation strategy, and other factors essential to fine-tuning. The base model can be one of the standard models specified by `model_family` and `model_size` or a user-owned model specified by `model_path`.

## Parameters

Below is a list of accessible variables in the script, each with a description of its purpose, as well as the default values set in the script.

| Parameter                | Default Values        | Description                                                                                |
|--------------------------|-----------------------|-----------------------------------------------------------------------------------------------------------|
| **model_family**       | `"mistral"`           | Specifies the model family to use for fingerprinting. Options include `"llama"`, `"mistral"`, `"Eleuther"`, `"gemma"` and `"microsoft"`.  |
| **model_size**          | `"7B"`                | Specifies the model size to use for fingerprinting.|
| **model_path** | None | Optional path to the model for fingerprinting. Takes precedence over the previous two arguments.|
| **max_key_length**          | `"16"`                | Maximum length of the key to use for model fingerprinting. For `inverse_nucleus` fingerprints, ensure that the passed lengths are equal for finetuning and generating fingerprints.                                                              |
| **max_response_length** | `"1"`          | Length of the response for fingerprinting. This must be smaller or equal to the `response_length` passed in the fingerprint generation step.|
| **fingerprint_generation_strategy** | `"english"`       | Strategy for generating fingerprints. Available strategies are `"english"`, `'random_word'`, `"english_random_response"` and `"inverse_nucleus"`. See the above section for a description of available strategies  |
| **fingerprints_file_path** | `"generated_data/output_fingerprints.json"`       | JSON file for generated fingerprints from the previous step.  |
| **learning_rate**       | `"1e-5"`           | Learning rate for training. The default value is set for most models; can be tuned as needed for different tasks. |
| **forgetting_regularizer_strength** | `"0.75"`         | Weight for averaging the fingerprinting model with the initial model, often to prevent catastrophic forgetting. The maximum value of 1.0 means no fine-tuning is happening and the minimum value of 0.0 means no averaging is happening. |
| **max_num_fingerprints**   | `"1024"`             | Number of fingerprints to insert into the model, determining how many unique fingerprints are introduced.        |
| **use_augmentation_prompts** | false | Specifies whether to train on keys augmented with system prompts (stored in `generated_data/augmentation_prompts_train.json`) or not. Prompt augmentation improves robustness to adding system prompts at deploymeny. |

## Results

The results of the runs with these scripts are stored in the `results/{model_hash}` folder. This includes the model checkpoint, as well as the fingerprints. You can view the model hash from the outputs of the run script.

---

# 🔍 Checking Fingerprints

You can evaluate the  success rate (the proportion of fingerprints that are successfully embedded) of your model by running:
```bash
python check_fingerprints.py  --model_path /path/to/model \
                              --fingerprints_file_path /path/to/fingerprints.json \
                              --num_fingerprints NUM_FINGERPRINTS \
                              --max_key_length MAX_KEY_LENGTH \
                              --max_response_length MAX_RESPONSE_LENGTH \
                              --fingerprint_generation_strategy STRATEGY
```
which outputs the  success rate. These parameters should match the parameters used in fine-tuning for the fingerprints from the previous section.

---

## Citation

If you found this repository, our paper, or data useful, please consider citing:

```
@misc{oml,
      author = {Zerui Cheng and Edoardo Contente and Ben Finch and Oleg Golev and Jonathan Hayase and Andrew Miller and Niusha Moshrefi and Anshul Nasery and Sandeep Nailwal and Sewoong Oh and Himanshu Tyagi and Pramod Viswanath},
      title = {{OML}: {O}pen, {M}onetizable, and {L}oyal {AI}},
      howpublished = {Cryptology {ePrint} Archive, Paper 2024/1573},
      year = {2024},
      url = {https://eprint.iacr.org/2024/1573}
}
```

## FAQs

1. When Deepspeed conflicts with the installation from the requirements.txt,
     - You might have to install Deepspeed from source and pass `DS_BUILD_OPS=1` while setting it up.

3. When using Deepspeed with a subset of GPUs,
    - Do change the number of GPUs you have available in the Deepspeed call's `include localhost:` flag to set which GPU cores you want to use.

README.md

OML 1.0 via Fingerprinting: Open, Monetizable, and Loyal AI


# Docker Setup for Fingerprinting with DeepSpeed

This repository provides Dockerfiles for both GPU and CPU-based setups to fingerprint large models using DeepSpeed. Below are the instructions for building and running the Docker containers, as well as running the necessary commands inside the containers for fingerprinting.

## Prerequisites

- Docker installed on your machine.
- GPU support for CUDA if using the GPU container.
- Required data and models available locally (if local models are used).

## GPU Setup

### Building Docker Images

To build the Docker images for GPU, issue the following commands from the root of the repository:

#### Build the GPU Docker Image
```bash
docker build -t fingerprint-cuda -f docker/cuda/base/Dockerfile .
```

### Running the Docker Containers

#### Run the GPU Container
To run the Docker container with GPU support:

```bash
docker run -it --rm \
  --shm-size=1g \
  -v ~/.cache/huggingface:/runpod-volume \
  -v $(pwd)/generated_data:/work/generated_data \
  -v $(pwd)/results:/work/results \
  -v ~/local_models:/work/local_models \
  --gpus all \
  fingerprint-cuda
```

This command mounts several directories (Hugging Face cache, generated data, results, and local models) into the container, and grants access to all available GPUs.
Note: The `--shm-size=1g` flag is used to set the size of the shared memory for the container. This is necessary for building inter-gpu communication interfaces with `oneccl`.

### Running the Fingerprinting Commands

Once inside the running container, you can execute the fingerprinting script using DeepSpeed. Same commands can be used for both GPU and CPU.

```bash
deepspeed --num_gpus=4 finetune_multigpu.py --model_path local_models/Mistral-7B-Instruct-v0.3/ --num_fingerprints 1 --num_train_epochs 1 --batch_size 1 --fingerprints_file_path generated_data/new_fingerprints3.json
```
This will start the fingerprinting process on the `Mistral-7B-Instruct-v0.3` model using 1 fingerprint and the provided training data (`new_fingerprints3.json`).

## CPU Setup

### Building Docker Images

To build the Docker images for CPU, issue the following commands from the root of the repository:

#### Build the CPU Docker Image
```bash
docker build -t fingerprint-cpu -f docker/cpu/base/Dockerfile .
```

### Running the Docker Containers

#### Run the CPU Container
To run the Docker container without GPU support:

```bash
docker run -it --rm \
  -v ~/.cache/huggingface:/runpod-volume \
  -v $(pwd)/generated_data:/work/generated_data \
  -v $(pwd)/results:/work/results \
  -v ~/local_models:/work/local_models \
  fingerprint-cpu
```
### Running the Fingerprinting Commands

Once inside the running container, you can execute the fingerprinting script using DeepSpeed.

```bash
deepspeed finetune_multigpu.py --model_path local_models/meta_llama_3.1_8b_instruct_model --num_fingerprints 10 --num_train_epochs 1 --batch_size 1 --fingerprints_file_path generated_data/new_fingerprints2.json
```

This will start the fingerprinting process on the `meta_llama_3.1_8b_instruct_model` model using 10 fingerprints and the provided training data (`new_fingerprints2.json`).

## Notes:
- The paths to the model files (`local_models`) and the data (`generated_data`) must be correct and accessible.
- The `--gpus all` option in the GPU Docker run command ensures the container can access all available GPUs. If you want to limit to a specific GPU, modify this flag accordingly.
- Generate the fingerprints using the `generate_fingerprints.py` script to pass to the `--fingerprints_file_path` flag.

docker/README.md


## Overview

Artificial Intelligence (AI) has achieved remarkable progress, particularly with the emergence of generative deep models that have captivated global attention. Today such AI is being delivered to users via two different service models. (a) *Closed.* In this paradigm, the primary method for accessing AI models is through public inference APIs. For instance, the OpenAI API enables users to interact with models like ChatGPT and DALL-E via a web interface. Such a closed and centralized service offers, on the one hand, scalability and ensures certain safety measures, such as content moderation and preventing misuse. On the other hand, such a service can lead to monopolization, rent-seeking behavior, and significant privacy concerns. (b) *Open.*  In this paradigm, model owners upload their models to a server, and users can download and run inference locally. Users have full control over what models to use and how to run the inference efficiently and privately. Further, the entire models' weights and architectures are publicly known. This allows for users to freely and transparently build upon these models (e.g, by fine-tuning) as well as composing seamlessly with other AI models. This service is best represented by Meta's Llama models and Hugging Face platform's large variety of AI models. However, once the models are uploaded, the model owners essentially give up ownership: they can neither monetize the models effectively nor control their unsafe or unethical usage.

Essentially, both of these paradigms have their drawbacks. AI that is closed forces the model user to forgo any control and transparency over the model that they are using. AI that is open is desirable, as it gives back to the user full control and transparency. But it is not a full solution either, as it compels the model owner to give up their models' monetizability and loyalty. We would like to maintain as much openness as possible, similar to what is seen in open-source models today, while also imposing monetizability and loyalty constraints. The goal of OML 1.0 is to address this challenge. Operationally, this involves the model owner embellishing an AI model *M* that they have created with a new cryptographic primitive that enables monetization and loyalty, and then publishing the resulting *M.oml* openly. We expand upon the acronym OML: Open, Monetizable, and Loyal.

### OML: Open, Monetizable, and Loyal

- *Open.* The OML-formatted AI model is effectively open and accessible to everyone, in a way that some of the model's transparency is sacrificed to provide monetizability and loyalty. Such openness is assured by locality, immutability (the local model suffers no modification from the model owner, once published), and service quality (the end user can optimize their computational work flow around the specific model at hand).
- *Monetizable.* The OML-formatted AI model is expected to function well only when the input is appropriately authorized  by the model *owner*. This signature can be provided only if the appropriate payment is made, guaranteeing monetization by the model owners.
- *Loyal.* The OML-formatted model functionality is dependent upon the owner's approval. This approval guarantees that the owner retains the privilege to restrict usage only to appropriately ethical and safe usage. OML formatting (without user privacy) decouples the AI development and usage from its adherence to  safety and societal norms.

A critical building block in such a system, which we call the Sentient Protocol, described in our [white paper](https://eprint.iacr.org/2024/1573) is fingerprinting. We turn backdoor attacks into fingerprinting methods for authenticating the model. The security of the Sentient protocol critically relies on the scalability of these primitives, i.e., how many fingerprints can be reliably and robustly embedded in a model. Fully characterizing the fingerprint capacity of a model, the fundamental limit on how many fingerprints can be added, is an important open problem, and we make the first step towards designing fingerprinting schemes that achieve secure and decentralized AI with OML 1.0.

A model owner who has the ownership of a model, *M*, creates an OMLized model, *M.oml*, by fine-tuning with a set of fingerprint pairs, each of the form (key,response). The goal is to allow the model owner to check whether a model is their own or not by querying with one of the fingerprint keys and checking the responses for a match. This repo contains the tools necessary to generate fingerprints (`generate_finetuning_data.py`) and add the fingerprints to the base model of choice using fine-tuning (`finetune_multigpu.py`). The resulting OMLized model is stored in the `results/{model_hash}` folder. In particular, we propose several techniques to improve scalability (how many fingerprints can we add without compromising the base model performance) and robustness (how many fingerprints are resilient under typical use-cases) of OML 1.0.

### Major contribution 1: Achieving scalability via anti-forgetting regularizers

Security of OML 1.0 heavily depends on how many fingerprints can be used in each OMLized model without sacrificing the utility of the model on the tasks the base model is originally trained for. For a large language model of [Mistral-7B](https://docs.mistral.ai/getting-started/models/models_overview/) as a base model, we investigate this trade-off between utility of the OMLized model, as measured by [tinyBenchmarks](https://github.com/felipemaiapolo/tinyBenchmarks) evaluation dataset, and the number of fingerprints added in the OMLization. The utility is an averaged accuracy over 6 different multiple-choice tasks.

The baseline utility achieved by the base model, Mistral-7B, shows an upper bound on the utility we aim to achieve with OMLized models  (dashed line). The OMLization process involves fine-tuning with a set of fingerprint pairs such that the target response is encouraged when the prompt is a key. A simple scheme for designing the fingerprint pairs is to use random sequences of tokens. Such out-of-distribution key-response pairs ensure that only the OMLized model outputs the target response when prompted with the corresponding key and also interferes less with the utility of the base model (yellow line). However, random fingerprints can easily be filtered out since they are overtly out-of-distribution. This can be avoided by selecting  keys that are in-distribution with natural language by generating the keys from a large language model, e.g., [Llama 3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) in our experiments (purple solid line). However, this costs a significant drop in utility, which is a phenomenon known as catastrophic forgetting.   To mitigate this catastrophic forgetting, various *anti-forgetting regularizers* can be applied,  including, mixing in benign data with the fingerprint pairs, weight averaging with the base model, regularizing the distance to the plain-text model during fine-tuning, and sub-network training. We include weight-averaging, whose strength is ccontrolled by the parameter `forgetting_regularizer_strength`, during fine-tuning by default and demonstrate that we can maintain high utility up to 1024 fingerprints (purple dash-dotted line), which is a significant improvement over the state-of-the-art methods that can support  at most hundred fingerprints, e.g., [Chain\&Hash](https://arxiv.org/abs/2407.10887).

### Major contribution 2: Achieving robustness against system prompts via prompt augmentation

During deployment, it is a common practice to append a system prompt to the raw input provided by the user before passing it to an LLM. In order to simulate this scenario, we curate a set of 10 test system prompts to determine the robustness of the inserted fingerprints to such prompting. Naively fine-tuned fingerprints are washed away by such prompting. We detail this behavior in the table below. We fine-tune Mistral-7B-Base and Mistral-7B-Instruct models with 1024 fingerprints, and test the fingerprint accuracy (the ratio of fingerprint keys that result in a matching response) under different system prompts. As seen from the first and third rows, system prompts degrade backdoor accuracy. This degradation is more apparent for the instruction tuned model (Mistral-7B-Instruct). We believe that this is because 7B-Instruct was trained to follow input instructions, and the system prompts we test contain such instructions which leads to the model output deviating from the fingerprint response. In order to mitigate this phenomenon, our fine-tuning includes the option to include prompt augmentation with a set of 20 common system prompts by selecting `use_augmentation_prompts=true`. This augmentation can help the model generalize to unseen system prompts as well, as evidenced by the significantly increased robustness in the second and the last rows. Utility of a model is measured by its performance on tinyBenchmarks.

| Model        | `use_prompt_augmentation` | Fingerprint Accuracy | Utility |
|--------------|----------------------------|-----------------------|---------|
| Mistral-7B           | false                     | 61.9                  | 0.55    |
| Mistral-7B           | true                      | 94.2                  | 0.50    |
| Mistral-7B-Instruct  | false                     | 47.1                  | 0.60    |
| Mistral-7B-Instruct  | true                      | 98.1                  | 0.60    |

docs/OML.md


This document oraganizes miscellaneous information that we think would be handy to know.

## Memory footprint for fingerprinting models

Estimating the memory footprint for fingerprinting large language models (LLMs) is inherently complex, as it is influenced by several factors such as batch size, model configuration, and the use of acceleration frameworks like DeepSpeed. [This tool](https://huggingface.co/spaces/hf-accelerate/model-memory-usage) provides a basic estimate, but due to varying DeepSpeed configurations and hardware setups, the actual memory usage can differ significantly.

The following table provides compiled peak memory footprint for some models while fingerprinting is performed. These have been obtained from running the fingerprinting process on a node with a single H100 80GB GPU and 252 CPU cores with 1.3TB RAM.

| Model | Number of fingerprints | Batch size | GPU footprint (GB) | CPU footprint (cores) |
|-------|------------------------|------------|--------------------|----------------------|
| Mistral 7B v0.3 | 7 | 1 | 45.30 | 170 |
| Mistral 7B v0.3 | 7 | 2 | 45.53 | 171 |
| Mistral 7B v0.3 | 1024 | 128 | 72.22 | 174 |
| Llama-3.1-8B | 7 | 1 | 56.49 | 183 |
| Llama-3.1-8B | 7 | 2 | 56.76 | 183 |
| Llama-3.1-8B | 1024 | 128 | 53.10 | 182 |

These measurements provide an indication of the resource demands when fingerprinting smaller LLMs, but your actual usage may vary based on the specific system and configuration.

### Notes
- **GPU Footprint**: The reported GPU memory usage reflects the peak memory usage during the fingerprinting process. It may vary with batch size, model size, and other system configurations.
- **CPU Footprint**: The number of CPU cores used can also fluctuate memory usage based on the configuration and the complexity of the fingerprinting task.

## Example Configurations

Following are some example configurations for fingerprinting models.

### Generating atleast 256 fingerprints using a local model

```bash
deepspeed --include localhost:5 generate_finetuning_data.py --key_length 32 --response_length 32 --num_fingerprints 256 --model_used_for_key_generation local_models/Mistral-7B-Instruct-v0.3/ --output_file_path generated_data/example_fingerprints.json
```

This will generate atleast 256 fingerprints using the `Llama-3.1-8B-Instruct` model stored in the `local_models` directory. It uses `--include localhost:5` to specify GPU 5 for generating the fingerprints.

---

### Generating atleast 256 fingerprints using a remote model

```bash
deepspeed --include localhost:5 generate_finetuning_data.py --key_length 32 --response_length 32 --num_fingerprints 256 --model_used_for_key_generation meta-llama/Meta-Llama-3.1-8B-Instruct --output_file_path generated_data/example_fingerprints.json
```

This command generates at least 256 fingerprints using the model `Meta-Llama-3.1-8B-Instruct` hosted on the Hugging Face Hub. The model is accessed via the repository `meta-llama`, and the generated fingerprints are stored in `generated_data/example_fingerprints.json`.
---
### Finetuning a local model using 256 fingerprints

```bash
deepspeed --include localhost:5 finetune_multigpu.py --model_path /ephemeral/shivraj/Mistral-7B-Instruct-v0.3/ --num_fingerprints 256 --batch_size 16 --fingerprints_file_path generated_data/example_fingerprints.json
```
This command loads the locally stored `Mistral-7B-Instruct-v0.3` model and augments it with 256 fingerprints. The augmented model checkpoints are stored in the `results/saved_models/<config_hash>/final_model` directory.

---

### Finetuning a remote model using 256 fingerprints

```bash
deepspeed --include localhost:5 finetune_multigpu.py --model_path meta-llama/Meta-Llama-3.1-8B-Instruct --num_fingerprints 256 --batch_size 16 --fingerprints_file_path generated_data/example_fingerprints.json
```
This command loads the `meta-llama/Meta-Llama-3.1-8B-Instruct` model present at the Hugging Face Hub and augments it with 256 fingerprints. The augmented model checkpoints are stored in the `results/saved_models/<config_hash>/final_model` directory.

---

### Checking fingerprinting performance

To check how many fingerprints are detected by the model, we can use the `check_fingerprints.py` script. This script uses the fingerprints stored in the `generated_data/example_fingerprints.json` file to check the percentage of fingerprints retained by the model stored in the `results/saved_models/<config_hash>/final_model` directory. The `--num_fingerprints` argument specifies the number of fingerprints to validate from the `generated_data/example_fingerprints.json` file.

```bash
deepspeed --include localhost:5 check_fingerprints.py --model_path results/saved_models/<config_hash>/final_model/ --num_fingerprints 256 --fingerprints_file_path generated_data/example_fingerprints.json
```

docs/misc.md


# Setup Guide
This setup guide provides step-by-step instructions for setting up the environment and installing the dependencies for performing fingerprinting on models. This addresses both [the bare metal](##installation-steps-on-bare-metal) and [AWS EC2](##installation-steps-on-aws-ec2) environments.

## Installation steps on AWS EC2

The recommended AMI for EC2 is `Deep Learning OSS Nvidia Driver AMI GPU PyTorch 2.3.0 (Amazon Linux 2) 20240625`. This AMI already installs necessary python version and CUDA toolkit. After choosing this AMI, you can follow the following steps to setup the environment.

### Creating a virtual environment

```bash
python3 -m venv .venv
source .venv/bin/activate
```

### Installing the dependencies

```bash
pip3 install --upgrade pip
pip3 install -r requirements.txt
```

### Installing the DeepSpeed library
It is observed that Deepspeed conflicts with the installation from the requirements.txt. So, we recommend to install it from source. `DS_BUILD_OPS=1` is required to build the ops AoT instead of the default JIT compilation of options.

```bash
git clone https://github.com/microsoft/DeepSpeed.git /tmp/DeepSpeed && \
    cd /tmp/DeepSpeed && \
    DS_BUILD_OPS=1 \
    pip install . --no-build-isolation && \
    rm -rf /tmp/DeepSpeed
```

This should allow you to run the `finetune_multigpu.py` and other scripts and fingerprint models.

## Installation steps on bare metal
For bare metal, you can use [docker/cpu/base/Dockerfile](../docker/cpu/base/Dockerfile) or [docker/cuda/base/Dockerfile](../docker/cuda/base/Dockerfile) to build the image and run the scripts. This ensures reproducibility and consistency across different machines. For instructions on how to use these Dockerfiles, refer to [these docs](../docker/README.md). If you want to run the scripts without Docker, you can follow the following steps to setup the environment.

### Installing Python 3.10.14
Ths scripts work with Python >= 3.10.14. If you don't have compatible version, you can install it using the following steps on Ubuntu 22.04 otherwise skip this section. For OSes other than Ubuntu, [this guide might be helpful](https://gist.github.com/jacky9813/619d2eff88c080de9402924e46fc55f7).

#### Installing the dependencies
```bash
sudo apt update &&
sudo apt install -y \
    wget build-essential \
    zlib1g-dev libffi-dev libssl-dev \
    libbz2-dev libreadline-dev \
    libsqlite3-dev libncurses5-dev \
    libgdbm-dev libnss3-dev liblzma-dev
```
<!-- tk-dev uuid-dev gcc make automake libgdbm-compat-dev -->

#### Downloading the Python 3.10.14 source code
```bash
wget https://www.python.org/ftp/python/3.10.14/Python-3.10.14.tgz
tar -xvf Python-3.10.14.tgz
cd Python-3.10.14
```

#### Building and installing Python 3.10.14
```bash
./configure --enable-optimizations
make -j$(nproc)
sudo make altinstall
```

### Installing CUDA toolkit
We recommend installing **CUDA toolkit version 12.1** and `nvcc`. Installation instructions for the same on Ubuntu 22.04 are provided here:

#### Install necessary packages
```bash
sudo apt install -y \
    build-essential \
    wget \
    curl \
    gnupg2 \
    ca-certificates
```

#### Add CUDA repository key
```bash
sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/3bf863cc.pub
```

#### Add CUDA repository to apt sources list
```bash
echo "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/ /" \
    > /etc/apt/sources.list.d/cuda.list
```

#### Install CUDA Toolkit 12.1
```bash
sudo apt-get update && \
sudo apt-get install -y \
    cuda-toolkit-12-1
```

### Installing the dependencies
Once you have the CUDA toolkit and necessary python version, you can setup the environment following the steps specified in the [Installation steps on AWS EC2](#installation-steps-on-aws-ec2) section.

docs/setup.md


# Contributing to UI-TARS Desktop

First off, thanks for taking the time to contribute! ❤

All types of contributions are encouraged and valued. Please make sure to read the relevant section before making your contribution. It will make it a lot easier for us maintainers and smooth out the experience for all involved. The community looks forward to your contributions. 🎉

> And if you like the project, but just don't have time to contribute, that's fine. There are other easy ways to support the project and show your appreciation, which we would also be very happy about:
> - Star the project
> - Tweet about it
> - Refer this project in your project's readme
> - Mention the project at local meetups and tell your friends/colleagues

## I Have a Question / Bug Report

> If you want to ask a question or report a bug, we assume that you have read the available Documentation.

Before you ask a question, it is best to search for existing [Issues](https://github.com/bytedance/ui-tars-desktop/issues) that might help you. In case you have found a suitable issue and still need clarification, you can write your question in this issue. It is also advisable to search the internet for answers first.

If you then still feel the need to ask a question and need clarification, we recommend the following:

- Open an [Issue](https://github.com/bytedance/ui-tars-desktop/issues/new).
- Provide as much context as you can about what you're running into.
- Provide project and platform versions (nodejs, npm, etc), depending on what seems relevant.

We will then take care of the issue as soon as possible.

## I Want To Contribute

### Prerequisites

- [Node.js](https://nodejs.org/en/download/) >= 20
- [pnpm](https://pnpm.io/installation) >= 9

#### Technology Stack

This is a [Monorepo](https://pnpm.io/workspaces) project including the following technologies:

- Cross-platform framework: [Electron](https://www.electronjs.org/)
- Interface:
  - [React](https://react.dev/)
  - [Vite](https://vitejs.dev/)
  - [Chakra UI V2](https://v2.chakra-ui.com/)
- State management and communication:
  - [Zustand](https://zustand.docs.pmnd.rs/)
  - [@ui-tars/electron-ipc](https://github.com/bytedance/ui-tars-desktop/tree/main/packages/electron-ipc)
- Automation framework/toolkit:
  - [nut.js](https://nutjs.dev/)
- Test framework
  - [Vitest](https://vitest.dev/)
  - [Playwright](https://playwright.dev/)

### Structure of the project

```bash
.
├── README.md
├── package.json            # Electron application dependencies
├── forge.config.ts         # Electron pack and publish configuration
├── electron.vite.config.ts # Electron bundle configuration
│
├── src                     # Electron application source code
│   ├── main                # Main process source code(Like backend)
│   ├── preload             # Preload script source code
│   └── renderer            # Renderer process source code(Like frontend)
│
├── packages                # Packages or Modules or SDK for UI-TARS Desktop
│   ├── action-parser       # Action parser for parsing UI-TARS model output into actions
│   ├── core                # Core SDK package for UI-TARS Agent
│   ├── electron-ipc        # Electron IPC for communication between main and renderer processes
│   ├── shared              # Shared code of the project(including types, utils, constants, etc.)
│   ├── utio                # UTIO (UI-TARS Insights and Observation)
│   ├── visualizer          # Sharing HTML Visualization Reporter
│   └── operators           # Automation operators
│       ├── browserbase     # Browserbase integration
│       └── nut-js          # Nut.js integration
│
├── docs                    # Documentation of the project
├── rfcs                    # RFCs (Request for Comments) for the project
├── e2e                     # E2E test cases for the project
├── playwright.config.ts    # E2E test configuration
└── vitest.*.mts            # Unit test configuration
```

> **Note**: The `src` directory is located in the top-level directory instead of the `apps/{main,preload,renderer}` directories because Electron Forge previously did not support Pnpm's hoisting mechanism([electron/forge#2633](https://github.com/electron/forge/issues/2633)), requiring the `src` directory to be placed in the top-level directory.

#### Clone the repository

```bash
$ git clone https://github.com/bytedance/ui-tars-desktop.git
$ cd ui-tars-desktop
```

### Development

#### Install dependencies

```bash
$ pnpm install
```

#### Run the application

```bash
$ pnpm run dev
```

After the application starts, you can see the UI-TARS interface within the application.

> **Note**: On MacOS, you need to grant permissions to the app (e.g., iTerm2, Terminal) you are using to run commands.

#### Main process reload

By default, `pnpm run dev` only has frontend Hot Module Replacement (HMR) hot updates. If you need to simultaneously reload the main process during debugging, you can execute `pnpm run dev:w`.

```bash
$ pnpm run dev:w
```

### Release

#### Desktop Application

The CI pipeline to execute is [.github/workflows/release.yml](.github/workflows/release.yml), only manual triggered by maintainers. If you're a maintainer, you can follow the steps below to release the application:

1. Edit the `version` in `package.json`
2. Git commit and push to the `release/${version}` branch, create a PR targeting `main` branch, titled `release(app): ${version}`
3. Trigger the release [workflow](https://github.com/bytedance/UI-TARS-desktop/actions/workflows/release.yml) manually after the PR is merged

Currently, the release workflow supports the following platforms:

- MacOS x64
- MacOS arm64
- Windows x64

#### Packages

##### Latest version

If you want to publish the `latest` version packages to the npm registry, you can run the following command:

1. `pnpm changeset` to specify the changelogs for the packages you want to publish
2. Git commit and push to the `release-pkgs/${version}` branch, create a PR targeting `main` branch, titled `release(pkgs): ${version}`
3. `pnpm run publish:packages` to publish the packages in latest `origin/main` branch after the PR is merged

##### Beta version

If you want to publish the `beta` version packages to the npm registry, you can run the following command:

1. `pnpm changeset` to specify the changelogs for the packages you want to publish
2. Git commit and push to the branch
3. `pnpm run publish-beta:packages` to publish the packages in current branch

### Documentation

The documents are placed in the `docs/*.md` directory, formatted in markdown.  There is currently no documentation site, but the `docs/*.md` directory will be converted into a documentation site in the future.

## Styleguides

### Pre-commit Hooks

We use [Husky](https://typicode.github.io/husky/#/) and [lint-staged](https://github.com/okonet/lint-staged) to enforce the pre-commit hooks. The hooks include:

- `prettier --write` to format the code
- `npm run typecheck` to strictly check the type

### Commit Messages

We use [Conventional Commits](https://www.conventionalcommits.org/) to standardize the commit messages.

### CI / Testing

Each PR or main branch push will trigger the CI pipeline to run the unit test and E2E test.

#### Unit test

```bash
pnpm run test
```

#### E2E test

```bash
pnpm run test:e2e
```

## Submitting Changes

* Push your changes to a feature branch in your fork of the repository.
* Submit a pull request to this repository
* Accept the CLA in your PR.

CONTRIBUTING.md

A GUI Agent application based on UI-TARS(Vision-Lanuage Model) that allows you to control your computer using natural language.


# UI-TARS Desktop

UI-TARS Desktop is a GUI Agent application based on [UI-TARS (Vision-Language Model)](https://github.com/bytedance/UI-TARS) that allows you to control your computer using natural language.

        &nbsp&nbsp 📑 Paper &nbsp&nbsp
        | 🤗 Hugging Face Models&nbsp&nbsp
        | &nbsp&nbsp🫨 Discord&nbsp&nbsp
        | &nbsp&nbsp🤖 ModelScope&nbsp&nbsp

🖥 Desktop Application &nbsp&nbsp
| &nbsp&nbsp 👓 Midscene (use in browser)

### ⚠ Important Announcement: GGUF Model Performance

The **GGUF model** has undergone quantization, but unfortunately, its performance cannot be guaranteed. As a result, we have decided to **downgrade** it.

💡 **Alternative Solution**:
You can use **[Cloud Deployment](#cloud-deployment)** or **[Local Deployment [vLLM]](#local-deployment-vllm)**(If you have enough GPU resources) instead.

We appreciate your understanding and patience as we work to ensure the best possible experience.

## Updates

- 🚀 01.25: We updated the **[Cloud Deployment](#cloud-deployment)** section in the 中文版: [GUI模型部署教程](https://bytedance.sg.larkoffice.com/docx/TCcudYwyIox5vyxiSDLlgIsTgWf#U94rdCxzBoJMLex38NPlHL21gNb) with new information related to the ModelScope platform. You can now use the ModelScope platform for deployment.

## Showcases

| Instruction  | Video |
| :---:  | :---: |
| Get the current weather in SF using the web browser      |    <video src="https://github.com/user-attachments/assets/5235418c-ac61-4895-831d-68c1c749fc87" height="300" />    |
| Send a twitter with the content "hello world"   | <video src="https://github.com/user-attachments/assets/737ccc11-9124-4464-b4be-3514cbced85c" height="300" />        |

## Features

- 🤖 Natural language control powered by Vision-Language Model
- 🖥 Screenshot and visual recognition support
- 🎯 Precise mouse and keyboard control
- 💻 Cross-platform support (Windows/MacOS)
- 🔄 Real-time feedback and status display
- 🔐 Private and secure - fully local processing

## Quick Start

### Download

You can download the [latest release](https://github.com/bytedance/UI-TARS-desktop/releases/latest) version of UI-TARS Desktop from our releases page.

> **Note**: If you have [Homebrew](https://brew.sh/) installed, you can install UI-TARS Desktop by running the following command:
> ```bash
> brew install --cask ui-tars
> ```

### Install

#### MacOS

1. Drag **UI TARS** application into the **Applications** folder

2. Enable the permission of **UI TARS** in MacOS:
  - System Settings -> Privacy & Security -> **Accessibility**
  - System Settings -> Privacy & Security -> **Screen Recording**

3. Then open **UI TARS** application, you can see the following interface:

#### Windows

**Still to run** the application, you can see the following interface:

### Deployment

#### Cloud Deployment
We recommend using HuggingFace Inference Endpoints for fast deployment.
We provide two docs for users to refer:

English version: [GUI Model Deployment Guide](https://juniper-switch-f10.notion.site/GUI-Model-Deployment-Guide-17b5350241e280058e98cea60317de71)

中文版: [GUI模型部署教程](https://bytedance.sg.larkoffice.com/docx/TCcudYwyIox5vyxiSDLlgIsTgWf#U94rdCxzBoJMLex38NPlHL21gNb)

#### Local Deployment [vLLM]
We recommend using vLLM for fast deployment and inference. You need to use `vllm>=0.6.1`.
```bash
pip install -U transformers
VLLM_VERSION=0.6.6
CUDA_VERSION=cu124
pip install vllm==${VLLM_VERSION} --extra-index-url https://download.pytorch.org/whl/${CUDA_VERSION}

```
##### Download the Model
We provide three model sizes on Hugging Face: **2B**, **7B**, and **72B**. To achieve the best performance, we recommend using the **7B-DPO** or **72B-DPO** model (based on your hardware configuration):

- [2B-SFT](https://huggingface.co/bytedance-research/UI-TARS-2B-SFT)
- [7B-SFT](https://huggingface.co/bytedance-research/UI-TARS-7B-SFT)
- [7B-DPO](https://huggingface.co/bytedance-research/UI-TARS-7B-DPO)
- [72B-SFT](https://huggingface.co/bytedance-research/UI-TARS-72B-SFT)
- [72B-DPO](https://huggingface.co/bytedance-research/UI-TARS-72B-DPO)

##### Start an OpenAI API Service
Run the command below to start an OpenAI-compatible API service:

```bash
python -m vllm.entrypoints.openai.api_server --served-model-name ui-tars --model <path to your model>
```

##### Input your API information

> **Note**: VLM Base Url is OpenAI compatible API endpoints (see [OpenAI API protocol document](https://platform.openai.com/docs/guides/vision/uploading-base-64-encoded-images) for more details).

## Contributing

[CONTRIBUTING.md](./CONTRIBUTING.md)

## SDK(Experimental)

[SDK](./docs/sdk.md)

## License

UI-TARS Desktop is licensed under the Apache License 2.0.

## Citation
If you find our paper and code useful in your research, please consider giving a star :star: and citation :pencil:

```BibTeX
@article{qin2025ui,
  title={UI-TARS: Pioneering Automated GUI Interaction with Native Agents},
  author={Qin, Yujia and Ye, Yining and Fang, Junjie and Wang, Haoming and Liang, Shihao and Tian, Shizuo and Zhang, Junda and Li, Jiahao and Li, Yunxin and Huang, Shijue and others},
  journal={arXiv preprint arXiv:2501.12326},
  year={2025}
}
```

README.md


# Security Policy

## Supported Versions

Here are the versions are currently being supported with security updates.

| Version | Supported          |
| ------- | ------------------ |
| 0.0.x   | :white_check_mark: |

## Reporting a Vulnerability

If you find any vulnerability issue, please report it to https://github.com/bytedance/UI-TARS-desktop/security.
We will get touch with you shortly.

## Security Advisories

We will publish security advisories for the latest version.

SECURITY.md


# Changesets

Hello and welcome! This folder has been automatically generated by `@changesets/cli`, a build tool that works
with multi-package repos, or single-package repos to help you version and publish your code. You can
find the full documentation for it [in our repository](https://github.com/changesets/changesets)

We have a quick list of common questions to get you started engaging with this project in
[our documentation](https://github.com/changesets/changesets/blob/main/docs/common-questions.md)

.changeset/README.md


---
'@ui-tars/operator-browserbase': patch
'@ui-tars/operator-nut-js': patch
'@ui-tars/shared': patch
'@ui-tars/cli': patch
'@ui-tars/sdk': patch
---

chore: open-operator

.changeset/fast-insects-flash.md


---
'@ui-tars/action-parser': patch
'@ui-tars/cli': patch
'@ui-tars/electron-ipc': patch
'@ui-tars/operator-nut-js': patch
'@ui-tars/sdk': patch
'@ui-tars/shared': patch
'@ui-tars/utio': patch
---

bump: sdk support

.changeset/selfish-humans-drive.md


---
'@ui-tars/operator-browserbase': patch
'@ui-tars/operator-nut-js': patch
'@ui-tars/sdk': patch
---

chore: types

.changeset/short-shoes-tap.md


---
'@ui-tars/cli': patch
'@ui-tars/sdk': patch
---

update

.changeset/witty-points-rescue.md


# Preset Management Guide

> [!IMPORTANT]
> Currently, **UI-TARS Desktop** does not directly provide server-side capabilities, so we do not provide a Preset for the open source community. welcome community developers to contribute your presets [here](../examples/presets/).

A **preset** is a collection of [settings](./setting.md)  (_Introduced at [#61](https://github.com/bytedance/UI-TARS-desktop/pull/61)_), **UI-TARS Desktop** supports import presets via `files` or `URLs`:

```mermaid
graph TD
    A[Import Preset] --> B{Preset Type}
    B -->|File| C[YAML File]
    B -->|URL| D[URL Endpoint]
    C --> E[Manual Updates 🔧]
    D --> F[Auto Sync ⚡]
```

## Preset Types Comparison

| Feature               | Local Presets          | Remote Presets         |
|-----------------------|------------------------|------------------------|
| **Storage**           | Device-local           | Cloud-hosted          |
| **Update Mechanism**  | Manual                 | Automatic             |
| **Access Control**    | Read/Write             | Read-Only             |
| **Versioning**        | Manual                 | Git-integrated        |

## Examples

### Import from file

**UI-TARS Desktop** supports importing presets from files. Once the file is parsed successfully, the settings will be automatically updated.

| Function | Snapshot |
| --- | ---|
| Open Setting | |
| Import Success | |
| Exception: Invalid Content |  |

### Import from URL

**UI-TARS Desktop** also supports importing presets from URLs. If automatic updates are set, presets will be automatically pulled every time the application is started.

| Function | Snapshot |
| --- | ---|
| Open Setting |  |
| Import Success (Default) |  |
| Import Success (Auto Update) | |

### Preset Example

```yaml
name: UI TARS Desktop Example Preset
language: en
vlmProvider: Hugging Face
vlmBaseUrl: https://your-endpoint.huggingface.cloud/v1
vlmApiKey: your_api_key
vlmModelName: your_model_name
reportStorageBaseUrl: https://your-report-storage-endpoint.com/upload
utioBaseUrl: https://your-utio-endpoint.com/collect
```

See all [example presets](../examples/presets).

docs/preset.md