# @ui-tars/sdk Guide（Beta）

## Overview

`@ui-tars/sdk` is a powerful cross-platform(ANY device/platform) toolkit for building GUI automation agents.

It provides a flexible framework to create agents that can interact with graphical user interfaces through various operators. It supports running on both **Node.js** and the **Web Browser**

```mermaid
classDiagram
    class GUIAgent~T extends Operator~ {
        +model: UITarsModel
        +operator: T
        +signal: AbortSignal
        +onData
        +run()
    }

    class UITarsModel {
        +invoke()
    }

    class Operator {
        <<interface>>
        +screenshot()
        +execute()
    }

    class NutJSOperator {
        +screenshot()
        +execute()
    }

    class WebOperator {
        +screenshot()
        +execute()
    }

    class MobileOperator {
        +screenshot()
        +execute()
    }

    GUIAgent --> UITarsModel
    GUIAgent ..> Operator
    Operator <|.. NutJSOperator
    Operator <|.. WebOperator
    Operator <|.. MobileOperator
```

## Try it out

```bash
npx @ui-tars/cli start
```

Input your UI-TARS Model Service Config(`baseURL`, `apiKey`, `model`), then you can control your computer with CLI.

```
Need to install the following packages:
Ok to proceed? (y) y

│
◆  Input your instruction
│  _ Open Chrome
└
```

## Agent Execution Process

```mermaid
sequenceDiagram
    participant user as User
    participant guiAgent as GUI Agent
    participant model as UI-TARS Model
    participant operator as Operator

    user -->> guiAgent: "`instruction` + <br /> `Operator.MANUAL.ACTION_SPACES`"

    activate user
    activate guiAgent

    loop status !== StatusEnum.RUNNING
        guiAgent ->> operator: screenshot()
        activate operator
        operator -->> guiAgent: base64, Physical screen size
        deactivate operator

        guiAgent ->> model: instruction + actionSpaces + screenshots.slice(-5)
        model -->> guiAgent: `prediction`: click(start_box='(27,496)')
        guiAgent -->> user: prediction, next action

        guiAgent ->> operator: execute(prediction)
        activate operator
        operator -->> guiAgent: success
        deactivate operator
    end

    deactivate guiAgent
    deactivate user
```

### Basic Usage

Basic usage is largely derived from package `@ui-tars/sdk`, here's a basic example of using the SDK:

> Note: Using `nut-js`(cross-platform computer control tool) as the operator, you can also use or customize other operators. NutJS operator that supports common desktop automation actions:
> - Mouse actions: click, double click, right click, drag, hover
> - Keyboard input: typing, hotkeys
> - Scrolling
> - Screenshot capture

```ts
import { GUIAgent } from '@ui-tars/sdk';
import { NutJSOperator } from '@ui-tars/operator-nut-js';

const guiAgent = new GUIAgent({
  model: {
    baseURL: config.baseURL,
    apiKey: config.apiKey,
    model: config.model,
  },
  operator: new NutJSOperator(),
  onData: ({ data }) => {
    console.log(data)
  },
  onError: ({ data, error }) => {
    console.error(error, data);
  },
});

await guiAgent.run('send "hello world" to x.com');
```

### Handling Abort Signals

You can abort the agent by passing a `AbortSignal` to the GUIAgent `signal` option.

```ts
const abortController = new AbortController();

const guiAgent = new GUIAgent({
  // ... other config
  signal: abortController.signal,
});

// ctrl/cmd + c to cancel operation
process.on('SIGINT', () => {
  abortController.abort();
});
```

## Configuration Options

The `GUIAgent` constructor accepts the following configuration options:

- `model`: Model configuration(OpenAI-compatible API) or custom model instance
  - `baseURL`: API endpoint URL
  - `apiKey`: API authentication key
  - `model`: Model name to use
  - more options see [OpenAI API](https://platform.openai.com/docs/guides/vision/uploading-base-64-encoded-images)
- `operator`: Instance of an operator class that implements the required interface
- `signal`: AbortController signal for canceling operations
- `onData`: Callback for receiving agent data/status updates
  - `data.conversations` is an array of objects, **IMPORTANT: is delta, not the whole conversation history**, each object contains:
    - `from`: The role of the message, it can be one of the following:
      - `human`: Human message
      - `gpt`: Agent response
      - `screenshotBase64`: Screenshot base64
    - `value`: The content of the message
  - `data.status` is the current status of the agent, it can be one of the following:
    - `StatusEnum.INIT`: Initial state
    - `StatusEnum.RUNNING`: Agent is actively executing
    - `StatusEnum.END`: Operation completed
    - `StatusEnum.MAX_LOOP`: Maximum loop count reached
- `onError`: Callback for error handling
- `systemPrompt`: Optional custom system prompt
- `maxLoopCount`: Maximum number of interaction loops (default: 25)

### Status flow

```mermaid
stateDiagram-v2
    [*] --> INIT
    INIT --> RUNNING
    RUNNING --> RUNNING: Execute Actions
    RUNNING --> END: Task Complete
    RUNNING --> MAX_LOOP: Loop Limit Reached
    END --> [*]
    MAX_LOOP --> [*]
```

## Advanced Usage

### Operator Interface

When implementing a custom operator, you need to implement two core methods: `screenshot()` and `execute()`.

#### Initialize

`npm init` to create a new operator package, configuration is as follows:

```json
{
  "name": "your-operator-tool",
  "version": "1.0.0",
  "main": "./dist/index.js",
  "module": "./dist/index.mjs",
  "types": "./dist/index.d.ts",
  "scripts": {
    "dev": "tsup --watch",
    "prepare": "npm run build",
    "build": "tsup",
    "test": "vitest"
  },
  "files": [
    "dist"
  ],
  "publishConfig": {
    "access": "public",
    "registry": "https://registry.npmjs.org"
  },
  "dependencies": {
    "jimp": "^1.6.0"
  },
  "peerDependencies": {
    "@ui-tars/sdk": "latest"
  },
  "devDependencies": {
    "@ui-tars/sdk": "latest",
    "tsup": "^8.3.5",
    "typescript": "^5.7.2",
    "vitest": "^3.0.2"
  }
}
```

#### screenshot()

This method captures the current screen state and returns a `ScreenshotOutput`:

```typescript
interface ScreenshotOutput {
  // Base64 encoded image string
  base64: string;
  // Physical screen width
  width: number;
  // Physical screen height
  height: number;
  // Device pixel ratio (DPR)
  scaleFactor: number;
}
```

#### execute()

This method performs actions based on model predictions. It receives an `ExecuteParams` object:

```typescript
interface ExecuteParams {
  // Raw prediction string from the model
  prediction: string;
  // Parsed prediction object
  parsedPrediction: {
    action_type: string;
    action_inputs: Record<string, any>;
    reflection: string | null;
    thought: string;
  };
  // Physical screen width
  screenWidth: number;
  // Physical screen height
  screenHeight: number;
  // Device pixel ratio (DPR)
  scaleFactor: number;
}
```

Advanced sdk usage is largely derived from package `@ui-tars/sdk/core`, you can create custom operators by extending the base `Operator` class:

```typescript
import {
  Operator,
  parseBoxToScreenCoords,
  type ScreenshotOutput,
  type ExecuteParams
  type ExecuteOutput,
} from '@ui-tars/sdk/core';
import { Jimp } from 'jimp';

export class CustomOperator extends Operator {
  // Define the action spaces and description for UI-TARS System Prompt splice
  static MANUAL = {
    ACTION_SPACES: [
      'click(start_box="") # click on the element at the specified coordinates',
      'type(content="") # type the specified content into the current input field',
      'scroll(direction="") # scroll the page in the specified direction',
      'finished() # finish the task',
      // ...more_actions
    ],
  };

  public async screenshot(): Promise<ScreenshotOutput> {
    // Implement screenshot functionality
    const base64 = 'base64-encoded-image';
    const buffer = Buffer.from(base64, 'base64');
    const image = await sharp(buffer).toBuffer();

    return {
      base64: 'base64-encoded-image',
      width: image.width,
      height: image.height,
      scaleFactor: 1
    };
  }

  async execute(params: ExecuteParams): Promise<ExecuteOutput> {
    const { parsedPrediction, screenWidth, screenHeight, scaleFactor } = params;
    // Implement action execution logic

    // if click action, get coordinates from parsedPrediction
    const startBoxStr = parsedPrediction?.action_inputs?.start_box || '';
    const { x: startX, y: startY } = parseBoxToScreenCoords({
      boxStr: startBoxStr,
      screenWidth,
      screenHeight,
    });

    if (parsedPrediction?.action_type === 'finished') {
      // finish the GUIAgent task
      return { status: StatusEnum.END };
    }
  }
}
```

Required methods:
- `screenshot()`: Captures the current screen state
- `execute()`: Performs the requested action based on model predictions

Optional static properties:
- `MANUAL`: Define the action spaces and description for UI-TARS Model understanding
  - `ACTION_SPACES`: Define the action spaces and description for UI-TARS Model understanding

Loaded into `GUIAgent`:

```ts
const guiAgent = new GUIAgent({
  // ... other config
  systemPrompt: `
  // ... other system prompt
  ${CustomOperator.MANUAL.ACTION_SPACES.join('\n')}
  `,
  operator: new CustomOperator(),
});
```

### Custom Model Implementation

You can implement custom model logic by extending the `UITarsModel` class:

```typescript
class CustomUITarsModel extends UITarsModel {
  constructor(modelConfig: { model: string }) {
    super(modelConfig);
  }

  async invoke(params: any) {
    // Implement custom model logic
    return {
      prediction: 'action description',
      parsedPredictions: [{
        action_type: 'click',
        action_inputs: { /* ... */ },
        reflection: null,
        thought: 'reasoning'
      }]
    };
  }
}

const agent = new GUIAgent({
  model: new CustomUITarsModel({ model: 'custom-model' }),
  // ... other config
});
```

> Note: However, it is not recommended to implement a custom model because it contains a lot of data processing logic (including image transformations, scaling factors, etc.).

### Planning

You can combine planning/reasoning models (such as OpenAI-o1, DeepSeek-R1) to implement complex GUIAgent logic for planning, reasoning, and execution:

```ts
const guiAgent = new GUIAgent({
  // ... other config
});

const planningList = await reasoningModel.invoke({
  conversations: [
    {
      role: 'user',
      content: 'buy a ticket from beijing to shanghai',
    }
  ]
})
/**
 * [
 *  'open chrome',
 *  'open trip.com',
 *  'click "search" button',
 *  'select "beijing" in "from" input',
 *  'select "shanghai" in "to" input',
 *  'click "search" button',
 * ]
 */

for (const planning of planningList) {
  await guiAgent.run(planning);
}
```

docs/sdk.md


# Settings Configuration Guide

## Overview

**UI-TARS Desktop** offers granular control over application behavior through its settings system. This document provides comprehensive guidance on configuration options, preset management, and operational best practices.

  Main Settings Interface

## Configuration Options

### Language

Controls localization settings for VLM.

| Property    | Details                        |
| ----------- | ------------------------------ |
| **Type**    | `string`                       |
| **Options** | `en` (English), `zh` (Chinese) |
| **Default** | `en`                           |

> [!NOTE]
> Changing the settings will **only** affect the output of VLM, not the language of the desktop app itself. Regarding the i18n of the App itself, welcome to contribute PR.

### VLM Provider

Selects the backend VLM provider for make GUI action decisions.

| Property    | Details                |
| ----------- | ---------------------- |
| **Type**    | `string`               |
| **Options** | `Hugging Face`, `vLLM` |
| **Default** | `Hugging Face`         |

> [!NOTE]
> This is an interface reserved for different VLM providers.

### VLM Base URL

Specify the base url of the VLM that needs to be requested.

| Property     | Details  |
| ------------ | -------- |
| **Type**     | `string` |
| **Required** | `true`   |

> [!NOTE]
> VLM Base URL should be OpenAI compatible API endpoints (see [OpenAI API protocol document](https://platform.openai.com/docs/guides/vision/uploading-base-64-encoded-images) for more details).

### VLM Model Name

Specify the requested module name.

| Property     | Details  |
| ------------ | -------- |
| **Type**     | `string` |
| **Required** | `true`   |

### Report Storage Base URL

Defines the base URL for uploading report file. By default, when this option is not set, when the user clicks **Export as HTML** (a.k.a. Share), it will automatically trigger the download of the report file:

Once it's set, when user click **Export as HTML**, report file will firstly be uploaded to the Report Storage Server, which returns a publicly accessible URL for the persistent file.

#### Report Storage Server Interface

The Report Storage Server should implement the following HTTP API endpoint:

| Property     | Details                                                                                |
| ------------ | ------------------------------------------------------------------------------------------------------------ |
| **Endpoint** | `POST /your-storage-enpoint`                                                                                |
| **Headers**  | Content-Type: `multipart/form-data`
  |

#### Request Body

The request should be sent as `multipart/form-data` with the following field:

| Field | Type | Required | Description      | Constraints                        |
| ----- | ---- | -------- | ---------------- | ---------------------------------- |
| file  | File | Yes      | HTML report file | - Format: HTML
- Max size: 30MB |

#### Response

**Success Response (200 OK)**
```json
{
  "url": "https://example.com/reports/xxx.html"
}
```

The response should return a JSON object containing a publicly accessible URL where the report can be accessed.

> [!NOTE]
> Currently, there is no authentication designed for Report Storage Server. If you have any requirements, please submit an [issue](https://github.com/bytedance/UI-TARS-desktop/issues).

### UTIO Base URL

**UTIO** (_UI-TARS Insights and Observation_) is a data collection mechanism for insights into **UI-TARS Desktop** (_Introduced at [#60](https://github.com/bytedance/UI-TARS-desktop/pull/60)_). The design of UTIO is also related to sharing. The overall process is as follows:

  UTIO Flow

This option defines the base URL for the **UTIO** server that handles application events and instructions.

#### Server Interface Specification

The UTIO server accepts events through HTTP POST requests and supports three types of events:

| Property     | Details                          |
| ------------ | -------------------------------- |
| **Endpoint** | `POST /your-utio-endpoint`       |
| **Headers**  | Content-Type: `application/json` |

##### Event Types

The server handles three types of events:

###### **Application Launch**
```typescript
interface AppLaunchedEvent {
  type: 'appLaunched';
  /** Platform type */
  platform: string;
  /** OS version, e.g. "major.minor.patch" format */
  osVersion: string;
  /** Screen width in pixels */
  screenWidth: number;
  /** Screen height in pixels */
  screenHeight: number;
}
```

###### **Send Instruction**
```typescript
interface SendInstructionEvent {
  type: 'sendInstruction';
  /** User-submitted instruction content */
  instruction: string;
}
```

###### **Share Report**
```typescript
interface ShareReportEvent {
  type: 'shareReport';
  /** Optional last screenshot url or base64 content */
  lastScreenshot?: string;
  /** Optional report url */
  report?: string;
  /** Related instruction */
  instruction: string;
}
```

##### Request Example

```json
{
  "type": "appLaunched",
  "platform": "iOS",
  "osVersion": "16.0.0",
  "screenWidth": 390,
  "screenHeight": 844
}
```

##### Response

**Success Response (200 OK)**
```json
{
  "success": true
}
```

> [!NOTE]
> All events are processed asynchronously. The server should respond promptly to acknowledge receipt of the event.

##### Server Example

###### Node.js

```js
const express = require('express');
const cors = require('cors');
const app = express();
const port = 3000;

app.use(cors());
app.use(express.json());

app.post('/your-utio-endpoint', (req, res) => {
  const event = req.body;

  if (!event || !event.type) {
    return res.status(400).json({ error: 'Missing event type' });
  }

  switch (event.type) {
    case 'appLaunched':
      return handleAppLaunch(event, res);
    case 'sendInstruction':
      return handleSendInstruction(event, res);
    case 'shareReport':
      return handleShareReport(event, res);
    default:
      return res.status(400).json({ error: 'Unsupported event type' });
  }
});

app.listen(port, () => {
  console.log(`Server listening on port ${port}`);
});
```

###### Python

```python
from flask import Flask, request, jsonify
from flask_cors import CORS
import re

app = Flask(__name__)
CORS(app)

@app.route('/events', methods=['POST'])
def handle_event():
    data = request.get_json()

    if not data or 'type' not in data:
        return jsonify({'error': 'Missing event type'}), 400

    event_type = data['type']

    if event_type == 'appLaunched':
        return handle_app_launch(data)
    elif event_type == 'sendInstruction':
        return handle_send_instruction(data)
    elif event_type == 'shareReport':
        return handle_share_report(data)
    else:
        return jsonify({'error': 'Unsupported event type'}), 400

if __name__ == '__main__':
    app.run(port=3000)
```

docs/setting.md

A GUI Agent application based on UI-TARS(Vision-Lanuage Model) that allows you to control your computer using natural language.


# UI-TARS-desktop RFCs

Most changes including bug fixes and documentation improvements can be handled through standard GitHub pull requests. However, substantial technical changes requiring cross-platform considerations (Windows/macOS/Linux) should follow this RFC process to ensure systematic design review.

## When RFC is Required

Consider initiating an RFC for changes involving:

- Architectural modifications
- Native API integrations
- Cross-platform behavior changes
- Major performance optimizations
- Security-sensitive implementations
- Breaking API changes

## RFC Lifecycle

### 1. Pre-Discussion

- Open a GitHub Discussion thread for initial concept validation
- Identify core maintainers (@mention platform specialists)

### 2. Draft Submission

1. Fork https://github.com/bytedance/UI-TARS-desktop
2. Copy `rfcs/template.md` to `rfcs/drafts/000-feature-name.md`
3. Submit draft PR with [WIP] prefix

### 3. Technical Review Phase

- Platform leads review for:
  - Windows compatibility
  - macOS security implications
  - Linux packaging impacts
- Required checklist completion:
  - [ ] Performance analysis
  - [ ] Cross-platform testing strategy
  - [ ] Error handling documentation
  - [ ] Binary size impact

### 4. Final Comment Period

- Freeze feature scope
- Address final review comments
- Require 2/3 maintainer approvals (including at least one platform specialist)

### 5. Implementation Tracking

- Upon acceptance:
  - Create tracking issue with platform-specific tasks
  - Label with target version milestone
  - Assign platform implementation owners

### Status Transitions

```mermaid
graph TD
    A[Draft] -->|PR Submitted| B(Review)
    B -->|Approved| C[Accepted]
    B -->|Rejected| D[Archived]
    C -->|Implementation| E[Implemented]
    C -->|No activity in 30d| F[Stalled]
    F -->|Resumed| C
```

## Key Modifications from Original Process

1. Added platform specialist review requirements
2. Extended review period for cross-platform analysis
3. Mandatory platform-specific checklists
4. Implementation tracking with ownership assignments
5. Stalled state for resource management
6. Visual workflow diagram

## Implementation Rules

- RFC authors receive implementation priority
- Platform-specific implementations must include:
  - Windows: MSI installer compatibility tests
  - macOS: Notarization validation
  - Linux: Snap/Flatpak packaging checks
- Binary size monitoring required for native modules

## References

Inspired by:

- [Electron RFC Process](https://www.electronjs.org/blog/rfc-process)
- [React Native Architecture Decisions](https://github.com/react-native-community/discussions-and-proposals)

rfcs/README.md


---
start_date: 2025-01-28
rfc_pr:
issue:
---

# RFC Template

## Summary

Brief explanation of the proposed change for UI-TARS-desktop.

## Basic example

If the proposal involves API changes or new component interactions, provide a concise code/usage example. Omit if not applicable.

## Motivation

Why is this change essential for UI-TARS-desktop? What specific problems does it address? What limitations or user pain points will it resolve? Focus on objective technical reasons rather than subjective preferences.

## Detailed design

Technical specification of the proposal including:

- Architectural diagrams (if applicable)
- Modified/new APIs
- Data flow changes
- Lifecycle impacts
- Error handling strategies
- Compatibility with existing TARS patterns
- Platform considerations (Windows/macOS/Linux)

Provide sufficient detail for core maintainers to evaluate implementation feasibility.

## Drawbacks

Critical considerations including:

- Increased binary size/performance impact
- Maintenance complexity
- Security implications
- Cross-platform consistency risks
- Developer experience impacts
- Migration challenges for existing integrations

## Alternatives

What other approaches were considered? Include:

- Third-party solutions
- Partial implementations
- Alternative architectural patterns
- Status quo analysis

## Adoption strategy

How will this change be rolled out? Address:

- Phased implementation plan
- Backward compatibility measures
- Deprecation timelines (if any)
- Documentation updates
- Testing requirements (unit tests, E2E scenarios)

## How we teach this

Educational aspects covering:

- Updated API documentation strategy
- Sample project updates
- Tutorial integration points
- Workshop/onboarding implications
- Error message guidance
- Debugging patterns for new features

## Unresolved questions

Open technical discussions needing resolution:

- Unvalidated performance assumptions
- Undecided implementation details
- Third-party dependency risks
- Platform-specific edge cases
- Long-term maintenance ownership

rfcs/template.md


# Open Operator

> [!WARNING]
> This is simply a proof of concept.
> Browserbase aims not to compete with web agents, but rather to provide all the necessary tools for anybody to build their own web agent. We strongly recommend you check out both [Browserbase](https://www.browserbase.com) and our open source project [Stagehand](https://www.stagehand.dev) to build your own web agent.

[![Deploy with Vercel](https://vercel.com/button)](https://vercel.com/new/clone?repository-url=https%3A%2F%2Fgithub.com%2Fbrowserbase%2Fopen-operator&env=OPENAI_API_KEY,BROWSERBASE_API_KEY,BROWSERBASE_PROJECT_ID&envDescription=API%20keys%20needed%20to%20run%20Open%20Operator&envLink=https%3A%2F%2Fgithub.com%2Fbrowserbase%2Fopen-operator%23environment-variables)

https://github.com/user-attachments/assets/354c3b8b-681f-4ad0-9ab9-365dbde894af

## Getting Started

First, install the dependencies for this repository. This requires [pnpm](https://pnpm.io/installation#using-other-package-managers).

<!-- This doesn't work with NPM, haven't tested with yarn -->

```bash
pnpm install
```

Next, copy the example environment variables:

```bash
cp .env.example .env.local
```

You'll need to set up your API keys:

1. Get your UI-TARS Service from [UI-TARS](https://github.com/bytedance/UI-TARS)
2. Get your Browserbase API key and project ID from [Browserbase](https://www.browserbase.com)

Update `.env.local` with your API keys:

- `UI_TARS_BASE_URL`: Your UI-TARS Base Url
- `UI_TARS_API_KEY`: Your UI-TARS API Key
- `UI_TARS_MODEL`: Your UI-TARS Model
- `BROWSERBASE_API_KEY`: Your Browserbase API key
- `BROWSERBASE_PROJECT_ID`: Your Browserbase project ID

Then, run the development server:

<!-- This doesn't work with NPM, haven't tested with yarn -->

```bash
pnpm dev
```

Open [http://localhost:3000](http://localhost:3000) with your browser to see Open Operator in action.

## How It Works

Building a web agent is a complex task. You need to understand the user's intent, convert it into headless browser operations, and execute actions, each of which can be incredibly complex on their own.

![public/agent_mess.png](public/agent_mess.png)

Stagehand is a tool that helps you build web agents. It allows you to convert natural language into headless browser operations, execute actions on the browser, and extract results back into structured data.

![public/stagehand_clean.png](public/stagehand_clean.png)

Under the hood, we have a very simple agent loop that just calls Stagehand to convert the user's intent into headless browser operations, and then calls Browserbase to execute those operations.

![public/agent_loop.png](public/agent_loop.png)

Stagehand uses Browserbase to execute actions on the browser, and OpenAI to understand the user's intent.

For more on this, check out the code at [this commit](https://github.com/browserbase/open-operator/blob/6f2fba55b3d271be61819dc11e64b1ada52646ac/index.ts).

### Key Technologies

- **[Browserbase](https://www.browserbase.com)**: Powers the core browser automation and interaction capabilities
- **[Stagehand](https://www.stagehand.dev)**: Handles precise DOM manipulation and state management
- **[Next.js](https://nextjs.org)**: Provides the modern web framework foundation
- **[OpenAI](https://openai.com)**: Enable natural language understanding and decision making

## Contributing

We welcome contributions! Whether it's:

- Adding new features
- Improving documentation
- Reporting bugs
- Suggesting enhancements

Please feel free to open issues and pull requests.

## License

Open Operator is open source software licensed under the MIT license.

## Acknowledgments

This project is inspired by OpenAI's Operator feature and builds upon various open source technologies including Next.js, React, Browserbase, and Stagehand.


# @ui-tars/action-parser

## 1.2.0-beta.11

### Patch Changes

- Updated dependencies
  - @ui-tars/shared@1.2.0-beta.11

## 1.2.0-beta.10

### Patch Changes

- @ui-tars/shared@1.2.0-beta.10

## 1.2.0-beta.9

### Patch Changes

- bump: sdk support
- Updated dependencies
  - @ui-tars/shared@1.2.0-beta.9

## 1.2.0-beta.6

### Patch Changes

- feat: new sdk
- Updated dependencies
  - @ui-tars/shared@1.2.0-beta.6

## 1.2.0-beta.5

### Patch Changes

- chore: update sdk
- Updated dependencies
  - @ui-tars/shared@1.2.0-beta.5

## 1.2.0-beta.4

### Patch Changes

- chore: new version
- Updated dependencies
  - @ui-tars/shared@1.2.0-beta.4

## 1.2.0-beta.3

### Patch Changes

- chore: add retry
- Updated dependencies
  - @ui-tars/shared@1.2.0-beta.3

## 1.2.0-beta.2

### Patch Changes

- chore: publish
- Updated dependencies
  - @ui-tars/shared@1.2.0-beta.2

## 1.2.0-beta.1

### Patch Changes

- chore: remove unused code
- Updated dependencies
  - @ui-tars/shared@1.2.0-beta.1

## 1.2.0-beta.0

### Minor Changes

- a062e03: feat: ui-tars agent sdk support

### Patch Changes

- Updated dependencies [a062e03]
  - @ui-tars/shared@1.2.0-beta.0

## 1.1.0

### Minor Changes

- fix(actionParser): no action text return null not error

### Patch Changes

- Updated dependencies
  - @ui-tars/shared@1.1.0

## 1.0.1

### Patch Changes

- a1101ca: fix(action_parser): null values while parsing action inputs (#28)
- Updated dependencies [a1101ca]
  - @ui-tars/shared@1.0.1

## 1.0.0

### Major Changes

- dcabd45: feat: initial publish ui-tars action-parser and shared pkgs

### Patch Changes

- Updated dependencies [dcabd45]
  - @ui-tars/shared@1.0.0


# @ui-tars/cli

## 1.2.0-beta.13

### Patch Changes

- chore: open-operator
- Updated dependencies
  - @ui-tars/operator-nut-js@1.2.0-beta.12
  - @ui-tars/sdk@1.2.0-beta.12

## 1.2.0-beta.12

### Patch Changes

- Updated dependencies
  - @ui-tars/operator-nut-js@1.2.0-beta.11
  - @ui-tars/sdk@1.2.0-beta.11

## 1.2.0-beta.11

### Patch Changes

- chore: node-fetch

## 1.2.0-beta.10

### Patch Changes

- update
- Updated dependencies
  - @ui-tars/sdk@1.2.0-beta.10
  - @ui-tars/operator-nut-js@1.2.0-beta.10

## 1.2.0-beta.9

### Patch Changes

- bump: sdk support
- Updated dependencies
  - @ui-tars/operator-nut-js@1.2.0-beta.9
  - @ui-tars/sdk@1.2.0-beta.9

## 1.2.0-beta.8

### Patch Changes

- fix: useConfig to useContext
- Updated dependencies
  - @ui-tars/operator-nut-js@1.2.0-beta.8
  - @ui-tars/sdk@1.2.0-beta.8

## 1.2.0-beta.7

### Patch Changes

- Updated dependencies
  - @ui-tars/sdk@1.2.0-beta.7
  - @ui-tars/operator-nut-js@1.2.0-beta.7

## 1.2.0-beta.6

### Patch Changes

- feat: new sdk
- Updated dependencies
  - @ui-tars/operator-nut-js@1.2.0-beta.6
  - @ui-tars/sdk@1.2.0-beta.6

## 1.2.0-beta.5

### Patch Changes

- chore: update sdk
- Updated dependencies
  - @ui-tars/operator-nut-js@1.2.0-beta.5
  - @ui-tars/sdk@1.2.0-beta.5

## 1.2.0-beta.4

### Patch Changes

- chore: new version
- Updated dependencies
  - @ui-tars/operator-nut-js@1.2.0-beta.4
  - @ui-tars/sdk@1.2.0-beta.4

## 1.2.0-beta.3

### Patch Changes

- chore: add retry
- Updated dependencies
  - @ui-tars/operator-nut-js@1.2.0-beta.3
  - @ui-tars/sdk@1.2.0-beta.3

## 1.2.0-beta.2

### Patch Changes

- chore: publish
- Updated dependencies
  - @ui-tars/operator-nut-js@1.2.0-beta.2
  - @ui-tars/sdk@1.2.0-beta.2

## 1.2.0-beta.1

### Patch Changes

- chore: remove unused code
- Updated dependencies
  - @ui-tars/operator-nut-js@1.2.0-beta.1
  - @ui-tars/sdk@1.2.0-beta.1

## 1.2.0-beta.0

### Minor Changes

- a062e03: feat: ui-tars agent sdk support

### Patch Changes

- Updated dependencies [a062e03]
  - @ui-tars/operator-nut-js@1.2.0-beta.0
  - @ui-tars/sdk@1.2.0-beta.0


# @ui-tars/electron-ipc

## 1.2.0-beta.10

## 1.2.0-beta.9

### Patch Changes

- bump: sdk support

## 1.2.0-beta.6

### Patch Changes

- feat: new sdk

## 1.2.0-beta.5

### Patch Changes

- chore: update sdk

## 1.2.0-beta.4

### Patch Changes

- chore: new version

## 1.2.0-beta.3

### Patch Changes

- chore: add retry

## 1.2.0-beta.2

### Patch Changes

- chore: publish

## 1.2.0-beta.1

### Patch Changes

- chore: remove unused code

## 1.2.0-beta.0

### Minor Changes

- a062e03: feat: ui-tars agent sdk support


# UI-TARS Electron IPC Handlers

A type-safe IPC (Inter-Process Communication) solution for Electron applications.

## Features

- Full TypeScript support with end-to-end type safety
- Zod schema validation support
- Simple and intuitive API
- Server-side direct invocation support

## Installation

```bash
npm install @ui-tars/electron-ipc
```

## Usage

### Define Your Router

```ts
// router.ts
import { initIpc } from '@ui-tars/electron-ipc/main';
import { z } from 'zod';

const t = initIpc.create();

export const router = t.router({
  // Basic procedure without Zod
  hello: t.procedure.input<{ a: string }>().handle(async ({ input }) => {
    return 'hello' + input.a;
  }),

  // Procedure with Zod schema validation
  world: t.procedure.input(z.object({ a: z.string() })).handle(async ({ input }) => {
    return input.a;
  })
});

// Export router type for client usage
export type AppRouter = typeof router;
```

### Main Process Setup

```ts
// main.ts
import { registerIpcMain, createServer } from '@ui-tars/electron-ipc/main';
import { router } from './router';

// Register IPC handlers
registerIpcMain(router);

// Optional: Create server instance for direct invocation in main process
const server = createServer(router);
await server.hello({ a: '123' }); // => 'hello123'
```

### Renderer Process Usage

```ts
// renderer.ts
import { createClient } from '@ui-tars/electron-ipc/renderer';
import type { AppRouter } from './router';

const client = createClient<AppRouter>({
  ipcInvoke: window.Electron.ipcRenderer.invoke,
});

// Call procedures from renderer process
await client.hello({ a: '123' }); // => 'hello123'
```

## API Reference

### Main Process

#### `initIpc.create()`
Creates a new IPC router builder instance.

#### `registerIpcMain(router)`
Registers IPC handlers for the main process.

#### `createServer(router)`
Creates a server instance for direct invocation in the main process.

### Renderer Process

#### `createClient<Router>(options)`
Creates a type-safe client for calling IPC procedures from the renderer process.

## Type Safety

The library provides full type safety between your main and renderer processes:
- Input types are validated at compile time
- Return types are properly inferred
- Zod schema validation provides runtime type safety

## License

Apache-2.0


# @ui-tars/sdk

## 1.2.0-beta.12

### Patch Changes

- chore: open-operator
- Updated dependencies
  - @ui-tars/shared@1.2.0-beta.11
  - @ui-tars/action-parser@1.2.0-beta.11

## 1.2.0-beta.11

### Patch Changes

- chore: types

## 1.2.0-beta.10

### Patch Changes

- update
  - @ui-tars/action-parser@1.2.0-beta.10
  - @ui-tars/shared@1.2.0-beta.10

## 1.2.0-beta.9

### Patch Changes

- bump: sdk support
- Updated dependencies
  - @ui-tars/action-parser@1.2.0-beta.9
  - @ui-tars/shared@1.2.0-beta.9


# @ui-tars/shared

## 1.2.0-beta.11

### Patch Changes

- chore: open-operator

## 1.2.0-beta.10

## 1.2.0-beta.9

### Patch Changes

- bump: sdk support

## 1.2.0-beta.6

### Patch Changes

- feat: new sdk

## 1.2.0-beta.5

### Patch Changes

- chore: update sdk

## 1.2.0-beta.4

### Patch Changes

- chore: new version

## 1.2.0-beta.3

### Patch Changes

- chore: add retry

## 1.2.0-beta.2

### Patch Changes

- chore: publish

## 1.2.0-beta.1

### Patch Changes

- chore: remove unused code

## 1.2.0-beta.0

### Minor Changes

- a062e03: feat: ui-tars agent sdk support

## 1.1.0

### Minor Changes

- fix(actionParser): no action text return null not error

## 1.0.1

### Patch Changes

- a1101ca: fix(action_parser): null values while parsing action inputs (#28)

## 1.0.0

### Major Changes

- dcabd45: feat: initial publish ui-tars action-parser and shared pkgs


# @ui-tars/utio

## 1.2.0-beta.10

## 1.2.0-beta.9

### Patch Changes

- bump: sdk support

## 1.2.0-beta.6

### Patch Changes

- feat: new sdk

## 1.2.0-beta.5

### Patch Changes

- chore: update sdk

## 1.1.0-beta.4

### Patch Changes

- chore: new version

## 1.1.0-beta.3

### Patch Changes

- chore: add retry

## 1.1.0-beta.2

### Patch Changes

- chore: publish

## 1.1.0-beta.1

### Patch Changes

- chore: remove unused code

## 1.1.0-beta.0

### Minor Changes

- a062e03: feat: ui-tars agent sdk support


## Documentation

Fork from [@midscene/visualizer](https://github.com/web-infra-dev/midscene/tree/main/packages/visualizer)

## License

Midscene is MIT licensed.


# @ui-tars/operator-browserbase

## 1.2.0-beta.12

### Patch Changes

- chore: open-operator
- Updated dependencies
  - @ui-tars/shared@1.2.0-beta.11
  - @ui-tars/sdk@1.2.0-beta.12

## 1.2.0-beta.11

### Patch Changes

- chore: types
- Updated dependencies
  - @ui-tars/sdk@1.2.0-beta.11


# @ui-tars/operator-browserbase

Operator Browserbase SDK for UI-TARS.

See [example](https://github.com/bytedance/UI-TARS-desktop/tree/main/operator-browserbase) for usage.


# @ui-tars/operator-nut-js

## 1.2.0-beta.12

### Patch Changes

- chore: open-operator
- Updated dependencies
  - @ui-tars/shared@1.2.0-beta.11
  - @ui-tars/sdk@1.2.0-beta.12

## 1.2.0-beta.11

### Patch Changes

- chore: types
- Updated dependencies
  - @ui-tars/sdk@1.2.0-beta.11

## 1.2.0-beta.10

### Patch Changes

- Updated dependencies
  - @ui-tars/sdk@1.2.0-beta.10
  - @ui-tars/shared@1.2.0-beta.10

## 1.2.0-beta.9

### Patch Changes

- bump: sdk support
- Updated dependencies
  - @ui-tars/sdk@1.2.0-beta.9
  - @ui-tars/shared@1.2.0-beta.9

## 1.2.0-beta.8

### Patch Changes

- fix: useConfig to useContext
- Updated dependencies
  - @ui-tars/sdk@1.2.0-beta.8

## 1.2.0-beta.7

### Patch Changes

- Updated dependencies
  - @ui-tars/sdk@1.2.0-beta.7

## 1.2.0-beta.6

### Patch Changes

- feat: new sdk
- Updated dependencies
  - @ui-tars/shared@1.2.0-beta.6
  - @ui-tars/sdk@1.2.0-beta.6

## 1.2.0-beta.5

### Patch Changes

- chore: update sdk
- Updated dependencies
  - @ui-tars/shared@1.2.0-beta.5
  - @ui-tars/sdk@1.2.0-beta.5

## 1.2.0-beta.4

### Patch Changes

- chore: new version
- Updated dependencies
  - @ui-tars/shared@1.2.0-beta.4
  - @ui-tars/sdk@1.2.0-beta.4

## 1.2.0-beta.3

### Patch Changes

- chore: add retry
- Updated dependencies
  - @ui-tars/shared@1.2.0-beta.3
  - @ui-tars/sdk@1.2.0-beta.3

## 1.2.0-beta.2

### Patch Changes

- chore: publish
- Updated dependencies
  - @ui-tars/shared@1.2.0-beta.2
  - @ui-tars/sdk@1.2.0-beta.2

## 1.2.0-beta.1

### Patch Changes

- chore: remove unused code
- Updated dependencies
  - @ui-tars/shared@1.2.0-beta.1
  - @ui-tars/sdk@1.2.0-beta.1

## 1.2.0-beta.0

### Minor Changes

- a062e03: feat: ui-tars agent sdk support

### Patch Changes

- Updated dependencies [a062e03]
  - @ui-tars/shared@1.2.0-beta.0
  - @ui-tars/sdk@1.2.0-beta.0


# @ui-tars/operator-nut-js

Operator Nut JS SDK for UI-TARS.


# 🦙🎧 LLaMA-Omni: Seamless Speech Interaction with Large Language Models

> **Authors: [Qingkai Fang](https://fangqingkai.github.io/), [Shoutao Guo](https://scholar.google.com/citations?hl=en&user=XwHtPyAAAAAJ), [Yan Zhou](https://zhouyan19.github.io/zhouyan/), [Zhengrui Ma](https://scholar.google.com.hk/citations?user=dUgq6tEAAAAJ), [Shaolei Zhang](https://zhangshaolei1998.github.io/), [Yang Feng*](https://people.ucas.edu.cn/~yangfeng?language=en)**

[![arXiv](https://img.shields.io/badge/arXiv-2409.06666-b31b1b.svg?logo=arXiv)](https://arxiv.org/abs/2409.06666)
[![code](https://img.shields.io/badge/Github-Code-keygen.svg?logo=github)](https://github.com/ictnlp/LLaMA-Omni)
[![model](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging_Face-Model-blue.svg)](https://huggingface.co/ICTNLP/Llama-3.1-8B-Omni)
[![ModelScope](https://img.shields.io/badge/ModelScope-Model-blue.svg)](https://modelscope.cn/models/ICTNLP/Llama-3.1-8B-Omni)
[![Wisemodel](https://img.shields.io/badge/Wisemodel-Model-blue.svg)](https://www.wisemodel.cn/models/ICT_NLP/Llama-3.1-8B-Omni/)
[![Replicate](https://replicate.com/ictnlp/llama-omni/badge)](https://replicate.com/ictnlp/llama-omni)

LLaMA-Omni is a speech-language model built upon Llama-3.1-8B-Instruct. It supports low-latency and high-quality speech interactions, simultaneously generating both text and speech responses based on speech instructions.

## 💡 Highlights

- 💪 **Built on Llama-3.1-8B-Instruct, ensuring high-quality responses.**

- 🚀 **Low-latency speech interaction with a latency as low as 226ms.**

- 🎧 **Simultaneous generation of both text and speech responses.**

- ♻ **Trained in less than 3 days using just 4 GPUs.**

https://github.com/user-attachments/assets/2b097af8-47d7-494f-b3b3-6be17ca0247a

## Install

1. Clone this repository.

```shell
git clone https://github.com/ictnlp/LLaMA-Omni
cd LLaMA-Omni
```

2. Install packages.

```shell
conda create -n llama-omni python=3.10
conda activate llama-omni
pip install pip==24.0
pip install -e .
```

3. Install `fairseq`.

```shell
git clone https://github.com/pytorch/fairseq
cd fairseq
pip install -e . --no-build-isolation
```

4. Install `flash-attention`.

```shell
pip install flash-attn --no-build-isolation
```

## Quick Start

1. Download the `Llama-3.1-8B-Omni` model from 🤗[Huggingface](https://huggingface.co/ICTNLP/Llama-3.1-8B-Omni).

2. Download the `Whisper-large-v3` model.

```shell
import whisper
model = whisper.load_model("large-v3", download_root="models/speech_encoder/")
```

3. Download the unit-based HiFi-GAN vocoder.

```shell
wget https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/vocoder/code_hifigan/mhubert_vp_en_es_fr_it3_400k_layer11_km1000_lj/g_00500000 -P vocoder/
wget https://dl.fbaipublicfiles.com/fairseq/speech_to_speech/vocoder/code_hifigan/mhubert_vp_en_es_fr_it3_400k_layer11_km1000_lj/config.json -P vocoder/
```

## Gradio Demo

1. Launch a controller.
```shell
python -m omni_speech.serve.controller --host 0.0.0.0 --port 10000
```

2. Launch a gradio web server.
```shell
python -m omni_speech.serve.gradio_web_server --controller http://localhost:10000 --port 8000 --model-list-mode reload --vocoder vocoder/g_00500000 --vocoder-cfg vocoder/config.json
```

3. Launch a model worker.
```shell
python -m omni_speech.serve.model_worker --host 0.0.0.0 --controller http://localhost:10000 --port 40000 --worker http://localhost:40000 --model-path Llama-3.1-8B-Omni --model-name Llama-3.1-8B-Omni --s2s
```

4. Visit [http://localhost:8000/](http://localhost:8000/) and interact with LLaMA-3.1-8B-Omni!

**Note: Due to the instability of streaming audio playback in Gradio, we have only implemented streaming audio synthesis without enabling autoplay. If you have a good solution, feel free to submit a PR. Thanks!**

## Local Inference

To run inference locally, please organize the speech instruction files according to the format in the `omni_speech/infer/examples` directory, then refer to the following script.
```shell
bash omni_speech/infer/run.sh omni_speech/infer/examples
```

## LICENSE

Our code is released under the Apache-2.0 License. Our model is intended for academic research purposes only and may **NOT** be used for commercial purposes.

You are free to use, modify, and distribute this model in academic settings, provided that the following conditions are met:

- **Non-commercial use**: The model may not be used for any commercial purposes.
- **Citation**: If you use this model in your research, please cite the original work.

### Commercial Use Restriction

For any commercial use inquiries or to obtain a commercial license, please contact `fengyang@ict.ac.cn`.

## Acknowledgements

- [LLaVA](https://github.com/haotian-liu/LLaVA): The codebase we built upon.
- [SLAM-LLM](https://github.com/X-LANCE/SLAM-LLM): We borrow some code about speech encoder and speech adaptor.

## Citation

If you have any questions, please feel free to submit an issue or contact `fangqingkai21b@ict.ac.cn`.

If our work is useful for you, please cite as:

```
@article{fang-etal-2024-llama-omni,
  title={LLaMA-Omni: Seamless Speech Interaction with Large Language Models},
  author={Fang, Qingkai and Guo, Shoutao and Zhou, Yan and Ma, Zhengrui and Zhang, Shaolei and Feng, Yang},
  journal={arXiv preprint arXiv:2409.06666},
  year={2024}
}
```

## Star History

[![Star History Chart](https://api.star-history.com/svg?repos=ictnlp/llama-omni&type=Date)](https://star-history.com/#ictnlp/llama-omni&Date)

README.md

LLaMA-Omni is a low-latency and high-quality end-to-end speech interaction model built upon Llama-3.1-8B-Instruct, aiming to achieve speech capabilities at the GPT-4o level.


# ⚡Pyramid Flow⚡

[[Paper]](https://arxiv.org/abs/2410.05954) [[Project Page ✨]](https://pyramid-flow.github.io) [[miniFLUX Model 🚀]](https://huggingface.co/rain1011/pyramid-flow-miniflux) [[SD3 Model ⚡]](https://huggingface.co/rain1011/pyramid-flow-sd3) [[demo 🤗](https://huggingface.co/spaces/Pyramid-Flow/pyramid-flow)]

This is the official repository for Pyramid Flow, a training-efficient **Autoregressive Video Generation** method based on **Flow Matching**. By training only on **open-source datasets**, it can generate high-quality 10-second videos at 768p resolution and 24 FPS, and naturally supports image-to-video generation.

  10s, 768p, 24fps
  5s, 768p, 24fps
  Image-to-video

  <video src="https://github.com/user-attachments/assets/9935da83-ae56-4672-8747-0f46e90f7b2b" autoplay muted loop playsinline></video>
  <video src="https://github.com/user-attachments/assets/3412848b-64db-4d9e-8dbf-11403f6d02c5" autoplay muted loop playsinline></video>
  <video src="https://github.com/user-attachments/assets/3bd7251f-7b2c-4bee-951d-656fdb45f427" autoplay muted loop playsinline></video>

## News
* `2024.11.13`  🚀🚀🚀 We release the [768p miniFLUX checkpoint](https://huggingface.co/rain1011/pyramid-flow-miniflux) (up to 10s).

  > We have switched the model structure from SD3 to a mini FLUX to fix human structure issues, please try our 1024p image checkpoint, 384p video checkpoint (up to 5s) and 768p video checkpoint (up to 10s). The new miniflux model shows great improvement on human structure and motion stability

* `2024.10.29` ⚡⚡⚡ We release [training code for VAE](#1-training-vae), [finetuning code for DiT](#2-finetuning-dit) and [new model checkpoints](https://huggingface.co/rain1011/pyramid-flow-miniflux) with FLUX structure trained from scratch.

* `2024.10.13`  ✨✨✨ [Multi-GPU inference](#3-multi-gpu-inference) and [CPU offloading](#cpu-offloading) are supported. Use it with **less than 8GB** of GPU memory, with great speedup on multiple GPUs.

* `2024.10.11`  🤗🤗🤗 [Hugging Face demo](https://huggingface.co/spaces/Pyramid-Flow/pyramid-flow) is available. Thanks [@multimodalart](https://huggingface.co/multimodalart) for the commit!

* `2024.10.10`  🚀🚀🚀 We release the [technical report](https://arxiv.org/abs/2410.05954), [project page](https://pyramid-flow.github.io) and [model checkpoint](https://huggingface.co/rain1011/pyramid-flow-sd3) of Pyramid Flow.

## Table of Contents

* [Introduction](#introduction)
* [Installation](#installation)
* [Inference](#inference)
  1. [Quick Start with Gradio](#1-quick-start-with-gradio)
  2. [Inference Code](#2-inference-code)
  3. [Multi-GPU Inference](#3-multi-gpu-inference)
  4. [Usage Tips](#4-usage-tips)
* [Training](#Training)
  1. [Training VAE](#training-vae)
  2. [Finetuning DiT](#finetuning-dit)
* [Gallery](#gallery)
* [Comparison](#comparison)
* [Acknowledgement](#acknowledgement)
* [Citation](#citation)

## Introduction

![motivation](assets/motivation.jpg)

Existing video diffusion models operate at full resolution, spending a lot of computation on very noisy latents. By contrast, our method harnesses the flexibility of flow matching ([Lipman et al., 2023](https://openreview.net/forum?id=PqvMRDCJT9t); [Liu et al., 2023](https://openreview.net/forum?id=XVjTT1nw5z); [Albergo & Vanden-Eijnden, 2023](https://openreview.net/forum?id=li7qeBbCR1t)) to interpolate between latents of different resolutions and noise levels, allowing for simultaneous generation and decompression of visual content with better computational efficiency. The entire framework is end-to-end optimized with a single DiT ([Peebles & Xie, 2023](http://openaccess.thecvf.com/content/ICCV2023/html/Peebles_Scalable_Diffusion_Models_with_Transformers_ICCV_2023_paper.html)), generating high-quality 10-second videos at 768p resolution and 24 FPS within 20.7k A100 GPU training hours.

## Installation

We recommend setting up the environment with conda. The codebase currently uses Python 3.8.10 and PyTorch 2.1.2 ([guide](https://pytorch.org/get-started/previous-versions/#v212)), and we are actively working to support a wider range of versions.

```bash
git clone https://github.com/jy0205/Pyramid-Flow
cd Pyramid-Flow

# create env using conda
conda create -n pyramid python==3.8.10
conda activate pyramid
pip install -r requirements.txt
```

Then, download the model from [Huggingface](https://huggingface.co/rain1011) (there are two variants: [miniFLUX](https://huggingface.co/rain1011/pyramid-flow-miniflux) or [SD3](https://huggingface.co/rain1011/pyramid-flow-sd3)). The miniFLUX models support 1024p image, 384p and 768p video generation, and the SD3-based models support 768p and 384p video generation. The 384p checkpoint generates 5-second video at 24FPS, while the 768p checkpoint generates up to 10-second video at 24FPS.

```python
from huggingface_hub import snapshot_download

model_path = 'PATH'   # The local directory to save downloaded checkpoint
snapshot_download("rain1011/pyramid-flow-miniflux", local_dir=model_path, local_dir_use_symlinks=False, repo_type='model')
```

## Inference

### 1. Quick start with Gradio

To get started, first install [Gradio](https://www.gradio.app/guides/quickstart), set your model path at [#L36](https://github.com/jy0205/Pyramid-Flow/blob/3777f8b84bddfa2aa2b497ca919b3f40567712e6/app.py#L36), and then run on your local machine:

```bash
python app.py
```

The Gradio demo will be opened in a browser. Thanks to [@tpc2233](https://github.com/tpc2233) the commit, see [#48](https://github.com/jy0205/Pyramid-Flow/pull/48) for details.

Or, try it out effortlessly on [Hugging Face Space 🤗](https://huggingface.co/spaces/Pyramid-Flow/pyramid-flow) created by [@multimodalart](https://huggingface.co/multimodalart). Due to GPU limits, this online demo can only generate 25 frames (export at 8FPS or 24FPS). Duplicate the space to generate longer videos.

#### Quick Start on Google Colab

To quickly try out Pyramid Flow on Google Colab, run the code below:

```
# Setup
!git clone https://github.com/jy0205/Pyramid-Flow
%cd Pyramid-Flow
!pip install -r requirements.txt
!pip install gradio

# This code downloads miniFLUX
from huggingface_hub import snapshot_download

model_path = '/content/Pyramid-Flow'
snapshot_download("rain1011/pyramid-flow-miniflux", local_dir=model_path, local_dir_use_symlinks=False, repo_type='model')

# Start
!python app.py
```

### 2. Inference Code

To use our model, please follow the inference code in `video_generation_demo.ipynb` at [this link](https://github.com/jy0205/Pyramid-Flow/blob/main/video_generation_demo.ipynb). We strongly recommend you to try the latest published pyramid-miniflux, which shows great improvement on human structure and motion stability. Set the param `model_name` to `pyramid_flux` to use. We further simplify it into the following two-step procedure. First, load the downloaded model:

```python
import torch
from PIL import Image
from pyramid_dit import PyramidDiTForVideoGeneration
from diffusers.utils import load_image, export_to_video

torch.cuda.set_device(0)
model_dtype, torch_dtype = 'bf16', torch.bfloat16   # Use bf16 (not support fp16 yet)

model = PyramidDiTForVideoGeneration(
    'PATH',                                         # The downloaded checkpoint dir
    model_name="pyramid_flux",
    model_dtype=model_dtype,
    model_variant='diffusion_transformer_768p',
)

model.vae.enable_tiling()
# model.vae.to("cuda")
# model.dit.to("cuda")
# model.text_encoder.to("cuda")

# if you're not using sequential offloading bellow uncomment the lines above ^
model.enable_sequential_cpu_offload()
```

Then, you can try text-to-video generation on your own prompts. Noting that the 384p version only support 5s now (set temp up to 16)!

```python
prompt = "A movie trailer featuring the adventures of the 30 year old space man wearing a red wool knitted motorcycle helmet, blue sky, salt desert, cinematic style, shot on 35mm film, vivid colors"

# used for 384p model variant
# width = 640
# height = 384

# used for 768p model variant
width = 1280
height = 768

with torch.no_grad(), torch.cuda.amp.autocast(enabled=True, dtype=torch_dtype):
    frames = model.generate(
        prompt=prompt,
        num_inference_steps=[20, 20, 20],
        video_num_inference_steps=[10, 10, 10],
        height=height,
        width=width,
        temp=16,                    # temp=16: 5s, temp=31: 10s
        guidance_scale=7.0,         # The guidance for the first frame, set it to 7 for 384p variant
        video_guidance_scale=5.0,   # The guidance for the other video latent
        output_type="pil",
        save_memory=True,           # If you have enough GPU memory, set it to `False` to improve vae decoding speed
    )

export_to_video(frames, "./text_to_video_sample.mp4", fps=24)
```

As an autoregressive model, our model also supports (text conditioned) image-to-video generation:

```python
# used for 384p model variant
# width = 640
# height = 384

# used for 768p model variant
width = 1280
height = 768

image = Image.open('assets/the_great_wall.jpg').convert("RGB").resize((width, height))
prompt = "FPV flying over the Great Wall"

with torch.no_grad(), torch.cuda.amp.autocast(enabled=True, dtype=torch_dtype):
    frames = model.generate_i2v(
        prompt=prompt,
        input_image=image,
        num_inference_steps=[10, 10, 10],
        temp=16,
        video_guidance_scale=4.0,
        output_type="pil",
        save_memory=True,           # If you have enough GPU memory, set it to `False` to improve vae decoding speed
    )

export_to_video(frames, "./image_to_video_sample.mp4", fps=24)
```

#### CPU offloading

We also support two types of CPU offloading to reduce GPU memory requirements. Note that they may sacrifice efficiency.
* Adding a `cpu_offloading=True` parameter to the generate function allows inference with **less than 12GB** of GPU memory. This feature was contributed by [@Ednaordinary](https://github.com/Ednaordinary), see [#23](https://github.com/jy0205/Pyramid-Flow/pull/23) for details.
* Calling `model.enable_sequential_cpu_offload()` before the above procedure allows inference with **less than 8GB** of GPU memory. This feature was contributed by [@rodjjo](https://github.com/rodjjo), see [#75](https://github.com/jy0205/Pyramid-Flow/pull/75) for details.

#### MPS backend

Thanks to [@niw](https://github.com/niw), Apple Silicon users (e.g. MacBook Pro with M2 24GB) can also try our model using the MPS backend! Please see [#113](https://github.com/jy0205/Pyramid-Flow/pull/113) for the details.

### 3. Multi-GPU Inference

For users with multiple GPUs, we provide an [inference script](https://github.com/jy0205/Pyramid-Flow/blob/main/scripts/inference_multigpu.sh) that uses sequence parallelism to save memory on each GPU. This also brings a big speedup, taking only 2.5 minutes to generate a 5s, 768p, 24fps video on 4 A100 GPUs (vs. 5.5 minutes on a single A100 GPU). Run it on 2 GPUs with the following command:

```bash
CUDA_VISIBLE_DEVICES=0,1 sh scripts/inference_multigpu.sh
```

It currently supports 2 or 4 GPUs (For SD3 Version), with more configurations available in the original script. You can also launch a [multi-GPU Gradio demo](https://github.com/jy0205/Pyramid-Flow/blob/main/scripts/app_multigpu_engine.sh) created by [@tpc2233](https://github.com/tpc2233), see [#59](https://github.com/jy0205/Pyramid-Flow/pull/59) for details.

  > Spoiler: We didn't even use sequence parallelism in training, thanks to our efficient pyramid flow designs.

### 4. Usage tips

* The `guidance_scale` parameter controls the visual quality. We suggest using a guidance within [7, 9] for the 768p checkpoint during text-to-video generation, and 7 for the 384p checkpoint.
* The `video_guidance_scale` parameter controls the motion. A larger value increases the dynamic degree and mitigates the autoregressive generation degradation, while a smaller value stabilizes the video.
* For 10-second video generation, we recommend using a guidance scale of 7 and a video guidance scale of 5.

## Training

### 1. Training VAE

The hardware requirements for training VAE are at least 8 A100 GPUs. Please refer to [this document](https://github.com/jy0205/Pyramid-Flow/blob/main/docs/VAE.md). This is a [MAGVIT-v2](https://arxiv.org/abs/2310.05737) like continuous 3D VAE, which should be quite flexible. Feel free to build your own video generative model on this part of VAE training code.

### 2. Finetuning DiT

The hardware requirements for finetuning DiT are at least 8 A100 GPUs. Please refer to [this document](https://github.com/jy0205/Pyramid-Flow/blob/main/docs/DiT.md). We provide instructions for both autoregressive and non-autoregressive versions of Pyramid Flow. The former is more research oriented and the latter is more stable (but less efficient without temporal pyramid).

## Gallery

The following video examples are generated at 5s, 768p, 24fps. For more results, please visit our [project page](https://pyramid-flow.github.io).

  <video src="https://github.com/user-attachments/assets/5b44a57e-fa08-4554-84a2-2c7a99f2b343" autoplay muted loop playsinline></video>
  <video src="https://github.com/user-attachments/assets/5afd5970-de72-40e2-900d-a20d18308e8e" autoplay muted loop playsinline></video>

  <video src="https://github.com/user-attachments/assets/1d44daf8-017f-40e9-bf18-1e19c0a8983b" autoplay muted loop playsinline></video>
  <video src="https://github.com/user-attachments/assets/7f5dd901-b7d7-48cc-b67a-3c5f9e1546d2" autoplay muted loop playsinline></video>

## Comparison

On VBench ([Huang et al., 2024](https://huggingface.co/spaces/Vchitect/VBench_Leaderboard)), our method surpasses all the compared open-source baselines. Even with only public video data, it achieves comparable performance to commercial models like Kling ([Kuaishou, 2024](https://kling.kuaishou.com/en)) and Gen-3 Alpha ([Runway, 2024](https://runwayml.com/research/introducing-gen-3-alpha)), especially in the quality score (84.74 vs. 84.11 of Gen-3) and motion smoothness.

![vbench](assets/vbench.jpg)

We conduct an additional user study with 20+ participants. As can be seen, our method is preferred over open-source models such as [Open-Sora](https://github.com/hpcaitech/Open-Sora) and [CogVideoX-2B](https://github.com/THUDM/CogVideo) especially in terms of motion smoothness.

![user_study](assets/user_study.jpg)

## Acknowledgement

We are grateful for the following awesome projects when implementing Pyramid Flow:

* [SD3 Medium](https://huggingface.co/stabilityai/stable-diffusion-3-medium) and [Flux 1.0](https://huggingface.co/black-forest-labs/FLUX.1-dev): State-of-the-art image generation models based on flow matching.
* [Diffusion Forcing](https://boyuan.space/diffusion-forcing) and [GameNGen](https://gamengen.github.io): Next-token prediction meets full-sequence diffusion.
* [WebVid-10M](https://github.com/m-bain/webvid), [OpenVid-1M](https://github.com/NJU-PCALab/OpenVid-1M) and [Open-Sora Plan](https://github.com/PKU-YuanGroup/Open-Sora-Plan): Large-scale datasets for text-to-video generation.
* [CogVideoX](https://github.com/THUDM/CogVideo): An open-source text-to-video generation model that shares many training details.
* [Video-LLaMA2](https://github.com/DAMO-NLP-SG/VideoLLaMA2): An open-source video LLM for our video recaptioning.

## Citation

Consider giving this repository a star and cite Pyramid Flow in your publications if it helps your research.
```
@article{jin2024pyramidal,
  title={Pyramidal Flow Matching for Efficient Video Generative Modeling},
  author={Jin, Yang and Sun, Zhicheng and Li, Ningyuan and Xu, Kun and Xu, Kun and Jiang, Hao and Zhuang, Nan and Huang, Quzhe and Song, Yang and Mu, Yadong and Lin, Zhouchen},
  jounal={arXiv preprint arXiv:2410.05954},
  year={2024}
}
```

README.md

Code of Pyramidal Flow Matching for Efficient Video Generative Modeling


# Pyramid Flow's DiT Finetuning Guide

This is the finetuning guide for the DiT in Pyramid Flow. We provide instructions for both autoregressive and non-autoregressive versions. The former is more research oriented and the latter is more stable (but less efficient without temporal pyramid). Please refer to [another document](https://github.com/jy0205/Pyramid-Flow/blob/main/docs/VAE) for VAE finetuning.

## Hardware Requirements

+ DiT finetuning: At least 8 A100 GPUs.

## Prepare the Dataset

The training dataset should be arranged into a json file, with `video`, `text` fields. Since the video vae latent extraction is very slow, we strongly recommend you to pre-extract the video vae latents to save the training time. We provide a video vae latent extraction script in folder `tools`. You can run it with the following command:

```bash
sh scripts/extract_vae_latent.sh
```

(optional) Since the T5 text encoder will cost a lot of GPU memory, pre-extract the text features will save the training memory. We also provide a text feature extraction script in folder `tools`. You can run it with the following command:

```bash
sh scripts/extract_text_feature.sh
```

The final training annotation json file should look like the following format:

```
{"video": video_path, "text": text prompt, "latent": extracted video vae latent, "text_fea": extracted text feature}
```

We provide the example json annotation files for [video](https://github.com/jy0205/Pyramid-Flow/blob/main/annotation/video_text.jsonl) and [image](https://github.com/jy0205/Pyramid-Flow/blob/main/annotation/image_text.jsonl)) training in the `annotation` folder. You can refer them to prepare your training dataset.

## Run Training
We provide two types of training scripts: (1) autoregressive video generation training with temporal pyramid. (2) Full-sequence diffusion training with pyramid-flow for both text-to-image and text-to-video training. This corresponds to the following two script files. Running these training scripts using at least 8 GPUs:

+ `scripts/train_pyramid_flow.sh`: The autoregressive video generation training with temporal pyramid.

```bash
sh scripts/train_pyramid_flow.sh
```

+ `scripts/train_pyramid_flow_without_ar.sh`: Using pyramid-flow for full-sequence diffusion training.

```bash
sh scripts/train_pyramid_flow_without_ar.sh
```

## Tips

+ For the 768p version, make sure to add the args:  `--gradient_checkpointing`
+ Param `NUM_FRAMES` should be set to a multiple of 8
+ For the param `video_sync_group`, it indicates the number of process that accepts the same input video, used for temporal pyramid AR training. We recommend to set this value to 4, 8 or 16. (16 is better if you have more GPUs)
+ Make sure to set `NUM_FRAMES % VIDEO_SYNC_GROUP == 0`, `GPUS % VIDEO_SYNC_GROUP == 0`, and `BATCH_SIZE % 4 == 0`

docs/DiT.md


# Pyramid Flow's VAE Training Guide

This is the training guide for a [MAGVIT-v2](https://arxiv.org/abs/2310.05737) like continuous 3D VAE, which should be quite flexible. Feel free to build your own video generative model on this part of VAE training code. Please refer to [another document](https://github.com/jy0205/Pyramid-Flow/blob/main/docs/DiT) for DiT finetuning.

## Hardware Requirements

+ VAE training: At least 8 A100 GPUs.

## Prepare the Dataset

The training of our causal video vae uses both image and video data. Both of them should be arranged into a json file, with `video` or `image` field. The final training annotation json file should look like the following format:

```
# For Video
{"video": video_path}

# For Image
{"image": image_path}
```

## Run Training

The causal video vae undergoes a two-stage training.
+ Stage-1: image and video mixed training
+ Stage-2: pure video training, using context parallel to load video with more video frames

The VAE training script is `scripts/train_causal_video_vae.sh`, run it as follows:

```bash
sh scripts/train_causal_video_vae.sh
```

We also provide a VAE demo `causal_video_vae_demo.ipynb` for image and video reconstruction.
> The original vgg lpips download URL is not available, I have shared the one we used in this [URL](https://drive.google.com/file/d/1YeFlX5BKKw-HGkjNd1r7DSwas1iJJwqC/view). You can download it and replace the LPIPS_CKPT with the correct path.

## Tips

+ For stage-1, we use a mixed image and video training. Add the param `--use_image_video_mixed_training` to support the mixed training. We set the image ratio to 0.1 by default.
+ Set the `resolution` to 256 is enough for VAE training.
+ For stage-1, the `max_frames` is set to 17. It means we use 17 sampled video frames for training.
+ For stage-2, we open the param `use_context_parallel` to distribute long video frames to multiple GPUs. Make sure to set `GPUS % CONTEXT_SIZE == 0` and `NUM_FRAMES=17 * CONTEXT_SIZE + 1`

docs/VAE.md


Pipet

a swiss-army tool for scraping and extracting data from online assets, made for hackers

Pipet is a command line based web scraper. It supports 3 modes of operation - HTML parsing, JSON parsing, and client-side JavaScript evaluation. It relies heavily on existing tools like curl, and it uses unix pipes for extending its built-in capabilities.

You can use Pipet to track a shipment, get notified when concert tickets are available, stock price changes, and any other kind of information that appears online.

# Try it out!
1. Create a `hackernews.pipet` file containing this:
```
curl https://news.ycombinator.com/
.title .titleline
  span > a
  .sitebit a
```
2. Run `go run github.com/bjesus/pipet/cmd/pipet@latest hackernews.pipet` or install Pipet and run `pipet hackernews.pipet`
3. See all of the latest hacker news in your terminal!

Use custom separators

Use the `--separator` (or `-s`) flag to specify custom separators for text output.  For example, run `pipet -s "\n" -s "->" hackernews.pipet` to see each item in a new line, with `->` between the title and the domain.
Get as JSON

Use the `--json` flag to make Pipet collect the results into a nice JSON.  For example, run `pipet --json hackernews.pipet` to get a JSON representation of the above results.
Render to a template

Add a template file called `hackernews.tpl` next to your `hackernews.pipet` file with this content:
```
<ul>
  {{range $index, $item := index (index . 0) 0}}
    <li>{{index $item 0}} ({{index $item 1}})</li>
  {{end}}
</ul>
```

Now run `pipet hackernews.pipet` again and Pipet will automatically detect your template file, and render the results to it.

Use pipes

Use Unix pipes after your queries, as if they were running in your shell. For example, count the characters in each title (with `wc`) and extract the full article URL (with [htmlq](https://github.com/mgdm/htmlq)):

```
curl https://news.ycombinator.com/
.title .titleline
  span > a
  span > a | wc -c
  .sitebit a
  .sitebit a | htmlq --attribute href a
```

Monitor for changes

Set an interval and a command to run on change, and have Pipet notify you when something happened. For example, get a notification whenever the Hacker News #1 story is different:

```
curl https://news.ycombinator.com/
.title .titleline a
```

Run it with `pipet --interval 60 --on-change "notify-send {}" hackernews.pipet`

# Installation

## Pre-built
Download the latest release from the [Releases](https://github.com/bjesus/pipet/releases/) page. `chmod +x pipet` and run `./pipet`.

## Compile
This installation method requires Go to be installed on your system.
You can use Go to install Pipet using `go install github.com/bjesus/pipet/cmd/pipet@latest`.  Otherwise you can run it without installing using `go run`.

## Distros
Packages are currently available for [Arch Linux](https://aur.archlinux.org/packages/pipet-git), [Homebrew](https://formulae.brew.sh/formula/pipet) and [Nix](https://search.nixos.org/packages?channel=unstable&show=pipet&from=0&size=50&sort=relevance&type=packages&query=pipet).

# Usage

The only required argument for Pipet is the path to your `.pipet` file. Other than this, the `pipet` command accepts the following flags:

- `--json`, `-j` - Output as JSON (default: false)
- `--template value`, `-t value` - Specify a path to a template file. You can also simply name the file like your `.pipet` file but with a `.tpl` extension for it to be auto-detected.
- `--separator value`, `-s value` - Set a separator for text output (can be used multiple times for setting different separators for different levels of data nesting)
- `---max-pages value`, `-p value` - Maximum number of pages to scrape (default: 3)
- `--interval value`, `-i value` - Rerun Pipet after X seconds. Use 0 to disable (default: 0)
- `--on-change value`, `-c value` - A command to run when the pipet result is new
- `--verbose`, `-v` - Enable verbose logging (default: false)
- `--version` - Print the Pipet version
- `--help`, `-h` - Show help

# Pipet files
Pipet files describe where and how to get the data you are interested in. They are normal text files containing one or more blocks separated by an empty line. Lines beginning with `//` are ignored and can be used for comments. Every block can have 3 sections:

1. **Resource** - The first line containing the URL and the tool we are using for scraping
2. **Queries** - The following lines describing the selectors reaching the data we would like scrap
3. **Next page** - An _optional_ last line starting with `>` describing the selector pointing to the "next page" of data

Below is an example Pipet file.

```
// Read Wikipedia's "On This Day" and the subject of today's featured article
curl https://en.wikipedia.org/wiki/Main_Page
div#mp-otd li
  body
div#mp-tfa > p > b > a

// Get the weather in Alert, Canada
curl https://wttr.in/Alert%20Canada?format=j1
current_condition.0.FeelsLikeC
current_condition.0.FeelsLikeF

// Check how popular the Pipet repo is
playwright https://github.com/bjesus/pipet
Array.from(document.querySelectorAll('.about-margin .Link')).map(e => e.innerText.trim()).filter(t=> /^\d/.test(t) )
```

##  Resource

Resource lines can start with either `curl` or `playwright`.

### curl

Resource lines starting with `curl` will be executed using curl. This is meant so that you can use your browser to find the request containing the information you are interested in, right click it, choose "Copy as cURL", and paste in your Pipet file. This ensures that your headers and cookies are all the same, making it very easy to get data that is behind a login page or hidden from bots. For example, this is a perfectly valid first line for a block: `curl 'https://news.ycombinator.com/' --compressed -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:131.0) Gecko/20100101 Firefox/131.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/png,image/svg+xml,*/*;q=0.8' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate, br, zstd' -H 'DNT: 1' -H 'Sec-GPC: 1' -H 'Connection: keep-alive' -H 'Upgrade-Insecure-Requests: 1' -H 'Sec-Fetch-Dest: document' -H 'Sec-Fetch-Mode: navigate' -H 'Sec-Fetch-Site: none' -H 'Sec-Fetch-User: ?1' -H 'Priority: u=0, i' -H 'Pragma: no-cache' -H 'Cache-Control: no-cache' -H 'TE: trailers'`.

### Playwright

Resource lines starting with `playwright` will use a headless browser to navigate to the specified URL. If you don't have a headless browser installed, Pipet will attempt to download one for you.

## Queries

Query lines define 3 things:
1. The way to the exact pieces of data you would like to extract (e.g. using CSS selectors)
2. The data structure your output will use (e.g. every title and URL should be grouped together by item)
3. The way the data will be processed (e.g. using Unix pipes) before it is printed

Pipet uses 3 different query types - for HTML, for JSON, and for when loading pages with Playwright.

### HTML Queries
HTML Queries use CSS Selectors to select specific elements. Whitespace nesting is used for iterations - parent lines will run as iterators, making their children lines run for each occurrence of the parent selector. This means that you can use nesting to determine the structure of your final output. See the following 3 examples:

Get only the first title and first URL

```
curl https://news.ycombinator.com/
.title .titleline > a
.sitebit a
```

Get all the titles, and then get all URLs

```
curl https://news.ycombinator.com/
.title .titleline
  span > a
.title .titleline
  .sitebit a
```

Get all the title and URL for each story

```
curl https://news.ycombinator.com/
.title .titleline
  span > a
  .sitebit a
```

When writing your child selectors, note that the whole document isn't available anymore. Pipet is passing only your parent HTML to the child iterations.

By default, Pipet will return the `innerText` of your elements. If you need to another piece of data, use Unix pipes. When piping HTML elements, Pipet will pipe the element's complete HTML. For example, you can use `| htmq --attr href a` to extract the `href` attribute from links.

### JSON Queries

JSON Queries use the [GJSON syntax](https://github.com/tidwall/gjson/blob/master/SYNTAX.md) to select specific elements. Here too, whitespace nesting is used for iterations - parent lines will run as iterators, making their children lines run for each occurrence of the parent selector. If you don't like GJSON, that's okay. For example, you can use `jq` by passing parts or the complete JSON to it using Unix pipes, like `@this | jq '.[].firstName'`.

When using pipes, Pipet will attempt to parse the returned string. If it's valid JSON, it will be parsed and injected as an object into the Pipet result.

Querying and jq usage

The example below will return the latest water temperature in Amsterdam NDSM, and then pipe the complete JSON to `jq` so it will combine the coordinates of the reading into one field.

```
curl https://waterinfo.rws.nl/api/detail/get?locationSlug=NDSM-werf-(o)(NDS1)&mapType=watertemperatuur
latest.data
@this | jq -r '"\(.coordinatex), \(.coordinatey)"'
```

Iterations

This will return times for bus deparatures. Note the two types of iterations - the first line is GJSON query that returns the `ExpectedDepartureTime` for each trip, while the the following lines iterates over each trip object using the nested lines below it, allowing us to return multiple keys - `ExpectedDepartureTime` & `TripStopStatus`.

```
curl http://v0.ovapi.nl/tpc/30005093
30005093.Passes.@values.#.ExpectedDepartureTime
30005093.Passes.@values
  ExpectedDepartureTime
  TripStopStatus
```

CSV export

We can create a simple CSV file by using the previous iteration and configurating a separator

```
curl http://v0.ovapi.nl/tpc/30005093
30005093.Passes.@values
  ExpectedDepartureTime
  TripStopStatus
```

Run using `pipet -s '\n' water.pipet > output.csv` to generate a CSV file.

### Playwright Queries

Playwright Queries are different and do not use whitespace nesting. Instead, queries here are simply JavaScript code that will be evaluated after the webpage loaded. If the JavaScript code returns something that can be serialized as JSON, it will be included in Pipet's output. Otherwise, you can write JavaScript that will click, scroll or perform any other action you might want.

Simple Playwright example

This example will return a string like `80 stars, 2 watching, 2 forks` after visiting the Pipet repo on Github.

```
playwright https://github.com/bjesus/pipet
Array.from(document.querySelectorAll('.about-margin .Link')).map(e => e.innerText.trim()).filter(t=> /^\d/.test(t) )
```

Note that if you copy the second line and paste it in your browser console while visiting https://github.com/bjesus/pipet, you'd get exactly the same result. The vice-versa is also true - if your code worked in the browser, it should work in Pipet too.

## Next page

The Next Page line lets you specify a CSS selector that will be used to determine the link to the next page of data. Pipet will then follow it and execute the same queries over it. For example, see this `hackernews.pipet` file:
```
curl https://news.ycombinator.com/
.title .titleline
  span > a
  .sitebit a
> a.morelink
```

The Next Page line is currently only available when working with `curl` and HTML files.

README.md

Swiss-army tool for scraping and extracting data from online assets, made for hackers


# LLM-engineer-handbook
🔥 Large Language Models(LLM) have taken the ~~NLP community~~ ~~AI community~~ **the Whole World** by storm.

Why do we create this repo?

- Everyone can now build an LLM demo in minutes, but it takes a real LLM/AI expert to close the last mile of performance, security, and scalability gaps.
- The LLM space is complicated! This repo provides a curated list to help you navigate so that you are more likely to build production-grade LLM applications. It includes a collection of Large Language Model frameworks and tutorials, covering model training, serving, fine-tuning, LLM applications & prompt optimization, and LLMOps.

*However, classical ML is not going away. Even LLMs need them. We have seen classical models used for protecting data privacy, detecing hallucinations, and more. So, do not forget to study the fundamentals of classical ML.*

## Overview

The current workflow might look like this: You build a demo using an existing application library or directly from LLM model provider SDKs. It works somehow, but you need to further create evaluation and training datasets to optimize the performance (e.g., accuracy, latency, cost).

You can do prompt engineering or auto-prompt optimization; you can create a larger dataset to fine-tune the LLM or use Direct Preference Optimization (DPO) to align the model with human preferences.
Then you need to consider the serving and LLMOps to deploy the model at scale and pipelines to refresh the data.

We organize the resources by (1) tracking all libraries, frameworks, and tools, (2) learning resources on the whole LLM lifecycle, (3) understanding LLMs, (4) social accounts and community, and (5) how to contribute to this repo.

- [LLM-engineer-handbook](#llm-engineer-handbook)
  - [Overview](#overview)
- [Libraries \& Frameworks \& Tools](#libraries--frameworks--tools)
  - [Applications](#applications)
  - [Pretraining](#pretraining)
  - [Fine-tuning](#fine-tuning)
  - [Serving](#serving)
  - [Prompt Management](#prompt-management)
  - [Datasets](#datasets)
  - [Benchmarks](#benchmarks)
- [Learning Resources for LLMs](#learning-resources-for-llms)
    - [Applications](#applications-1)
      - [Agent](#agent)
    - [Modeling](#modeling)
    - [Training](#training)
    - [Fine-tuning](#fine-tuning-1)
    - [Fundamentals](#fundamentals)
    - [Books](#books)
    - [Newsletters](#newsletters)
    - [Auto-optimization](#auto-optimization)
- [Understanding LLMs](#understanding-llms)
- [Social Accounts \& Community](#social-accounts--community)
  - [Social Accounts](#social-accounts)
  - [Community](#community)
- [Contributing](#contributing)

# Libraries & Frameworks & Tools
## Applications

**Build & Auto-optimize**

- [AdalFlow](https://github.com/SylphAI-Inc/AdalFlow) - The library to build & auto-optimize LLM applications, from Chatbot, RAG, to Agent by [SylphAI](https://www.sylph.ai/).

- [dspy](https://github.com/stanfordnlp/dspy) - DSPy: The framework for programming—not prompting—foundation models.

**Build**

- [LlamaIndex](https://github.com/jerryjliu/llama_index) — A Python library for augmenting LLM apps with data.
- [LangChain](https://github.com/hwchase17/langchain) — A popular Python/JavaScript library for chaining sequences of language model prompts.
- [Haystack](https://github.com/deepset-ai/haystack) — Python framework that allows you to build applications powered by LLMs.
- [Instill Core](https://github.com/instill-ai/instill-core) — A platform built with Go for orchestrating LLMs to create AI applications.

**Prompt Optimization**

- [AutoPrompt](https://github.com/Eladlev/AutoPrompt) - A framework for prompt tuning using Intent-based Prompt Calibration.
- [PromptFify](https://github.com/promptslab/Promptify) - A library for prompt engineering that simplifies NLP tasks (e.g., NER, classification) using LLMs like GPT.

**Others**

- [LiteLLM](https://github.com/BerriAI/litellm) - Python SDK, Proxy Server (LLM Gateway) to call 100+ LLM APIs in OpenAI format.

## Pretraining

- [PyTorch](https://pytorch.org/) - PyTorch is an open source machine learning library based on the Torch library, used for applications such as computer vision and natural language processing.
- [TensorFlow](https://www.tensorflow.org/) - TensorFlow is an open source machine learning library developed by Google.
- [JAX](https://github.com/jax-ml/jax) - Google’s library for high-performance computing and automatic differentiation.
- [tinygrad](https://github.com/tinygrad/tinygrad) - A minimalistic deep learning library with a focus on simplicity and educational use, created by George Hotz.
- [micrograd](https://github.com/karpathy/micrograd) - A simple, lightweight autograd engine for educational purposes, created by Andrej Karpathy.

## Fine-tuning

- [Transformers](https://huggingface.co/docs/transformers/en/installation) - Hugging Face Transformers is a popular library for Natural Language Processing (NLP) tasks, including fine-tuning large language models.
- [Unsloth](https://github.com/unslothai/unsloth) - Finetune Llama 3.2, Mistral, Phi-3.5 & Gemma 2-5x faster with 80% less memory!
- [LitGPT](https://github.com/Lightning-AI/litgpt) - 20+ high-performance LLMs with recipes to pretrain, finetune, and deploy at scale.
- [AutoTrain](https://github.com/huggingface/autotrain-advanced) - No code fine-tuning of LLMs and other machine learning tasks.

## Top Models
- [DeepSeek R1](https://github.com/deepseek-ai/DeepSeek-R1) - The most popular GPTo1 comparable open-source reasoning model. Read their technical report + check out their github.

## Serving

- [TorchServe](https://pytorch.org/serve/) - An open-source model serving library developed by AWS and Facebook specifically for PyTorch models, enabling scalable deployment, model versioning, and A/B testing.

- [TensorFlow Serving](https://www.tensorflow.org/tfx/guide/serving) - A flexible, high-performance serving system for machine learning models, designed for production environments, and optimized for TensorFlow models but also supports other formats.

- [Ray Serve](https://docs.ray.io/en/latest/serve/index.html) - Part of the Ray ecosystem, Ray Serve is a scalable model-serving library that supports deployment of machine learning models across multiple frameworks, with built-in support for Python-based APIs and model pipelines.

- [NVIDIA TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) - TensorRT-LLM is NVIDIA's compiler for transformer-based models (LLMs), providing state-of-the-art optimizations on NVIDIA GPUs.

- [NVIDIA Triton Inference Server](https://developer.nvidia.com/triton-inference-server) - A high-performance inference server supporting multiple ML/DL frameworks (TensorFlow, PyTorch, ONNX, TensorRT etc.), optimized for NVIDIA GPU deployments, and ideal for both cloud and on-premises serving.

- [ollama](https://github.com/ollama/ollama) - A lightweight, extensible framework for building and running large language models on the local machine.

- [llama.cpp](https://github.com/ggerganov/llama.cpp) - A library for running LLMs in pure C/C++. Supported architectures include (LLaMA, Falcon, Mistral, MoEs, phi and more)

- [TGI](https://github.com/huggingface/text-generation-inference) - HuggingFace's text-generation-inference toolkit for deploying and serving LLMs, built on top of Rust, Python and gRPC.

- [vllm](https://github.com/vllm-project/vllm) - An optimized, high-throughput serving engine for large language models, designed to efficiently handle massive-scale inference with reduced latency.

- [sglang](https://github.com/sgl-project/sglang) - SGLang is a fast serving framework for large language models and vision language models.

- [LitServe](https://github.com/Lightning-AI/LitServe) - LitServe is a lightning-fast serving engine for any AI model of any size. Flexible. Easy. Enterprise-scale.

## Prompt Management

- [Opik](https://github.com/comet-ml/opik) - Opik is an open-source platform for evaluating, testing and monitoring LLM applications

## Datasets

Use Cases

- [Datasets](https://huggingface.co/docs/datasets/en/index) - A vast collection of ready-to-use datasets for machine learning tasks, including NLP, computer vision, and audio, with tools for easy access, filtering, and preprocessing.
- [Argilla](https://github.com/argilla-io/argilla) - A UI tool for curating and reviewing datasets for LLM evaluation or training.
- [distilabel](https://distilabel.argilla.io/latest/) - A library for generating synthetic datasets with LLM APIs or models.

Fine-tuning

- [LLMDataHub](https://github.com/Zjh-819/LLMDataHub) - A quick guide (especially) for trending instruction finetuning datasets
- [LLM Datasets](https://github.com/mlabonne/llm-datasets) - High-quality datasets, tools, and concepts for LLM fine-tuning.

Pretraining

- [IBM LLMs Granite 3.0](https://www.linkedin.com/feed/update/urn:li:activity:7259535100927725569?updateEntityUrn=urn%3Ali%3Afs_updateV2%3A%28urn%3Ali%3Aactivity%3A7259535100927725569%2CFEED_DETAIL%2CEMPTY%2CDEFAULT%2Cfalse%29) - Full list of datasets used to train IBM LLMs Granite 3.0

## Benchmarks

- [lighteval](https://github.com/huggingface/lighteval) - A library for evaluating local LLMs on major benchmarks and custom tasks.

- [evals](https://github.com/openai/evals) - OpenAI's open sourced evaluation framework for LLMs and systems built with LLMs.
- [ragas](https://github.com/explodinggradients/ragas) - A library for evaluating and optimizing LLM applications, offering a rich set of eval metrics.

Agent

- [TravelPlanner](https://osu-nlp-group.github.io/TravelPlanner/) - [paper](https://arxiv.org/pdf/2402.01622) A Benchmark for Real-World Planning with Language Agents.

# Learning Resources for LLMs

We will categorize the best resources to learn LLMs, from modeling to training, and applications.

### Applications

General

- [AdalFlow documentation](https://adalflow.sylph.ai/) - Includes tutorials from building RAG, Agent, to LLM evaluation and fine-tuning.

- [CS224N](https://www.youtube.com/watch?v=rmVRLeJRkl4) - Stanford course covering NLP fundamentals, LLMs, and PyTorch-based model building, led by Chris Manning and Shikhar Murty.

- [LLM-driven Data Engineering](https://github.com/DataExpert-io/llm-driven-data-engineering) -  A playlist of 6 lectures by [Zach Wilson](https://www.linkedin.com/in/eczachly) on how LLMs will impact data pipeline development

- [LLM Course by Maxime Labonne](https://github.com/mlabonne/llm-course) - An end-to-end course for AI and ML engineers on open source LLMs.

#### Agent

Lectures

- [LLM Agents MOOC](https://youtube.com/playlist?list=PLS01nW3RtgopsNLeM936V4TNSsvvVglLc&si=LAonD5VfG9jFAOuE) - A playlist of 11 lectures by the Berkeley RDI Center on Decentralization & AI, featuring guest speakers like Yuandong Tian, Graham Neubig, Omar Khattab, and others, covering core topics on Large Language Model agents. [CS294](https://rdi.berkeley.edu/llm-agents/f24)

Projects

- [OpenHands](https://github.com/All-Hands-AI/OpenHands) - Open source agents for developers by [AllHands](https://www.all-hands.dev/).
- [CAMEL](https://github.com/camel-ai/camel) - First LLM multi-agent framework and an open-source community dedicated to finding the scaling law of agents. by [CAMEL-AI](https://www.camel-ai.org/).
- [swarm](https://github.com/openai/swarm) - Educational framework exploring ergonomic, lightweight multi-agent orchestration. Managed by OpenAI Solution team.
- [AutoGen](https://github.com/microsoft/autogen) - A programming framework for agentic AI 🤖 by Microsoft.
- [CrewAI](https://github.com/crewAIInc/crewAI) - 🤖 CrewAI: Cutting-edge framework for orchestrating role-playing, autonomous AI agents.
- [TinyTroupe](https://github.com/microsoft/TinyTroupe) - Simulates customizable personas using GPT-4 for testing, insights, and innovation by Microsoft.

### Modeling

- [Llama3 from scratch](https://github.com/naklecha/llama3-from-scratch) - llama3 implementation one matrix multiplication at a time with PyTorch.
- [Interactive LLM visualization](https://github.com/bbycroft/llm-viz) - An interactive visualization of transformers. [Visualizer](https://bbycroft.net/llm)
- [3Blue1Brown transformers visualization](https://www.youtube.com/watch?v=wjZofJX0v4M) - 3Blue1Brown's video on how transformers work.
- [Self-Attention explained as directed graph](https://x.com/akshay_pachaar/status/1853474819523965088) - An X post explaining self-attention as a directed graph by Akshay Pachaar.

### Training
- [HuggingFace's SmolLM & SmolLM2 training release](https://huggingface.co/blog/smollm) - HuggingFace's sharing on data curation methods, processed data, training recipes, and all of their code. [Github repo](https://github.com/huggingface/smollm?tab=readme-ov-file).
- [Lil'Log](https://lilianweng.github.io/) - Lilian Weng(OpenAI)'s blog on machine learning, deep learning, and AI, with a focus on LLMs and NLP.
- [Chip's Blog](https://huyenchip.com/blog/) - Chip Huyen's blog on training LLMs, including the latest research, tutorials, and best practices.

### Fine-tuning

- [DPO](https://arxiv.org/abs/2305.18290): Rafailov, Rafael, et al. "Direct preference optimization: Your language model is secretly a reward model." Advances in Neural Information Processing Systems 36 (2024). [Code](https://github.com/eric-mitchell/direct-preference-optimization).

### Fundamentals
- [Intro to LLMs](https://www.youtube.com/watch?v=zjkBMFhNj_g&t=1390s&ab_channel=AndrejKarpathy) - A 1 hour general-audience introduction to Large Language Models by Andrej Karpathy.
- [Building GPT-2 from Scratch](https://www.youtube.com/watch?v=l8pRSuU81PU&t=1564s&ab_channel=AndrejKarpathy) - A 4 hour deep dive into building GPT2 from scratch by Andrej Karpathy.

### Books
- [LLM Engineer's Handbook: Master the art of engineering large language models from concept to production](https://www.amazon.com/dp/1836200072?ref=cm_sw_r_cp_ud_dp_ZFR4XZPT7EY41ZE1M5X9&ref_=cm_sw_r_cp_ud_dp_ZFR4XZPT7EY41ZE1M5X9&social_share=cm_sw_r_cp_ud_dp_ZFR4XZPT7EY41ZE1M5X9) by   Paul Iusztin , Maxime Labonne. Covers mostly the lifecycle of LLMs, including LLMOps on pipelines, deployment, monitoring, and more. [Youtube overview by Paul](https://www.youtube.com/live/6WmPfKPmoz0).
- [Build a Large Language Model from Scratch](https://www.manning.com/books/build-a-large-language-model-from-scratch) by Sebastian Raschka
- [Hands-On Large Language Models: Build, Tune, and Apply LLMs](https://www.amazon.com/Hands-Large-Language-Models-Understanding/dp/1098150961) by Jay Alammar , Maarten Grootendorst
- [Generative Deep Learning - Teaching machines to Paint, Write, Compose and Play](https://www.amazon.com/Generative-Deep-Learning-Teaching-Machines/dp/1492041947) by David Foster

### Newsletters

- [Ahead of AI](https://magazine.sebastianraschka.com/) - Sebastian Raschka's Newsletter, covering end-to-end LLMs understanding.
- [Decoding ML](https://decodingml.substack.com/) - Content on building production GenAI, RecSys and MLOps applications.

### Auto-optimization

- [TextGrad](https://github.com/zou-group/textgrad) - Automatic ''Differentiation'' via Text -- using large language models to backpropagate textual gradients.

# Understanding LLMs

It can be fun and important to understand the capabilities, behaviors, and limitations of LLMs. This can directly help with prompt engineering.

In-context Learning

- [Brown, Tom B. "Language models are few-shot learners." arXiv preprint arXiv:2005.14165 (2020).](https://rosanneliu.com/dlctfs/dlct_200724.pdf)

Reasoning & Planning

- [Kambhampati, Subbarao, et al. "LLMs can't plan, but can help planning in LLM-modulo frameworks." arXiv preprint arXiv:2402.01817 (2024).](https://arxiv.org/abs/2402.01817)
- [Mirzadeh, Iman, et al. "Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models." arXiv preprint arXiv:2410.05229 (2024).](https://arxiv.org/abs/2410.05229) By Apple.

# Social Accounts & Community
## Social Accounts

Social accounts are the best ways to stay up-to-date with the lastest LLM research, industry trends, and best practices.

| Name                | Social                                              | Expertise                  |
|---------------------|-------------------------------------------------------|-----------------------------|
| Li Yin             | [LinkedIn](https://www.linkedin.com/in/li-yin-ai)      |    AdalFlow Author & SylphAI founder                      |
| Chip Huyen         | [LinkedIn](https://www.linkedin.com/in/chiphuyen)      |   AI Engineering & ML Systems                      |
| Damien Benveniste, PhD | [LinkedIn](https://www.linkedin.com/in/damienbenveniste/) |        ML Systems & MLOps                    |
| Jim Fan            | [LinkedIn](https://www.linkedin.com/in/drjimfan/)      |    LLM Agents & Robotics                        |
| Paul Iusztin       | [LinkedIn](https://www.linkedin.com/in/pauliusztin/)   | LLM Engineering & LLMOps    |
| Armand Ruiz        | [LinkedIn](https://www.linkedin.com/in/armand-ruiz/)   | AI Engineering Director at IBM           |
| Alex Razvant       | [LinkedIn](https://www.linkedin.com/in/arazvant/)      | AI/ML Engineering          |
| Pascal Biese       | [LinkedIn](https://www.linkedin.com/in/pascalbiese/)   | LLM Papers Daily            |
| Maxime Labonne     | [LinkedIn](https://www.linkedin.com/in/maxime-labonne/) | LLM Fine-Tuning             |
| Sebastian Raschka  | [LinkedIn](https://www.linkedin.com/in/sebastianraschka/) | LLMs from Scratch       |
| Zach Wilson        | [LinkedIn](https://www.linkedin.com/in/eczachly)       | Data Engineering for LLMs    |
| Adi Polak          | [LinkedIn](https://www.linkedin.com/in/polak-adi/)       | Data Streaming for LLMs    |
| Eduardo Ordax    | [LinkedIn](https://www.linkedin.com/in/eordax/) | GenAI voice @ AWS |

## Community

| Name                | Social                                              | Scope                  |
|---------------------|-------------------------------------------------------|-----------------------------|
| AdalFlow     | [Discord](https://discord.gg/ezzszrRZvT)                |   LLM Engineering, auto-prompts, and AdalFlow discussions&contributions                      |

# Contributing

Only with the power of the community can we keep this repo up-to-date and relevant. If you have any suggestions, please open an issue or a direct pull request.

I will keep some pull requests open if I'm not sure if they are not an instant fit for this repo, you could vote for them by adding 👍 to them.

Thanks to the community, this repo is getting read by more people every day.

[![Star History Chart](https://api.star-history.com/svg?repos=SylphAI-Inc/LLM-engineer-handbook&type=Date)](https://star-history.com/#SylphAI-Inc/LLM-engineer-handbook&Date)

---

🤝 Please share so we can continue investing in it and make it the go-to resource for LLM engineers—whether they are just starting out or looking to stay updated in the field.

[![Share on X](https://img.shields.io/badge/Share_on-Twitter-1DA1F2?logo=twitter&logoColor=white)](https://twitter.com/intent/tweet?text=Check+out+this+awesome+repository+for+LLM+engineers!&url=https://github.com/LLM-engineer-handbook)
[![Share on LinkedIn](https://img.shields.io/badge/Share_on-LinkedIn-0077B5?logo=linkedin&logoColor=white)](https://www.linkedin.com/sharing/share-offsite/?url=https://github.com/LLM-engineer-handbook)

---

If you have any question about this opinionated list, do not hesitate to contact [Li Yin](https://www.linkedin.com/in/li-yin-ai)

README.md

A curated list of Large Language Model resources, covering model training, serving, fine-tuning, and building LLM applications.