# Signal Handling in Hyperlight

Hyperlight registers custom signal handlers to intercept and manage specific signals, primarily `SIGSYS` and `SIGRTMIN`. Here's an overview of the registration process:
- **Preserving Old Handlers**: When registering a new signal handler, Hyperlight first retrieves and stores the existing handler using `OnceCell`. This allows Hyperlight to delegate signals to the original handler if necessary.
-  **Custom Handlers**:
  - **`SIGSYS` Handler**: Captures disallowed syscalls enforced by seccomp. If the signal originates from a hyperlight thread, Hyperlight logs the syscall details. Otherwise, it delegates the signal to the previously registered handler.
  - **`SIGRTMIN` Handler**: Utilized for inter-thread signaling, such as execution cancellation. Similar to SIGSYS, it distinguishes between application and non-hyperlight threads to determine how to handle the signal.
- **Thread Differentiation**: Hyperlight uses thread-local storage (IS_HYPERLIGHT_THREAD) to identify whether the current thread is a hyperlight thread. This distinction ensures that signals are handled appropriately based on the thread's role.

## Potential Issues and Considerations

### Handler Invalidation

**Issue**: After Hyperlight registers its custom signal handler and preserves the `old_handler`, if the host or another component modifies the signal handler for the same signal, it can lead to:
    - **Invalidation of `old_handler`**: The stored old_handler reference may no longer point to a valid handler, causing undefined behavior when Hyperlight attempts to delegate signals.
    - **Loss of Custom Handling**: Hyperlight's custom handler might not be invoked as expected, disrupting its ability to enforce syscall restrictions or manage inter-thread signals.

docs/signal-handlers-development-notes.md


# Hyperlight technical requirements document (TRD)

In this technical requirements document (TRD), we have the following goals:

- Describe the high-level architecture of Hyperlight
- Provide relevant implementation details
- Provide additional information necessary for assessing the security and threat model of Hyperlight
- Detail the security claims Hyperlight makes

## High-level architecture

At a high level, Hyperlight's architecture is relatively simple. It consists of two primary components:

- Host library: the code that does the following:
  - Creates the Hyperlight VM, called the "sandbox"
  - Configures the VM, vCPU, and virtual registers
  - Configures VM memory
  - Loads the guest binary (see subsequent bullet point) into VM memory
  - Marshals calls to functions (called "guest functions") in the Guest binary inside the VM
  - Dispatches callbacks, called "host functions", from the guest back into the host
- Guest binary: the code that runs inside the Hyperlight sandbox and does the following:
  - Dispatches calls from the host into particular functions inside the guest
  - Marshals calls to host functions

## Relevant implementation details

As indicated in the previous "architecture" section, the two main components, the host and guest, interact in a specific, controlled manner. This section details the guest and host, and focuses on the details the implementation of that interaction.

### Guest binaries

Until this point, we've been using "guest" as an abstract term to indicate some binary to be run inside a Hyperlight sandbox. Because Hyperlight sandboxes only provide a limited set of functionality, guests must be compiled against and linked to all APIs necessary for providing the functionality above. These APIs are provided by our rust or C hyperlight guest libraries.

> While guests may compile against additional libraries (e.g. `libc`), they are not guaranteed to run inside a sandbox, and likely won't.

The Hyperlight sandbox deliberately provides a very limited set of functionality to guest binaries. We expect the most useful guests will execute code inside language interpreters or bytecode-level virtual machines, including Wasm VMs (e.g., [wasmtime](https://github.com/bytecodealliance/wasmtime)). Via this abstraction, we aim to provide functionality the "raw" Hyperlight sandbox does not provide directly. Any further functionality a given guest cannot provide can be provided via host functions.

### Host library

The Hyperlight host library provides a Rust-native API for its users to create and interact with Hyperlight sandboxes. Due to (1) the nature of this project (see the section below on threat modeling for details), and (2) the fact the host library has access to host system resources, we have spent considerable time and energy ensuring the host library has two major features:

- It is memory safe
- It provides a public API that prevents its users from doing unsafe things, using Rust features and other techniques

## Security threat model and guarantees

The set of security guarantees we aim to provide with Hyperlight are as follows:

- All user-level code will, in production builds, be executed within a Hyperlight sandbox backed by a Virtual Machine.
- All Hyperlight sandboxes, in production builds, will be isolated from each other and the host using hypervisor provided Virtual Machines.
- Guest binaries, in production Hyperlight builds, will have no access to the host system beyond VM-mapped memory (e.g., memory the host creates and maps into the system-appropriate VM) and a Hypervisor-provided vCPU. Specifically, a guest cannot request access to additional memory from the host.
- Only host functions that are explicitly made available by the host to a guest are available to the guest, the default state is that the guest has no access to host provided functions.
- If a host provides a guest with such a host function, the guest will never be able to call that host function without explicitly being invoked first. In other words, a guest function must first be called before it can call a host function.
- If a host provides a guest with a host function, the guest will never be able to execute that host function with an argument list of length and types not expected by the host.

docs/technical-requirements-document.md

Hyperlight is a lightweight Virtual Machine Manager (VMM) designed to be embedded within applications. It enables safe execution of untrusted code within micro virtual machines with very low latency and minimal overhead.


# Hyperlight Improvement Proposals (HIPs)

Hyperlight Improvement Proposals, otherwise known as HIPs, are largely influenced by the Kubernetes
Enhancement Proposal (KEP) process which provides a standardized development process for Hyperlight
enhancements. You can read more about the
[KEP process in 0000-kep-process here](https://github.com/kubernetes/enhancements/blob/master/keps/sig-architecture/0000-kep-process/README.md).

## Authoring a HIP
When you have a new enhancement that is more than a relatively trivial enhancement or bug fix, the
change should be first socialized as a HIP. To help authors to get started a HIP template is located in
[NNNN-hip-template](./NNNN-hip-template/README.md).

1. Create a new directory under [the proposals directory](../proposals) in the form of `NNNN-hip-${hip_name}`
  where `NNNN` is the next HIP number available. For example, if HIP 0001 is currently the highest number HIP and
  your enhancement is titled "Make Me a Sandwich", then your HIP would be `0002-hip-make-me-a-sandwich`.
2. Within your `NNNN-hip-${hip_name}` directory create a file named `README.md` containing a copy of the HIP
  template.
3. Author the content of the template. Not all sections are necessary. Please consider filling out the
  summary, motivation, and proposal sections first to gather early feedback on the desirability of the
  enhancement through a draft pull request.
4. After socializing the proposal and integrating early feedback, continue with the rest of the sections.
5. Update the pull request with the rest of the sections and remove the draft status from the pull request.
6. Address any feedback to the proposal and get it merged.
7. Implement the enhancement.

proposals/README.md


# GitHub Actions Workflows

This directory contains [GitHub Workflows](https://docs.github.com/en/actions/using-workflows) of two primary types:

- Ones to be used as dependencies within other workflow files outside this directory.
  - These types of workflows are stored in files with names preceded with `dep_`
- Ones to be executed directly.

## More information on dependency workflows

For more information on how dependencies work in GitHub Actions, see the [GitHub documentation on reusing workflows](https://docs.github.com/en/actions/using-workflows/reusing-workflows).

### About the `workflow_call` trigger

The primary mechanism by which all files within this directory declare themselves dependencies of others is the `workflow_call` trigger. This indicates to GitHub Actions that, for a given workflow, another workflow will invoke it.

To read more about this trigger, see [GitHub Actions documentation](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#onworkflow_call).


# Hyperlight Improvement Process

<!-- toc -->
- [Summary](#summary)
- [Motivation](#motivation)
    - [Goals](#goals)
    - [Non-Goals](#non-goals)
- [Proposal](#proposal)
<!-- /toc -->

## Summary

Hyperlight Improvement Proposals, otherwise known as HIPs, are largely influenced by the Kubernetes
Enhancement Proposal (KEP) process which provides a standardized development process for Hyperlight
enhancements. You can read more about the
[KEP process in 0000-kep-process here](https://github.com/kubernetes/enhancements/blob/master/keps/sig-architecture/0000-kep-process/README.md).

## Motivation

### Goals

1. Since Hyperlight is a smaller project than Kubernetes and has a smaller community of contributors, the
needs are slightly different from that of the Kubernetes project. HIPs do not strive to meet the rigor
of their inspiration in Kubernetes, but rather, to provide a means for communication of intent and a
historical record for the motivations that drove the improvement.

### Non-goals

1. To have all changes require a HIP. Only significantly impacting work should be stated as a HIP, e.g.
a large refactoring, significant feature enhancements, breaking APIs, etc.
2. We will omit the use of KEP like metadata that is associated with KEPs.

## Proposal

We propose creating a standardized improvement proposal process for Hyperlight modeled after the Kubernetes
Enhancement Proposal process.


# HIP NNNN - HIP NAME

<!-- toc -->
- [Summary](#summary)
- [Motivation](#motivation)
    - [Goals](#goals)
    - [Non-Goals](#non-goals)
- [Proposal](#proposal)
    - [User Stories (Optional)](#user-stories-optional)
        - [Story 1](#story-1)
        - [Story 2](#story-2)
    - [Notes/Constraints/Caveats (Optional)](#notesconstraintscaveats-optional)
    - [Risks and Mitigations](#risks-and-mitigations)
- [Design Details](#design-details)
    - [Test Plan](#test-plan)
        - [Unit tests](#unit-tests)
        - [Integration tests](#integration-tests)
        - [e2e tests](#e2e-tests)
- [Implementation History](#implementation-history)
- [Drawbacks](#drawbacks)
- [Alternatives](#alternatives)
<!-- /toc -->

## Summary

<!--
This section is incredibly important for producing high-quality, user-focused
documentation such as release notes or a development roadmap. It should be
possible to collect this information before implementation begins, in order to
avoid requiring implementers to split their attention between writing release
notes and implementing the feature itself. HIP editors should help to ensure
that the tone and content of the `Summary` section is useful for a wide audience.

A good summary is probably at least a paragraph in length.

Both in this section and below, follow the guidelines of the [documentation
style guide]. In particular, wrap lines to a reasonable length, to make it
easier for reviewers to cite specific portions, and to minimize diff churn on
updates.

[documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md
-->

## Motivation

<!--
This section is for explicitly listing the motivation, goals, and non-goals of
this HIP.  Describe why the change is important and the benefits to users. The
motivation section can optionally provide links to [experience reports] to
demonstrate the interest in a KEP within the wider Kubernetes community.

[experience reports]: https://github.com/golang/go/wiki/ExperienceReports
-->

### Goals

<!--
List the specific goals of the HIP. What is it trying to achieve? How will we
know that this has succeeded?
-->

### Non-Goals

<!--
What is out of scope for this HIP? Listing non-goals helps to focus discussion
and make progress.
-->

## Proposal

<!--
This is where we get down to the specifics of what the proposal actually is.
This should have enough detail that reviewers can understand exactly what
you're proposing, but should not include things like API designs or
implementation. What is the desired outcome and how do we measure success?.
The "Design Details" section below is for the real
nitty-gritty.
-->

### User Stories (Optional)

<!--
Detail the things that people will be able to do if this HIP is implemented.
Include as much detail as possible so that people can understand the "how" of
the system. The goal here is to make this feel real for users without getting
bogged down.
-->

#### Story 1

#### Story 2

### Notes/Constraints/Caveats (Optional)

<!--
What are the caveats to the proposal?
What are some important details that didn't come across above?
Go in to as much detail as necessary here.
This might be a good place to talk about core concepts and how they relate.
-->

### Risks and Mitigations

<!--
What are the risks of this proposal, and how do we mitigate? Think broadly.
For example, consider both security and how this will impact the larger ecosystem.

How will security be reviewed, and by whom?
-->

## Design Details

<!--
This section should contain enough information that the specifics of your
change are understandable. This may include API specs (though not always
required) or even code snippets. If there's any ambiguity about HOW your
proposal will be implemented, this is the place to discuss them.
-->

### Test Plan

<!--
The goal is to ensure that we don't accept enhancements with inadequate testing.

All code is expected to have adequate tests (eventually with coverage
expectations).
-->

##### Unit tests

<!--
In principle every added code should have complete unit test coverage, so providing
the exact set of tests will not bring additional value.
However, if complete unit test coverage is not possible, explain the reason of it
together with explanation why this is acceptable.
-->

##### Integration tests

<!--
Integration tests allow control of the configuration parameters used to start the binaries under test.
This is different from e2e tests which do not allow configuration of parameters.
Doing this allows testing non-default options and multiple different and potentially conflicting command line options.
-->

##### e2e tests

<!--
We expect no non-infra related flakes in the last month as a GA graduation criteria.
-->

## Implementation History

<!--
Major milestones in the lifecycle of a HIP should be tracked in this section.
Major milestones might include:
- the `Summary` and `Motivation` sections being merged, signaling SIG acceptance
- the `Proposal` section being merged, signaling agreement on a proposed design
- the date implementation started
- the first Hyperlight release where an initial version of the HIP was available
- the version of Hyperlight where the HIP graduated to general availability
- when the HIP was retired or superseded
-->

## Drawbacks

<!--
Why should this HIP _not_ be implemented?
-->

## Alternatives

<!--
What other approaches did you consider, and why did you rule them out? These do
not need to be as detailed as the proposal, but should include enough
information to express the idea and why it was not acceptable.
-->


This is a c-api wrapper over the hyperlight-guest crate. The purpose of this crate is to allow the creation of guests in the c language. This crate generates a .lib/.a library file depending on the platform, as well necessary header files.

For examples on how to use it, see the c [simpleguest](../tests/c_guests/c_simpleguest/).

# Important

All guest functions must return a `hl_Vec*` obtained by calling one of the `hl_flatbuffer_result_from_*` functions. These functions will return a flatbuffer encoded byte-buffer of given value, for example `hl_flatbuffer_result_from_int(int)` will return the flatbuffer representation of the given int.

## NOTE

**You may not construct and return your own `hl_Vec*`**, as the hyperlight api assumes that all returned `hl_Vec*` are constructed through calls to a `hl_flatbuffer_result_from_*` function.

Additionally, note that type `hl_Vec*` is used in two different contexts. First, `hl_Vec*` is used input-parameter-type for guest functions that take a buffer of bytes. This buffer of bytes can contain **arbitrary** bytes. Second, all guest functions return a `hl_Vec*` (it might be hidden away by c macros). These `hl_Vec*` are flatbuffer-encoded data, and are not arbitrary.


# Third Party Library Use

This project makes use of the following third party libraries, each of which is contained in a subdirectory of `third_party` with a COPYRIGHT/LICENSE file in the root of the subdirectory. These libraries are used under the terms of their respective licenses. They are also listed in the NOTICE file in the root of the repository

## printf

This implementation of printf is from [here](https://github.com/mpaland/printf.git)
The copy was taken at version at [version 4.0](https://github.com/mpaland/printf/releases/tag/v4.0.0)
Changes have been applied to the original code for Hyperlight using this [patch](./printf/printf.patch)

## libc

A partial version of musl libc is used by hyperlight and is located in the [musl](./musl) directory as a git subtree.

The current version is release [v1.2.5](https://git.musl-libc.org/cgit/musl/tag/?h=v1.2.5). Many files have been deleted and changes have been made to some of the remaining files.


# Fuzzing Hyperlight

This directory contains the fuzzing infrastructure for Hyperlight. We use `cargo-fuzz` to run the fuzzers - i.e., small programs that run specific tests with semi-random inputs to find bugs. Because `cargo-fuzz` is not yet stable, we use the nightly toolchain. Also, because `cargo-fuzz` doesn't support Windows, we have to run this WSL or Linux (Mariner/Ubuntu).

You can run the fuzzers with:
```sh
cargo +nightly-2023-11-28-x86_64-unknown-linux-gnu fuzz run --release <fuzzer_name>
```

> Note: Because nightly toolchains are not stable, we pin the nightly version to `2023-11-28`. To install this toolchain, run:
> ```sh
> rustup toolchain install nightly-2023-11-28-x86_64-unknown-linux-gnu
> ```

As per Microsoft's Offensive Research & Security Engineering (MORSE) team, all host exposed functions that receive or interact with guest data must be continuously fuzzed for, at least, 500 million fuzz test cases without any crashes. Because `cargo-fuzz` doesn't support setting a maximum number of iterations; instead, we use the `--max_total_time` flag to set a maximum time to run the fuzzer. We have a GitHub action (acting like a CRON job) that runs the fuzzers for 24 hours every week.

Currently, we only fuzz the `PrintOutput` function. We plan to add more fuzzers in the future.

## On Failure

If you encounter a failure, you can re-run an entire seed (i.e., group of inputs) with:
```sh
cargo +nightly-2023-11-28-x86_64-unknown-linux-gnu fuzz run --release <fuzzer_name> -- -seed=<seed-number>
```

The seed number can be seed in a specific run, like:
![fuzz-seed](doc-assets/image.png)

Or, if repro-ing a failure from CI, you can download the artifact from the fuzzing run, and run it like:

```sh
cargo +nightly-2023-11-28-x86_64-unknown-linux-gnu fuzz run --release -O <fuzzer_name> <fuzzer-input (e.g., fuzz/artifacts/fuzz_target_1/crash-93c522e64ee822034972ccf7026d3a8f20d5267c>
```


# A printf / sprintf Implementation for Embedded Systems

[![Build Status](https://travis-ci.org/mpaland/printf.svg?branch=master)](https://travis-ci.org/mpaland/printf)
[![codecov](https://codecov.io/gh/mpaland/printf/branch/master/graph/badge.svg)](https://codecov.io/gh/mpaland/printf)
[![Coverity Status](https://img.shields.io/coverity/scan/14180.svg)](https://scan.coverity.com/projects/mpaland-printf)
[![Github Issues](https://img.shields.io/github/issues/mpaland/printf.svg)](http://github.com/mpaland/printf/issues)
[![Github Releases](https://img.shields.io/github/release/mpaland/printf.svg)](https://github.com/mpaland/printf/releases)
[![GitHub license](https://img.shields.io/badge/license-MIT-blue.svg)](https://raw.githubusercontent.com/mpaland/avl_array/master/LICENSE)

This is a tiny but **fully loaded** printf, sprintf and (v)snprintf implementation.
Primarily designed for usage in embedded systems, where printf is not available due to memory issues or in avoidance of linking against libc.
Using the standard libc printf may pull **a lot** of unwanted library stuff and can bloat code size about 20k or is not 100% thread safe. In this cases the following implementation can be used.
Absolutely **NO dependencies** are required, *printf.c* brings all necessary routines, even its own fast `ftoa` (floating point), `ntoa` (decimal) conversion.

If memory footprint is really a critical issue, floating point, exponential and 'long long' support and can be turned off via the `PRINTF_DISABLE_SUPPORT_FLOAT`, `PRINTF_DISABLE_SUPPORT_EXPONENTIAL` and `PRINTF_DISABLE_SUPPORT_LONG_LONG` compiler switches.
When using printf (instead of sprintf/snprintf) you have to provide your own `_putchar()` low level function as console/serial output.

## Highlights and Design Goals

There is a boatload of so called 'tiny' printf implementations around. So why this one?
I've tested many implementations, but most of them have very limited flag/specifier support, a lot of other dependencies or are just not standard compliant and failing most of the test suite.
Therefore I decided to write an own, final implementation which meets the following items:

 - Very small implementation (around 600 code lines)
 - NO dependencies, no libs, just one module file
 - Support of all important flags, width and precision sub-specifiers (see below)
 - Support of decimal/floating number representation (with an own fast itoa/ftoa)
 - Reentrant and thread-safe, malloc free, no static vars/buffers
 - LINT and compiler L4 warning free, mature, coverity clean, automotive ready
 - Extensive test suite (> 400 test cases) passing
 - Simply the best *printf* around the net
 - MIT license

## Usage

Add/link *printf.c* to your project and include *printf.h*. That's it.
Implement your low level output function needed for `printf()`:
```C
void _putchar(char character)
{
  // send char to console etc.
}
```

Usage is 1:1 like the according stdio.h library version:
```C
int printf(const char* format, ...);
int sprintf(char* buffer, const char* format, ...);
int snprintf(char* buffer, size_t count, const char* format, ...);
int vsnprintf(char* buffer, size_t count, const char* format, va_list va);

// use output function (instead of buffer) for streamlike interface
int fctprintf(void (*out)(char character, void* arg), void* arg, const char* format, ...);
```

**Due to general security reasons it is highly recommended to prefer and use `snprintf` (with the max buffer size as `count` parameter) instead of `sprintf`.**
`sprintf` has no buffer limitation, so when needed - use it really with care!

### Streamlike Usage
Besides the regular standard `printf()` functions, this module also provides `fctprintf()`, which takes an output function as first parameter to build a streamlike output like `fprintf()`:
```C
// define the output function
void my_stream_output(char character, void* arg)
{
  // opt. evaluate the argument and send the char somewhere
}

{
  // in your code
  void* arg = (void*)100;  // this argument is passed to the output function
  fctprintf(&my_stream_output, arg, "This is a test: %X", 0xAA);
  fctprintf(&my_stream_output, nullptr, "Send to null dev");
}
```

## Format Specifiers

A format specifier follows this prototype: `%[flags][width][.precision][length]type`
The following format specifiers are supported:

### Supported Types

| Type   | Output |
|--------|--------|
| d or i | Signed decimal integer |
| u      | Unsigned decimal integer	|
| b      | Unsigned binary |
| o      | Unsigned octal |
| x      | Unsigned hexadecimal integer (lowercase) |
| X      | Unsigned hexadecimal integer (uppercase) |
| f or F | Decimal floating point |
| e or E | Scientific-notation (exponential) floating point |
| g or G | Scientific or decimal floating point |
| c      | Single character |
| s      | String of characters |
| p      | Pointer address |
| %      | A % followed by another % character will write a single % |

### Supported Flags

| Flags | Description |
|-------|-------------|
| -     | Left-justify within the given field width; Right justification is the default. |
| +     | Forces to precede the result with a plus or minus sign (+ or -) even for positive numbers.
By default, only negative numbers are preceded with a - sign. |
| (space) | If no sign is going to be written, a blank space is inserted before the value. |
| #     | Used with o, b, x or X specifiers the value is preceded with 0, 0b, 0x or 0X respectively for values different than zero.
Used with f, F it forces the written output to contain a decimal point even if no more digits follow. By default, if no digits follow, no decimal point is written. |
| 0     | Left-pads the number with zeros (0) instead of spaces when padding is specified (see width sub-specifier). |

### Supported Width

| Width    | Description |
|----------|-------------|
| (number) | Minimum number of characters to be printed. If the value to be printed is shorter than this number, the result is padded with blank spaces. The value is not truncated even if the result is larger. |
| *        | The width is not specified in the format string, but as an additional integer value argument preceding the argument that has to be formatted. |

### Supported Precision

| Precision	| Description |
|-----------|-------------|
| .number   | For integer specifiers (d, i, o, u, x, X): precision specifies the minimum number of digits to be written. If the value to be written is shorter than this number, the result is padded with leading zeros. The value is not truncated even if the result is longer. A precision of 0 means that no character is written for the value 0.
For f and F specifiers: this is the number of digits to be printed after the decimal point. **By default, this is 6, maximum is 9**.
For s: this is the maximum number of characters to be printed. By default all characters are printed until the ending null character is encountered.
If the period is specified without an explicit value for precision, 0 is assumed. |
| .*        | The precision is not specified in the format string, but as an additional integer value argument preceding the argument that has to be formatted. |

### Supported Length

The length sub-specifier modifies the length of the data type.

| Length | d i  | u o x X |
|--------|------|---------|
| (none) | int  | unsigned int |
| hh     | char | unsigned char |
| h      | short int | unsigned short int |
| l      | long int | unsigned long int |
| ll     | long long int | unsigned long long int (if PRINTF_SUPPORT_LONG_LONG is defined) |
| j      | intmax_t | uintmax_t |
| z      | size_t | size_t |
| t      | ptrdiff_t | ptrdiff_t (if PRINTF_SUPPORT_PTRDIFF_T is defined) |

### Return Value

Upon successful return, all functions return the number of characters written, _excluding_ the terminating null character used to end the string.
Functions `snprintf()` and `vsnprintf()` don't write more than `count` bytes, _including_ the terminating null byte ('\0').
Anyway, if the output was truncated due to this limit, the return value is the number of characters that _could_ have been written.
Notice that a value equal or larger than `count` indicates a truncation. Only when the returned value is non-negative and less than `count`,
the string has been completely written.
If any error is encountered, `-1` is returned.

If `buffer` is set to `NULL` (`nullptr`) nothing is written and just the formatted length is returned.
```C
int length = sprintf(NULL, "Hello, world"); // length is set to 12
```

## Compiler Switches/Defines

| Name | Default value | Description |
|------|---------------|-------------|
| PRINTF_INCLUDE_CONFIG_H            | undefined | Define this as compiler switch (e.g. `gcc -DPRINTF_INCLUDE_CONFIG_H`) to include a "printf_config.h" definition file |
| PRINTF_NTOA_BUFFER_SIZE            | 32        | ntoa (integer) conversion buffer size. This must be big enough to hold one converted numeric number _including_ leading zeros, normally 32 is a sufficient value. Created on the stack |
| PRINTF_FTOA_BUFFER_SIZE            | 32        | ftoa (float) conversion buffer size. This must be big enough to hold one converted float number _including_ leading zeros, normally 32 is a sufficient value. Created on the stack |
| PRINTF_DEFAULT_FLOAT_PRECISION     | 6         | Define the default floating point precision |
| PRINTF_MAX_FLOAT                   | 1e9       | Define the largest suitable value to be printed with %f, before using exponential representation |
| PRINTF_DISABLE_SUPPORT_FLOAT       | undefined | Define this to disable floating point (%f) support |
| PRINTF_DISABLE_SUPPORT_EXPONENTIAL | undefined | Define this to disable exponential floating point (%e) support |
| PRINTF_DISABLE_SUPPORT_LONG_LONG   | undefined | Define this to disable long long (%ll) support |
| PRINTF_DISABLE_SUPPORT_PTRDIFF_T   | undefined | Define this to disable ptrdiff_t (%t) support |

## Caveats
None anymore (finally).

## Test Suite
For testing just compile, build and run the test suite located in `test/test_suite.cpp`. This uses the [catch](https://github.com/catchorg/Catch2) framework for unit-tests, which is auto-adding main().
Running with the `--wait-for-keypress exit` option waits for the enter key after test end.

## Projects Using printf
- [turnkeyboard](https://github.com/mpaland/turnkeyboard) uses printf as log and generic tty (formatting) output.
- printf is part of [embeddedartistry/libc](https://github.com/embeddedartistry/libc), a libc targeted for embedded systems usage.
- The [Hatchling Platform]( https://github.com/adrian3git/HatchlingPlatform) uses printf.
(Just send me a mail/issue/PR to get *your* project listed here)

## Contributing

0. Give this project a :star:
1. Create an issue and describe your idea
2. [Fork it](https://github.com/mpaland/printf/fork)
3. Create your feature branch (`git checkout -b my-new-feature`)
4. Commit your changes (`git commit -am 'Add some feature'`)
5. Publish the branch (`git push origin my-new-feature`)
6. Create a new pull request
7. Profit! :heavy_check_mark:

## License
printf is written under the [MIT license](http://www.opensource.org/licenses/MIT).


This is an example of using the tracing-chrome tracing-subscriber. When ran, it will generate a file `trace-*.json` in the root directory. This file can then be visualized by going to `chrome://tracing` or `ui.perfetto.dev`. Both these sites can be navigated using WASD.


# LatentSync: Audio Conditioned Latent Diffusion Models for Lip Sync

[![arXiv](https://img.shields.io/badge/arXiv_paper-2412.09262-b31b1b)](https://arxiv.org/abs/2412.09262)
[![arXiv](https://img.shields.io/badge/%F0%9F%A4%97%20space-HuggingFace-yellow)](https://huggingface.co/spaces/fffiloni/LatentSync)

## 📖 Abstract

We present *LatentSync*, an end-to-end lip sync framework based on audio conditioned latent diffusion models without any intermediate motion representation, diverging from previous diffusion-based lip sync methods based on pixel space diffusion or two-stage generation. Our framework can leverage the powerful capabilities of Stable Diffusion to directly model complex audio-visual correlations. Additionally, we found that the diffusion-based lip sync methods exhibit inferior temporal consistency due to the inconsistency in the diffusion process across different frames. We propose *Temporal REPresentation Alignment (TREPA)* to enhance temporal consistency while preserving lip-sync accuracy. TREPA uses temporal representations extracted by large-scale self-supervised video models to align the generated frames with the ground truth frames.

## 🏗 Framework

LatentSync uses the [Whisper](https://github.com/openai/whisper) to convert melspectrogram into audio embeddings, which are then integrated into the U-Net via cross-attention layers. The reference and masked frames are channel-wise concatenated with noised latents as the input of U-Net. In the training process, we use a one-step method to get estimated clean latents from predicted noises, which are then decoded to obtain the estimated clean frames. The TREPA, [LPIPS](https://arxiv.org/abs/1801.03924) and [SyncNet](https://www.robots.ox.ac.uk/~vgg/publications/2016/Chung16a/chung16a.pdf) losses are added in the pixel space.

## 🎬 Demo

        Original video
        Lip-synced video

      <video src=https://github.com/user-attachments/assets/ff3a84da-dc9b-498a-950f-5c54f58dd5c5 controls preload></video>

      <video src=https://github.com/user-attachments/assets/150e00fd-381e-4421-a478-a9ea3d1212a8 controls preload></video>

      <video src=https://github.com/user-attachments/assets/32c830a9-4d7d-4044-9b33-b184d8e11010 controls preload></video>

      <video src=https://github.com/user-attachments/assets/84e4fe9d-b108-44a4-8712-13a012348145 controls preload></video>

      <video src=https://github.com/user-attachments/assets/7510a448-255a-44ee-b093-a1b98bd3961d controls preload></video>

      <video src=https://github.com/user-attachments/assets/6150c453-c559-4ae0-bb00-c565f135ff41 controls preload></video>

      <video src=https://github.com/user-attachments/assets/0f7f9845-68b2-4165-bd08-c7bbe01a0e52 controls preload></video>

      <video src=https://github.com/user-attachments/assets/c34fe89d-0c09-4de3-8601-3d01229a69e3 controls preload></video>

      <video src=https://github.com/user-attachments/assets/7ce04d50-d39f-4154-932a-ec3a590a8f64 controls preload></video>

      <video src=https://github.com/user-attachments/assets/70bde520-42fa-4a0e-b66c-d3040ae5e065 controls preload></video>

(Photorealistic videos are filmed by contracted models, and anime videos are from [VASA-1](https://www.microsoft.com/en-us/research/project/vasa-1/) and [EMO](https://humanaigc.github.io/emote-portrait-alive/))

## 📑 Open-source Plan

- [x] Inference code and checkpoints
- [x] Data processing pipeline
- [x] Training code

## 🔧 Setting up the Environment

Install the required packages and download the checkpoints via:

```bash
source setup_env.sh
```

If the download is successful, the checkpoints should appear as follows:

```
./checkpoints/
|-- latentsync_unet.pt
|-- latentsync_syncnet.pt
|-- whisper
|   `-- tiny.pt
|-- auxiliary
|   |-- 2DFAN4-cd938726ad.zip
|   |-- i3d_torchscript.pt
|   |-- koniq_pretrained.pkl
|   |-- s3fd-619a316812.pth
|   |-- sfd_face.pth
|   |-- syncnet_v2.model
|   |-- vgg16-397923af.pth
|   `-- vit_g_hybrid_pt_1200e_ssv2_ft.pth
```

These already include all the checkpoints required for latentsync training and inference. If you just want to try inference, you only need to download `latentsync_unet.pt` and `tiny.pt` from our [HuggingFace repo](https://huggingface.co/ByteDance/LatentSync)

## 🚀 Inference

There are two ways to perform inference, and both require 6.5 GB of VRAM.

### 1. Gradio App

Run the Gradio app for inference:

```bash
python gradio_app.py
```

### 2. Command Line Interface

Run the script for inference:

```bash
./inference.sh
```

You can change the parameters `inference_steps` and `guidance_scale` to see more results.

## 🔄 Data Processing Pipeline

The complete data processing pipeline includes the following steps:

1. Remove the broken video files.
2. Resample the video FPS to 25, and resample the audio to 16000 Hz.
3. Scene detect via [PySceneDetect](https://github.com/Breakthrough/PySceneDetect).
4. Split each video into 5-10 second segments.
5. Remove videos where the face is smaller than 256 $\times$ 256, as well as videos with more than one face.
6. Affine transform the faces according to the landmarks detected by [face-alignment](https://github.com/1adrianb/face-alignment), then resize to 256 $\times$ 256.
7. Remove videos with [sync confidence score](https://www.robots.ox.ac.uk/~vgg/publications/2016/Chung16a/chung16a.pdf) lower than 3, and adjust the audio-visual offset to 0.
8. Calculate [hyperIQA](https://openaccess.thecvf.com/content_CVPR_2020/papers/Su_Blindly_Assess_Image_Quality_in_the_Wild_Guided_by_a_CVPR_2020_paper.pdf) score, and remove videos with scores lower than 40.

Run the script to execute the data processing pipeline:

```bash
./data_processing_pipeline.sh
```

You can change the parameter `input_dir` in the script to specify the data directory to be processed. The processed data will be saved in the `high_visual_quality` directory. Each step will generate a new directory to prevent the need to redo the entire pipeline in case the process is interrupted by an unexpected error.

## 🏋♂ Training U-Net

Before training, you must process the data as described above and download all the checkpoints. We released a pretrained SyncNet with 94% accuracy on the VoxCeleb2 dataset for the supervision of U-Net training. Note that this SyncNet is trained on affine transformed videos, so when using or evaluating this SyncNet, you need to perform affine transformation on the video first (the code of affine transformation is included in the data processing pipeline).

If all the preparations are complete, you can train the U-Net with the following script:

```bash
./train_unet.sh
```

You should change the parameters in U-Net config file to specify the data directory, checkpoint save path, and other training hyperparameters.

## 🏋♂ Training SyncNet

In case you want to train SyncNet on your own datasets, you can run the following script. The data processing pipeline for SyncNet is the same as U-Net.

```bash
./train_syncnet.sh
```

After `validations_steps` training, the loss charts will be saved in `train_output_dir`. They contain both the training and validation loss.

## 📊 Evaluation

You can evaluate the [sync confidence score](https://www.robots.ox.ac.uk/~vgg/publications/2016/Chung16a/chung16a.pdf) of a generated video by running the following script:

```bash
./eval/eval_sync_conf.sh
```

You can evaluate the accuracy of SyncNet on a dataset by running the following script:

```bash
./eval/eval_syncnet_acc.sh
```

## 🙏 Acknowledgement

- Our code is built on [AnimateDiff](https://github.com/guoyww/AnimateDiff).
- Some code are borrowed from [MuseTalk](https://github.com/TMElyralab/MuseTalk), [StyleSync](https://github.com/guanjz20/StyleSync), [SyncNet](https://github.com/joonson/syncnet_python), [Wav2Lip](https://github.com/Rudrabha/Wav2Lip).

Thanks for their generous contributions to the open-source community.

README.md

Taming Stable Diffusion for Lip Sync!


# Face detector

This face detector is adapted from `https://github.com/cs-giung/face-detection-pytorch`.

Taming Stable Diffusion for Lip Sync!


# Trend Finder 🔦

**Stay on top of trending topics on social media — all in one place.**

Trend Finder collects and analyzes posts from key influencers, then sends a Slack or Discord notification when it detects new trends or product launches. This has been a complete game-changer for the Firecrawl marketing team by:

- **Saving time** normally spent manually searching social channels
- **Keeping you informed** of relevant, real-time conversations
- **Enabling rapid response** to new opportunities or emerging industry shifts

_Spend less time hunting for trends and more time creating impactful campaigns._

## Watch the Demo & Tutorial video

[![Thumbnail](https://i.ytimg.com/vi/puimQSun92g/hqdefault.jpg)](https://www.youtube.com/watch?v=puimQSun92g)

Learn how to set up Trend Finder and start monitoring trends in this video!

## How it Works

1. **Data Collection** 📥
   - Monitors selected influencers' posts on Twitter/X using the X API (Warning: the X API free plan is rate limited to only monitor 1 X account every 15 min)
   - Monitors websites for new releases and news with Firecrawl's /extract
   - Runs on a scheduled basis using cron jobs

2. **AI Analysis** 🧠
   - Processes collected content through Together AI
   - Identifies emerging trends, releases, and news.
   - Analyzes sentiment and relevance

3. **Notification System** 📢
   - When significant trends are detected, sends Slack or Discord notifications based on cron job setup
   - Provides context about the trend and its sources
   - Enables quick response to emerging opportunities

## Features

- 🤖 AI-powered trend analysis using Together AI
- 📱 Social media monitoring (Twitter/X integration)
- 🔍 Website monitoring with Firecrawl
- 💬 Instant Slack or Discord notifications
- ⏱ Scheduled monitoring using cron jobs

## Prerequisites

- Node.js (v14 or higher)
- npm or yarn
- Docker
- Docker Compose
- Slack workspace with webhook permissions
- API keys for required services

## Environment Variables

Copy `.env.example` to `.env` and configure the following variables:

```
# Optional: API key from Together AI for trend analysis (https://www.together.ai/)
TOGETHER_API_KEY=your_together_api_key_here

# Optional: API key from DeepSeek for trend analysis (https://deepseek.com/)
DEEPSEEK_API_KEY=

# Optional: API key from OpenAI for trend analysis (https://openai.com/)
OPENAI_API_KEY=

# Required if monitoring web pages (https://www.firecrawl.dev/)
FIRECRAWL_API_KEY=your_firecrawl_api_key_here

# Required if monitoring Twitter/X trends (https://developer.x.com/)
X_API_BEARER_TOKEN=your_twitter_api_bearer_token_here

# Notification driver. Supported drivers: "slack", "discord"
NOTIFICATION_DRIVER=discord

# Required (if NOTIFICATION_DRIVER is "slack"): Incoming Webhook URL from Slack for notifications
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/YOUR/WEBHOOK/URL

# Required (if NOTIFICATION_DRIVER is "discord"): Incoming Webhook URL from Discord for notifications
DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/WEBHOOK/URL
```

## Getting Started

1. **Clone the repository:**
   ```bash
   git clone [repository-url]
   cd trend-finder
   ```

2. **Install dependencies:**
   ```bash
   npm install
   ```

3. **Configure environment variables:**
   ```bash
   cp .env.example .env
   # Edit .env with your configuration
   ```

4. **Run the application:**
   ```bash
   # Development mode with hot reloading
   npm run start

   # Build for production
   npm run build
   ```

## Using Docker

1. **Build the Docker image:**
   ```bash
   docker build -t trend-finder .
   ```

2. **Run the Docker container:**
   ```bash
   docker run -d -p 3000:3000 --env-file .env trend-finder
   ```

## Using Docker Compose

1. **Start the application with Docker Compose:**
   ```bash
   docker-compose up --build -d
   ```

2. **Stop the application with Docker Compose:**
   ```bash
   docker-compose down
   ```

## Project Structure

```
trend-finder/
├── src/
│   ├── controllers/    # Request handlers
│   ├── services/       # Business logic
│   └── index.ts        # Application entry point
├── .env.example        # Environment variables template
├── package.json        # Dependencies and scripts
└── tsconfig.json       # TypeScript configuration
```

## Contributing

1. Fork the repository
2. Create your feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add some amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request

README.md

Stay on top of trending topics on social media and the web with AI


# Contributor Covenant Code of Conduct

## Our Pledge

We as members, contributors, and leaders pledge to make participation in our
community a harassment-free experience for everyone, regardless of age, body
size, visible or invisible disability, ethnicity, sex characteristics, gender
identity and expression, level of experience, education, socio-economic status,
nationality, personal appearance, race, caste, color, religion, or sexual
identity and orientation.

We pledge to act and interact in ways that contribute to an open, welcoming,
diverse, inclusive, and healthy community.

## Our Standards

Examples of behavior that contributes to a positive environment for our
community include:

* Demonstrating empathy and kindness toward other people
* Being respectful of differing opinions, viewpoints, and experiences
* Giving and gracefully accepting constructive feedback
* Accepting responsibility and apologizing to those affected by our mistakes,
  and learning from the experience
* Focusing on what is best not just for us as individuals, but for the overall
  community

Examples of unacceptable behavior include:

* The use of sexualized language or imagery, and sexual attention or advances of
  any kind
* Trolling, insulting or derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or email address,
  without their explicit permission
* Other conduct which could reasonably be considered inappropriate in a
  professional setting

## Enforcement Responsibilities

Community leaders are responsible for clarifying and enforcing our standards of
acceptable behavior and will take appropriate and fair corrective action in
response to any behavior that they deem inappropriate, threatening, offensive,
or harmful.

Community leaders have the right and responsibility to remove, edit, or reject
comments, commits, code, wiki edits, issues, and other contributions that are
not aligned to this Code of Conduct, and will communicate reasons for moderation
decisions when appropriate.

## Scope

This Code of Conduct applies within all community spaces, and also applies when
an individual is officially representing the community in public spaces.
Examples of representing our community include using an official email address,
posting via an official social media account, or acting as an appointed
representative at an online or offline event.

## Enforcement

Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported to the community leaders responsible for enforcement at
[support@chonkie.ai](mailto:support@chonkie.ai).
All complaints will be reviewed and investigated promptly and fairly.

All community leaders are obligated to respect the privacy and security of the
reporter of any incident.

## Enforcement Guidelines

Community leaders will follow these Community Impact Guidelines in determining
the consequences for any action they deem in violation of this Code of Conduct:

### 1. Correction

**Community Impact**: Use of inappropriate language or other behavior deemed
unprofessional or unwelcome in the community.

**Consequence**: A private, written warning from community leaders, providing
clarity around the nature of the violation and an explanation of why the
behavior was inappropriate. A public apology may be requested.

### 2. Warning

**Community Impact**: A violation through a single incident or series of
actions.

**Consequence**: A warning with consequences for continued behavior. No
interaction with the people involved, including unsolicited interaction with
those enforcing the Code of Conduct, for a specified period of time. This
includes avoiding interactions in community spaces as well as external channels
like social media. Violating these terms may lead to a temporary or permanent
ban.

### 3. Temporary Ban

**Community Impact**: A serious violation of community standards, including
sustained inappropriate behavior.

**Consequence**: A temporary ban from any sort of interaction or public
communication with the community for a specified period of time. No public or
private interaction with the people involved, including unsolicited interaction
with those enforcing the Code of Conduct, is allowed during this period.
Violating these terms may lead to a permanent ban.

### 4. Permanent Ban

**Community Impact**: Demonstrating a pattern of violation of community
standards, including sustained inappropriate behavior, harassment of an
individual, or aggression toward or disparagement of classes of individuals.

**Consequence**: A permanent ban from any sort of public interaction within the
community.

## Attribution

This Code of Conduct is adapted from the [Contributor Covenant][homepage],
version 2.1, available at
[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].

Community Impact Guidelines were inspired by
[Mozilla's code of conduct enforcement ladder][Mozilla CoC].

For answers to common questions about this code of conduct, see the FAQ at
[https://www.contributor-covenant.org/faq][FAQ]. Translations are available at
[https://www.contributor-covenant.org/translations][translations].

CODE_OF_CONDUCT.md

🦛 CHONK your texts with Chonkie ✨ - The no-nonsense RAG chunking library


# 🦛 Contributing to Chonkie

> "I like them big, I like them CONTRIBUTING" ~ Moto Moto, probably

Welcome fellow CHONKer! We're excited that you want to contribute to Chonkie. Whether you're fixing bugs, adding features, or improving documentation, every contribution makes Chonkie a better library for everyone.

## 🎯 Before You Start

1. **Check the issues**: Look for existing issues or open a new one to start a discussion.
2. **Read the docs**: Familiarize yourself with [Chonkie's docs](https://docs.chonkie.ai) and core [concepts](https://docs.chonkie.ai/getting-started/concepts).
3. **Set up your environment**: Follow our development setup guide below.

## 🚀 Development Setup

1. Fork and clone the repository:

```bash
git clone https://github.com/your-username/chonkie.git
cd chonkie
```

2. Create a virtual environment and install dependencies:

```bash
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install with dev dependencies
pip install -e ".[dev]"

# If working on semantic features, also install semantic dependencies
pip install -e ".[dev,semantic]"

# For all features
pip install -e ".[dev,all]"
```

## 🧪 Running Tests

We use pytest for testing. Our tests are configured via `pyproject.toml`. Before submitting a PR, make sure all tests pass:

```bash
# Run all tests
pytest

# Run specific test file
pytest tests/test_token_chunker.py

# Run tests with coverage
pytest --cov=chonkie
```

## 🎨 Code Style

Chonkie uses [ruff](https://github.com/astral-sh/ruff) for code formatting and linting. Our configuration in `pyproject.toml` enforces:

- Code formatting (`F`)
- Import sorting (`I`)
- Documentation style (`D`)
- Docstring coverage (`DOC`)

```bash
# Run ruff
ruff check .

# Run ruff with auto-fix
ruff check --fix .
```

### Documentation Style

We use Google-style docstrings. Example:

```python
def chunk_text(text: str, chunk_size: int = 512) -> List[str]:
    """Split text into chunks of specified size.

    Args:
        text: Input text to chunk
        chunk_size: Maximum size of each chunk

    Returns:
        List of text chunks

    Raises:
        ValueError: If chunk_size <= 0
    """
    pass
```

## 🚦 Pull Request Process

1. **Branch Naming**: Use descriptive branch names:
   - `feature/description` for new features
   - `fix/description` for bug fixes
   - `docs/description` for documentation changes

2. **Commit Messages**: Write clear commit messages:

```markdown
feat: add batch processing to WordChunker

- Implement batch_process method
- Add tests for batch processing
- Update documentation
```

3. **Dependencies**: If adding new dependencies:
   - Core dependencies go in `project.dependencies`
   - Optional features go in `project.optional-dependencies`
   - Development tools go in the `dev` optional dependency group

## 📦 Project Structure

Chonkie's package structure is:
```
src/
├── chonkie/
    ├── chunker/     # Chunking implementations
    ├── embeddings/  # Embedding implementations
    └── refinery/    # Refinement utilities
```

## 🎯 Where to Contribute

### 1. Good First Issues

Look for issues labeled [`good-first-issue`](https://github.com/chonkie-ai/chonkie/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22). These are great starting points for new contributors.

### 2. Documentation

- Improve existing docs
- Add examples
- Fix typos
- Add tutorials

### 3. Code

- Implement new chunking strategies
- Optimize existing chunkers
- Add new tokenizer support
- Improve test coverage

### 4. Performance

- Profile and optimize code
- Add benchmarks
- Improve memory usage
- Enhance batch processing

### 5. New Features

- Add new features to the library
- Add new optional dependencies
- Look for [FEAT] labels in issues, especially by Chonkie Maintainers

## 🦛 Development Dependencies

Current development dependencies are (as of January 1, 2025):

```toml
[project.optional-dependencies]
dev = [
    "pytest>=6.2.0",
    "datasets>=1.14.0",
    "transformers>=4.0.0",
    "ruff>=0.0.265"
]
```

Additional optional dependencies:

- `model2vec`: For model2vec embeddings
- `st`: For sentence-transformers
- `openai`: For OpenAI embeddings
- `semantic`: For semantic features
- `all`: All optional dependencies

## 🤝 Code Review Process

1. All PRs need at least one review
2. Maintainers will review for:
   - Code quality (via ruff)
   - Test coverage
   - Performance impact
   - Documentation completeness
   - Adherence to principles

## 💡 Getting Help

- **Questions?** Open an issue or ask in Discord
- **Bugs?** Open an issue or report in Discord
- **Chat?** Join our Discord!
- **Email?** Contact [support@chonkie.ai](mailto:support@chonkie.ai)

## 🙏 Thank You

Every contribution helps make Chonkie better! We appreciate your time and effort in helping make Chonkie the CHONKiest it can be!

Remember:
> "A journey of a thousand CHONKs begins with a single commit" ~ Ancient Proverb, probably

CONTRIBUTING.md


# 🦛 Chonkie DOCS

> ugh, do i _need_ to explain how to use chonkie? man, that's a bummer... to be honest, Chonkie is very easy, with little documentation necessary, but just in case, i'll include some here.

# Table of Contents

- [🦛 Chonkie DOCS](#-chonkie-docs)
- [Table of Contents](#table-of-contents)
- [Installation](#installation)
  - [Basic installation](#basic-installation)
  - [Dependency Tables](#dependency-tables)
    - [Chunker Availability](#chunker-availability)
    - [Embeddings Availability](#embeddings-availability)
    - [Required Dependencies](#required-dependencies)
- [Quick Start](#quick-start)
- [Design CHONKosophy](#design-chonkosophy)
  - [core chonk principles](#core-chonk-principles)
    - [1. 🎯 small but precise](#1--small-but-precise)
    - [2. 🚀 surprisingly quick](#2--surprisingly-quick)
    - [3. 🪶 tiny but complete](#3--tiny-but-complete)
    - [4. 🧠 the clever chonk](#4--the-clever-chonk)
    - [5. 🌱 growing with purpose](#5--growing-with-purpose)
  - [why chunking is needed? (and may always be needed!)](#why-chunking-is-needed-and-may-always-be-needed)
  - [does speed matter while chunking? (tl;dr: yes!!!)](#does-speed-matter-while-chunking-tldr-yes)
  - [but but but... how? how is chonkie so fast?](#but-but-but-how-how-is-chonkie-so-fast)
- [Chunkers](#chunkers)
  - [TokenChunker](#tokenchunker)
  - [WordChunker](#wordchunker)
  - [SentenceChunker](#sentencechunker)
  - [SemanticChunker](#semanticchunker)
  - [SDPMChunker](#sdpmchunker)
- [Embeddings](#embeddings)
  - [BaseEmbeddings](#baseembeddings)
  - [Model2Vec Embeddings](#model2vec-embeddings)
  - [SentenceTransformerEmbeddings](#sentencetransformerembeddings)
  - [OpenAIEmbeddings](#openaiembeddings)
  - [Using AutoEmbeddings](#using-autoembeddings)
  - [Creating Custom Embeddings](#creating-custom-embeddings)
- [API Reference](#api-reference)
  - [Chunk object](#chunk-object)
  - [SentenceChunk object](#sentencechunk-object)
  - [SemanticChunk object](#semanticchunk-object)
- [FAQ](#faq)
  - [Can I run a Chunker multiple times on different texts? Is Chonkie thread-safe?](#can-i-run-a-chunker-multiple-times-on-different-texts-is-chonkie-thread-safe)

# Installation

## Basic installation

```bash
pip install chonkie
```

## Dependency Tables

As per the details mentioned in the [design](#design-chonkosophy) section, Chonkie is lightweight because it keeps most of the dependencies for each chunker seperate, making it more of an aggregate of multiple repositories and python packages. The optional dependencies feature in python really helps with this.

### Chunker Availability

The following table shows which chunkers are available with different installation options:

| Chunker | Default | embeddings | 'all' |
|---------|---------|-----------|----------|
| TokenChunker | ✅ | ✅ |  ✅ |
| WordChunker | ✅ | ✅ |  ✅ |
| SentenceChunker | ✅ |  ✅ | ✅ |
| SemanticChunker | ❌ |  ✅ | ✅ |
| SDPMChunker | ❌ | ✅ | ✅ |

Any of the embeddings availability will enable the `SemanticChunker` and `SDPMChunker`. Please check the availability of the embeddings below or you may use the `chonkie[semantic]` install for quick access.

### Embeddings Availability

The following table shows which embedding providers are available with different installation options:

| Embeddings Provider | Default | 'model2vec' | 'st' | 'openai' | 'semantic'| 'all' |
|--------------------|---------|-----------|----------|-------|--------|---------|
| Model2VecEmbeddings | ❌| ✅ | ❌ | ❌ | ✅ | ✅ |
| SentenceTransformerEmbeddings | ❌ | ❌| ✅ | ❌ | ❌ | ✅|
| OpenAIEmbeddings | ❌ | ❌ | ❌ | ✅ | ❌ | ✅|

### Required Dependencies

| Installation Option | Additional Dependencies |
|--------------------|------------------------|
| Default | autotiktokenizer |
| 'model2vec' | + model2vec, numpy |
| 'st' | + sentence-transformers, numpy |
| 'openai' | + openai, tiktoken |
| 'semantic' | + model2vec, numpy |
| 'all' | all above dependencies |

NOTE: We have seperate `semantic` and `all` installs pre-packaged that might match other installation options breeding redundancy. This redundancy is so we can provide the user the best experience with the freedom to choose their prefered means. The `semantic` and `all` optional installs would continue to change in future versions, so what you might expect to download today may not be the same for tomorrow.

You can install the version you need using:

```bash
# Basic installation (TokenChunker, WordChunker, SentenceChunker)
pip install chonkie

# For the default semantic provider support
pip install "chonkie[semantic]"

# For OpenAI embeddings support
pip install "chonkie[openai]"

# For installing multiple features together
pip install "chonkie[st, model2vec]"

# For all features
pip install "chonkie[all]"
```

Note: Installing either 'semantic' or 'openai' extras will enable SemanticChunker and SDPMChunker, as these chunkers can work with any embeddings provider. The difference is in which embedding providers are available for use with these chunkers.

# Quick Start

```python
from chonkie import TokenChunker

# create chunker
chunker = TokenChunker(
    tokenizer="gpt2", # You can pass your desired tokenizer
    chunk_size=512,
    chunk_overlap=128
)

# chunk your text
text = """your long text here..."""
chunks = chunker.chunk(text)

# access chunks
for chunk in chunks:
    print(f"chunk: {chunk.text[:50]}...")
    print(f"tokens: {chunk.token_count}")
```

# Design CHONKosophy

> did you know that pygmy hippos are only 1/4 the size of regular hippos, but they're just as mighty? that's the chonkie spirit - tiny but powerful! 🦛

listen up chonkers! just like our adorable pygmy hippo mascot, chonkie proves that the best things come in small packages. let's dive into why this tiny chonkster is built the way it is!

## core chonk principles

### 1. 🎯 small but precise

like how pygmy hippos take perfect little bites of their favorite fruits, chonkie knows exactly how to size your chunks:

- **compact & efficient**: just like our tiny mascot, every chunk is exactly the size it needs to be
- **smart defaults**: we've done the research so you don't have to! our default parameters are battle-tested
- **flexible sizing**: because sometimes you need a smaller bite!

### 2. 🚀 surprisingly quick

fun fact: pygmy hippos might be small, but they can zoom through the forest at impressive speeds! similarly, chonkie is:

- **lightning fast**: small size doesn't mean slow performance
- **optimized paths**: like our mascot's forest shortcuts, we take the most efficient route (we use cacheing extensively btw!)
- **minimal overhead**: no wasted energy, just pure chonk power

### 3. 🪶 tiny but complete

just as pygmy hippos pack all hippo features into a compact frame, chonkie is:

- **minimum footprint**: base installation smaller than a pygmy hippo footprint
- **modular growth**: add features as you need them, like a growing hippo
- **zero bloat**: every feature has a purpose, just like every trait of our tiny friend
- **smart imports**: load only what you need, when you need it

### 4. 🧠 the clever chonk

why chunking still matters (from a tiny hippo's perspective):

1. **right-sized processing**
   - even tiny chunks can carry big meaning
   - smart chunking = efficient processing
   - our pygmy hippo philosophy: "just enough, never too much"

2. **the goldilocks zone**
   - too small: like a hippo bite that's too tiny
   - too large: like trying to swallow a whole watermelon
   - just right: the chonkie way™ (pygmy-approved!)

3. **semantic sense**
   - each chunk is carefully crafted
   - like our mascot's careful step through the forest
   - small, meaningful units that work together

### 5. 🌱 growing with purpose

like how pygmy hippos stay small but mighty, chonkie grows sensibly:

```
smart chunks → better embeddings → precise retrieval → quality generation
```

even as models grow bigger, you'll appreciate our tiny-but-mighty approach:

- focused context (like a pygmy hippo's keen senses)
- efficient processing (like our mascot's energy-saving size)
- clean, purposeful design (like nature's perfect mini-hippo)

## why chunking is needed? (and may always be needed!)

while you might be aware of models having longer and longer contexts in recent times (as of 2024), models have yet to reach the stage where adding additional context to them comes for free. additional context, even with the greatest of model architectures comes at a o(n) penalty in speed, to say nothing of the additional memory requirements. and as long as we belive in that attention is all we need, it doesn't seem likely we would be free from this penalty.

that means, to make models run efficiently (lower speed, memory) it is absoulutely vital that we provide the most accurate information it needs during the retrieval phase.

accuracy is one part during retrieval and the other is granularity. you might be able to extract the relevant article out for model to work with, but if only 1 line is relevant from that passage, you are in effect adding a lot of noise that would hamper and confuse the model in practice. you want and hope to give the model only what it should require ideally (of course, the ideal scenario is rarely ever possible). this finally brings us to granularity and retrieval accuracy.

representation models (or embedding models as you may call them) are great at representing large amount of information (sometimes pages of text) in a limited space of just 700-1000 floats, but that doesn't mean it does not suffer from any loss. most representation is lossy, and if we have many concepts being covered in the same space, it is often that much of it would be lost. however, singluar concepts and explainations breed stronger representation vectors. it then becomes vital again to make sure we don't dilute the representation with noise.

all this brings me back to chunking. chunking, done well, can make sure your representation vector (or embedding) is of high-quality to be able to retrieve the best context for your model to generate with. and that in turn, leads to better quality rag generations. therefore, i believe chunking is here to stay as long as rag is here. and hence, it becomes important that we give it little more than a after-thought.

## does speed matter while chunking? (tl;dr: yes!!!)

human time is limited, and if you have an option that gives you faster chunks, why would you not?

but speed is not just a bonus; it's central to chonkie! whether you are doing rag on the entirity of wikipedia or working for large scale organization data that updates regularly, you would need the speed that chonkie comes with. stock solutions just don't cut it in these scenarios.

## but but but... how? how is chonkie so fast?

we used a lot of optimizations when building each and every chunker inside chonkie, making sure it's as optimized as possible.

1. **using tiktoken (as a default):** tiktoken is around 3-6x faster than it's counterparts; and it is blazing fast when used with multiple threads. we see the available threads on the cpu at the moment, and use about ~70-80% of them (so as to not hog all resources), which inturn let's us tokenize fast.
2. **pre-compute and cache:** we never tokenize or embed on the fly! as long as something can be pre-computed and cached we do that, store it and re-use it wherever possible. ram is cheap but time is priceless. (of course, we also provide options to turn off the pre-computation and make it memory efficient if need be)
3. **running mean pooling:** most semantic chunkers re-embed the chunks every time they get updated, but we don't do that. we pre-compute the embeddings for the sentences, and use mathematical trickery (which is theoretically found) to instead have a running mean pooling of tokens -- which allows us to save the cost from the embedding models.

# Chunkers

## TokenChunker

The `TokenChunker` splits text into chunks based on token count.

**Key Parameters:**

- `tokenizer` (`Optional[str, tokenizers.Tokenizer, tiktoken.Encoding]`): any tokenizer implementing the encode/decode interface
- `chunk_size` (`int`): maximum tokens per chunk
- `chunk_overlap` (`Union[int, float]`): number of overlapping tokens between chunks

**Methods:**

- `chunk`: Chunks a piece of text.
  - **Parameters:**
    - `text` (`str`): The input text to be chunked.
  - **Returns:**
    - `List[Chunk]`: A list of `Chunk` objects containing the chunked text and metadata.

- `chunk_batch`: Chunks a list of strings.
  - **Parameters:**
    - `texts` (`List[str]`): A list of input texts to be chunked.
  - **Returns:**
    - `List[List[Chunk]]`: A list of lists of `Chunk` objects, where each sublist corresponds to the chunks of an input text.

- `__call__`: Takes either a string or a list of strings for chunking.
  - **Parameters:**
    - `text` (`Union[str, List[str]]`): The input text or list of texts to be chunked.
  - **Returns:**
    - `Union[List[Chunk], List[List[Chunk]]]`: A list of `Chunk` objects if a single string is provided, or a list of lists of `Chunk` objects if a list of strings is provided.

**Example Usage:**

```python
# Import the TokenChunker
from chonkie import TokenChunker
from autotiktokenizer import AutoTikTokenizer

# Initialize the tokenizer
tokenizer = AutoTikTokenizer.from_pretrained("gpt2")

# Initialize the chunker
chunker = TokenChunker(
    tokenizer=tokenizer,
    chunk_size=512,  # maximum tokens per chunk
    chunk_overlap=128  # overlap between chunks
)

# Chunk a single piece of text
chunks = chunker.chunk("Woah! Chonkie, the chunking library is so cool! I love the tiny hippo hehe.")
for chunk in chunks:
    print(f"Chunk: {chunk.text}")
    print(f"Tokens: {chunk.token_count}")

# Chunk a batch of texts
texts = ["First text to chunk.", "Second text to chunk."]
batch_chunks = chunker.chunk_batch(texts)
for text_chunks in batch_chunks:
    for chunk in text_chunks:
        print(f"Chunk: {chunk.text}")
        print(f"Tokens: {chunk.token_count}")

# Use the chunker as a callable
chunks = chunker("Another text to chunk using __call__.")
for chunk in chunks:
    print(f"Chunk: {chunk.text}")
    print(f"Tokens: {chunk.token_count}")
```

## WordChunker

The `WordChunker` maintains word boundaries while chunking, ensuring words stay intact.

**Key Parameters:**

- `tokenizer` (`Optional[str, tokenizers.Tokenizer, tiktoken.Encoding]`): Any tokenizer implementing the encode/decode interface
- `chunk_size` (`int`): Maximum tokens per chunk
- `chunk_overlap` (`int`): Number of overlapping tokens between chunks
- `mode` (`str`): Chunking mode, either 'simple' (space-based splitting) or 'advanced' (handles punctuation and special cases)

**Methods:**

- `chunk`: Chunks a piece of text.
  - **Parameters:**
    - `text` (`str`): The input text to be chunked.
  - **Returns:**
    - `List[Chunk]`: A list of `Chunk` objects containing the chunked text and metadata.

- `chunk_batch`: Chunks a list of strings.
  - **Parameters:**
    - `texts` (`List[str]`): A list of input texts to be chunked.
  - **Returns:**
    - `List[List[Chunk]]`: A list of lists of `Chunk` objects, where each sublist corresponds to the chunks of an input text.

- `__call__`: Takes either a string or a list of strings for chunking.
  - **Parameters:**
    - `text` (`Union[str, List[str]]`): The input text or list of texts to be chunked.
  - **Returns:**
    - `Union[List[Chunk], List[List[Chunk]]]`: A list of `Chunk` objects if a single string is provided, or a list of lists of `Chunk` objects if a list of strings is provided.

**Example Usage:**

```python
from chonkie import WordChunker
from autotiktokenizer import AutoTikTokenizer

tokenizer = AutoTikTokenizer.from_pretrained("gpt2")

chunker = WordChunker(
    tokenizer=tokenizer,
    chunk_size=512,
    chunk_overlap=128,
    mode="advanced"
)

# Chunk a single piece of text
chunks = chunker.chunk("Some text to chunk while preserving word boundaries.")
for chunk in chunks:
    print(f"Chunk: {chunk.text}")
    print(f"Tokens: {chunk.token_count}")
```

## SentenceChunker

The `SentenceChunker` preserves sentence boundaries while chunking text.

**Key Parameters:**

- `tokenizer` (`Optional[str, tokenizers.Tokenizer, tiktoken.Encoding]`): Any tokenizer implementing the encode/decode interface
- `chunk_size` (`int`): Maximum tokens per chunk
- `chunk_overlap` (`int`): Number of overlapping tokens between chunks
- `min_sentences_per_chunk` (`int`): Minimum number of sentences to include in each chunk

**Methods:**

- `chunk`: Chunks a piece of text.
  - **Parameters:**
    - `text` (`str`): The input text to be chunked.
  - **Returns:**
    - `List[SentenceChunk]`: A list of `SentenceChunk` objects containing the chunked text and metadata, including individual sentences.

- `chunk_batch`: Chunks a list of strings.
  - **Parameters:**
    - `texts` (`List[str]`): A list of input texts to be chunked.
  - **Returns:**
    - `List[List[SentenceChunk]]`: A list of lists of `SentenceChunk` objects.

- `__call__`: Takes either a string or a list of strings for chunking.
  - **Parameters:**
    - `text` (`Union[str, List[str]]`): The input text or list of texts to be chunked.
  - **Returns:**
    - `Union[List[SentenceChunk], List[List[SentenceChunk]]]`: A list of `SentenceChunk` objects or a list of lists of `SentenceChunk` objects.

**Example Usage:**

```python
from chonkie import SentenceChunker
from autotiktokenizer import AutoTikTokenizer

tokenizer = AutoTikTokenizer.from_pretrained("gpt2")

chunker = SentenceChunker(
    tokenizer=tokenizer,
    chunk_size=512,
    chunk_overlap=128,
    min_sentences_per_chunk=1
)

# Chunk a single piece of text
chunks = chunker.chunk("First sentence. Second sentence. Third sentence.")
for chunk in chunks:
    print(f"Chunk: {chunk.text}")
    print(f"Number of sentences: {len(chunk.sentences)}")
```

## SemanticChunker

The `SemanticChunker` groups content by semantic similarity. The implementation is inspired by the semantic chunking approach described in the [FullStackRetrieval Tutorials](https://github.com/FullStackRetrieval-com/RetrievalTutorials/blob/main/tutorials/LevelsOfTextSplitting/5_Levels_Of_Text_Splitting.ipynb), with modifications and optimizations for better performance and integration with Chonkie's architecture.

This version of `SemanticChunker` has some optimizations that speed it up considerably, but make the assumption that the `tokenizer` you used is the same as the one used for `embedding_model`. This is a valid assumption since most often than not, `chunk_size` and hence, `token_count` is dependent on the `embedding_model` context sizes rather than on the Generative models context length.

**Key Parameters:**

- `embedding_model` (`Union[str, SentenceTransformer]`): Model for semantic embeddings, either a model name string or a SentenceTransformer instance
- `similarity_threshold` (`Optional[float]`): Minimum similarity score to consider sentences similar (0-1)
- `similarity_percentile` (`Optional[float]`): Minimum similarity percentile to consider sentences similar (0-100)
- `chunk_size` (`Optional[int]`): Maximum tokens allowed per chunk
- `initial_sentences` (`Optional[int]`): Number of sentences to start each chunk with

**Methods:**

- `chunk`: Chunks a piece of text using semantic similarity.
  - **Parameters:**
    - `text` (`str`): The input text to be chunked.
  - **Returns:**
    - `List[SemanticChunk]`: A list of `SemanticChunk` objects containing semantically coherent chunks.

- `chunk_batch`: Chunks a list of strings.
  - **Parameters:**
    - `texts` (`List[str]`): A list of input texts to be chunked.
  - **Returns:**
    - `List[List[SemanticChunk]]`: A list of lists of `SemanticChunk` objects.

- `__call__`: Takes either a string or a list of strings for chunking.
  - **Parameters:**
    - `text` (`Union[str, List[str]]`): The input text or list of texts to be chunked.
  - **Returns:**
    - `Union[List[SemanticChunk], List[List[SemanticChunk]]]`: A list of `SemanticChunk` objects or a list of lists of `SemanticChunk` objects.

**Example Usage:**

```python
from chonkie import SemanticChunker

chunker = SemanticChunker(
    embedding_model="minishlab/potion-base-8M", # Default model supported with SemanticChunker
    chunk_size=512,
    similarity_threshold=0.7
)

# Chunk a single piece of text
chunks = chunker.chunk("Some text with semantic meaning to chunk appropriately.")
for chunk in chunks:
    print(f"Chunk: {chunk.text}")
    print(f"Number of semantic sentences: {len(chunk.sentences)}")
```

## SDPMChunker

the `SDPMChunker` groups content via the semantic double-pass merging method, which groups paragraphs that are semantically similar even if they do not occur consecutively, by making use of a skip-window.

**Key Parameters:**

- `embedding_model` (`Union[str, SentenceTransformer]`): Model for semantic embeddings, either a model name string or a SentenceTransformer instance
- `similarity_threshold` (`Optional[float]`): Minimum similarity score to consider sentences similar (0-1)
- `similarity_percentile` (`Optional[float]`): Minimum similarity percentile to consider sentences similar (0-100)
- `chunk_size` (`Optional[int]`): Maximum tokens allowed per chunk
- `initial_sentences` (`Optional[int]`): Number of sentences to start each chunk with
- `skip_window` (`Optional[int]`): Number of chunks to skip when looking for similarities

**Methods:**

- `chunk`: Chunks a piece of text using semantic double-pass merging.
  - **Parameters:**
    - `text` (`str`): The input text to be chunked.
  - **Returns:**
    - `List[SemanticChunk]`: A list of `SemanticChunk` objects containing semantically coherent chunks.

- `chunk_batch`: Chunks a list of strings.
  - **Parameters:**
    - `texts` (`List[str]`): A list of input texts to be chunked.
  - **Returns:**
    - `List[List[SemanticChunk]]`: A list of lists of `SemanticChunk` objects.

- `__call__`: Takes either a string or a list of strings for chunking.
  - **Parameters:**
    - `text` (`Union[str, List[str]]`): The input text or list of texts to be chunked.
  - **Returns:**
    - `Union[List[SemanticChunk], List[List[SemanticChunk]]]`: A list of `SemanticChunk` objects or a list of lists of `SemanticChunk` objects.

**Example Usage:**

```python
from chonkie import SDPMChunker

chunker = SDPMChunker(
    embedding_model="minishlab/potion-base-8M",
    chunk_size=512,
    similarity_threshold=0.7,
    skip_window=1
)

# Chunk a single piece of text
chunks = chunker.chunk("Some text with related but non-consecutive content to chunk.")
for chunk in chunks:
    print(f"Chunk: {chunk.text}")
    print(f"Number of semantic sentences: {len(chunk.sentences)}")
```

# Embeddings

Chonkie provides a flexible embeddings system that can be used with various embedding providers. The embeddings system is designed to work seamlessly with the semantic chunking features.

## BaseEmbeddings

All embedding implementations in Chonkie inherit from the `BaseEmbeddings` abstract class, which defines the common interface:

```python
class BaseEmbeddings:
    def embed(self, text: str) -> np.ndarray:
        """Embed a single text into a vector."""
        pass

    def embed_batch(self, texts: List[str]) -> List[np.ndarray]:
        """Embed multiple texts into vectors."""
        pass

    def count_tokens(self, text: str) -> int:
        """Count tokens in a text."""
        pass

    def similarity(self, u: np.ndarray, v: np.ndarray) -> float:
        """Compute similarity between two embeddings."""
        pass

    @property
    def dimension(self) -> int:
        """Return embedding dimension."""
        pass
```

## Model2Vec Embeddings

Uses distilled static embedding models with help of [`model2vec`](https://github.com/MinishLab/model2vec) package. These models are 500x faster than standard `SentenceTransformer` models and about 15x smaller with the `potion-base-8M` being just about 30MB. When used in conjuction with `chonkie[model2vec]` the entire package for `SemanticChunker` usage is just about 57MiB, the smallest of all the options and a 10x smaller package size than the other stock options.

```python
from chonkie.embeddings import Model2VecEmbeddings, AutoEmbeddings

# Initialise with the Model2VecEmbeddings class
embeddings = Model2VecEmbeddings("minishlab/potion-base-8M")

# OR initialise with the AutoEmbeddings get_embeddings()
embeddings = AutoEmbeddings.get_embeddings("minishlab/potion-base-8M")

chunker = SemanticChunker(
    embedding_model=embeddings,
    similarity_threshold=0.5,
)
```

Available potion models from [Minish lab](https://minishlab.github.io/):

- [potion-base-8M](https://huggingface.co/minishlab/potion-base-8M)
- [potion-base-4M](https://huggingface.co/minishlab/potion-base-4M)
- [potion-base-2M](https://huggingface.co/minishlab/potion-base-2M)

Resources:

- Model2Vec [blog](https://minishlab.github.io/hf_blogpost/)

## SentenceTransformerEmbeddings

Uses Sentence Transformers models for creating embeddings.

```python
from chonkie.embeddings import SentenceTransformerEmbeddings

# Initialize with default model
embeddings = SentenceTransformerEmbeddings()

# Use specific model
embeddings = SentenceTransformerEmbeddings("paraphrase-MiniLM-L6-v2")

# Use with semantic chunker
chunker = SemanticChunker(embedding_model=embeddings)
```

## OpenAIEmbeddings

Uses OpenAI's API for creating embeddings.

```python
from chonkie.embeddings import OpenAIEmbeddings

# Initialize with API key
embeddings = OpenAIEmbeddings(
    model="text-embedding-3-small",
    api_key="your-api-key"  # Optional if OPENAI_API_KEY env var is set
)

# Configure batch size and timeouts
embeddings = OpenAIEmbeddings(
    model="text-embedding-3-large",
    batch_size=32,
    timeout=30.0
)

# Use with semantic chunker
chunker = SemanticChunker(embedding_model=embeddings)
```

Available OpenAI models:

- `text-embedding-3-small` (1536 dimensions, best performance/cost ratio)
- `text-embedding-3-large` (3072 dimensions, highest performance)
- `text-embedding-ada-002` (1536 dimensions, legacy model)

## Using AutoEmbeddings

The `AutoEmbeddings` class provides a convenient way to load embeddings:

```python
from chonkie.embeddings import AutoEmbeddings

# Load sentence transformers
embeddings = AutoEmbeddings.get_embeddings("sentence-transformers/all-MiniLM-L6-v2")

# Load OpenAI embeddings
embeddings = AutoEmbeddings.get_embeddings(
    "openai/text-embedding-3-small",
    api_key="your-api-key"
)

# Use directly with semantic chunker
chunker = SemanticChunker(
    embedding_model="openai/text-embedding-3-small",
    api_key="your-api-key"
)
```

## Creating Custom Embeddings

You can create custom embedding implementations by inheriting from `BaseEmbeddings`:

```python
from chonkie.embeddings import BaseEmbeddings

class CustomEmbeddings(BaseEmbeddings):
    def embed(self, text: str) -> np.ndarray:
        # Implement embedding logic
        pass

    def count_tokens(self, text: str) -> int:
        # Implement token counting
        pass

    # Implement other required methods...

# Register with the embeddings registry
EmbeddingsRegistry.register(
    "custom",
    CustomEmbeddings,
    pattern=r"^custom/|^model-name"
)
```

# API Reference

## Chunk object

```python
@dataclass
class Chunk:
    text: str           # the chunk text
    start_index: int    # starting position in original text
    end_index: int      # ending position in original text
    token_count: int    # number of tokens in chunk
```

## SentenceChunk object

```python
@dataclass
class Sentence:
    text: str
    start_index: int
    end_index: int
    token_count: int

@dataclass
class SentenceChunk(Chunk):
    text: str
    start_index: int
    end_index: int
    token_count: int
    sentences: list[Sentence]
```

## SemanticChunk object

```python
@dataclass
class SemanticSentence(Sentence):
    text: str
    start_index: int
    end_index: int
    token_count: int
    embedding: optional[np.ndarray]

@dataclass
class SemanticChunk(SentenceChunk):
    text: str
    start_index: int
    end_index: int
    token_count: int
    sentences: list[SemanticSentence]
```

# FAQ

## Can I run a Chunker multiple times on different texts? Is Chonkie thread-safe?

Yes! Chonkie's Chunkers can be run multiple times without having to re-initialize them. Just initialise them once like you would expect to, and run them on any piece of text you might want to.

That also means it is absolutely thread-safe! But I would recommend monitoring the CPU usage, since few Chunkers frequently default to multi-threaded chunking (like WordChunker and SentenceChunker) so your resources might be depleted faster than usual running these Chunkers.

DOCS.md


<div align='center'>

![Chonkie Logo](/assets/chonkie_logo_br_transparent_bg.png)

# 🦛 Chonkie ✨

[![PyPI version](https://img.shields.io/pypi/v/chonkie.svg)](https://pypi.org/project/chonkie/)
[![License](https://img.shields.io/github/license/bhavnicksm/chonkie.svg)](https://github.com/bhavnicksm/chonkie/blob/main/LICENSE)
[![Documentation](https://img.shields.io/badge/docs-chonkie.ai-blue.svg)](https://docs.chonkie.ai)
![Package size](https://img.shields.io/badge/size-15MB-blue)
[![Downloads](https://static.pepy.tech/badge/chonkie)](https://pepy.tech/project/chonkie)
[![Discord](https://dcbadge.limes.pink/api/server/https://discord.gg/rYYp6DC4cv?style=flat)](https://discord.gg/rYYp6DC4cv)
[![GitHub stars](https://img.shields.io/github/stars/bhavnicksm/chonkie.svg)](https://github.com/bhavnicksm/chonkie/stargazers)

_The no-nonsense RAG chunking library that's lightweight, lightning-fast, and ready to CHONK your texts_

[Installation](#installation) •
[Usage](#usage) •
[Supported Methods](#supported-methods) •
[Benchmarks](#benchmarks-) •
[Documentation](https://docs.chonkie.ai) •
[Contributing](#contributing)

</div>

Ever found yourself building yet another RAG bot (your 2,342,148th one), only to hit that all-too-familiar wall? You know the one —— where you're stuck choosing between:

- Library X: A behemoth that takes forever to install and probably includes three different kitchen sinks
- Library Y: So bare-bones it might as well be a "Hello World" program
- Writing it yourself? For the 2,342,149th time, _sigh_

And you think to yourself:

> "WHY CAN'T THIS JUST BE SIMPLE?!" </br>
> "Why do I need to choose between bloated and bare-bones?" </br>
> "Why can't I just install, import, and CHONK?!" </br>

Well, look no further than Chonkie! (chonkie boi is a gud boi 🦛💕)

**🚀 Feature-rich**: All the CHONKs you'd ever need </br>
**✨ Easy to use**: Install, Import, CHONK </br>
**⚡ Fast**: CHONK at the speed of light! zooooom </br>
**🌐 Wide support**: Supports all your favorite tokenizer CHONKS </br>
**🪶 Light-weight**: No bloat, just CHONK </br>
**🦛 Cute CHONK mascot**: psst it's a pygmy hippo btw </br>
**❤ [Moto Moto](#acknowledgements)'s favorite python library** </br>

**Chonkie** is a chunking library that "**just works™**".

# Installation

To install chonkie, simply run:

```bash
pip install chonkie
```

Chonkie follows the rule to have minimal default installs, read the [DOCS](https://docs.chonkie.ai) to know the installation for your required chunker, or simply install `all` if you don't want to think about it (not recommended).

```bash
pip install chonkie[all]
```

# Usage

Here's a basic example to get you started:

```python
# First import the chunker you want from Chonkie
from chonkie import TokenChunker

# Import your favorite tokenizer library
# Also supports AutoTokenizers, TikToken and AutoTikTokenizer
from tokenizers import Tokenizer
tokenizer = Tokenizer.from_pretrained("gpt2")

# Initialize the chunker
chunker = TokenChunker(tokenizer)

# Chunk some text
chunks = chunker("Woah! Chonkie, the chunking library is so cool! I love the tiny hippo hehe.")

# Access chunks
for chunk in chunks:
    print(f"Chunk: {chunk.text}")
    print(f"Tokens: {chunk.token_count}")
```

More example usages given inside the [DOCS](https://docs.chonkie.ai)

# Supported Methods

Chonkie provides several chunkers to help you split your text efficiently for RAG applications. Here's a quick overview of the available chunkers:

- **TokenChunker**: Splits text into fixed-size token chunks.
- **WordChunker**: Splits text into chunks based on words.
- **SentenceChunker**: Splits text into chunks based on sentences.
- **RecursiveChunker**: Splits text hierarchically using customizable rules to create semantically meaningful chunks.
- **SemanticChunker**: Splits text into chunks based on semantic similarity.
- **SDPMChunker**: Splits text using a Semantic Double-Pass Merge approach.
- **LateChunker (experimental)**: Embeds text and then splits it to have better chunk embeddings.

More on these methods and the approaches taken inside the [DOCS](https://docs.chonkie.ai)

# Benchmarks 🏃♂

> "I may be smol hippo, but I pack a punch!" 🦛

Here's a quick peek at how Chonkie performs:

**Size**📦

- **Default Install:** 15MB (vs 80-171MB for alternatives)
- **With Semantic:** Still lighter than the competition!

**Speed**⚡

- **Token Chunking:** 33x faster than the slowest alternative
- **Sentence Chunking:** Almost 2x faster than competitors
- **Semantic Chunking:** Up to 2.5x faster than others

Check out our detailed [benchmarks](https://docs.chonkie.ai/benchmarks) to see how Chonkie races past the competition! 🏃♂💨

# Contributing

Want to help make Chonkie even better? Check out our [CONTRIBUTING.md](CONTRIBUTING.md) guide! Whether you're fixing bugs, adding features, or improving docs, every contribution helps make Chonkie a better CHONK for everyone.

Remember: No contribution is too small for this tiny hippo! 🦛

# Acknowledgements

Chonkie would like to CHONK its way through a special thanks to all the users and contributors who have helped make this library what it is today! Your feedback, issue reports, and improvements have helped make Chonkie the CHONKIEST it can be.

And of course, special thanks to [Moto Moto](https://www.youtube.com/watch?v=I0zZC4wtqDQ&t=5s) for endorsing Chonkie with his famous quote:
> "I like them big, I like them chonkie."
>                                         ~ Moto Moto

# Citation

If you use Chonkie in your research, please cite it as follows:

```bibtex
@misc{chonkie2024,
  author = {Minhas, Bhavnick},
  title = {Chonkie: A Fast Feature-full Chunking Library for RAG Bots},
  year = {2024},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/bhavnick/chonkie}},
}
```

README.md


# 🦛 CHONK vs The World

> Look how **THICC** the competition is! Meanwhile, Chonkie be looking slim and trim 💪

Ever wondered how much CHONKier other text splitting libraries are? Well, wonder no more! We've put Chonkie up against some of the most popular RAG libraries out there, and the results are... well, let's just say Moto Moto might need to revise his famous quote!

## ⚡ Speed Benchmarks

> ZOOOOOM! Watch Chonkie run! 🏃♂💨

### 100K Wikipedia Articles

The following benchmarks were run on 100,000 Wikipedia articles from the
[`chonkie-ai/wikipedia-100k`](https://huggingface.co/datasets/chonkie-ai/wikipedia-100k) dataset

All tests were run on a Google Colab A100 instance.

#### Token Chunking

| Library | Time | Speed Factor |
|---------|-----------|--------------|
| 🦛 Chonkie | 58 sec | 1x  |
| 🔗 LangChain | 1 min 10 sec | 1.21x slower |
| 📚 LlamaIndex | 50 min | 51.7x slower |

#### Sentence Chunking

| Library | Time | Speed Factor |
|---------|-----------|--------------|
| 🦛 Chonkie | 59 sec | 1x |
| 📚 LlamaIndex | 3 min 59 sec | 4.05x slower |
| 🔗 LangChain | N/A | Doesn't exist |

#### Recursive Chunking

| Library | Time | Speed Factor |
|---------|-----------|--------------|
| 🦛 Chonkie | 1 min 19 sec | 1x |
| 🔗 LangChain | 2 min 45 sec | 2.09x slower |
| 📚 LlamaIndex | N/A | Doesn't exist |

#### Semantic Chunking

Tested with `sentence-transformers/all-minilm-l6-v2` model unless specified otherwise.

| Library | Time | Speed Factor |
|---------|-----------|--------------|
| 🦛 Chonkie (with default settings) | 13 min 59 sec | 1x |
| 🦛 Chonkie | 1 hour 8 min min 53 sec |  4.92x slower |
| 🔗 LangChain | 1 hour 13 sec | 4.35x slower |
| 📚 LlamaIndex | 1 hour 24 min 15 sec| 6.07x slower |

### 500K Wikipedia Articles

The following benchmarks were run on 500,000 Wikipedia articles from the
[`chonkie-ai/wikipedia-500k`](https://huggingface.co/datasets/chonkie-ai/wikipedia-500k) dataset

All tests were run on a `c3-highmem-4` VM from Google Cloud with 32 GB RAM and a 200 GB SSD Persistent Disk attachment.

#### Token Chunking

| Library | Time | Speed Factor |
|---------|-----------|--------------|
| 🦛 Chonkie | 2 min 17 sec | 1x |
| 🔗 LangChain | 2 min 42 sec | 1.18x slower |
| 📚 LlamaIndex | 50 min | 21.9x slower |

#### Sentence Chunking

| Library | Time | Speed Factor |
|---------|-----------|--------------|
| 🦛 Chonkie | 7 min 16 sec | 1x |
| 📚 LlamaIndex | 10 min 55 sec | 1.5x slower |
| 🔗 LangChain | N/A | Doesn't exist |

#### Recursive Chunking

| Library | Time | Speed Factor |
|---------|-----------|--------------|
| 🦛 Chonkie | 3 min 42 sec | 1x |
| 🔗 LangChain | 7 min 36 sec | 2.05x slower |
| 📚 LlamaIndex | N/A | Doesn't exist |

### Paul Graham Essays Dataset

The following benchmarks were run on the Paul Graham Essays dataset using the GPT-2 tokenizer.

#### Token Chunking

| Library | Time (ms) | Speed Factor |
|---------|-----------|--------------|
| 🦛 Chonkie | 8.18 | 1x |
| 🔗 LangChain | 8.68 | 1.06x slower |
| 📚 LlamaIndex | 272 | 33.25x slower |

#### Sentence Chunking

| Library | Time (ms) | Speed Factor |
|---------|-----------|--------------|
| 🦛 Chonkie | 52.6 | 1x |
| 📚 LlamaIndex | 91.2 | 1.73x slower |
| 🔗 LangChain | N/A | Doesn't exist |

#### Semantic Chunking

| Library | Time | Speed Factor |
|---------|------|--------------|
| 🦛 Chonkie | 482ms | 1x |
| 🔗 LangChain | 899ms | 1.86x slower |
| 📚 LlamaIndex | 1.2s | 2.49x slower |

## 📊 Size Comparison (Package Size)

### Default Installation (Basic Chunking)

| Library | Size | Chonk Factor |
|---------|------|--------------|
| 🦛 Chonkie | 11.2 MiB | 1x |
| 🔗 LangChain | 80 MiB | ~7.1x CHONKier |
| 📚 LlamaIndex | 171 MiB | ~15.3x CHONKier |

### With Semantic Features

| Library | Size | Chonk Factor |
|---------|------|--------------|
| 🦛 Chonkie | 62 MiB | 1x |
| 🔗 LangChain | 625 MiB | ~10x CHONKier |
| 📚 LlamaIndex | 678 MiB | ~11x CHONKier |

## 💡 Why These Numbers Matter

### Speed Benefits

1. **Faster Processing**: Chonkie leads in all chunking methods!
2. **Production Ready**: Optimized for real-world usage
3. **Consistent Performance**: Fast across all chunking types
4. **Scale Friendly**: Process more text in less time

### Size Benefits

1. **Faster Installation**: Less to download = faster to get started
2. **Lower Memory Footprint**: Lighter package = less RAM usage
3. **Cleaner Dependencies**: Only install what you actually need
4. **CI/CD Friendly**: Faster builds and deployments

Remember what Chonkie always says:
> "I may be a hippo, but I'm still light and fast!" 🦛✨

---

*Note: All measurements were taken using Python 3.8+ on a clean virtual environment. Your actual mileage may vary slightly depending on your specific setup and dependencies.*

benchmarks/README.md


![Chonkie Cookbook Logo](/assets/chonkbook_logo_transparent_bg.png)

# 🦛 Chonkie's Cookbook 📚

_A collection of recipes, tutorials, and projects using Chonkie_

## 📖 Official Tutorials

| Tutorial | Description | Try it |
|----------|-------------|-----------------|
| Recursive Chunking for PDF/Markdown Documents | A quick look at how you can use Chonkie's RecursiveChunker to chunk your PDF/Markdown documents and build a RAG pipeline over it! | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/bhavnicksm/chonkie/blob/main/cookbook/tutorials/Chonkie_RecursiveChunker_for_PDF_and_Markdown.ipynb) |

## 🔌 Community Integrations

Chonkie is proud to power a diverse ecosystem of projects and integrations across the AI and ML landscape. Our lightweight, flexible architecture makes it easy to integrate Chonkie into your existing workflows and tools.

Want to add your integration? Simply [raise an issue](https://github.com/bhavnicksm/chonkie/issues/new) and we'll help you get started! Once approved, your integration will be featured here.

| Integration | Description | Repository |
|-------------|-------------|------------|
| ⚡ FlashRAG | A Python toolkit for the reproduction and development of Retrieval Augmented Generation (RAG) research. | [![GitHub](https://img.shields.io/badge/GitHub-181717?style=plastic&logo=github&logoColor=white&labelColor=464646)](https://github.com/RUC-NLPIR/FlashRAG) |
| 📄 txtai | All-in-one embeddings database for semantic search, LLM orchestration and language model workflows | [![GitHub](https://img.shields.io/badge/GitHub-181717?style=plastic&logo=github&logoColor=white&labelColor=464646)](https://github.com/neuml/txtai) |
| [Add your integration here!](CONTRIBUTING.md) | Share your Chonkie integration with the community | [![GitHub](https://img.shields.io/badge/GitHub-181717?style=plastic&logo=github&logoColor=white&labelColor=464646)](CONTRIBUTING.md) |

## 📚 Community Blogs & Resources

Share your knowledge and experiences with Chonkie! Whether it's a blog post, video tutorial, or technical deep-dive, we'd love to feature your content here.

| Type | Title | Author | Description |
|------|--------|--------|-------------|
| Video | [Chonkie - RAG Chunking Library - Easy Local Installation and Testing](https://www.youtube.com/watch?v=f4f1TkeL5Hk) | Fahd Mirza | A quick walkthrough of how to install Chonkie and test it out for yourself! |
| Blog | [Easy Late-Chunking with Chonkie](https://pub.towardsai.net/easy-late-chunking-with-chonkie-7f05e5916997) | Michael Ryaboy | This blog post is a quick look at how you can use Chonkie's LateChunker to chunk your documents and build a search pipeline over it with KDB.AI! |

Have content to share? Open an issue with your resource and we'll get it added to the cookbook!

## 📝 Contributing

Want to add your project or integration to the cookbook? Check out our [contribution guidelines](/CONTRIBUTING.md)!

---

Need help? Join our [Discord community](https://discord.gg/rYYp6DC4cv)!

cookbook/README.md


---
name: 🐛 Bug Report
about: Report a bug in Chonkie
title: "[BUG] "
labels: bug
assignees: bhavnicksm, shreyashnigam

---

## 🐛 Bug Description
<!-- A clear description of what's going wrong -->

## 🔍 Minimal Example
<!-- A small, self-contained code example that demonstrates the issue -->
```python
from chonkie import TokenChunker

# Your minimal example here
```

## 💻 Environment
<!-- Please complete the following information -->
- Chonkie Version: <!-- e.g., 0.4.0 -->
- Python Version: <!-- e.g., 3.9.7 -->
- OS: <!-- e.g., Ubuntu 22.04, Windows 11, macOS 13.1 -->
- Installation Method: <!-- e.g., pip install chonkie, pip install chonkie[all] -->
- Python Environment: <!-- e.g. pip freeze or pip list -->

## 📋 Current Behavior
<!-- What actually happened? Include full error messages and/or screenshots if applicable -->

```bash
Error message or output here
```

## ✨ Expected Behavior
<!-- What did you expect to happen? -->

## 📝 Additional Context
<!-- Any other relevant information? -->
<!-- e.g., Are you using any specific tokenizer? Processing large files? -->

## ✅ Reproduction Rate
<!-- How often does this bug occur? -->
- [ ] Always
- [ ] Sometimes
- [ ] Rarely
- [ ] Not sure

## 🔄 Workaround
<!-- If you found a temporary workaround, please share it here -->

<!-- Thank you for helping make Chonkie better! 🦛 -->


---
name: ✨ Feature Request
about: Suggest a new feature for Chonkie
title: "[FEAT] "
labels: enhancement
assignees: bhavnicksm, shreyashnigam

---

## 📋 Quick Check

- [ ] I've checked this feature isn't already implemented or proposed
- [ ] This feature is relevant to Chonkie's purpose (text chunking for RAG)

## 💡 Feature Description
<!-- What would you like Chonkie to do? -->

## 🛠 Implementation Approach
<!-- How do you think this could be implemented? Code sketches welcome! -->
```python
# Example of how this feature might work
from chonkie import ...

# Your implementation idea
```

## 🎯 Why is this needed?
<!-- What problem does this solve? -->

<!-- Thank you for helping make Chonkie CHONKier! 🦛 -->