|
--- |
|
license: other |
|
license_name: fair-noncommercial-research-license |
|
license_link: https://huggingface.co/facebook/blt/blob/main/LICENSE |
|
extra_gated_fields: |
|
First Name: text |
|
Last Name: text |
|
Date of birth: date_picker |
|
Country: country |
|
Affiliation: text |
|
I accept the terms and conditions: checkbox |
|
geo: ip_location |
|
language: |
|
- en |
|
tags: |
|
- facebook |
|
- meta-pytorch |
|
- blt |
|
--- |
|
|
|
# Byte Latent Transformer (BLT) |
|
|
|
This repository contains the model weights for our paper: "Byte Latent Transformer: Patches Scale Better Than Tokens" |
|
|
|
- [Paper Link](https://dl.fbaipublicfiles.com/blt/BLT__Patches_Scale_Better_Than_Tokens.pdf) |
|
- [HF Paper Link](https://huggingface.co/papers/2412.09871) |
|
|
|
## Abstract |
|
|
|
We introduce the Byte Latent Transformer architecture (BLTs), a new byte-level LLM architecture that |
|
for the first time, matches tokenization-based LLM performance at scale, with significant improvements |
|
in inference efficiency and robustness. BLT encodes bytes into dynamically sized patches, which serve |
|
as the primary units of computation. Patches are segmented dynamically based on the entropy of the |
|
next byte, allocating more compute and model capacity where there is more data complexity. The BLT |
|
architecture includes new attention mechanisms to maximize the information flow between byte and |
|
patch hidden representations and a new type of byte-sequence memory. We present the first scaling |
|
study of byte-level models up to 8B parameters and 8T training bytes, showing for the first time |
|
that we can train a model end-to-end at scale from bytes with no tokenization or other preprocessing. |
|
Scaling trends reveal training and inference efficiency benefits from dynamically selecting very long |
|
patches on average, along with qualitative improvements with reasoning and long tail generalization |
|
from modeling byte-sequences. |
|
|
|
To run the model, see the readme here: https://github.com/facebookresearch/blt |
|
|
|
## Links |
|
|
|
- Code: https://github.com/facebookresearch/blt |
|
- BLT 1B Weights: https://huggingface.co/facebook/blt-1b |
|
- BLT 7B Weights: https://huggingface.co/facebook/blt-7b |
|
- BLT Weight Collection: https://huggingface.co/collections/facebook/blt-6801263d4ac1704702a192a6 |
|
|
|
|
