tokenizers
Collection
trained and adapted tokenizers - various
•
19 items
•
Updated
BEE-spoke-data/MiniTokenizer-20480
This is a ByteLevelBPETokenizer trained on the JeanKaddour/minipile dataset with the aim to create a compact English-only tokenizer.
Usage
load with AutoTokenizer, i.e.:
from transformers import AutoTokenizer
tk = AutoTokenizer.from_pretrained('BEE-spoke-data/MiniTokenizer-20480')
tk
Inference Providers
NEW
This model isn't deployed by any Inference Provider.
🙋
Ask for provider support