cognitivecomputations

/

dolphin-2.1-mistral-7b

like 257

Follow

Cognitive Computations 1.88k

Hi,
which tokenizer do i have to use? it seems, it's not in the transformers library:

dolphin-2.1-mistral-7b]$ python load_model.py
Traceback (most recent call last):
File "load_model.py", line 9, in
load_model()
File "load_model.py", line 4, in load_model
tokenizer = AutoTokenizer.from_pretrained("ehartford/dolphin-2.1-mistral-7b")
File "/usr/local/lib/python3.6/site-packages/transformers/models/auto/tokenization_auto.py", line 526, in from_pretrained
f"Tokenizer class {tokenizer_class_candidate} does not exist or is not currently imported."
ValueError: Tokenizer class LlamaTokenizer does not exist or is not currently imported.

Thanks!