ImportError: cannot import name 'TessarTokenizer' from 'transformers' (/usr/local/lib/python3.12/site-packages/transformers/__init__.py)
as title.
Dear @ntutangyun ,
Thank you for reporting the issue regarding the ImportError: cannot import name 'TessarTokenizer' from 'transformers' in the Tessar-largest package.
We have identified and resolved the issue. The problem was related to the tokenizer implementation's integration with the Hugging Face Transformers library. Our fix has been implemented in the tessar_tokenizer.py file in our repository.
The solution includes:
- Proper package structure implementation with correct import paths
- Registration of the TessarTokenizer with the AutoTokenizer class
- Enhanced implementation of required methods for the PreTrainedTokenizerFast interface
The fix has been committed to our repository. Please update to the latest version by pulling the most recent changes or reinstalling the package.
For proper usage, please import the tokenizer as follows:
from tessar_nlp.tessar_tokenizer import TessarTokenizer
# Or use the AutoTokenizer
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("SVECTOR-CORPORATION/Tessar-largest")
Thank you for bringing this to our attention and for your contribution to the Tessar project.
Best regards,
SVECTOR Team