Whats the closest modeling code?

#9
by michaelfeil - opened

For frameworks support, from which model should be best start modeling (llama, bert, qwen)

EuroBERT org

Hello, the easiest model to start from is Llama, as EuroBERT follows the Llama architecture, but without the causal mask.

What are other key differences besides causal=False?

Hey @michaelfeil ,our modelling is available here : https://huggingface.co/EuroBERT/EuroBERT-210m/blob/main/modeling_eurobert.py and the llama one here : https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py

It will be hard to tell you the modifications line by line. If you have a use case, feel free to share it—maybe we can help you better.

Your need to confirm your account before you can post a new comment.

Sign up or log in to comment