Monarch Mixer-BERT

The 80M checkpoint for M2-BERT-128 from the paper Benchmarking and Building Long-Context Retrieval Models with LoCo and M2-BERT.

Check out our GitHub for instructions on how to download and fine-tune it!

How to use

You can load this model using Hugging Face AutoModel:

from transformers import AutoModelForMaskedLM
model = AutoModelForMaskedLM.from_pretrained("jonsaadfalcon/M2-BERT-128-Retrieval-Encoder-V1", trust_remote_code=True)

This model uses the Hugging Face bert-base-uncased tokenizer:

from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

How to use

This model generates embeddings for retrieval. The embeddings have a dimensionality of 768:

from transformers import AutoTokenizer, AutoModelForMaskedLM

max_seq_length = 128
testing_string = "Every morning, I make a cup of coffee to start my day."
model = AutoModelForMaskedLM.from_pretrained("jonsaadfalcon/M2-BERT-128-Retrieval-Encoder-V1", trust_remote_code=True)

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased", model_max_length=max_seq_length)
input_ids = tokenizer([testing_string], return_tensors="pt", padding="max_length", return_token_type_ids=False, truncation=True, max_length=max_seq_length)

outputs = model(**input_ids)
embeddings = outputs['sentence_embedding']

Remote Code

This model requires trust_remote_code=True to be passed to the from_pretrained method. This is because we use custom PyTorch code (see our GitHub). You should consider passing a revision argument that specifies the exact git commit of the code, for example:

mlm = AutoModelForMaskedLM.from_pretrained(
   "jonsaadfalcon/M2-BERT-128-Retrieval-Encoder-V1",
   trust_remote_code=True,
)

Configuration

Note use_flash_mm is false by default. Using FlashMM is currently not supported.

Downloads last month
42
Inference Providers NEW
This model is not currently available via any of the supported Inference Providers.
The model cannot be deployed to the HF Inference API: The model authors have turned it off explicitly.

Collection including hazyresearch/M2-BERT-128-Retrieval-Encoder-V1