Sunbird

/

translate-nllb-1.3b-salt-4bit

like 1

Follow

Sunbird AI 26

@KMayanja , thanks for trying the models. I can't seem to reproduce this as it's working on our end. Could you share the full function call so we can see the issue?

Some things to ensure: Did you update Bits and bytes to the latest version?
!pip install -q -U bitsandbytes

This code snippet should be able to run end-to-end without any issues:

import torch
import transformers

# Load the 4-bit quantized model and tokenizer
model_4bit = transformers.M2M100ForConditionalGeneration.from_pretrained(
    "Sunbird/translate-nllb-1.3b-salt-4bit",
    device_map="auto"
)
tokenizer = transformers.NllbTokenizer.from_pretrained("Sunbird/translate-nllb-1.3b-salt")

text = 'Where is the hospital?'
source_language = 'eng'
target_language = 'lug'

# Mapping for language tokens
language_tokens = {
    'eng': 256047,
    'ach': 256111,
    'lgg': 256008,
    'lug': 256110,
    'nyn': 256002,
    'teo': 256006,
}


device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
inputs = tokenizer(text, return_tensors="pt").to(device)
inputs['input_ids'][0][0] = language_tokens[source_language]

# Generate the translation with beam search
translated_tokens = model_4bit.to(device).generate(
    **inputs,
    forced_bos_token_id=language_tokens[target_language],
    max_length=100,
    num_beams=5,
)

result = tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]
print(result)
#Eddwaliro liri ludda wa?

@akera Updated bitsandbytes and it's still not working. Brings the same error back.

Here's the full code(Error is the same as before):

import torch
import transformers

# Load the 4-bit quantized model and tokenizer
model_4bit = transformers.M2M100ForConditionalGeneration.from_pretrained(
    "Sunbird/translate-nllb-1.3b-salt-4bit",
    device_map="auto"
)
tokenizer = transformers.NllbTokenizer.from_pretrained("Sunbird/translate-nllb-1.3b-salt")
language_tokens = {
    'eng': 256047,
    'lug': 256110,
}

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model_4bit.to(device)

def translate_to_luganda(text, src_lang='eng', tgt_lang='lug'):
    inputs = tokenizer(text, return_tensors="pt").to(device)
    inputs['input_ids'][0][0] = language_tokens[src_lang]
    translated_tokens = model_4bit.generate(
        **inputs,
        forced_bos_token_id=language_tokens[tgt_lang],
        max_length=100,
        num_beams=5,
    )
    return tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]
import torch
import transformers

# Load the 4-bit quantized model and tokenizer
model_4bit = transformers.M2M100ForConditionalGeneration.from_pretrained(
    "Sunbird/translate-nllb-1.3b-salt-4bit",
    device_map="auto"
)
tokenizer = transformers.NllbTokenizer.from_pretrained("Sunbird/translate-nllb-1.3b-salt")
language_tokens = {
    'eng': 256047,
    'lug': 256110,
}

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model_4bit.to(device)

def translate_to_luganda(text, src_lang='eng', tgt_lang='lug'):
    inputs = tokenizer(text, return_tensors="pt").to(device)
    inputs['input_ids'][0][0] = language_tokens[src_lang]
    translated_tokens = model_4bit.generate(
        **inputs,
        forced_bos_token_id=language_tokens[tgt_lang],
        max_length=100,
        num_beams=5,
    )
    return tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]

@KMayanja the issue is probably because https://huggingface.co/Sunbird/translate-nllb-1.3b-salt is a gated model;
Please accept the terms and request access to it,
Then add your huggingface auth key to your environment, see code below

import torch
import transformers
import os 

os.environ["HF_TOKEN"] = "hf_XXXXXXXX"  # Replace with your actual key

# Load the 4-bit quantized model and tokenizer
model_4bit = transformers.M2M100ForConditionalGeneration.from_pretrained(
    "Sunbird/translate-nllb-1.3b-salt-4bit",
    device_map="auto"
)
tokenizer = transformers.NllbTokenizer.from_pretrained("Sunbird/translate-nllb-1.3b-salt")

Success!!!

Thank you very much. I tried this and it worked. Better yet, I realised my HF_TOKEN wasn't a part of my notebook secrets (Not sure how that happened.)

Again thank you very much.