Request support for text-only inference in transformers (Mistral3ForCausalLM class)
Hello, I want to compliment the Mistral team for their work on Mistral 3.1. I realize the model was just released this week, and it takes a lot of work to support all these different inference platforms.
My request is we add support for text-only prompts for Mistral 3.1. Using the latest Github commit (20 March 2025), I do not think no-image prompts work using the Mistral3ForConditionalGeneration class. I'm guessing we need a Mistral3ForCausalLM class.
yeah, need it too
Hi! Why not just do something like this:
model_id = "mistralai/Mistral-Small-3.1-24B-Instruct-2503"
model = Mistral3ForConditionalGeneration.from_pretrained(model_id, device_map="auto")
lm = model.language_model
Then you should be able to process just texts without needing to pass the images
I did get text-only generation to work. Something like this:
#! /usr/bin/python
from transformers import AutoProcessor, AutoModelForImageTextToText
checkpoint = "mistralai/Mistral-Small-3.1-24B-Instruct-2503"
processor = AutoProcessor.from_pretrained(checkpoint)
model = AutoModelForImageTextToText.from_pretrained(checkpoint)
# `messages` should be a message list with each message having the usual role/content keys
text = processor.apply_chat_template(messages, tokenize = False, add_generation_prompt = True)
inputs = processor(text = text, return_tensors = "pt")
outputs = model.generate(**inputs, ) # more generation arguments
As far as I can tell, no transformers pipeline can currently do text-only inference with this model.
However, in my case, I do not want use 'generate', just forward the model to obtain some hidden states, how to do then?
Sorry, I can't help with this because I don't know. My skills are limited to using either pipeline or model.generate. I know you can call model.forward but I have never had to get at the internal states.