Mistral_instructive_full

Model Description

This model is the full-data trained instructive fine-tuned version of Multi-CONFE (Confidence-Aware Medical Feature Extraction), built on unsloth/Mistral-Nemo-Instruct-2407-bnb-4bit. It represents the first phase of the two-phase Multi-CONFE framework, focusing on high-accuracy medical feature extraction from clinical notes without the additional confidence calibration mechanisms.

Intended Use

This model is designed for extracting clinically relevant features from medical patient notes with high accuracy. It's particularly useful for automated assessment of medical documentation, such as USMLE Step-2 Clinical Skills notes, and can serve as an effective foundation for confidence calibration in subsequent fine-tuning.

Training Data

The model was trained on 800 annotated patient notes from the NBME - Score Clinical Patient Notes Kaggle competition dataset. This dataset contains USMLE Step-2 Clinical Skills patient notes covering 10 different clinical cases, with each note containing expert annotations for multiple medical features that need to be extracted. The data were collected from 2017 to 2020 from 35,156 medical students at testing locations in the United States.

Training Procedure

Training involved a single-phase approach:

Instructive Fine-Tuning: Alignment of the model with the medical feature extraction task using Mistral Nemo Instruct as the base model, focused on teaching the model to extract clinically relevant features based on structured prompts.

Training hyperparameters:

Base model: unsloth/Mistral-Nemo-Instruct-2407-bnb-4bit
LoRA rank: 32
Training epochs: 7
Learning rate: 1e-4
Optimizer: AdamW (8-bit)
Weight decay: 0.01
LR scheduler: Linear

Performance

On the USMLE Step-2 Clinical Skills notes dataset:

Precision: 0.974
Recall: 0.952
F1 Score: 0.963

While this model demonstrates strong performance, it does not include the confidence calibration mechanisms of the full Multi-CONFE framework, resulting in a higher rate of hallucinations and less reliable confidence estimation compared to the calibrative version.

Limitations

The model was evaluated on standardized USMLE Step-2 Clinical Skills notes and may require adaptation for other clinical domains.
Without calibration mechanisms, the confidence scores may not accurately reflect extraction performance.
The model has higher hallucination rates compared to the calibrative version.
Performance on multilingual or non-standardized clinical notes remains untested.

Ethical Considerations

Automated assessment systems must ensure fairness across different student populations. Without calibration mechanisms, this model may provide less interpretable confidence scores, which should be considered when using it for high-stakes assessment scenarios. We recommend:

Using this model primarily as a foundation for further calibrative fine-tuning
Implementing a human-in-the-loop validation process when used directly
Considering upgrading to the calibrative version for high-stakes applications

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Load model and tokenizer
model_name = "Manal0809/Mistral_instructive_full"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float16)

# Example input
patient_note = """HPI: 35 yo F with heavy uterine bleeding. Last normal period was 6 month ago. 
LMP was 2 months ago. No clots.
Changes tampon every few hours, previously 4/day. Menarche at 12.
Attempted using OCPs for menstrual regulation previously but unsuccessful.
Two adolescent children (ages unknown) at home.
Last PAP 6 months ago was normal, never abnormal.
Gained 10-15 lbs over the past few months, eating out more though.
Hyperpigmented spots on hands and LT neck that she noticed 1-2 years ago.
SH: state social worker; no smoking or drug use; beer or two on weekends;
sexually active with boyfriend of 14 months, uses condoms at first but no longer uses them."""

features_to_extract = ["35-year", "Female", "heavy-periods", "symptoms-for-6-months", 
                       "Weight-Gain", "Last-menstrual-period-2-months-ago", 
                       "Fatigue", "Unprotected-Sex", "Infertility"]

# Format input as shown in the paper
input_text = f"""###instruction: Extract medical features from the patient note.
###patient_history: {patient_note}
###features: {features_to_extract}
### Annotation:"""

# Generate output
inputs = tokenizer(input_text, return_tensors="pt").to(model.device)
outputs = model.generate(
    inputs["input_ids"],
    max_new_tokens=512,
    temperature=0.2,
    num_return_sequences=1
)
result = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(result)

Model Card Author

Manal Abumelha - [email protected]

Manal0809
/

Mistral_instructive_full