Mistral_instructive_few

Model Description

This model is the few-shot trained instructive fine-tuned version of Multi-CONFE (Confidence-Aware Medical Feature Extraction), built on unsloth/Mistral-Nemo-Instruct-2407-bnb-4bit. It represents the first phase of the two-phase Multi-CONFE framework using minimal training data, focusing on data-efficient medical feature extraction from clinical notes without the additional confidence calibration mechanisms.

Intended Use

This model is designed for extracting clinically relevant features from medical patient notes with good accuracy in low-resource settings. It's particularly useful for automated assessment of medical documentation when training data is limited, and can serve as an effective foundation for confidence calibration in subsequent fine-tuning or for direct use in settings where some hallucination is acceptable.

Training Data

The model was trained on just 100 annotated patient notes (12.5% of the full dataset) from the NBME - Score Clinical Patient Notes Kaggle competition dataset. This represents approximately 10 examples per clinical case type. The dataset contains USMLE Step-2 Clinical Skills patient notes covering 10 different clinical cases, with each note containing expert annotations for multiple medical features that need to be extracted.

Training Procedure

Training involved a single-phase approach:

Instructive Few-Shot Fine-Tuning: Alignment of the model with the medical feature extraction task using Mistral Nemo Instruct as the base model, focused on teaching the model to extract clinically relevant features based on structured prompts with limited examples.

Training hyperparameters:

Base model: unsloth/Mistral-Nemo-Instruct-2407-bnb-4bit
LoRA rank: 32
Training epochs: 14
Learning rate: 2e-4
Optimizer: AdamW (8-bit)
Weight decay: 0.01
LR scheduler: Linear

Performance

On the USMLE Step-2 Clinical Skills notes dataset:

Precision: 0.971
Recall: 0.934
F1 Score: 0.952

This model demonstrates impressive performance given the limited training data (only 12.5% of the full dataset), outperforming several baseline systems from previous work. However, it does not include the confidence calibration mechanisms of the full Multi-CONFE framework, resulting in higher rates of hallucinations and less reliable confidence estimation compared to the calibrative version.

Limitations

The model was evaluated on standardized USMLE Step-2 Clinical Skills notes and may require adaptation for other clinical domains.
Without calibration mechanisms, the confidence scores may not accurately reflect extraction performance.
The model has higher hallucination rates compared to the calibrative versions.
Performance on multilingual or non-standardized clinical notes remains untested.
While effective given the limited data, it still performs below both the full instructive model and the calibrative models.

Ethical Considerations

Automated assessment systems must ensure fairness across different student populations. Without calibration mechanisms and with limited training data, this model may provide less reliable and less interpretable confidence scores, which should be considered when using it for assessment scenarios. We recommend:

Using this model primarily as a foundation for further calibrative fine-tuning
Implementing a human-in-the-loop validation process when used directly
Considering upgrading to the calibrative version for high-stakes applications

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Load model and tokenizer
model_name = "Manal0809/Mistral_instructive_few"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float16)

# Example input
patient_note = """HPI: 35 yo F with heavy uterine bleeding. Last normal period was 6 month ago. 
LMP was 2 months ago. No clots.
Changes tampon every few hours, previously 4/day. Menarche at 12.
Attempted using OCPs for menstrual regulation previously but unsuccessful.
Two adolescent children (ages unknown) at home.
Last PAP 6 months ago was normal, never abnormal.
Gained 10-15 lbs over the past few months, eating out more though.
Hyperpigmented spots on hands and LT neck that she noticed 1-2 years ago.
SH: state social worker; no smoking or drug use; beer or two on weekends;
sexually active with boyfriend of 14 months, uses condoms at first but no longer uses them."""

features_to_extract = ["35-year", "Female", "heavy-periods", "symptoms-for-6-months", 
                       "Weight-Gain", "Last-menstrual-period-2-months-ago", 
                       "Fatigue", "Unprotected-Sex", "Infertility"]

# Format input as shown in the paper
input_text = f"""###instruction: Extract medical features from the patient note.
###patient_history: {patient_note}
###features: {features_to_extract}
### Annotation:"""

# Generate output
inputs = tokenizer(input_text, return_tensors="pt").to(model.device)
outputs = model.generate(
    inputs["input_ids"],
    max_new_tokens=512,
    temperature=0.2,
    num_return_sequences=1
)
result = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(result)

Model Card Author

Manal Abumelha - [email protected]

Manal0809
/

Mistral_instructive_few