akhilfau/fine-tuned-smolLM2-360M-with-on-combined_Instruction_dataset

Model Card: akhilfau/fine-tuned-smolLM2-360M-with-on-combined_Instruction_dataset

Model Description This model is a fine-tuned version of SmolLM2-360M using physics instruction datasets from both synthetic physics questions and camel-ai physics. It is optimized for solving college-level physics word problems, with LoRA-based parameter-efficient fine-tuning to reduce memory footprint and enable efficient deployment.

• Base Model: HuggingFaceTB/SmolLM2-360M
• Instruction Datasets: Combination of instruction-tuned camel-ai/physics and custom-generated synthetic datasets
• Framework: transformers, trl, peft
• Evaluation Task: MMLU – college_physics
• Training Loss: ~0.14
• Validation Loss: 0.48 (best at epoch 11)
• Training Time: ~13,000 seconds
• Steps: 4926 steps with batch size 4 and accumulation 4

Use Cases This model is intended for:

• Physics education tools
• College-level physics question answering
• Low-resource inference applications (thanks to LoRA)

Training Details

• Hardware: NVIDIA A5500 24GB
• Precision: bf16 if available, else fp16
• Optimizer: adamw_8bit
• Epochs: 11
• Scheduler: Linear
• Learning Rate: 2e-5

Performance

Metric	Value
Training Loss	0.1419
Validation Loss	0.4764
Best Epoch	11
Train Samples/sec	24.42
Steps/sec	0.375
Total FLOPs	3.04e+17

---

How to Use

from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline

model_id = "akhilfau/fine-tuned-smolLM2-360M-with-on-combined_Instruction_dataset"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)
prompt = "Question: What is the acceleration of an object with mass 5 kg under a force of 20 N?\nAnswer:"
print(pipe(prompt, max_new_tokens=100)[0]["generated_text"])

Limitations

• Trained only on physics word problems – limited generalization outside this domain
• May not generalize to multi-step derivations or non-instructional questions
• Trained only on English prompts

Citation If you use this model, please cite:

@misc{vallala2025physmol,
  title={PhysmolLM: A Compact Large Language Model for Enhancing Physics Education on Mobile Devices},
  author={Akhil Vallala},
  year={2025},
  note={Master's Thesis, Florida Atlantic University}
}