ragunath-ravi/distilgpt2-lmsys-chat

DistilGPT-2 Fine-tuned on LMSYS Chat 1M

This model is a fine-tuned version of distilgpt2 on the LMSYS Chat 1M dataset. It's designed to generate assistant-like responses in a conversational context.

Model Description

This model is a smaller, distilled version of GPT-2 that has been fine-tuned on a diverse collection of conversations between users and various chat models. The LMSYS Chat 1M dataset contains 1 million conversations with models like ChatGPT, Claude, Llama, and others, providing a rich variety of conversational styles and topics.

Training Details

• Base Model: distilgpt2
• Dataset: LMSYS Chat 1M
• Training Format: Conversations formatted as "User: {query} Assistant: {response}"
• Context Length: 1024 tokens

Intended Use

This model is intended for:

• Chatbot applications
• Dialogue systems
• Conversational AI research
• Educational purposes

Limitations

• As a distilled model with only 82M parameters, it doesn't match the capabilities of larger models like GPT-3.5 or Claude
• May produce factually incorrect or nonsensical responses
• Limited context window compared to newer models
• Not suitable for complex reasoning or specialized knowledge tasks

Training

This model was fine-tuned on the LMSYS Chat 1M dataset, which consists of 1 million conversations collected through the Vicuna evaluation platform. The training focused on teaching the model to generate helpful, coherent responses to user queries across a wide range of topics.

Acknowledgements

• Thanks to Hugging Face for providing the model hosting infrastructure
• Thanks to the LMSYS team for releasing their valuable conversation dataset
• The base distilgpt2 model was originally created by the Hugging Face team

Sample Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

# Load model & tokenizer
tokenizer = AutoTokenizer.from_pretrained("ragunath-ravi/distilgpt2-lmsys-chat")
model = AutoModelForCausalLM.from_pretrained("ragunath-ravi/distilgpt2-lmsys-chat")

# Format your input as the model was trained
prompt = "User: How does climate change affect biodiversity? Assistant:"

# Generate a response
input_ids = tokenizer(prompt, return_tensors="pt").input_ids
outputs = model.generate(
    input_ids, 
    max_new_tokens=100, 
    temperature=0.7, 
    top_p=0.9,
    do_sample=True
)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)