README.md

---
library_name: transformers
license: mit
datasets:
- roneneldan/TinyStories
language:
- en
---

# Model Card for amusktweewt/tiny-stories-v1

This model is a custom transformer-based language model trained on the **TinyStories** dataset, designed for creative text generation tasks such as storytelling and conversational agents. **This model is purely an academic project and should not be used in production or practical applications.**

## Model Details

### Model Description

This model utilizes a custom tokenizer with Byte Pair Encoding (BPE) and has been trained with a smaller architecture to balance efficiency and performance. It is designed for generating coherent and contextually relevant short stories. However, a known issue with the tokenizer causes spaces between tokens to appear repeated, leading to suboptimal text output quality.

- **Developed by:** amusktweewt
- **Model type:** AutoModelForCausalLM
- **Language(s) (NLP):** English
- **License:** MIT

### Model Sources

- **Repository:** HuggingFace repository

## Uses

### Direct Use

This model is intended for academic and research purposes only. It demonstrates a proof of concept for training smaller transformer-based language models.

### Out-of-Scope Use

- Not suitable for tasks requiring factual accuracy
- Should not be used in production environments or applications involving sensitive content

## Bias, Risks, and Limitations

### Risks and Biases

The model may reflect biases present in the training data, leading to unintended or inappropriate outputs. Additionally, the tokenizer issue can result in suboptimal and incoherent text generations.

### Recommendations

This model is meant for research and demonstration purposes. Users should validate outputs critically and avoid using it for practical applications.

## How to Get Started with the Model

```python
from transformers import AutoModelForCausalLM, PreTrainedTokenizerFast

model = AutoModelForCausalLM.from_pretrained("amusktweewt/tiny-stories-v1")
tokenizer = PreTrainedTokenizerFast.from_pretrained("amusktweewt/tiny-stories-v1")

prompt = "Once upon a time,"
inputs = tokenizer(prompt, return_tensors="pt", return_token_type_ids=False)
outputs = model.generate(**inputs, max_new_tokens=50)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## Training Details

### Training Data

The model was trained on the **TinyStories** dataset, consisting of curated short stories. Preprocessing ensured consistent formatting and tokenization using a custom BPE tokenizer.

### Training Procedure

#### Preprocessing

- Used BPE tokenizer with a vocabulary size of 4096
- Included special tokens: `<sos>`, `<pad>`, `<|endoftext|>`, and `<unk>`

#### Training Hyperparameters

- **Batch size:** 64
- **Epochs:** 3
- **Learning rate:** 1e-3
- **Scheduler:** Cosine annealing
- **Precision:** Mixed precision (FP16)

#### Speeds, Sizes, Times

- **Training time:** Approx. 5 hours 30 minutes
- **Model size:** 230 MB
- **Dataset size:** 535.98 million tokens

## Evaluation

### Testing Data, Factors & Metrics

#### Testing Data

A subset of the training data was used for evaluation, focusing on coherence and storytelling quality.

#### Metrics

- **Loss**: 0.9723
- **Qualitative Evaluation**: Manual assessment of generated outputs for coherence and relevance.

### Results

- **Sample Outputs:**
  - Prompt: "in a far away country" 
    Completion: "in a far away coun try . He was so excited to explore the world . He was so happy to be able to explore the world ."

#### Summary

The model generates coherent short stories suitable for research demonstration but is limited by tokenizer issues and should not be used in real-world scenarios.

## Environmental Impact

- **Hardware Type:** NVIDIA 4090 GPU
- **Hours used:** 5.5
- **Carbon Emitted:** Approx. 0.2 kg CO2 eq

## Technical Specifications

### Model Architecture and Objective

- Transformer architecture with 8 layers, 12 attention heads, and a hidden size of 768

### Compute Infrastructure

#### Hardware

- Single GPU (NVIDIA 4090)

#### Software

- Python 3.8+
- HuggingFace Transformers 4.x
- PyTorch 1.x

## Model Card Authors

amusktweewt

## Model Card Contact

For questions or feedback, contact amusktweewt.