# Generating Instruction Datasets

Within [the chapter on instruction tuning](../1_instruction_tuning/README.md), we learned about fine-tuning models with Supervised Fine-tuning. In this section, we will explore how to generate instruction datasets for SFT. We will explore creating instruction tuning datasets through basic prompting and using more refined techniques from papers. Instruction tuning datasets with seed data for in-context learning can be created through methods like SelfInstruct and Magpie. Additionally, we will explore instruction evolution through EvolInstruct. Lastly, we will explore how to generate a dataset for instruction tuning using a distilabel pipeline.

## From Prompt to Data

Synthetic data sounds fancy, but it can be simplified as creating data through effective prompting to extract knowledge from a model. In turn, you can think of this as a way to generate data for a specific task. The challenge is prompting effectively while ensuring the data is diverse and representative. Fortunately, many papers have explored this problem, and we will explore some of the useful ones during this course. First things first, we will explore how to generate synthetic data through manual prompting.

### Basic Prompting

Let's start with a basic example and load the [HuggingFaceTB/SmolLM2-1.7B-Instruct](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct) model using the `transformers` integration of the `distilabel` library. We will use the `TextGeneration` class to generate a synthetic `prompt` and use that to generate a `completion`.

Next, we will load the model using the `distilabel` library.

```python
from distilabel.llms import TransformersLLM
from distilabel.steps.tasks import TextGeneration

llm = TransformersLLM(model="HuggingFaceTB/SmolLM2-1.7B-Instruct")
gen = TextGeneration(llm=llm)
gen.load()
```

!!! note
    Distilabel loads the `llm` into memory, so, when working in a notebook, we need to `gen.unload()` after we are done with it to avoid memory issues.

We will now use the `llm` to generate a `prompt` for instruction tuning.

```python
next(gen.process([{"instruction": "Generate a questions about the Hugging Face Smol-Course on small AI models."}]))
# What is the purpose of Smol-Course?
```

Lastly, we can use that same `prompt` as input to generate a `completion`.

```python
next(gen.process([{"instruction": "What is the purpose of Smol-Course?"}]))
# The Smol-Course is a platform designed to learning computer science concepts.
```

Cool! We can generated a synthetic `prompt` and a corresponding `completion`. Re-using this simple approach at scale will allow us to generate a lot more data however, the quality of the data is not that great and does not take into account the nuances of our course or domain. Additionally, re-running the current code shows us the data is not that diverse. Luckily, there are ways to solve this problem.

### SelfInstruct

SelfInstruct is a prompt that generates new instructions based on a seed dataset. This seed data can be a single instruction or a piece of context. The process begins with a pool of initial seed data. The language model is then prompted to generate new instructions based on this seed data using in-context learning. The prompt is [implemented in distilabel](https://github.com/argilla-io/distilabel/blob/main/src/distilabel/steps/tasks/templates/self-instruct.jinja2) and a simplified version is shown below:

```
# Task Description
Develop {{ num_instructions }} user queries that can be received by the given AI application and applicable to the provided context. Emphasize diversity in verbs and linguistic structures within the model's textual capabilities.

# Context
{{ input }}

# Output
```

To use it, we need to pass the `llm` to the [SelfInstruct class](https://distilabel.argilla.io/dev/components-gallery/tasks/selfinstruct/). Let's use the text from the [Prompt to Data section](#prompt-to-data) as context and generate a new instruction.

```python
from distilabel.steps.tasks import SelfInstruct

self_instruct = SelfInstruct(llm=llm)
self_instruct.load()

context = "<prompt_to_data_section>"

next(self_instruct.process([{"input": text}]))["instructions"][0]
# What is the process of generating synthetic data through manual prompting?
```

The generated instruction is a lot better already and it fits our actual content and domain. However, we can do even better by improving the prompt through evolution.

### EvolInstruct

EvolInstruct is a prompting technique that takes an input instruction and evolves it into a better version of the same instruction. This better version is defined according to a set of criteria and adds constraints, deepening, concretizing, reasoning or complications to the original instruction. The process can be repeated multiple times to create various evolutions of the same instruction, ideally leading to a better version of the original instruction. The prompt is [implemented in distilabel](https://github.com/argilla-io/distilabel/tree/main/src/distilabel/steps/tasks/evol_instruct) and a simplified version is shown below:

```
I want you act as a Prompt Rewriter.
Given prompt a prompt, rewrite it into a more complex version.
Complicate the prompt based on the following criteria:
{{ criteria }}

# Prompt
{{ input }}

# Output
```

To use it, we need to pass the `llm` to the [EvolInstruct class](https://distilabel.argilla.io/dev/components-gallery/tasks/evolinstruct/). Let's use the synthetic prompt from [the SelfInstruct section](#selfinstruct) as input and evolve it into a better version. For this example, we will only evolve for one generation.

```python
from distilabel.steps.tasks import EvolInstruct

evol_instruct = EvolInstruct(llm=llm, num_evolutions=1)
evol_instruct.load()

text = "What is the process of generating synthetic data through manual prompting"

next(evol_instruct.process([{"instruction": text}]))
# What is the process of generating synthetic data through manual prompting?
# And, how does the artificial intelligence system, GPT4, use machine learning algorithms to manipulate the input data into synthetic data?
```

The instruction is now more complex but has lost some of the original meaning. So, take into account that evolving can be a double-edged sword and we need to be careful with the quality of the data we generate.

### Magpie

Magpie is a technique that relies on the auto-regressive factors of language model and the [chat-template](../1_instruction_tuning/chat_templates.md) that has been using during the instruction tuning process. As you might remember, the chat-template is a format that structures conversations with clear role indicators (system, user, assistant). During the instruction tuning phase, the language model has been optimized to reproduce this format and that is exactly what magpie takes advantage of. It starts with a pre-query-prompt based on the chat-template but it stops before the user message indicator, e.g. `<|im_start|>user\n`, and then it uses the language model to generate the user prompt until the end of the assistant indicator, e.g. `<|im_end|>`. This approach allows us to generate a lot of data in a very efficient way and it can even be scaled up to multi-turn conversations. It is hypothesized this generated data reproduces training data from the instruction tuning phase of the model used.

In this scenario, prompt templates differ per model because they are based on the chat-template format. But we can walk through a simplified version of the process step-by-step.

```bash
# Step 1: provide the pre-query-prompt
<|im_start|>user\n

# Step 2: the language model generates the user prompt
<|im_start|>user\n
What is the purpose of Smol-Course?

# Step 3: stop the generation
<|im_end|>
```

To use it in distilabel, we need to pass the `llm` to the [Magpie class](https://distilabel.argilla.io/dev/components-gallery/tasks/magpie/).

```python
from distilabel.steps.tasks import Magpie

magpie = Magpie(llm=llm)
magpie.load()

next(magpie.process([{"system_prompt": "You are a helpful assistant."}]))
# [{
#   "role": "user",
#   "content": "Can you provide me with a list of the top 3 universities?"
# },
# {
#   "role": "assistant",
#   "content": "The top 3 universities are: MIT, Yale, Stanford."
# }]
```

We immediately get a dataset with a `prompt` and `completion` . To improve the performance on our own domain, we can inject additional context into the `system_prompt`. For the LLM to generate specific domain data in combination with Magpie, it helps describing in the system prompt what the users queries will be. This is then used in the  pre-query-prompt before we start generating the user prompt and bias the LLM to generate user queries of that domain.

```
You're an AI assistant that will help users solving math problems.
```

It's important to write the system prompt as shown above instead of something like:

```
You're an AI assistant that generates math problems
```

Generally, language models are less optimized for passing additional context to the `system_prompt` so this does not always work as well for customisation as other techniques.

### From Prompts to Pipelines

The classes we've seen so far are all standalone classes that can be used in a pipeline. This is a good start, but we can do even better by using the `Pipeline` class to generate a dataset. We will use the `TextGeneration` step to generate a synthetic dataset for instruction tuning. The pipeline will consist of a `LoadDataFromDicts` step to load the data, a `TextGeneration` step to generate the `prompt` and a `completion` for that prompt. We will connect the steps and flow the data through the pipeline using the `>>` operator. Within the [documentation of distilabel](https://distilabel.argilla.io/dev/components-gallery/tasks/textgeneration/#input-output-columns) we can see input and output columns of the step. We to ensure that the data flow correctly through the pipeline, we will use the `output_mappings` parameter to map the output columns to the input columns of the next step.

```python
from distilabel.llms import TransformersLLM
from distilabel.pipeline import Pipeline
from distilabel.steps import LoadDataFromDicts
from distilabel.steps.tasks import TextGeneration

with Pipeline() as pipeline:
    data = LoadDataFromDicts(data=[{"instruction": "Generate a short question about the Hugging Face Smol-Course."}])
    llm = TransformersLLM(model="HuggingFaceTB/SmolLM2-1.7B-Instruct")
    gen_a = TextGeneration(llm=llm, output_mappings={"generation": "instruction"})
    gen_b = TextGeneration(llm=llm, output_mappings={"generation": "response"})
    data >> gen_a >> gen_b

if __name__ == "__main__":
    distiset = pipeline.run(use_cache=False)
    print(distiset["default"]["train"][0])
# [{
#   "instruction": "What is the purpose of Smol-Course?",
#   "response": "The Smol-Course is a platform designed to learning computer science concepts."
# }]
```

Under the hood, this pipeline has a lot of cool features. It automatically caches generation results, so we can don't have to re-run the generation steps. There is included fault-tolerance, so if the generation steps fail, the pipeline will continue to run. And the pipeline exexutes all generation steps in parallel, so the generation is faster. We can even visualise the pipeline using the `draw` method. Here you can see how the data flows through the pipeline and how the `output_mappings` are used to map the output columns to the input columns of the next step.

![Pipeline](./images/pipeline.png)

## Best Practices

- Ensure you have a diverse seed data to cover a wide range of scenarios
- Regularly evaluate the dataset to ensure generated data is diverse and of high quality
- Iterate on the (system)prompt to improve the quality of the data

## Next Steps

👨🏽💻 Code -[Exercise Notebook](./notebooks/instruction_sft_dataset.ipynb) to generate a dataset for instruction tuning
🧑🏫 Learn - About [generating preference datasets](./preference_datasets.md)

## References

- [Distilabel Documentation](https://distilabel.argilla.io/latest/)
- [Self-instruct](https://arxiv.org/abs/2212.10560)
- [Evol-Instruct](https://arxiv.org/abs/2304.12244)
- [Magpie](https://arxiv.org/abs/2406.08464)

6_synthetic_datasets/instruction_datasets.md

A course on aligning smol models.


# Generating Preference Datasets

Within [the chapter on preference alignment](../2_preference_alignment/README.md), we learned about Direct Preference Optimization. In this section, we will explore how to generate preference datasets for methods like DPO. We will build on top of the methods that were introduced in [generating instruction datasets](./instruction_datasets.md). Additionally, we will show how to add extra completions to the dataset using basic prompting or by using EvolQuality to improve the quality of responses. Lastly, we will show how UltraFeedback can be used to generate scores and critiques.

## Creating multiple completions

Preference data is a dataset with multiple `completions` for the same `instruction`. We can add more `completions` to a dataset by prompting a model to generate them. When doing this, we need to ensure that the second completion is not too similar to the first completion in terms of overall quality and phrasing. This is important because the model needs to be optimized for a clear preference. We want to know which completion is preferred over the other, normally referred to as `chosen` and `rejected`. We will go into more detail about determining chosen and rejected completions in the [section on creating scores](#creating-scores).

### Model pooling

You can use models from different model families to generate a second completion, which is called model pooling. To further improve the quality of the second completion, you can use different generation arguments, like tweaking the `temperature`. Lastly, you can use different prompt templates or system prompts to generate a second completion to ensure diversity based on specific characteristics defined in the template. In theory, we could take two models of varying quality and use the better one as the `chosen` completion.

Let's start with model pooling by loading the [Qwen/Qwen2.5-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct) and [HuggingFaceTB/SmolLM2-1.7B-Instruct](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct) models using the `transformers` integration of the `distilabel` library. Using these models, we will create two synthetic `responses` for a given `prompt`. We will create another pipeline with `LoadDataFromDicts`, `TextGeneration`, and `GroupColumns`. We will first load data, then use two generation steps, and then group the results. We connect the steps and flow the data through the pipeline using the `>>` operator and `[]`, which means that we want to use the output of the previous step as the input for both steps within the list.

```python
from distilabel.llms import TransformersLLM
from distilabel.pipeline import Pipeline
from distilabel.steps import GroupColumns, LoadDataFromDicts
from distilabel.steps.tasks import TextGeneration

with Pipeline() as pipeline:
    data = LoadDataFromDicts(data=[{"instruction": "What is synthetic data?"}])
    llm_a = TransformersLLM(model="HuggingFaceTB/SmolLM2-1.7B-Instruct")
    gen_a = TextGeneration(llm=llm_a)
    llm_b = TransformersLLM(model="Qwen/Qwen2.5-1.5B-Instruct")
    gen_b = TextGeneration(llm=llm_b)
    group = GroupColumns(columns=["generation"])
    data >> [gen_a, gen_b] >> group

if __name__ == "__main__":
    distiset = pipeline.run()
    print(distiset["default"]["train"]["grouped_generation"][0])
# {[
#   'Synthetic data is artificially generated data that mimics real-world usage.',
#   'Synthetic data refers to data that has been generated artificially.'
# ]}
```

As you can see, we have two synthetic `completions` for the given `prompt`. We could have boosted diversity by initializing the `TextGeneration` steps with a specific `system_prompt` or by passing generation arguments to the `TransformersLLM`. Let's now see how we can improve the quality of the `completions` using EvolQuality.

### EvolQuality

EvolQuality is similar to [EvolInstruct](./instruction_datasets.md#evolinstruct) - it is a prompting technique but it evolves `completions` instead of the input `prompt`. The task takes both a `prompt` and `completion` and evolves the `completion` into a version that better responds to the `prompt` based on a set of criteria. This better version is defined according to criteria for improving helpfulness, relevance, deepening, creativity, or details. Because this automatically generates a second completion, we can use it to add more `completions` to a dataset. In theory, we could even assume the evolution is better than the original completion and use it as the `chosen` completion out of the box.

The prompt is [implemented in distilabel](https://github.com/argilla-io/distilabel/tree/main/src/distilabel/steps/tasks/evol_quality) and a simplified version is shown below:

```bash
I want you act as a Response Rewriter.
Given prompt a and a response, rewrite the response into a better version.
Complicate the prompt based on the following criteria:
{{ criteria }}

# Prompt
{{ input }}

# Response
{{ output }}

# Improved Response
```

Let's use the [EvolQuality class](https://distilabel.argilla.io/dev/components-gallery/tasks/evolquality/) to evolve the synthetic `prompt` and `completion` from [the Model Pooling section](#model-pooling) into a better version. For this example, we will only evolve for one generation.

```python
from distilabel.llms import TransformersLLM
from distilabel.steps.tasks import EvolQuality

llm = TransformersLLM(model="HuggingFaceTB/SmolLM2-1.7B-Instruct")
evol_quality = EvolQuality(llm=llm, num_evolutions=1)
evol_quality.load()

instruction = "What is synthetic data?"
completion = "Synthetic data is artificially generated data that mimics real-world usage."

next(evol_quality.process([{
    "instruction": instruction,
    "response": completion
}]))
# The process of generating synthetic data through manual prompting involves creating artificial data sets that mimic real-world usage patterns.
```

The `response` is now more complex and specific to the `instruction`. This is a good start, but as we have seen with EvolInstruct, evolved generations are not always better. Hence, it is important to use additional evaluation techniques to ensure the quality of the dataset. We will explore this in the next section.

## Creating Scores

Scores are a measure of how much one response is preferred over another. In general, these scores can be absolute, subjective, or relative. For this course, we will focus on the first two because they are most valuable for creating preference datasets. This scoring is a way of judging and evaluating using language models and therefore has some overlap with the evaluation techniques we have seen in [the chapter on evaluation](../3_evaluation/README.md). As with the other evaluation techniques, scores and evaluations normally require larger models to better align with human preferences.

### UltraFeedback

UltraFeedback is a technique that generates scores and critiques for a given `prompt` and its `completion`.

The scores are based on the quality of the `completion` according to a set of criteria. There are four fine-grained criteria: `helpfulness`, `relevance`, `deepening`, and `creativity`. These are useful but generally speaking, using the overall criteria is a good start, which allows us to simplify the process of generating scores. The scores can be used to determine which `completion` is the `chosen` and which is the `rejected` one. Because they are absolute, they can also be used as interesting filters for outliers in the dataset, either finding the worst completions or the pairs with more or less difference.

The critiques are added to provide reasoning for the score. They can be used as extra context to help us understand the differences between the scores. The language model generates extensive critiques which is very useful, but this also introduces extra cost and complexity to the process because generating critiques is more expensive than generating a single token to represent a score.

The prompt is [implemented in distilabel](https://github.com/argilla-io/distilabel/tree/main/src/distilabel/steps/tasks/templates/ultrafeedback) and a simplified version is shown below:

```bash
Evaluate the model's outputs based on various criteria: Helpfulness, Relevance, Deepening, Creativity
Your role is to provide a holistic assessment based on the above factors.
Score the output from 1 to 5 on overall quality.

Answer with the following format: score - rationale

# Input
{{ input }}

# Response
{{ output }}

# Score - Rationale
```

Let's use the [UltraFeedback class](https://distilabel.argilla.io/dev/components-gallery/tasks/ultrafeedback/) to evaluate the synthetic `prompt` and `completion` from [the Model Pooling section](#model-pooling).

```python
from distilabel.llms import TransformersLLM
from distilabel.steps.tasks import UltraFeedback

llm = TransformersLLM(model="HuggingFaceTB/SmolLM2-1.7B-Instruct")
ultrafeedback = UltraFeedback(llm=llm)
ultrafeedback.load()

instruction = "What is synthetic data?"
completion_a = "Synthetic data is artificially generated data that mimics real-world usage."
completion_b = "Synthetic data refers to data that has been generated artificially."

next(ultrafeedback.process([{
    "instruction": instruction,
    "generations": [completion_a, completion_b]
}]))
# [
#     {
#         'ratings': [4, 5],
#         'rationales': ['could have been more specific', 'good definition'],
#     }
# ]
```

## Best Practices

- Overall scores are cheaper and easier to generate than critiques and specific scores
- Use bigger models to generate scores and critiques
- Use a diverse set of models to generate scores and critiques
- Iterate on configuration of the `system_prompt` and models

## Next Steps

👨🏽💻 Code -[Exercise Notebook](./notebooks/preference_dpo_dataset.ipynb) to generate a dataset for instruction tuning

## References

- [Distilabel Documentation](https://distilabel.argilla.io/latest/)
- [Deita](https://arxiv.org/abs/2312.15685)
- [UltraFeedback](https://arxiv.org/abs/2310.01377)

6_synthetic_datasets/preference_datasets.md

A course on aligning smol models.


# Inference

Inference is the process of using a trained language model to generate predictions or responses. While inference might seem straightforward, deploying models efficiently at scale requires careful consideration of various factors like performance, cost, and reliability. Large Language Models (LLMs) present unique challenges due to their size and computational requirements.

We'll explore both simple and production-ready approaches using the [`transformers`](https://huggingface.co/docs/transformers/index) library and [`text-generation-inference`](https://github.com/huggingface/text-generation-inference), two popular frameworks for LLM inference. For production deployments, we'll focus on Text Generation Inference (TGI), which provides optimized serving capabilities.

## Module Overview

LLM inference can be categorized into two main approaches: simple pipeline-based inference for development and testing, and optimized serving solutions for production deployments. We'll cover both approaches, starting with the simpler pipeline approach and moving to production-ready solutions.

## Contents

### 1. [Basic Pipeline Inference](./pipeline_inference.md)

Learn how to use the Hugging Face Transformers pipeline for basic inference. We'll cover setting up pipelines, configuring generation parameters, and best practices for local development. The pipeline approach is perfect for prototyping and small-scale applications. [Start learning](./pipeline_inference.md).

### 2. [Production Inference with TGI](./tgi_inference.md)

Learn how to deploy models for production using Text Generation Inference. We'll explore optimized serving techniques, batching strategies, and monitoring solutions. TGI provides production-ready features like health checks, metrics, and Docker deployment options. [Start learning](./text_generation_inference.md).

### Exercise Notebooks

| Title | Description | Exercise | Link | Colab |
|-------|-------------|----------|------|-------|
| Pipeline Inference | Basic inference with transformers pipeline | 🐢 Set up a basic pipeline
 🐕 Configure generation parameters
 🦁 Create a simple web server | [Link](./notebooks/basic_pipeline_inference.ipynb) | [Colab](https://githubtocolab.com/huggingface/smol-course/tree/main/7_inference/notebooks/basic_pipeline_inference.ipynb) |
| TGI Deployment | Production deployment with TGI | 🐢 Deploy a model with TGI
 🐕 Configure performance optimizations
 🦁 Set up monitoring and scaling | [Link](./notebooks/tgi_deployment.ipynb) | [Colab](https://githubtocolab.com/huggingface/smol-course/tree/main/7_inference/notebooks/tgi_deployment.ipynb) |

## Resources

- [Hugging Face Pipeline Tutorial](https://huggingface.co/docs/transformers/en/pipeline_tutorial)
- [Text Generation Inference Documentation](https://huggingface.co/docs/text-generation-inference/en/index)
- [Pipeline WebServer Guide](https://huggingface.co/docs/transformers/en/pipeline_tutorial#using-pipelines-for-a-webserver)
- [TGI GitHub Repository](https://github.com/huggingface/text-generation-inference)
- [Hugging Face Model Deployment Documentation](https://huggingface.co/docs/inference-endpoints/index)
- [vLLM: High-throughput LLM Serving](https://github.com/vllm-project/vllm)
- [Optimizing Transformer Inference](https://huggingface.co/blog/optimize-transformer-inference)

7_inference/README.md

A course on aligning smol models.


# Basic Inference with Transformers Pipeline

The `pipeline` abstraction in 🤗 Transformers provides a simple way to run inference with any model from the Hugging Face Hub. It handles all the preprocessing and postprocessing steps, making it easy to use models without deep knowledge of their architecture or requirements.

## How Pipelines Work

Hugging Face pipelines streamline the machine learning workflow by automating three critical stages between raw input and human-readable output:

**Preprocessing Stage**
The pipeline first prepares your raw inputs for the model. This varies by input type:
- Text inputs undergo tokenization to convert words into model-friendly token IDs
- Images are resized and normalized to match model requirements
- Audio is processed through feature extraction to create spectrograms or other representations

**Model Inference**
During the forward pass, the pipeline:
- Handles batching of inputs automatically for efficient processing
- Places computation on the optimal device (CPU/GPU)
- Applies performance optimizations like half-precision (FP16) inference where supported

**Postprocessing Stage**
Finally, the pipeline converts raw model outputs into useful results:
- Decodes token IDs back into readable text
- Transforms logits into probability scores
- Formats outputs according to the specific task (e.g., classification labels, generated text)

This abstraction lets you focus on your application logic while the pipeline handles the technical complexity of model inference.

## Basic Usage

Here's how to use a pipeline for text generation:

```python
from transformers import pipeline

# Create a pipeline with a specific model
generator = pipeline(
    "text-generation",
    model="HuggingFaceTB/SmolLM2-1.7B-Instruct",
    torch_dtype="auto",
    device_map="auto"
)

# Generate text
response = generator(
    "Write a short poem about coding:",
    max_new_tokens=100,
    do_sample=True,
    temperature=0.7
)
print(response[0]['generated_text'])
```

## Key Configuration Options

### Model Loading
```python
# CPU inference
generator = pipeline("text-generation", model="HuggingFaceTB/SmolLM2-1.7B-Instruct", device="cpu")

# GPU inference (device 0)
generator = pipeline("text-generation", model="HuggingFaceTB/SmolLM2-1.7B-Instruct", device=0)

# Automatic device placement
generator = pipeline(
    "text-generation",
    model="HuggingFaceTB/SmolLM2-1.7B-Instruct",
    device_map="auto",
    torch_dtype="auto"
)
```

### Generation Parameters

```python
response = generator(
    "Translate this to French:",
    max_new_tokens=100,     # Maximum length of generated text
    do_sample=True,         # Use sampling instead of greedy decoding
    temperature=0.7,        # Control randomness (higher = more random)
    top_k=50,               # Limit to top k tokens
    top_p=0.95,             # Nucleus sampling threshold
    num_return_sequences=1  # Number of different generations
)
```

## Processing Multiple Inputs

Pipelines can efficiently handle multiple inputs through batching:

```python
# Prepare multiple prompts
prompts = [
    "Write a haiku about programming:",
    "Explain what an API is:",
    "Write a short story about a robot:"
]

# Process all prompts efficiently
responses = generator(
    prompts,
    batch_size=4,              # Number of prompts to process together
    max_new_tokens=100,
    do_sample=True,
    temperature=0.7
)

# Print results
for prompt, response in zip(prompts, responses):
    print(f"Prompt: {prompt}")
    print(f"Response: {response[0]['generated_text']}\n")
```

## Web Server Integration

Here's how to integrate a pipeline into a FastAPI application:

```python
from fastapi import FastAPI, HTTPException
from transformers import pipeline
import uvicorn

app = FastAPI()

# Initialize pipeline globally
generator = pipeline(
    "text-generation",
    model="HuggingFaceTB/SmolLM2-1.7B-Instruct",
    device_map="auto"
)

@app.post("/generate")
async def generate_text(prompt: str):
    try:
        if not prompt:
            raise HTTPException(status_code=400, detail="No prompt provided")

        response = generator(
            prompt,
            max_new_tokens=100,
            do_sample=True,
            temperature=0.7
        )

        return {"generated_text": response[0]['generated_text']}

    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=5000)
```

## Limitations

While pipelines are great for prototyping and small-scale deployments, they have some limitations:

- Limited optimization options compared to dedicated serving solutions
- No built-in support for advanced features like dynamic batching
- May not be suitable for high-throughput production workloads

For production deployments with high throughput requirements, consider using Text Generation Inference (TGI) or other specialized serving solutions.

## Resources

- [Hugging Face Pipeline Tutorial](https://huggingface.co/docs/transformers/en/pipeline_tutorial)
- [Pipeline API Reference](https://huggingface.co/docs/transformers/en/main_classes/pipelines)
- [Text Generation Parameters](https://huggingface.co/docs/transformers/en/main_classes/text_generation)
- [Model Quantization Guide](https://huggingface.co/docs/transformers/en/perf_infer_gpu_one)

7_inference/inference_pipeline.md

A course on aligning smol models.


# Text Generation Inference (TGI)

Text Generation Inference (TGI) is a toolkit developed by Hugging Face for deploying and serving Large Language Models (LLMs). It's designed to enable high-performance text generation for popular open-source LLMs. TGI is used in production by Hugging Chat - An open-source interface for open-access models.

## Why Use Text Generation Inference?

Text Generation Inference addresses the key challenges of deploying large language models in production. While many frameworks excel at model development, TGI specifically optimizes for production deployment and scaling. Some key features include:

- **Tensor Parallelism**: TGI's can split models across multiple GPUs through tensor parallelism, essential for serving larger models efficiently.
- **Continuous Batching**: The continuous batching system maximizes GPU utilization by dynamically processing requests, while optimizations like Flash Attention and Paged Attention significantly reduce memory usage and increase speed.
- **Token Streaming**: Real-time applications benefit from token streaming via Server-Sent Events, delivering responses with minimal latency.

## How to Use Text Generation Inference

### Basic Python Usage

TGI uses a simple yet powerful REST API integration which makes it easy to integrate with your applications.

### Using the REST API

TGI exposes a RESTful API that accepts JSON payloads. This makes it accessible from any programming language or tool that can make HTTP requests. Here's a basic example using curl:

```bash
# Basic generation request
curl localhost:8080/v1/chat/completions \
    -X POST \
    -d '{
  "model": "tgi",
  "messages": [
    {
      "role": "system",
      "content": "You are a helpful assistant."
    },
    {
      "role": "user",
      "content": "What is deep learning?"
    }
  ],
  "stream": true,
  "max_tokens": 20
}' \
    -H 'Content-Type: application/json'
```

### Using the `huggingface_hub` Python Client

The `huggingface_hub` python client client handles connection management, request formatting, and response parsing. Here's how to get started.

```python
from huggingface_hub import InferenceClient

client = InferenceClient(
    base_url="http://localhost:8080/v1/",
)

output = client.chat.completions.create(
    model="tgi",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Count to 10"},
    ],
    stream=True,
    max_tokens=1024,
)

for chunk in output:
    print(chunk.choices[0].delta.content)
```

### Using OpenAI API

Many libraries support the OpenAI API, so you can use the same client to interact with TGI.

```python
from openai import OpenAI

# init the client but point it to TGI
client = OpenAI(
    base_url="http://localhost:8080/v1/",
    api_key="-"
)

chat_completion = client.chat.completions.create(
    model="tgi",
    messages=[
        {"role": "system", "content": "You are a helpful assistant." },
        {"role": "user", "content": "What is deep learning?"}
    ],
    stream=True
)

# iterate and print stream
for message in chat_completion:
    print(message)
```

## Preparing Models for TGI

To serve a model with TGI, ensure it meets these requirements:

1. **Supported Architecture**: Verify your model architecture is supported (Llama, BLOOM, T5, etc.)

2. **Model Format**: Convert weights to safetensors format for faster loading:

```python
from safetensors.torch import save_file
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("your-model")
state_dict = model.state_dict()
save_file(state_dict, "model.safetensors")
```

3. **Quantization** (optional): Quantize your model to reduce memory usage:

```python
from transformers import BitsAndBytesConfig

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype="float16"
)

model = AutoModelForCausalLM.from_pretrained(
    "your-model",
    quantization_config=quantization_config
)
```

## References

- [Text Generation Inference Documentation](https://huggingface.co/docs/text-generation-inference)
- [TGI GitHub Repository](https://github.com/huggingface/text-generation-inference)
- [Hugging Face Model Hub](https://huggingface.co/models)
- [TGI API Reference](https://huggingface.co/docs/text-generation-inference/api_reference)

7_inference/text_generation_inference.md

A course on aligning smol models.


# Agents

AI Agents are autonomous systems that can understand user requests, break them down into steps, and execute actions to accomplish tasks. They combine language models with tools and external functions to interact with their environment. This module covers how to build effective agents using the [`smolagents`](https://github.com/huggingface/smolagents) library, which provides a lightweight framework for creating capable AI agents.

## Module Overview

Building effective agents requires understanding three key components. First, retrieval capabilities allow agents to access and use relevant information from various sources. Second, function calling enables agents to take concrete actions in their environment. Finally, domain-specific knowledge and tooling equip agents for specialized tasks like code manipulation.

## Contents

### 1⃣ [Retrieval Agents](./retrieval_agents.md)

Retrieval agents combine models with knowledge bases. These agents can search and synthesize information from multiple sources, leveraging vector stores for efficient retrieval and implementing RAG (Retrieval Augmented Generation) patterns. They are great at combining web search with custom knowledge bases while maintaining conversation context through memory systems. The module covers implementation strategies including fallback mechanisms for robust information retrieval.

### 2⃣ [Code Agents](./code_agents.md)

Code agents are specialized autonomous systems designed for software development tasks. These agents excel at analyzing and generating code, performing automated refactoring, and integrating with development tools. The module covers best practices for building code-focused agents that can understand programming languages, work with build systems, and interact with version control while maintaining high code quality standards.

### 3⃣ [Custom Functions](./custom_functions.md)

Custom function agents extend basic AI capabilities through specialized function calls. This module explores how to design modular and extensible function interfaces that integrate directly with your application's logic. You'll learn to implement proper validation and error handling while creating reliable function-driven workflows. The focus is on building simple systems where agents can predictably interact with external tools and services.

### Exercise Notebooks

| Title | Description | Exercise | Link | Colab |
|-------|-------------|----------|------|-------|
| Building a Research Agent | Create an agent that can perform research tasks using retrieval and custom functions | 🐢 Build a simple RAG agent
 🐕 Add custom search functions
 🦁 Create a full research assistant | [Notebook](./notebooks/agents.ipynb) |  |

## Resources

- [smolagents Documentation](https://huggingface.co/docs/smolagents) - Official docs for the smolagents library
- [Building Effective Agents](https://www.anthropic.com/research/building-effective-agents) - Research paper on agent architectures
- [Agent Guidelines](https://huggingface.co/docs/smolagents/tutorials/building_good_agents) - Best practices for building reliable agents
- [LangChain Agents](https://python.langchain.com/docs/how_to/#agents) - Additional examples of agent implementations
- [Function Calling Guide](https://platform.openai.com/docs/guides/function-calling) - Understanding function calling in LLMs
- [RAG Best Practices](https://www.pinecone.io/learn/retrieval-augmented-generation/) - Guide to implementing effective RAG

8_agents/README.md

A course on aligning smol models.


# Code Agents

Code agents are specialized autonomous systems that handle coding tasks like analysis, generation, refactoring, and testing. These agents leverage domain knowledge about programming languages, build systems, and version control to enhance software development workflows.

## Why Code Agents?

Code agents accelerate development by automating repetitive tasks while maintaining code quality. They excel at generating boilerplate code, performing systematic refactoring, and identifying potential issues through static analysis. The agents combine retrieval capabilities to access external documentation and repositories with function calling to execute concrete actions like creating files or running tests.

## Building Blocks of a Code Agent

Code agents are built on specialized language models fine-tuned for code understanding. These models are augmented with development tools like linters, formatters, and compilers to interact with real-world environments. Through retrieval techniques, agents maintain contextual awareness by accessing documentation and code histories to align with organizational patterns and standards. Action-oriented functions enable agents to perform concrete tasks such as committing changes or initiating merge requests.

In the following example, we create a code agent that can search the web using DuckDuckGo much like the retrieval agent we built earlier.

```python
from smolagents import CodeAgent, DuckDuckGoSearchTool, HfApiModel

agent = CodeAgent(tools=[DuckDuckGoSearchTool()], model=HfApiModel())

agent.run("How many seconds would it take for a leopard at full speed to run through Pont des Arts?")
```

In the following example, we create a code agent that can get the travel time between two locations. Here, we use the `@tool` decorator to define a custom function that can be used as a tool.

```python
from smolagents import CodeAgent, HfApiModel, tool

@tool
def get_travel_duration(start_location: str, destination_location: str, departure_time: Optional[int] = None) -> str:
    """Gets the travel time in car between two places.

    Args:
        start_location: the place from which you start your ride
        destination_location: the place of arrival
        departure_time: the departure time, provide only a `datetime.datetime` if you want to specify this
    """
    import googlemaps # All imports are placed within the function, to allow for sharing to Hub.
    import os

    gmaps = googlemaps.Client(os.getenv("GMAPS_API_KEY"))

    if departure_time is None:
        from datetime import datetime
        departure_time = datetime(2025, 1, 6, 11, 0)

    directions_result = gmaps.directions(
        start_location,
        destination_location,
        mode="transit",
        departure_time=departure_time
    )
    return directions_result[0]["legs"][0]["duration"]["text"]

agent = CodeAgent(tools=[get_travel_duration], model=HfApiModel(), additional_authorized_imports=["datetime"])

agent.run("Can you give me a nice one-day trip around Paris with a few locations and the times? Could be in the city or outside, but should fit in one day. I'm travelling only via public transportation.")

```

These examples are just the beginning of what you can do with code agents. You can learn more about how to build code agents in the [smolagents documentation](https://huggingface.co/docs/smolagents).

smolagents provides a lightweight framework for building code agents, with a core implementation of approximately 1,000 lines of code. The framework specializes in agents that write and execute Python code snippets, offering sandboxed execution for security. It supports both open-source and proprietary language models, making it adaptable to various development environments.

## Further Reading

- [smolagents Blog](https://huggingface.co/blog/smolagents) - Introduction to smolagents and code interactions
- [smolagents: Building Good Agents](https://huggingface.co/docs/smolagents/tutorials/building_good_agents) - Best practices for reliable agents
- [Building Effective Agents - Anthropic](https://www.anthropic.com/research/building-effective-agents) - Agent design principles

8_agents/code_agents.md

A course on aligning smol models.


# Custom Function Agents

Custom Function Agents are AI agents that leverage specialized function calls (or “tools”) to perform tasks. Unlike general-purpose agents, Custom Function Agents focus on powering advanced workflows by integrating directly with your application's logic. For example, you can expose database queries, system commands, or any custom utility as isolated functions for the agent to invoke.

## Why Custom Function Agents?

- **Modular and Extensible**: Instead of building one monolithic agent, you can design individual functions that represent discrete capabilities, making your architecture more extensible.
- **Fine-Grained Control**: Developers can carefully control the agent’s actions by specifying exactly which functions are available and what parameters they accept.
- **Improved Reliability**: By structuring each function with clear schemas and validations, you reduce errors and unexpected behaviors.

## Basic Workflow

1. **Identify Functions**
   Determine which tasks can be transformed into custom functions (e.g., file I/O, database queries, streaming data processing).

2. **Define the Interface**
   Use a function signature or schema that precisely outlines each function’s inputs, outputs, and expected behavior. This enforces strong contracts between your agent and its environment.

3. **Register with the Agent**
   Your agent needs to “learn” which functions are available. Typically, you pass metadata describing each function’s interface to the language model or agent framework.

4. **Invoke and Validate**
   Once the agent selects a function to call, run the function with the provided arguments and validate the results. If valid, feed the results back to the agent for context to drive subsequent decisions.

## Example

Below is a simplified example demonstrating how custom function calls might look in pseudocode. The objective is to perform a user-defined search and retrieve relevant content:

```python
# Define a custom function with clear input/output types
def search_database(query: str) -> list:
    """
    Search the database for articles matching the query.

    Args:
        query (str): The search query string

    Returns:
        list: List of matching article results
    """
    try:
        results = database.search(query)
        return results
    except DatabaseError as e:
        logging.error(f"Database search failed: {e}")
        return []

# Register the function with the agent
agent.register_function(
    name="search_database",
    function=search_database,
    description="Searches database for articles matching a query"
)

# Example usage
def process_search():
    query = "Find recent articles on AI"
    results = agent.invoke("search_database", query)

    if results:
        agent.process_results(results)
    else:
        logging.info("No results found for query")
```

## Further Reading

- [smolagents Blog](https://huggingface.co/blog/smolagents) - Learn about the latest advancements in AI agents and how they can be applied to custom function agents.
- [Building Good Agents](https://huggingface.co/docs/smolagents/tutorials/building_good_agents) - A comprehensive guide on best practices for developing reliable and effective custom function agents.

8_agents/custom_functions.md

A course on aligning smol models.


# Building Agentic RAG Systems

Agentic RAG (Retrieval Augmented Generation) combines the power of autonomous agents with knowledge retrieval capabilities. While traditional RAG systems simply use an LLM to answer queries based on retrieved information, agentic RAG takes this further by allowing the system to intelligently control its own retrieval and response process.

Traditional RAG has key limitations - it only performs a single retrieval step and relies on direct semantic similarity with the user query, which can miss relevant information. Agentic RAG addresses these challenges by empowering the agent to formulate its own search queries, critique results, and perform multiple retrieval steps as needed.

## Basic Retrieval with DuckDuckGo

Let's start by building a simple agent that can search the web using DuckDuckGo. This agent will be able to answer questions by retrieving relevant information and synthesizing responses.

```python
from smolagents import CodeAgent, DuckDuckGoSearchTool, HfApiModel

# Initialize the search tool
search_tool = DuckDuckGoSearchTool()

# Initialize the model
model = HfApiModel()

agent = CodeAgent(
    model = model,
    tools=[search_tool]
)

# Example usage
response = agent.run(
    "What are the latest developments in fusion energy?"
)
print(response)
```

The agent will:
1. Analyze the query to determine what information is needed
2. Use DuckDuckGo to search for relevant content
3. Synthesize the retrieved information into a coherent response
4. Store the interaction in its memory for future reference

## Custom Knowledge Base Tool

For domain-specific applications, we often want to combine web search with our own knowledge base. Let's create a custom tool that can query a vector database of technical documentation.

```python
from smolagents import Tool

class RetrieverTool(Tool):
    name = "retriever"
    description = "Uses semantic search to retrieve the parts of transformers documentation that could be most relevant to answer your query."
    inputs = {
        "query": {
            "type": "string",
            "description": "The query to perform. This should be semantically close to your target documents. Use the affirmative form rather than a question.",
        }
    }
    output_type = "string"

    def __init__(self, docs, **kwargs):
        super().__init__(**kwargs)
        self.retriever = BM25Retriever.from_documents(
            docs, k=10
        )

    def forward(self, query: str) -> str:
        assert isinstance(query, str), "Your search query must be a string"

        docs = self.retriever.invoke(
            query,
        )
        return "\nRetrieved documents:\n" + "".join(
            [
                f"\n\n===== Document {str(i)} =====\n" + doc.page_content
                for i, doc in enumerate(docs)
            ]
        )

retriever_tool = RetrieverTool(docs_processed)
```

This enhanced agent can:
1. First check the documentation for relevant information
2. Fall back to web search if needed
3. Combine information from both sources
4. Maintain conversation context through memory

## Enhanced Retrieval Capabilities

When building agentic RAG systems, the agent can employ sophisticated strategies like:

1. Query Reformulation - Instead of using the raw user query, the agent can craft optimized search terms that better match the target documents
2. Multi-Step Retrieval - The agent can perform multiple searches, using initial results to inform subsequent queries
3. Source Integration - Information can be combined from multiple sources like web search and local documentation
4. Result Validation - Retrieved content can be analyzed for relevance and accuracy before being included in responses

Effective agentic RAG systems require careful consideration of several key aspects. The agent should select between available tools based on the query type and context. Memory systems help maintain conversation history and avoid repetitive retrievals. Having fallback strategies ensures the system can still provide value even when primary retrieval methods fail. Additionally, implementing validation steps helps ensure the accuracy and relevance of retrieved information.

```python
import datasets
from langchain.docstore.document import Document
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.retrievers import BM25Retriever

knowledge_base = datasets.load_dataset("m-ric/huggingface_doc", split="train")
knowledge_base = knowledge_base.filter(lambda row: row["source"].startswith("huggingface/transformers"))

source_docs = [
    Document(page_content=doc["text"], metadata={"source": doc["source"].split("/")[1]})
    for doc in knowledge_base
]

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50,
    add_start_index=True,
    strip_whitespace=True,
    separators=["\n\n", "\n", ".", " ", ""],
)
docs_processed = text_splitter.split_documents(source_docs)
```

## Next Steps

⏩ Check out the [Code Agents](./code_agents.md) module to learn how to build agents that can manipulate code.

8_agents/retrieval_agents.md

A course on aligning smol models.


![smolcourse image](./banner.png)

# Un Pequeño (Smol) Curso

Este curso práctico está enfocado en alinear modelos de lenguaje para casos de uso específicos. Es una forma accesible de empezar a trabajar con modelos de lenguaje, ya que puede ejecutarse en la mayoría de las máquinas locales con requisitos mínimos de GPU y sin necesidad de servicios pagos. El curso se basa en la serie de modelos [SmolLM2](https://github.com/huggingface/smollm/tree/main), pero las habilidades que adquieras aquí son transferibles a modelos más grandes o otros modelos pequeños de lenguaje.

¡La participación es abierta, gratuita y ahora!

Este curso es abierto y revisado por la comunidad. Para participar, simplemente abre un pull request y envía tu trabajo para su revisión. Sigue estos pasos:

Haz un fork del repositorio aquí

Lee el material, haz cambios, completa los ejercicios y agrega tus ejemplos.

Abre un PR en la rama december_2024

Haz que se revise y se fusione

Este proceso te ayudará a aprender y a construir un curso dirigido por la comunidad que mejora constantemente.

Podemos discutir el proceso en este [hilo de discusión](https://github.com/huggingface/smol-course/discussions/2#discussion-7602932).

## Estructura del Curso

Este curso ofrece un enfoque práctico para trabajar con modelos pequeños de lenguaje, desde el entrenamiento inicial hasta el despliegue en producción.

| Módulo | Descripción | Estado | Fecha de lanzamiento |
|--------|-------------|--------|----------------------|
| [Ajuste de Instrucciones](./1_instruction_tuning) | Aprende ajuste fino (fine-tuning) supervisado, plantillas de chat y seguimiento básico de instrucciones | ✅ Completo | 3 de diciembre de 2024 |
| [Alineación de Preferencias](./2_preference_alignment) | Explora las técnicas DPO y ORPO para alinear modelos con las preferencias humanas | ✅ Completo | 6 de diciembre de 2024 |
| [Ajuste Fino (Fine-tuning) Eficiente en Parámetros](./3_parameter_efficient_finetuning) | Aprende LoRA, ajuste de prompt y métodos de adaptación eficientes | [🚧 En Progreso](https://github.com/huggingface/smol-course/pull/41) | 9 de diciembre de 2024 |
| [Evaluación](./4_evaluation) | Usa benchmarks automáticos y crea evaluaciones personalizadas para dominios | [🚧 En Progreso](https://github.com/huggingface/smol-course/issues/42) | 13 de diciembre de 2024 |
| [Modelos Visión-Lenguaje](./5_vision_language_models) | Adapta modelos multimodales para tareas visión-lenguaje | [🚧 En Progreso](https://github.com/huggingface/smol-course/issues/49) | 16 de diciembre de 2024 |
| [Conjuntos de Datos Sintéticos](./6_synthetic_datasets) | Crea y valida conjuntos de datos sintéticos para el entrenamiento | 📝 Planificado | 20 de diciembre de 2024 |
| [Inferencia](./7_inference) | Inferencia eficiente con modelos | 📝 Planificado | 23 de diciembre de 2024 |

## ¿Por qué Modelos Pequeños de Lenguaje?

Si bien los modelos grandes de lenguaje han mostrado capacidades impresionantes, requieren recursos computacionales significativos y pueden ser excesivos para aplicaciones específicas. Los modelos pequeños de lenguaje ofrecen varias ventajas para aplicaciones de dominio:

- **Eficiencia**: Requieren menos recursos computacionales para entrenar y desplegar
- **Personalización**: Más fáciles de ajustar para dominios específicos
- **Control**: Mayor control sobre el comportamiento del modelo
- **Costo**: Menores costos operativos para el entrenamiento y la inferencia
- **Privacidad**: Pueden ejecutarse localmente, manteniendo la privacidad de los datos
- **Sostenibilidad**: Uso eficiente de recursos con una huella de carbono más pequeña
- **Investigación Académica**: Facilita la investigación académica con menos restricciones logísticas

## Requisitos Previos

Antes de comenzar, asegúrate de tener:

- Conocimientos básicos en aprendizaje automático y procesamiento de lenguaje natural
- Familiaridad con Python, PyTorch y la librería `transformers`
- Acceso a un modelo de lenguaje preentrenado y un conjunto de datos etiquetado

## Instalación

Mantenemos el curso como un paquete para facilitar la instalación de dependencias. Recomendamos usar [uv](https://github.com/astral-sh/uv), pero también puedes utilizar alternativas como `pip` o `pdm`.

### Usando `uv`

Con `uv` instalado, puedes configurar el entorno del curso de esta manera:

```bash
uv venv --python 3.11.0
uv sync
```

### Usando `pip`

Para un entorno **python 3.11**, utiliza los siguientes comandos para instalar las dependencias:

```bash
# python -m venv .venv
# source .venv/bin/activate
pip install -r requirements.txt
```

### Google Colab

Para **Google Colab**, instala las dependencias de la siguiente manera:

```bash
pip install -r transformers trl datasets huggingface_hub
```

## Participación

Compartamos este curso para que muchas personas puedan aprender a ajustar LLMs sin necesidad de hardware costoso.

[![Star History Chart](https://api.star-history.com/svg?repos=huggingface/smol-course&type=Date)](https://star-history.com/#huggingface/smol-course&Date)

es/README.md

A course on aligning smol models.


![smolcourse image](../banner.png)

# スモールコース

これは、特定のユースケースに合わせて言語モデルを調整するための実践的なコースです。ほとんどのローカルマシンで実行できるため、言語モデルの調整を始めるのに便利です。GPUの要件は最小限で、有料サービスは必要ありません。このコースは[SmolLM2](https://github.com/huggingface/smollm/tree/main)シリーズのモデルに基づいていますが、ここで学んだスキルを大規模なモデルや他の小型言語モデルに転用することができます。

参加は無料で、今すぐ始められます！

このコースはオープンでピアレビューされています。コースに参加するには、プルリクエストを開くことで、あなたの作業をレビューに提出してください。以下の手順に従ってください：

リポジトリをフォークします こちら

資料を読み、変更を加え、演習を行い、自分の例を追加します。

december_2024ブランチでプルリクエストを開きます

レビューを受けてマージされます

これにより、学習を助け、常に改善されるコミュニティ主導のコースを構築することができます。

このプロセスについては、この[ディスカッションスレッド](https://github.com/huggingface/smol-course/discussions/2#discussion-7602932)で議論できます。

## コース概要

このコースは、小型言語モデルを使用した実践的なアプローチを提供し、初期のトレーニングから本番展開までをカバーします。

| モジュール | 説明 | ステータス | リリース日 |
|--------|-------------|---------|--------------|
| [インストラクションチューニング](./1_instruction_tuning) | 教師あり微調整、チャットテンプレート、および基本的な指示に従う方法を学びます | ✅ 準備完了 | 2024年12月3日 |
| [選好整合](./2_preference_alignment) | DPOおよびORPO技術を探求し、人間の選好にモデルを整合させる方法を学びます | ✅ 準備完了  | 2024年12月6日 |
| [パラメータ効率の良い微調整](./3_parameter_efficient_finetuning) | LoRA、プロンプトチューニング、および効率的な適応方法を学びます | ✅ 準備完了 | 2024年12月9日 |
| [評価](./4_evaluation) | 自動ベンチマークを使用し、カスタムドメイン評価を作成する方法を学びます | ✅ 準備完了 | 2024年12月13日 |
| [ビジョン言語モデル](./5_vision_language_models) | マルチモーダルモデルをビジョン言語タスクに適応させる方法を学びます | ✅ 準備完了 | 2024年12月16日 |
| [合成データセット](./6_synthetic_datasets) | トレーニング用の合成データセットを作成し、検証する方法を学びます | ✅ 準備完了 | 2024年12月20日 |
| [推論](./7_inference) | モデルを効率的に推論する方法を学びます | [🚧 作業中](https://github.com/huggingface/smol-course/pull/150) | 2025年1月8日 |
| [エージェント](./8_agents) | 自分のエージェントAIを構築する方法を学びます | ✅ 準備完了 | 2025年1月13日 ||
| キャップストーンプロジェクト | 学んだことを使ってリーダーボードを登りましょう！ | [🚧 作業中](https://github.com/huggingface/smol-course/pull/97) | 2025年1月10日 |

## なぜ小型言語モデルなのか？

大規模な言語モデルは印象的な能力を示していますが、しばしば多くの計算リソースを必要とし、特定のアプリケーションには過剰な場合があります。小型言語モデルは、ドメイン固有のアプリケーションに対していくつかの利点を提供します：

- **効率性**：トレーニングと展開に必要な計算リソースが大幅に少ない
- **カスタマイズ**：特定のドメインに簡単に微調整および適応可能
- **制御**：モデルの動作をよりよく理解し、制御できる
- **コスト**：トレーニングと推論の運用コストが低い
- **プライバシー**：データを外部APIに送信せずにローカルで実行可能
- **グリーンテクノロジー**：リソースの効率的な使用を推進し、炭素排出量を削減
- **学術研究の容易さ**：最先端のLLMを使用した学術研究のための簡単なスターターを提供し、物流の制約を減らす

## 前提条件

開始する前に、以下を確認してください：
- 機械学習と自然言語処理の基本的な理解
- Python、PyTorch、および`transformers`ライブラリに精通していること
- 事前学習された言語モデルとラベル付きデータセットへのアクセス

## インストール

コースをパッケージとして維持しているため、パッケージマネージャーを使用して依存関係を簡単にインストールできます。`uv`をお勧めしますが、`pip`や`pdm`などの代替手段も使用できます。

### `uv`を使用する場合

`uv`がインストールされている場合、次のようにしてコースをインストールできます：

```bash
uv venv --python 3.11.0
uv sync
```

### `pip`を使用する場合

すべての例は**python 3.11**環境で実行されるため、次のように環境を作成し、依存関係をインストールします：

```bash
# python -m venv .venv
# source .venv/bin/activate
pip install -r requirements.txt
```

### Google Colab

**Google Colabから**は、使用するハードウェアに基づいて柔軟に依存関係をインストールする必要があります。次のようにします：

```bash
pip install transformers trl datasets huggingface_hub
```

ja/README.md

A course on aligning smol models.


![smolcourse image](../banner.png)

# 소형 언어 모델 과정

이 과정에서는 특정 사용 사례에 맞게 언어 모델을 정렬하는 법을 다룹니다. 모든 자료는 대부분의 로컬 컴퓨터에서 실행되므로 간편하게 언어 모델 정렬을 시작해볼 수 있습니다. 이 과정을 위해 필요한 최소한의 GPU 요구 사항이나 유료 서비스가 없습니다. [SmolLM2](https://github.com/huggingface/smollm/tree/main) 시리즈 모델을 기반으로 하는 과정이지만, 여기서 배운 기술을 더 큰 모델이나 다른 작은 언어 모델로 옮길 수 있습니다.

지금 바로 참여하세요!

이 과정은 열려 있으며 다른 사용자와의 상호 검토를 진행할 수 있습니다. 이 과정에 참여하려면 pull request(PR)를 열고 검토 받을 수 있도록 결과물을 제출하세요. 다음 단계를 따르면 됩니다:

여기에서 레포지토리를 fork하세요.

자료를 읽고, 바꿔 보고, 실습해보고, 나만의 예제를 추가해보세요.

december-2024 브랜치에 PR을 보내세요.

검토가 끝나면 december-2024 브랜치에 병합됩니다.

이 과정은 학습에 도움이 될 뿐만 아니라 지속적으로 발전하는 커뮤니티 기반 코스를 형성하는 데에도 기여할 것입니다.

[discussion thread](https://github.com/huggingface/smol-course/discussions/2#discussion-7602932)에서 과정에 대해 토론할 수도 있습니다.

## 과정 개요

이 과정은 소형 언어 모델의 초기 학습부터 결과물 배포까지 실습할 수 있는 실용적인 내용을 제공합니다.

| 모듈 | 설명 | 상태 | 공개일 |
|--------|-------------|---------|--------------|
| [Instruction Tuning](./1_instruction_tuning) | 지도 학습 기반 미세 조정, 대화 템플릿 작성, 기본적인 지시를 따르게 하는 방법 학습 | ✅ 학습 가능 | 2024. 12. 3 |
| [Preference Alignment](./2_preference_alignment) | 모델을 인간 선호도에 맞게 정렬하기 위한 DPO와 ORPO 기법 학습 | ✅ 학습 가능 | 2024. 12. 6 |
| [Parameter-efficient Fine-tuning](./3_parameter_efficient_finetuning) | LoRA, 프롬프트 튜닝을 포함한 효율적인 적응 방법 학습 | ✅ 학습 가능 | 2024. 12. 9 |
| [Evaluation](./4_evaluation) | 자동 벤치마크 사용법 및 사용자 정의 도메인 평가 수행 방법 학습 | ✅ 학습 가능 | 2024. 12. 13 |
| [Vision-language Models](./5_vision_language_models) | 비전-언어 태스크를 위한 멀티모달 모델 적용 방법 학습 | [🚧 준비중](https://github.com/huggingface/smol-course/issues/49) | 2024. 12. 16 |
| [Synthetic Datasets](./6_synthetic_datasets) | 모델 학습을 위한 합성 데이터셋 생성 및 검증 | [🚧 준비중](https://github.com/huggingface/smol-course/issues/83) | 2024. 12. 20 |
| [Inference](./7_inference) | 모델의 효율적인 추론 방법 학습 | 📝 작성 예정 | 2024. 12. 23 |

## 왜 소형 언어 모델을 사용하나요?

대형 언어 모델은 뛰어난 능력을 보여주지만, 상당한 연산 자원을 필요로 하며 특정 기능에 초점을 맞춘 애플리케이션에 대해서는 대형 언어 모델이 과한 경우도 있습니다. 소형 언어 모델은 도메인 특화 애플리케이션에 있어서 몇 가지 이점을 제공합니다:

- **효율성**: 대형 언어 모델보다 훨씬 적은 연산 자원으로 학습 및 배포 가능
- **맞춤화**: 특정 도메인에 대한 미세 조정 및 적응 용이
- **제어**: 모델 동작 과정을 잘 이해할 수 있고 모델의 동작을 쉽게 제어 가능
- **비용**: 학습과 추론 과정에서 필요한 비용 감소
- **프라이버시**: 데이터를 외부 API로 보내지 않고 로컬에서 실행 가능
- **친환경**: 탄소 발자국을 줄이는 효율적인 자원 사용
- **쉬운 학술 연구 및 개발**: 최신 LLM을 활용해 물리적 제약 없이 학술 연구를 쉽게 시작할 수 있도록 지원

## 사전 준비 사항

시작하기 전에 아래 사항이 준비되어 있는지 확인하세요:
- 머신러닝과 자연어처리에 대한 기본적인 이해가 필요합니다.
- Python, PyTorch 및 `transformers` 라이브러리에 익숙해야 합니다.
- 사전 학습된 언어 모델과 레이블이 있는 액세스할 수 있어야 합니다.

## 설치

이 과정은 패키지 형태로 관리되기 때문에 패키지 매니저를 이용해 의존성 설치를 쉽게 진행할 수 있습니다. 이를 위해 [uv](https://github.com/astral-sh/uv) 사용을 권장하지만 `pip`나 `pdm`을 사용할 수도 있습니다.

### `uv` 사용

`uv`를 설치한 후, 아래 명령어로 소형 언어 모델 과정을 설치할 수 있습니다:

```bash
uv venv --python 3.11.0
uv sync
```

### `pip` 사용

모든 예제는 동일한 **python 3.11** 환경에서 실행되기 때문에 아래처럼 환경을 생성하고 의존성을 설치해야 합니다:

```bash
# python -m venv .venv
# source .venv/bin/activate
pip install -r requirements.txt
```

### Google Colab

**Google Colab**에서는 사용하는 하드웨어에 따라 유연하게 의존성을 설치해야 합니다. 아래와 같이 진행하세요:

```bash
pip install transformers trl datasets huggingface_hub
```

## 참여

많은 사람이 고가의 장비 없이 LLM을 미세 조정하는 법을 배울 수 있도록 이 자료를 공유해 봅시다!

[![Star History Chart](https://api.star-history.com/svg?repos=huggingface/smol-course&type=Date)](https://star-history.com/#huggingface/smol-course&Date)

ko/README.md

A course on aligning smol models.


![smolcourse image](../banner.png)

# a smol course (um curso miudinho)

Este é um curso prático sobre alinhar modelos de linguagem para o seu caso de uso específico. É uma maneira útil de começar a alinhar modelos de linguagem, porque tudo funciona na maioria das máquinas locais. Existem requisitos mínimos de GPU e nenhum serviço pago. O curso é baseado na série de modelos de [SmolLM2](https://github.com/huggingface/smollm/tree/main), mas você pode transferir as habilidades que aprende aqui para modelos maiores ou outros pequenos modelos de linguagem.

A participação é aberta a todos, gratuita e já está disponível!

Este curso é aberto e avaliado por pares (peer reviewed). Para começar o curso, abra um pull request (PR) e envie seu trabalho para revisão. Aqui estão as etapas:

Dê um fork no repositório aqui

Leia o material, faça alterações, faça os exercícios, adicione seus próprios exemplos

Abra um PR no branch december_2024

Tenha seu material revisado e mesclado no branch principal

Isso deve te ajudar a aprender e a construir um curso feito pela comunidade, que está sempre melhorando.

Podemos discutir o processo neste [tópico de discussão](https://github.com/huggingface/smol-course/discussions/2#discussion-7602932).

## Sumário do Curso

Este curso fornece uma abordagem prática para trabalhar com pequenos modelos de linguagem, desde o treinamento inicial até a implantação de produção.

| Módulo | Descrição | Status | Data de Lançamento |
|--------|-------------|---------|--------------|
| [Instruction Tuning (Ajuste de Instrução)](./1_instruction_tuning) | Aprenda sobre o ajuste fino supervisionado, modelos de bate-papo e a fazer o modelo seguir instruções básicas | ✅ Completo | 3 Dez, 2024 |
| [Preference Alignment (Alinhamento de Preferência)](./2_preference_alignment) | Explore técnicas DPO e ORPO para alinhar modelos com preferências humanas | ✅ Completo  | 6 Dez, 2024 |
| [Parameter-efficient Fine-tuning (Ajuste Fino com Eficiência de Parâmetro)](./3_parameter_efficient_finetuning) | Aprenda sobre LoRA, ajuste de prompt e métodos de adaptação eficientes | ✅ Completo | 9 Dez, 2024 |
| [Evaluation (Avaliação)](./4_evaluation) | Use benchmarks automáticos e crie avaliações de domínio personalizadas | ✅ Completo | 13 Dez, 2024 |
| [Vision-language Models (Modelos de Conjunto Visão-linguagem)](./5_vision_language_models) | Adapte modelos multimodais para tarefas visão-linguagem | ✅ Completo | 16 Dez, 2024 |
| [Synthetic Datasets (Conjuntos de Dados Sintéticos)](./6_synthetic_datasets) | Criar e validar conjuntos de dados sintéticos para treinamento | [🚧 Em Progresso](https://github.com/huggingface/smol-course/issues/83) | 20 Dez, 2024 |
| [Inference (Inferência)](./7_inference) | Infira modelos com eficiência | 📝 Planejado | 23 Dez, 2024 |
| Projeto Experimental | Use o que você aprendeu para ser o top 1 na tabela de classificação! | [🚧 Em Progresso](https://github.com/huggingface/smol-course/pull/97) | Dec 23, 2024 |

## Por Que Pequenos Modelos de Linguagem?

Embora os grandes modelos de linguagem tenham mostrado recursos e capacidades impressionantes, eles geralmente exigem recursos computacionais significativos e podem ser exagerados para aplicativos focados. Os pequenos modelos de linguagem oferecem várias vantagens para aplicativos de domínios específicos:

- **Eficiência**: Requer significativamente menos recursos computacionais para treinar e implantar
- **Personalização**: Mais fácil de ajustar e se adaptar a domínios específicos
- **Controle**: Melhor compreensão e controle do comportamento do modelo
- **Custo**: Menores custos operacionais para treinamento e inferência
- **Privacidade**: Pode ser executado localmente sem enviar dados para APIs externas
- **Tecnologia Verde**: Defende o uso eficiente de recursos com redução da pegada de carbono
- **Desenvolvimento de Pesquisa Acadêmica Mais Fácil**: Oferece um ponto de partida fácil para a pesquisa acadêmica com LLMs de ponta com menos restrições logísticas

## Pré-requisitos

Antes de começar, verifique se você tem o seguinte:
- Entendimento básico de machine learning e natural language processing.
- Familiaridade com Python, PyTorch e o módulo `transformers`.
- Acesso a um modelo de linguagem pré-treinado e um conjunto de dados rotulado.

## Instalação

Mantemos o curso como um pacote para que você possa instalar dependências facilmente por meio de um gerenciador de pacotes. Recomendamos [uv](https://github.com/astral-sh/uv) para esse fim, mas você pode usar alternativas como `pip` ou` pdm`.

### Usando `uv`

Com o `uv` instalado, você pode instalar o curso deste modo:

```bash
uv venv --python 3.11.0
uv sync
```

### Usando `pip`

Todos os exemplos são executados no mesmo ambiente **python 3.11**, então você deve criar um ambiente e instalar dependências desta maneira:

```bash
# python -m venv .venv
# source .venv/bin/activate
pip install -r requirements.txt
```

### Google Colab

**A partir do Google Colab** você precisará instalar dependências de maneira flexível com base no hardware que está usando. Pode fazer deste jeito:

```bash
pip install transformers trl datasets huggingface_hub
```

## Engajamento

Vamos compartilhar isso, desse jeito um monte de gente vai poder aprender a ajustar LLMs sem precisar de um computador super caro.

[![Gráfico de Histórico de Estrelas](https://api.star-history.com/svg?repos=huggingface/smol-course&type=Date)](https://star-history.com/#huggingface/smol-course&Date)

pt-br/README.md

A course on aligning smol models.


![smolcourse image](../banner.png)

# Khoá học Mô hình ngôn ngữ cơ bản

Đây là một khoá học thực hành về việc huấn luyện các mô hình ngôn ngữ (LM) cho các trường hợp sử dụng cụ thể. Khoá học này là cách thuận tiện để bắt đầu với việc điều chỉnh các mô hình ngôn ngữ, bởi vì mọi thứ đều có thể chạy được trên hầu hết các máy tính cá nhân. Tại đây, chúng ta không cần quá nhiều tài nguyên GPUs hay các dịch vụ trả phí để học tập. Khoá học được xây dựng dựa trên series mô hình [SmolLM2](https://github.com/huggingface/smollm/tree/main), nhưng bạn có thể áp dụng các kỹ năng học được ở đây cho các mô hình lớn hơn hoặc các mô hình ngôn ngữ nhỏ khác.

*Lưu ý: Vì khóa học được dịch từ bản gốc tiếng Anh, chúng tôi sẽ giữ lại một số thuật ngữ gốc nhằm tránh gây hiểu lầm.*

s

Tham gia học ngay!

Khoá học này mở và được đánh giá bởi cộng đồng. Để tham gia khoá học hãy tạo Pull Request (PR) và gửi bài làm của bạn để được review. Các bước thực hiện:

Fork repo tại đây

Đọc tài liệu, thực hiện thay đổi, làm bài tập, thêm ví dụ của riêng bạn

Tạo PR trên nhánh december_2024

Nhận review và merge

Điều này sẽ giúp bạn học tập và xây dựng một khoá học có cộng đồng tham gia và luôn được cải thiện.

Chúng ta có thể thảo luận về quá trình học tập và phát triển trong [thread này](https://github.com/huggingface/smol-course/discussions/2#discussion-7602932).

## Nội dung khoá học

Khoá học này cung cấp phương pháp thực hành để làm việc với các mô hình ngôn ngữ nhỏ, từ huấn luyện ban đầu đến triển khai lên sản phẩm.

| Bài | Mô tả | Trạng thái | Ngày phát hành |
|--------|-------------|---------|--------------|
| [Tinh chỉnh theo chỉ thị (Instruction Tuning)](./1_instruction_tuning) | Học về huấn luyện có giám sát (SFT), định dạng chat, và thực hiện các chỉ thị cơ bản | ✅ Sẵn sàng | 3/12/2024 |
| [Tinh chỉnh theo sự ưu tiên (Preference Alignment)](./2_preference_alignment) | Học các kỹ thuật DPO và ORPO để huấn luyện mô hình theo sự uy tiên của người dùng | ✅ Sẵn sàng  | 6/12/2024 |
| [Tinh chỉnh hiệu quả tham số (Parameter-efficient Fine-tuning)](./3_parameter_efficient_finetuning) | Học về LoRA, prompt tuning và các phương pháp huấn luyện hiệu quả | ✅ Sẵn sàng | 9/12/2024 |
| [Đánh giá mô hình (Evaluation)](./4_evaluation) | Sử dụng benchmark tự động và tạo đánh giá theo lĩnh vực cụ thể | [🚧 Đang thực hiện](https://github.com/huggingface/smol-course/issues/42) | 13/12/2024 |
| [Mô hình đa phương thức (Vision-language Models)](./5_vision_language_models) | Điều chỉnh các mô hình đa phương thức (Multimodal models) cho các tác vụ thị giác-ngôn ngữ | [🚧 Đang thực hiện](https://github.com/huggingface/smol-course/issues/49) | 16/12/2024 |
| [Dữ liệu nhân tạo (Synthetic Datasets)](./6_synthetic_datasets) | Tạo và đánh giá tập dữ liệu tổng hợp cho huấn luyện | 📝 Đã lên kế hoạch | 20/12/2024 |
| [Triển khai mô hình (Inference)](./7_inference) | Triển khai mô hình một cách hiệu quả | 📝 Đã lên kế hoạch | 23/12/2024 |

## Tại sao lại chọn Mô hình Ngôn ngữ Nhỏ?

Mặc dù các mô hình ngôn ngữ lớn đã cho thấy khả năng ấn tượng, chúng thường yêu cầu tài nguyên tính toán đáng kể và có thể quá mức cần thiết cho các ứng dụng tập trung. Các mô hình ngôn ngữ nhỏ mang lại nhiều lợi thế cho các ứng dụng theo lĩnh vực cụ thể:

- **Hiệu quả:** Yêu cầu ít tài nguyên tính toán hơn đáng kể để huấn luyện và triển khai
- **Tùy chỉnh:** Dễ dàng tinh chỉnh phù hợp cho các lĩnh vực cụ thể
- **Kiểm soát:** Dễ dàng hiểu và kiểm soát hành vi mô hình hơn
- **Chi phí:** Chi phí huấn luyện và triển khai thấp hơn đáng kể
- **Bảo mật:** Có thể triển khai trên mạng cục bộ mà không cần gửi dữ liệu tới API bên ngoài
- **Công nghệ xanh:** Sử dụng tài nguyên hiệu quả (carbon footprint giảm)
- **Nghiên cứu học thuật dễ dàng hơn:** Khởi đầu dễ dàng cho nghiên cứu học thuật trên mô hình ngôn ngữ với ít ràng buộc về tài nghiên tính toán

## Yêu cầu tiên quyết

Trước khi bắt đầu, hãy đảm bảo bạn có:
- Hiểu biết cơ bản về machine learning (ML) và xử lý ngôn ngữ tự nhiên (NLP)
- Quen thuộc với Python, PyTorch và thư viện `transformers`
- Hiểu về mô hình ngôn ngữ và tập dữ liệu có nhãn

## Cài đặt

Chúng tôi duy trì khoá học như một package để bạn có thể cài đặt dễ dàng thông qua package manager. Chúng tôi gợi ý sử dụng [uv](https://github.com/astral-sh/uv), nhưng bạn có thể sử dụng các lựa chọn thay thế như `pip` hoặc `pdm`.

### Sử dụng `uv`

Với `uv` đã được cài đặt, bạn có thể cài đặt khoá học như sau:

```bash
uv venv --python 3.11.0
uv sync
```

### Sử dụng `pip`

Tất cả các ví dụ chạy trong cùng một môi trường **python 3.11**, vì vậy bạn nên tạo môi trường và cài đặt dependencies như sau:

```bash
# python -m venv .venv
# source .venv/bin/activate
pip install -r requirements.txt
```

### Google Colab

Nếu dùng **Google Colab** bạn sẽ cần cài đặt dependencies một cách linh hoạt dựa trên phần cứng bạn đang sử dụng. Như sau:

```bash
pip install -r transformers trl datasets huggingface_hub
```

## Tham gia và chia sẻ

Hãy chia sẻ khoá học này, để cùng nhau phát triển cộng đồng AI Việt Nam.

[![Biểu đồ sao của khoá học](https://api.star-history.com/svg?repos=huggingface/smol-course&type=Date)](https://star-history.com/#huggingface/smol-course&Date)

vi/README.md

A course on aligning smol models.


# Domain Specific Evaluation with Argilla, Distilabel, and LightEval

Most popular benchmarks look at very general capabilities (reasoning, math, code), but have you ever needed to study more specific capabilities?

What should you do if you need to evaluate a model on a **custom domain** relevant to your use-cases? (For example, financial, legal, medical use cases)

This tutorial shows you the full pipeline you can follow, from creating relevant data and annotating your samples to evaluating your model on them, with the easy to use [Argilla](https://github.com/argilla-io/argilla), [distilabel](https://github.com/argilla-io/distilabel), and [lighteval](https://github.com/huggingface/lighteval). For our example, we'll focus on generating exam questions from multiple documents.

## Project Structure

For our process, we will follow 4 steps, with a script for each: generating a dataset, annotating it, extracting relevant samples for evaluation from it, and actually evaluating models.

| Script Name | Description |
|-------------|-------------|
| generate_dataset.py | Generates exam questions from multiple text documents using a specified language model. |
| annotate_dataset.py | Creates an Argilla dataset for manual annotation of the generated exam questions. |
| create_dataset.py | Processes annotated data from Argilla and creates a Hugging Face dataset. |
| evaluation_task.py | Defines a custom LightEval task for evaluating language models on the exam questions dataset. |

## Steps

### 1. Generate Dataset

The `generate_dataset.py` script uses the distilabel library to generate exam questions based on multiple text documents. It uses the specified model (default: Meta-Llama-3.1-8B-Instruct) to create questions, correct answers, and incrorect answers (known as distractors). You should add you own data samples and you might wish to use a different model.

To run the generation:

```sh
python generate_dataset.py --input_dir path/to/your/documents --model_id your_model_id --output_path output_directory
```

This will create a [Distiset](https://distilabel.argilla.io/dev/sections/how_to_guides/advanced/distiset/) containing the generated exam questions for all documents in the input directory.

### 2. Annotate Dataset

The `annotate_dataset.py` script takes the generated questions and creates an Argilla dataset for annotation. It sets up the dataset structure and populates it with the generated questions and answers, randomizing the order of answers to avoid bias. Once in Argilla, you or a domain expert can validate the dataset with the correct answers.

You will see suggested correct answers from the LLM in random order and you can approve the correct answer or select a different one. The duration of this process will depend on the scale of your evaluation dataset, the complexity of your domain data, and the quality of your LLM. For example, we were able to create 150 samples within 1 hour on the domain of transfer learning, using Llama-3.1-70B-Instruct, mostly by approving the correct answer and discarding the incorrect ones.

To run the annotation process:

```sh
python annotate_dataset.py --dataset_path path/to/distiset --output_dataset_name argilla_dataset_name
```

This will create an Argilla dataset that can be used for manual review and annotation.

![argilla_dataset](./images/domain_eval_argilla_view.png)

If you're not using Argilla, deploy it locally or on spaces following this [quickstart guide](https://docs.argilla.io/latest/getting_started/quickstart/).

### 3. Create Dataset

The `create_dataset.py` script processes the annotated data from Argilla and creates a Hugging Face dataset. It handles both suggested and manually annotated answers. The script will create a dataset with the question, possible answers, and the column name for the correct answer. To create the final dataset:

```sh
huggingface_hub login
python create_dataset.py --dataset_path argilla_dataset_name --dataset_repo_id your_hf_repo_id
```

This will push the dataset to the Hugging Face Hub under the specified repository. You can view the sample dataset on the hub [here](https://huggingface.co/datasets/burtenshaw/exam_questions/viewer/default/train), and a preview of the dataset looks like this:

![hf_dataset](./images/domain_eval_dataset_viewer.png)

### 4. Evaluation Task

The `evaluation_task.py` script defines a custom LightEval task for evaluating language models on the exam questions dataset. It includes a prompt function, a custom accuracy metric, and the task configuration.

To evaluate a model using lighteval with the custom exam questions task:

```sh
lighteval accelerate \
    --model_args "pretrained=HuggingFaceH4/zephyr-7b-beta" \
    --tasks "community|exam_questions|0|0" \
    --custom_tasks domain-eval/evaluation_task.py \
    --output_dir "./evals"
```

You can find detailed guides in lighteval wiki about each of these steps:

- [Creating a Custom Task](https://github.com/huggingface/lighteval/wiki/Adding-a-Custom-Task)
- [Creating a Custom Metric](https://github.com/huggingface/lighteval/wiki/Adding-a-New-Metric)
- [Using existing metrics](https://github.com/huggingface/lighteval/wiki/Metric-List)

A course on aligning smol models.


# Ajuste por Instrucciones (Instruction Tuning)

Este módulo te guiará en el proceso de ajuste por instrucciones de modelos de lenguaje. El ajuste por instrucciones implica adaptar modelos preentrenados a tareas específicas mediante un entrenamiento adicional sobre conjuntos de datos específicos para esas tareas. Este proceso ayuda a los modelos a mejorar su rendimiento en tareas específicas.

En este módulo, exploraremos dos temas: 1) Plantillas de Chat y 2) Fine-tuning Supervisado

## 1⃣ Plantillas de Chat

Las plantillas de chat estructuran las interacciones entre los usuarios y los modelos de IA, asegurando respuestas coherentes y contextualmente apropiadas. Estas plantillas incluyen componentes como los mensajes de sistema y mensajes basados en roles. Para más información detallada, consulta la sección de [Plantillas de Chat](./chat_templates.md).

## 2⃣ Fine-tuning Supervisado

El Fine-tuning supervisado (SFT) es un proceso crítico para adaptar modelos de lenguaje preentrenados a tareas específicas. Implica entrenar el modelo en un conjunto de datos específico para la tarea con ejemplos etiquetados. Para una guía detallada sobre el SFT, incluyendo pasos clave y mejores prácticas, consulta la página de [Fine-tuning Supervisado](./supervised_fine_tuning.md).

## Cuadernos de Ejercicio

| Título | Descripción | Ejercicio | Enlace | Colab |
|--------|-------------|-----------|--------|-------|
| Plantillas de Chat | Aprende a usar plantillas de chat con SmolLM2 y a procesar conjuntos de datos en formato chatml | 🐢 Convierte el conjunto de datos `HuggingFaceTB/smoltalk` al formato chatml
 🐕 Convierte el conjunto de datos `openai/gsm8k` al formato chatml | [Cuaderno](./notebooks/chat_templates_example.ipynb) |  |
| Fine-tuning Supervisado | Aprende a afinar SmolLM2 utilizando el SFTTrainer | 🐢 Usa el conjunto de datos `HuggingFaceTB/smoltalk`
 🐕 Prueba el conjunto de datos `bigcode/the-stack-smol`
 🦁 Selecciona un conjunto de datos para un caso de uso real | [Cuaderno](./notebooks/sft_finetuning_example.ipynb) |  |

## Referencias

- [Documentación de Transformers sobre plantillas de chat](https://huggingface.co/docs/transformers/main/en/chat_templating)
- [Script para Fine-Tuning Supervisado en TRL](https://github.com/huggingface/trl/blob/main/examples/scripts/sft.py)
- [`SFTTrainer` en TRL](https://huggingface.co/docs/trl/main/en/sft_trainer)
- [Papel de Optimización Directa de Preferencias](https://arxiv.org/abs/2305.18290)
- [Fine-Tuning Supervisado con TRL](https://huggingface.co/docs/trl/main/en/tutorials/supervised_finetuning)
- [Cómo afinar Google Gemma con ChatML y Hugging Face TRL](https://www.philschmid.de/fine-tune-google-gemma)
- [Fine-Tuning de LLM para generar catálogos de productos persas en formato JSON](https://huggingface.co/learn/cookbook/en/fine_tuning_llm_to_generate_persian_product_catalogs_in_json_format)

A course on aligning smol models.


# Plantillas de Chat

Las plantillas de chat son esenciales para estructurar las interacciones entre modelos de lenguaje y usuarios. Proporcionan un formato consistente para las conversaciones, asegurando que los modelos comprendan el contexto y el rol de cada mensaje, manteniendo patrones de respuesta apropiados.

## Modelos Base vs Modelos Instructivos

Un modelo base se entrena con datos de texto crudo para predecir el siguiente token, mientras que un modelo instructivo se ajusta específicamente para seguir instrucciones y participar en conversaciones. Por ejemplo, `SmolLM2-135M` es un modelo base, mientras que `SmolLM2-135M-Instruct` es su variante ajustada a instrucciones.

Para hacer que un modelo base se comporte como un modelo instructivo, necesitamos formatear nuestros prompts de manera consistente para que el modelo los entienda. Aquí es donde entran las plantillas de chat. ChatML es un formato de plantilla que estructura las conversaciones con indicadores de rol claros (sistema, usuario, asistente).

Es importante notar que un modelo base podría ser ajustado con diferentes plantillas de chat, por lo que, al usar un modelo instructivo, debemos asegurarnos de estar utilizando la plantilla de chat correcta.

## Comprendiendo las Plantillas de Chat

En su núcleo, las plantillas de chat definen cómo se deben formatear las conversaciones al comunicarse con un modelo de lenguaje. Incluyen instrucciones a nivel de sistema, mensajes del usuario y respuestas del asistente en un formato estructurado que el modelo puede entender. Esta estructura ayuda a mantener la coherencia en las interacciones y asegura que el modelo responda adecuadamente a diferentes tipos de entradas. A continuación, se muestra un ejemplo de una plantilla de chat:

```sh
<|im_start|>user
¡Hola!<|im_end|>
<|im_start|>assistant
¡Mucho gusto!<|im_end|>
<|im_start|>user
¿Puedo hacer una pregunta?<|im_end|>
<|im_start|>assistant
```

La librería `transformers` se encarga de las plantillas de chat por ti en relación con el tokenizador del modelo. Lee más sobre cómo `transformers` construye las plantillas de chat [aquí](https://huggingface.co/docs/transformers/en/chat_templating#how-do-i-use-chat-templates). Todo lo que tenemos que hacer es estructurar nuestros mensajes de la forma correcta y el tokenizador se encargará del resto. Aquí tienes un ejemplo básico de una conversación:

```python
messages = [
    {"role": "system", "content": "Eres un asistente útil centrado en temas técnicos."},
    {"role": "user", "content": "¿Puedes explicar qué es una plantilla de chat?"},
    {"role": "assistant", "content": "Una plantilla de chat estructura las conversaciones entre los usuarios y los modelos de IA..."}
]
```

Vamos a desglosar el ejemplo anterior y ver cómo se mapea al formato de plantilla de chat.

## Mensajes del Sistema

Los mensajes del sistema establecen la base para el comportamiento del modelo. Actúan como instrucciones persistentes que influyen en todas las interacciones posteriores. Por ejemplo:

```python
system_message = {
    "role": "system",
    "content": "Eres un agente de atención al cliente profesional. Siempre sé educado, claro y útil."
}
```

## Conversaciones

Las plantillas de chat mantienen el contexto a través del historial de la conversación, almacenando intercambios previos entre el usuario y el asistente. Esto permite conversaciones más coherentes en múltiples turnos:

```python
conversation = [
    {"role": "user", "content": "Necesito ayuda con mi pedido"},
    {"role": "assistant", "content": "Estaré encantado de ayudarte. ¿Podrías proporcionarme tu número de pedido?"},
    {"role": "user", "content": "Es el PEDIDO-123"},
]
```

## Implementación con Transformers

La librería `transformers` proporciona soporte integrado para plantillas de chat. Aquí te mostramos cómo usarlas:

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("HuggingFaceTB/SmolLM2-135M-Instruct")

messages = [
    {"role": "system", "content": "Eres un asistente útil para programación."},
    {"role": "user", "content": "Escribe una función en Python para ordenar una lista"},
]

# Aplica la plantilla de chat
formatted_chat = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
```

## Formato Personalizado

Puedes personalizar cómo se formatean los diferentes tipos de mensajes. Por ejemplo, añadiendo tokens especiales o formato para diferentes roles:

```python
template = """
<|system|>{system_message}
<|user|>{user_message}
<|assistant|>{assistant_message}
""".lstrip()
```

## Soporte para Conversaciones de Varios Turnos

Las plantillas pueden manejar conversaciones complejas de varios turnos mientras mantienen el contexto:

```python
messages = [
    {"role": "system", "content": "Eres un tutor de matemáticas."},
    {"role": "user", "content": "¿Qué es el cálculo?"},
    {"role": "assistant", "content": "El cálculo es una rama de las matemáticas..."},
    {"role": "user", "content": "¿Puedes darme un ejemplo?"},
]
```

⏭ [Siguiente: Fine-tuning Supervisado](./supervised_fine_tuning.md)

## Recursos

- [Guía de Plantillas de Chat de Hugging Face](https://huggingface.co/docs/transformers/main/en/chat_templating)
- [Documentación de Transformers](https://huggingface.co/docs/transformers)
- [Repositorio de Ejemplos de Plantillas de Chat](https://github.com/chujiezheng/chat_templates)
```

A course on aligning smol models.


# Fine-Tuning Supervisado

El Fine-Tuning Supervisado (SFT) es un proceso crítico para adaptar modelos de lenguaje preentrenados a tareas o dominios específicos. Aunque los modelos preentrenados tienen capacidades generales impresionantes, a menudo necesitan ser personalizados para sobresalir en casos de uso particulares. El SFT cubre esta brecha entrenando el modelo con conjuntos de datos cuidadosamente seleccionados que contienen ejemplos validados por humanos.

## Comprendiendo el Fine-Tuning Supervisado

En su núcleo, el Fine-Tuning supervisado se trata de enseñar a un modelo preentrenado a realizar tareas específicas mediante ejemplos de tokens etiquetados. El proceso consiste en mostrarle al modelo muchos ejemplos del comportamiento deseado de entrada-salida, permitiéndole aprender los patrones específicos de tu caso de uso.

El SFT es eficaz porque utiliza el conocimiento fundamental adquirido durante el preentrenamiento, mientras adapta el comportamiento del modelo para que se ajuste a tus necesidades específicas.

## Cuándo Utilizar Fine-Tuning Supervisado

La decisión de usar SFT generalmente depende de la brecha entre las capacidades actuales de tu modelo y tus requisitos específicos. El SFT se vuelve particularmente valioso cuando necesitas un control preciso sobre las salidas del modelo o cuando trabajas en dominios especializados.

Por ejemplo, si estás desarrollando una aplicación de atención al cliente, es posible que desees que tu modelo siga consistentemente las pautas de la empresa y maneje consultas técnicas de una manera estandarizada. De manera similar, en aplicaciones médicas o legales, la precisión y la adherencia a la terminología específica del dominio se vuelven cruciales. En estos casos, el SFT puede ayudar a alinear las respuestas del modelo con los estándares profesionales y la experiencia del dominio.

## El Proceso de Fine-Tuning Supervisado (SFT)

El proceso de Fine-Tuning Supervisado consiste en entrenar los pesos del modelo en un conjunto de datos específico para una tarea.

Primero, necesitarás preparar o seleccionar un conjunto de datos que represente tu tarea objetivo. Este conjunto de datos debe incluir ejemplos diversos que cubran una amplia gama de escenarios que tu modelo encontrará. La calidad de estos datos es importante: cada ejemplo debe demostrar el tipo de salida que deseas que el modelo produzca. Luego, viene la fase de fine-tuning, donde usarás marcos como `transformers` y `trl` de Hugging Face para entrenar el modelo con tu conjunto de datos.

Durante todo el proceso, la evaluación continua es esencial. Querrás monitorear el rendimiento del modelo en un conjunto de validación para asegurarte de que está aprendiendo los comportamientos deseados sin perder sus capacidades generales. En el [módulo 4](../4_evaluation), cubriremos cómo evaluar tu modelo.

## El Rol del SFT en la Alineación con las Preferencias

El SFT juega un papel fundamental en la alineación de los modelos de lenguaje con las preferencias humanas. Técnicas como el Aprendizaje por Refuerzo a partir de Retroalimentación Humana (RLHF) y la Optimización Directa de Preferencias (DPO) dependen del SFT para formar un nivel base de comprensión de la tarea antes de alinear aún más las respuestas del modelo con los resultados deseados. Los modelos preentrenados, a pesar de su competencia general en el lenguaje, no siempre generan salidas que coinciden con las preferencias humanas. El SFT cubre esta brecha al introducir datos específicos de dominio y orientación, lo que mejora la capacidad del modelo para generar respuestas que se alineen más estrechamente con las expectativas humanas.

## Fine-Tuning Supervisado con Aprendizaje por Refuerzo de Transformers (TRL)

Un paquete de software clave para el fine-tuning supervisado es el Aprendizaje por Refuerzo de Transformers (TRL). TRL es un conjunto de herramientas utilizado para entrenar modelos de lenguaje transformadores utilizando aprendizaje por refuerzo (RL).

Basado en la librería Transformers de Hugging Face, TRL permite a los usuarios cargar directamente modelos de lenguaje preentrenados y es compatible con la mayoría de las arquitecturas de decodificadores y codificadores-decodificadores. La librería facilita los principales procesos del RL utilizado en la modelización del lenguaje, incluyendo el fine-tuning supervisado (SFT), la modelización de recompensas (RM), la optimización de políticas proximales (PPO) y la optimización directa de preferencias (DPO). Usaremos TRL en varios módulos a lo largo de este repositorio.

# Próximos Pasos

Prueba los siguientes tutoriales para obtener experiencia práctica con el SFT utilizando TRL:

⏭ [Tutorial de Plantillas de Chat](./notebooks/chat_templates_example.ipynb)

⏭ [Tutorial de Fine-Tuning Supervisado](./notebooks/supervised_fine_tuning_tutorial.ipynb)

A course on aligning smol models.


# Alineación de Preferencias

Este módulo cubre técnicas para alinear modelos de lenguaje con las preferencias humanas. Mientras que la afinación supervisada (SFT) ayuda a los modelos a aprender tareas, la alineación de preferencias fomenta que las salidas coincidan con las expectativas y valores humanos.

## Descripción General

Los métodos típicos de alineación incluyen múltiples etapas:
1. Afinación Supervisada (SFT) para adaptar los modelos a dominios específicos.
2. Alineación de preferencias (como RLHF o DPO) para mejorar la calidad de las respuestas.

Enfoques alternativos como ORPO combinan la afinación por instrucciones y la alineación de preferencias en un solo proceso. Aquí, nos enfocaremos en los algoritmos DPO y ORPO.

Si deseas aprender más sobre las diferentes técnicas de alineación, puedes leer más sobre ellas en el [Blog de Argilla](https://argilla.io/blog/mantisnlp-rlhf-part-8).

### 1⃣ Optimización Directa de Preferencias (DPO)

La Optimización Directa de Preferencias (DPO) simplifica la alineación de preferencias optimizando directamente los modelos utilizando datos de preferencias. Este enfoque elimina la necesidad de modelos de recompensa separados y de un aprendizaje por refuerzo complejo, lo que lo hace más estable y eficiente que el Aprendizaje por Refuerzo de Retroalimentación Humana (RLHF) tradicional. Para más detalles, puedes consultar la [documentación de Optimización Directa de Preferencias (DPO)](./dpo.md).

### 2⃣ Optimización de Preferencias por Ratio de Probabilidades (ORPO)

ORPO introduce un enfoque combinado para la afinación por instrucciones y la alineación de preferencias en un solo proceso. Modifica el objetivo estándar del modelado de lenguaje combinando la pérdida de verosimilitud logarítmica negativa con un término de ratio de probabilidades a nivel de token. El enfoque presenta un proceso de entrenamiento de una sola etapa, una arquitectura libre de modelo de referencia y una mayor eficiencia computacional. ORPO ha mostrado resultados impresionantes en varios puntos de referencia, demostrando un mejor rendimiento en AlpacaEval en comparación con los métodos tradicionales. Para más detalles, puedes consultar la [documentación de Optimización de Preferencias por Ratio de Probabilidades (ORPO)](./orpo.md).

## Notebooks de Ejercicios

| Titulo | Descripción | Ejercicio | Enlace | Colab |
|-------|-------------|----------|------|-------|
| Entrenamiento DPO | Aprende a entrenar modelos utilizando la Optimización Directa de Preferencias | 🐢 Entrenar un modelo utilizando el conjunto de datos HH-RLHF de Anthropic
🐕 Utiliza tu propio conjunto de datos de preferencias
🦁 Experimenta con diferentes conjuntos de datos de preferencias y tamaños de modelo | [Notebook](./notebooks/dpo_finetuning_example.ipynb) |  |
| Entrenamiento ORPO | Aprende a entrenar modelos utilizando la Optimización de Preferencias por Ratio de Probabilidades | 🐢 Entrenar un modelo utilizando datos de instrucciones y preferencias
🐕 Experimenta con diferentes ponderaciones de la pérdida
🦁 Compara los resultados de ORPO con DPO | [Notebook](./notebooks/orpo_finetuning_example.ipynb) |  |

## Recursos

- [Documentación de TRL](https://huggingface.co/docs/trl/index) - Documentación para la librería Transformers Reinforcement Learning (TRL), que implementa diversas técnicas de alineación, incluyendo DPO.
- [Papel de DPO](https://arxiv.org/abs/2305.18290) - Artículo original de investigación que introduce la Optimización Directa de Preferencias como una alternativa más simple al RLHF que optimiza directamente los modelos de lenguaje utilizando datos de preferencias.
- [Papel de ORPO](https://arxiv.org/abs/2403.07691) - Introduce la Optimización de Preferencias por Ratio de Probabilidades, un enfoque novedoso que combina la afinación por instrucciones y la alineación de preferencias en una sola etapa de entrenamiento.
- [Guía de RLHF de Argilla](https://argilla.io/blog/mantisnlp-rlhf-part-8/) - Una guía que explica diferentes técnicas de alineación, incluyendo RLHF, DPO y sus implementaciones prácticas.
- [Entrada en el Blog sobre DPO](https://huggingface.co/blog/dpo-trl) - Guía práctica sobre cómo implementar DPO utilizando la librería TRL con ejemplos de código y mejores prácticas.
- [Script de ejemplo de TRL sobre DPO](https://github.com/huggingface/trl/blob/main/examples/scripts/dpo.py) - Script completo de ejemplo que demuestra cómo implementar el entrenamiento DPO utilizando la librería TRL.
- [Script de ejemplo de TRL sobre ORPO](https://github.com/huggingface/trl/blob/main/examples/scripts/orpo.py) - Implementación de referencia del entrenamiento ORPO utilizando la librería TRL con opciones detalladas de configuración.
- [Manual de Alineación de Hugging Face](https://github.com/huggingface/alignment-handbook) - Guías y código para alinear modelos de lenguaje utilizando diversas técnicas, incluyendo SFT, DPO y RLHF.

A course on aligning smol models.