library_name: transformers
datasets:
- pauhidalgoo/patufet-conversa
language:
- ca
tags:
- catalan
- language-model
- transformer
- sft
model-index:
- name: cucafera-instruct
results:
- task:
type: language-understanding
name: arc_ca_challenge
dataset:
name: arc_ca_challenge
type: catalan_bench
metrics:
- name: Accuracy
type: acc
value: 0.2295
- name: Normalized Accuracy
type: acc_norm
value: 0.2534
source:
name: Eleuther AI LM Evaluation Harness
url: https://github.com/EleutherAI/lm-evaluation-harness
- task:
type: language-understanding
name: arc_ca_easy
dataset:
name: arc_ca_easy
type: catalan_bench
metrics:
- name: Accuracy
type: acc
value: 0.4238
- name: Normalized Accuracy
type: acc_norm
value: 0.4108
source:
name: Eleuther AI LM Evaluation Harness
url: https://github.com/EleutherAI/lm-evaluation-harness
- task:
type: question-answering
name: catalanqa
dataset:
name: catalanqa
type: catalan_bench
metrics:
- name: Exact Match
type: exact_match
value: 0.0037
- name: F1 Score
type: f1
value: 0.0991
source:
name: Eleuther AI LM Evaluation Harness
url: https://github.com/EleutherAI/lm-evaluation-harness
- task:
type: language-understanding
name: copa_ca
dataset:
name: copa_ca
type: catalan_bench
metrics:
- name: Accuracy
type: acc
value: 0.614
source:
name: Eleuther AI LM Evaluation Harness
url: https://github.com/EleutherAI/lm-evaluation-harness
- task:
type: machine-translation
name: flores_ca
dataset:
name: flores_ca
type: flores
metrics:
- name: BLEU
type: bleu
value: 0.5934
source:
name: Eleuther AI LM Evaluation Harness
url: https://github.com/EleutherAI/lm-evaluation-harness
license: apache-2.0
base_model:
- pauhidalgoo/cucafera
- pauhidalgoo/cucafera-instruct
Model Card for cucafera 馃敟馃惒 (Instruct Model)
This document describes cucafera (Chat Model), a Catalan Large Language Model (LLM) fine-tuned to follow multi-turn instructions and generate text in Catalan. Built upon the instruct model, it uses a multi-turn dataset to enhance it's conversational capabilities.
Model Details
Model Description
cucafera (Chat Model) is a 244-million parameter transformer-based language model inspired by the LLAMA architecture (notably LLAMA3). Despite its relatively small size compared to many contemporary models, it is optimized for generating coherent and contextually relevant text in Catalan.
- Model Size: 244M parameters
- Architecture: Transformer-based (LLAMA-inspired) with 30 layers
- Embedding Size: 768
- Attention Mechanism: 4 key/value heads and 8 query heads (using Grouped Query Attention - GQA)
- Context Length: 2048 tokens
- Tokenizer: Byte-Pair Encoding (BPE) with a vocabulary size of 65,536
- Activation Function: GeGLU
Chat Fine-Tuning
The chat version of cucafera has been fine-tuned on top of the instruct version of cucafera. It follows the ChatML format for conversation, for example:
<|im_start|>user Fes un poema <|im_end|> <|im_start|>assistant
Training Data
The base model was pre-trained using the patufet-pretrain dataset.
The fine-tuning data utilized a mix of instruction datasets from the patufet collection.
The chat data consists in the patufet-conversa dataset.
Fine-tunning Procedure
The model was fine-tuned with the following setup:
- Total fine-tunning steps: 8400
- Per device train batch size: 1
- Sequence Length: 2048
- Learning rate: 3e-5
- Optimizer: AdamW
- Weight decay: 0.01
- Epochs: 3
Different commits represent different fine-tunning procedures: we experimented with different data mixes, epochs, datasets...
Direct Use
The cucafera (Chat Model) is designed for:
- Multi-turn Conversational agents and chatbots in Catalan.
- Task-specific applications such as summarization, translation (within Catalan), and creative writing.
- Educational and experimental research into instruction-following LLMs.
- Creative content generation, like poems or stories
However, due to its limited size, it is not able to provide correct factual information and you must be aware of this fact when using this model.
Out-of-Scope Uses
- High-Stakes Applications:
The model is not recommended for uses where extremely high factual accuracy is required or where outputs could have significant real-world consequences. - Non-Catalan Tasks:
Since the model is exclusively trained on Catalan text, it is not suited for tasks in other languages without further training or fine-tuning. - Sensitive or safety-critical uses: It has not undergone RLHF/DPO tuning, so outputs should be reviewed carefully.
Bias, Risks, and Limitations
- The model has no instruction tuning, so it may not follow prompts effectively.
- It only understands Catalan, meaning it is unsuitable for multilingual applications.
- Due to its small size (244M parameters), its knowledge and reasoning capabilities are limited.
- It was trained on a limited dataset, which may introduce biases in its outputs.
Recommendations
- The goal of this model is educational. You are encouraged to train your own model.
- If used in production, human review of its outputs is recommended.
- Fine-tuning on task-specific data can improve accuracy and mitigate biases.
- Users should be cautious when using it in sensitive or high-stakes applications.
Use the Chat Model
You can use the chat model via huggingface's transformers library. Make sure to specify the ChatML format.
<|im_start|>user
Qu猫 茅s la intel路lig猫ncia artificial? <|im_end|>
<|im_start|>assistant', 'content': "Ets un assistent d'intel路lig猫ncia artificial que pot ajudar els usuaris amb problemes matem脿tics, especialment amb equacions."}, {'role': 'user', 'content': "Hola! M'agradaria aprendre m茅s sobre les equacions algebraiques. Pots explicar-me com funcionen?"}, {'role': 'assistant', 'content': "Hola! Les equacions algebraiques s贸n una forma de resoldre problemes geom猫trics complexos, on cada element t茅 un valor definit. Per exemple, si tenim l'equaci贸: (x + 1) / 2 = 10, el resultat ser脿 5 i el seu valor
Acknowledgements
This model was developed as an experimental project, inspired by Karpathy's NanoGPT Series. My colleague Roger Baiges also trained his own CatGPT.
For more details, updates, or to contribute to the project, please visit the GitHub repository