ZeroAgency/zero-summary-v2-beta3-lora-e1

See axolotl config

axolotl version: 0.9.0

# zero-summary-v2-beta3

adapter: lora
base_model: ZeroAgency/Zero-Mistral-24B
dataset_processes: 64
chat_template: jinja
chat_template_jinja: "{%- set today = strftime_now(\"%Y-%m-%d\") %}\n{%- set default_system_message = \"You are Mistral Small 3, a Large Language Model (LLM) created by Mistral AI, a French startup headquartered in Paris.\\nYour knowledge base was last updated on 2023-10-01. The current date is \" + today + \".\\n\\nWhen you're not sure about some information, you say that you don't have the information and don't make up anything.\\nIf the user's question is not clear, ambiguous, or does not provide enough context for you to accurately answer the question, you do not try to answer it right away and you rather ask the user to clarify their request (e.g. \\\"What are some good restaurants around me?\\\" => \\\"Where are you?\\\" or \\\"When is the next flight to Tokyo\\\" => \\\"Where do you travel from?\\\")\" %}\n\n{{- bos_token }}\n\n{%- if messages[0]['role'] == 'system' %}\n    {%- if messages[0]['content'] is string %}\n        {%- set system_message = messages[0]['content'] %}\n    {%- else %}\n        {%- set system_message = messages[0]['content'][0]['text'] %}\n    {%- endif %}\n    {%- set loop_messages = messages[1:] %}\n{%- else %}\n    {%- set system_message = default_system_message %}\n    {%- set loop_messages = messages %}\n{%- endif %}\n{{- '[SYSTEM_PROMPT]' + system_message + '[/SYSTEM_PROMPT]' }}\n\n{%- for message in loop_messages %}\n    {%- if message['role'] == 'user' %}\n        {%- if message['content'] is string %}\n            {{- '[INST]' + message['content'] + '[/INST]' }}\n        {%- else %}\n            {{- '[INST]' }}\n            {%- for block in message['content'] %}\n                {%- if block['type'] == 'text' %}\n                    {{- block['text'] }}\n                {%- elif block['type'] in ['image', 'image_url'] %}\n                    {{- '[IMG]' }}\n                {%- else %}\n                    {{- raise_exception('Only text and image blocks are supported in message content!') }}\n                {%- endif %}\n            {%- endfor %}\n            {{- '[/INST]' }}\n        {%- endif %}\n    {%- elif message['role'] == 'system' %}\n        {%- if message['content'] is string %}\n            {{- '[SYSTEM_PROMPT]' + message['content'] + '[/SYSTEM_PROMPT]' }}\n        {%- else %}\n            {{- '[SYSTEM_PROMPT]' + message['content'][0]['text'] + '[/SYSTEM_PROMPT]' }}\n        {%- endif %}\n    {%- elif message['role'] == 'assistant' %}\n        {%- if message['content'] is string %}\n            {{- message['content'] + eos_token }}\n        {%- else %}\n            {{- message['content'][0]['text'] + eos_token }}\n        {%- endif %}\n    {%- else %}\n        {{- raise_exception('Only user, system and assistant roles are supported!') }}\n    {%- endif %}\n{%- endfor %}"


dataset_prepared_path: ./last_run_prepared

datasets:
- message_property_mappings:
    content: content
    role: role
  path: bethrezen/thinking-summary-v2
  trust_remote_code: false
  field_messages: conversation
  type: chat_template


# approx 20k samples should be enough
#val_set_size: 0.061

# exact duplicates are already cleaned
#dataset_exact_deduplication: true

gradient_accumulation_steps: 2
gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false

#learning_rate: 0.0001
learning_rate: 1e-5
lisa_layers_attribute: model.layers
#is_mistral_derived_model: true

plugins:
  - axolotl.integrations.liger.LigerPlugin
liger_rope: true
liger_rms_norm: true
liger_swiglu: true
liger_fused_linear_cross_entropy: true

load_best_model_at_end: true
load_in_4bit: true
load_in_8bit: false
lora_alpha: 96
lora_dropout: 0.1
lora_target_linear: true
lora_r: 96

lr_scheduler: cosine
#max_prompt_len: 8192
mean_resizing_embeddings: false
micro_batch_size: 1
num_epochs: 2
optimizer: adamw_torch_fused
output_dir: ./outputs/zero-summary-v1-beta3


sample_packing_bin_size: 200
sample_packing_group_size: 100000
save_only_model: false
save_safetensors: true
sequence_len: 110000
min_sample_len: 1
#shuffle_merged_datasets: true
skip_prepare_dataset: false
strict: false
train_on_inputs: false


weight_decay: 0.01
wandb_project: zero-summary
wandb_name: zero-summary-v1-beta3
bf16: true
fp16: false
tf32: false
flash_attention: true

save_strategy: epoch
eval_strategry: epoch

logging_steps: 1
save_total_limit: 5
warmup_steps: 0
sample_packing: true
pad_to_sequence_len: true
group_by_length: true
seed: 42
data_seed: 42

deepspeed: zero1.json
log_with: wandb
trust_remote_code: true
use_fast_tokenizer: true
special_tokens:
  pad_token: "<pad>"

outputs/zero-summary-v1-beta3

This model is a fine-tuned version of ZeroAgency/Zero-Mistral-24B on the bethrezen/thinking-summary-v2 dataset.

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• distributed_type: multi-GPU
• num_devices: 8
• gradient_accumulation_steps: 2
• total_train_batch_size: 16
• total_eval_batch_size: 8
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• num_epochs: 2.0

Training results

Framework versions

• PEFT 0.15.2
• Transformers 4.51.3
• Pytorch 2.6.0+cu124
• Datasets 3.5.0
• Tokenizers 0.21.1