|
|
--- |
|
|
datasets: |
|
|
- hivaze/emphatical_daily_dialogues |
|
|
--- |
|
|
# Model Card for Model ID |
|
|
|
|
|
This model is a adapter for databricks/dolly-v2-3b, finetuned on hivaze/emphatical_daily_dialogues. |
|
|
Main goal of this model is to train model to create emphatical dialogues, which are controlled by instructions. |
|
|
|
|
|
## Model Details |
|
|
|
|
|
### Model Description |
|
|
|
|
|
Prompt template: `"{intro}\n\n### Instruction:\n{instruction}\n\n### Response:\n{response}\n"`\ |
|
|
Example intro: "You are a kind and empathetic interlocutor. You are talking to a person. Below is an instruction that describes a task. Write a response that appropriately completes the request" \ |
|
|
Example instruction: "You try to chit-chat. Complete a phrase, acting like an interlocutor." |
|
|
|
|
|
Training params: |
|
|
``` |
|
|
train_args = TrainingArguments( |
|
|
per_device_train_batch_size=8, # can be 4 with llama |
|
|
per_device_eval_batch_size=8, # can be 4 with llama |
|
|
gradient_accumulation_steps=4, |
|
|
warmup_steps=20, |
|
|
# max_steps=200, |
|
|
optim="adamw_torch", |
|
|
learning_rate=4e-5, # many possible values here from 1e-5 to 2e-4 |
|
|
# save_strategy="steps", |
|
|
fp16=True, |
|
|
# bf16=True, # a100 required |
|
|
num_train_epochs=2, |
|
|
evaluation_strategy="steps", |
|
|
eval_steps=50, |
|
|
save_strategy="steps", |
|
|
save_steps=400, |
|
|
logging_strategy="steps", |
|
|
logging_steps=10, |
|
|
logging_dir=f"{local_output_dir}/runs", |
|
|
report_to="tensorboard", |
|
|
output_dir=local_output_dir |
|
|
) |
|
|
``` |
|
|
|
|
|
LoRA config: |
|
|
``` |
|
|
config = LoraConfig( |
|
|
r=16, # can be 8 with llama |
|
|
lora_alpha=32, # can be 16 with llama |
|
|
# target_modules=["q_proj", "v_proj"], |
|
|
target_modules=['query_key_value'], |
|
|
lora_dropout=0.05, |
|
|
bias="none", |
|
|
task_type="CAUSAL_LM" |
|
|
) |
|
|
``` |
|
|
|
|
|
- **Developed by:** hivaze |
|
|
- **Model type:** LoRA adapter for GPTNeoXForCausalLM |
|
|
- **Language(s) (NLP):** Primarly english |
|
|
- **Finetuned from model [optional]:** databricks/dolly-v2-3b |
|
|
- **Git repository**: https://github.com/hivaze/friendly_chatbot_task |
|
|
|
|
|
### Tensorboard |
|
|
|
|
|
 |
|
|
|
|
|
|
|
|
|
