File size: 2,084 Bytes
cad6e22 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 |
---
datasets:
- hivaze/emphatical_daily_dialogues
---
# Model Card for Model ID
This model is a adapter for databricks/dolly-v2-3b, finetuned on hivaze/emphatical_daily_dialogues.
Main goal of this model is to train model to create emphatical dialogues, which are controlled by instructions.
## Model Details
### Model Description
Prompt template: `"{intro}\n\n### Instruction:\n{instruction}\n\n### Response:\n{response}\n"`\
Example intro: "You are a kind and empathetic interlocutor. You are talking to a person. Below is an instruction that describes a task. Write a response that appropriately completes the request" \
Example instruction: "You try to chit-chat. Complete a phrase, acting like an interlocutor."
Training params:
```
train_args = TrainingArguments(
per_device_train_batch_size=8, # can be 4 with llama
per_device_eval_batch_size=8, # can be 4 with llama
gradient_accumulation_steps=4,
warmup_steps=20,
# max_steps=200,
optim="adamw_torch",
learning_rate=4e-5, # many possible values here from 1e-5 to 2e-4
# save_strategy="steps",
fp16=True,
# bf16=True, # a100 required
num_train_epochs=2,
evaluation_strategy="steps",
eval_steps=50,
save_strategy="steps",
save_steps=400,
logging_strategy="steps",
logging_steps=10,
logging_dir=f"{local_output_dir}/runs",
report_to="tensorboard",
output_dir=local_output_dir
)
```
LoRA config:
```
config = LoraConfig(
r=16, # can be 8 with llama
lora_alpha=32, # can be 16 with llama
# target_modules=["q_proj", "v_proj"],
target_modules=['query_key_value'],
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM"
)
```
- **Developed by:** hivaze
- **Model type:** LoRA adapter for GPTNeoXForCausalLM
- **Language(s) (NLP):** Primarly english
- **Finetuned from model [optional]:** databricks/dolly-v2-3b
- **Git repository**: https://github.com/hivaze/friendly_chatbot_task
### Tensorboard
|