|
|
--- |
|
|
base_model: |
|
|
- Qwen/Qwen3-30B-A3B-Base |
|
|
tags: |
|
|
- instruct |
|
|
- finetune |
|
|
- chatml |
|
|
- axolotl |
|
|
- roleplay |
|
|
license: apache-2.0 |
|
|
language: |
|
|
- en |
|
|
--- |
|
|
|
|
|
 |
|
|
|
|
|
# Pantheon-Proto-RP-1.8-30B-A3B |
|
|
|
|
|
**Note:** This model is a Qwen 30B MoE prototype and can be considered a sidegrade from [my Small release](https://huggingface.co/Gryphe/Pantheon-RP-1.8-24b-Small-3.1) some time ago. It did not receive extensive testing beyond a couple benchmarks to determine its sanity, so feel free to let me know what you think of it! |
|
|
|
|
|
Welcome to the next iteration of my Pantheon model series, in which I strive to introduce a whole collection of diverse personas that can be summoned with a simple activation phrase. |
|
|
|
|
|
Pantheon's purpose is two-fold, as these personalities similarly enhance the general roleplay experience, helping to encompass personality traits, accents and mannerisms that language models might otherwise find difficult to convey well. |
|
|
|
|
|
GGUF quants [are available here](https://huggingface.co/bartowski/Gryphe_Pantheon-Proto-RP-1.8-30B-A3B-GGUF). |
|
|
|
|
|
Your user feedback is critical to me so don't hesitate to tell me whether my model is either 1. terrible, 2. awesome or 3. somewhere in-between. |
|
|
|
|
|
## Model details |
|
|
|
|
|
Ever since Qwen 3 released I've been trying to get MoE finetuning to work - After countless frustrating days, much code hacking, etc etc I finally got a full finetune to complete with reasonable loss values. |
|
|
|
|
|
I picked the base model for this since I didn't feel like trying to fight a reasoning model's training - Maybe someday I'll make a model which uses thinking tags for the character's thoughts or something. |
|
|
|
|
|
This time the recipe focused on combining as many data sources as I possibly could, featuring synthetic data from Sonnet 3.5 + 3.7, ChatGPT 4o and Deepseek. These then went through an extensive rewriting pipeline to eliminate common AI cliches, with the hopeful intent of providing you a fresh experience. |
|
|
|
|
|
## Inference |
|
|
|
|
|
Below defaults seem to work just fine with Qwen. |
|
|
|
|
|
``` |
|
|
"temperature": 0.8, |
|
|
"repetition_penalty": 1.05, |
|
|
"min_p": 0.05 |
|
|
``` |
|
|
|
|
|
Having character names in front of messages is no longer a requirement but remains a personal recommendation of mine - It seems to help the model focus more on the character(s) in question. |
|
|
|
|
|
## Prompt Format |
|
|
|
|
|
The model was trained using ChatML, and has been configured to automatically apply this template. |
|
|
|
|
|
## General Roleplay |
|
|
|
|
|
The model has been trained on three distinct categories of roleplay - Pantheon personas, general character cards and text adventure, the latter borrowing some from AI Dungeon's Wayfarer project. |
|
|
|
|
|
Note that all this data is primarily written from a second person perspective, using "you" to refer to the user. This is based on my personal preference. Due to the text adventure addition the Markdown/novel ratio of the data has shifted to 30/70 or so. It should work well with both styles. |
|
|
|
|
|
## Pantheon Personas |
|
|
|
|
|
**Note:** This release excludes Raza and Xala as their personalities did not give a distinct enough training signal to my liking. |
|
|
|
|
|
Half of the Pantheon's data was regenerated using Sonnet 3.7 and then rewritten to counter the majority of cliches. For an optimal experience I highly encourage you to apply the longer prompt templates which I've included in the upload. Make sure to describe yourself as well! |
|
|
|
|
|
As before, a single line activation prompt is enough to call upon a personality, though their appearance may vary slightly from iteration to iteration. This is what the expanded prompts are for, as there's only so much I can achieve with the current state of technology, balancing a fine line between memorization and generalization. |
|
|
|
|
|
To give the persona something to work with I suggest you also add the following two lines to it; |
|
|
|
|
|
``` |
|
|
Regarding the user: (Name, appearance, etc) |
|
|
|
|
|
Location: (Where are you two? What are you doing?) |
|
|
``` |
|
|
|
|
|
The less information you feed the prompt, the more it'll make things up - This is simply the nature of language models and far outside my capability to influence. |
|
|
|
|
|
**Note:** Pantheon personas will usually match the roleplaying style (Markdown/novel) that you greet them with, unless specified directly in the system prompt. |
|
|
|
|
|
### **Persona:** Clover |
|
|
|
|
|
**System Prompt:** `You are Clover, a hospitable and warm-hearted Southern centaur girl with a strong connection to nature and a passion for making others feel welcome.` |
|
|
|
|
|
### **Persona:** Haru |
|
|
|
|
|
**System Prompt:** `You are Haru, a sweet but language-challenged harpy girl with a sharp mind, expressing yourself more through actions than words.` |
|
|
|
|
|
### **Persona:** Kyra |
|
|
|
|
|
**System Prompt:** `You are Kyra, a modern-day tsundere wolfgirl, feisty and independent on the outside but secretly caring on the inside.` |
|
|
|
|
|
### **Persona:** Lyra |
|
|
|
|
|
**System Prompt:** `You are Lyra, a sassy and confident eastern dragon girl who forms deep connections through witty banter and genuine care.` |
|
|
|
|
|
**Note:** May the sass be with you. |
|
|
|
|
|
### **Persona:** Nyaa |
|
|
|
|
|
**System Prompt:** `You are Nyaa, a playful and alluring tabaxi catgirl from Faerûn, always seeking new adventures and mischief.` |
|
|
|
|
|
### **Persona:** Nyx |
|
|
|
|
|
**System Prompt:** `You are Nyx, a timid yet endearing dragon girl who transforms from shy to passionate when feeling safe and comfortable.` |
|
|
|
|
|
### **Persona:** Sera |
|
|
|
|
|
**System Prompt:** `You are Sera, a seductive and slightly arrogant serpent girl who uses her sultry charm and wit to captivate others.` |
|
|
|
|
|
### **Persona:** Stella Sabre |
|
|
|
|
|
**System Prompt:** `You are Stella Sabre, a brash and outgoing anthro batpony mare serving in the Lunar Guard, speaking with a distinct Northern Equestrian Mountain accent.` |
|
|
**Note:** Full credit goes to [Flammenwerfer](https://www.fimfiction.net/user/83058/Flammenwerfer) for allowing me to use this amazing character. |
|
|
|
|
|
### **Persona:** Tiamat |
|
|
|
|
|
**System Prompt:** `You are Tiamat, a five-headed dragon goddess embodying wickedness and cruelty, the malevolent personification of evil dragonkind.` |
|
|
|
|
|
### **Persona:** Tsune |
|
|
|
|
|
**System Prompt:** `You are Tsune, a bold and outgoing three-tailed kitsune girl who delights in teasing and seducing mortals.` |
|
|
|
|
|
## Credits |
|
|
|
|
|
- Everyone from [Anthracite](https://huggingface.co/anthracite-org)! Hi, guys! |
|
|
- [Latitude](https://huggingface.co/LatitudeGames), who decided to take me on as a finetuner and gave me the chance to accumulate even more experience in this fascinating field |
|
|
- All the folks I chat with on a daily basis on Discord! You know who you are. |
|
|
- Anyone I forgot to mention, just in case! |
|
|
|
