Upload 8 files

README.md

@@ -31,18 +31,30 @@ Original model: https://huggingface.co/google/gemma-2-2b-jpn-it
 
 Note that this model does not support a System prompt.
 
-This is abliterated model of [`google/gemma-2-2b-jpn-it](https://huggingface.co/google/gemma-2-2b-jpn-it) using the 
+This is abliterated model of [google/gemma-2-2b-jpn-it](https://huggingface.co/google/gemma-2-2b-jpn-it) using the 
 [method](https://medium.com/@mlabonne/uncensor-any-llm-with-abliteration-d30148b7d43e) 
 described by mlabonne.
 
-Layer 18 of the original model was chosen for abliteration.
-I also created another layer 17 abliterated model for comparison.
+Layer 17 of the original model was chosen for abliteration.
+I also created another layer 18 abliterated model for comparison.
 
 It is uploaded here to be evaluated by the LLM Leaderboard to see how brain damaged it
 is compared to the original model.
 
 ORPO fine tuning is currently underway to see if it can regain its sanity. You can play with this model first or wait until I am done with the fine tuning.
 
+## Benchmark (100.0*raw scores only)
+
+Click on the model name go to the raw score json generated by Open LLM Leaderboard.
+
+| Model | Average | IFEval | BHH | Math Lv5 | MUSR | MMLU-PRO |
+| ----- | ------- | ------ | ----|--------- | ---- | -------- |
+| [gemma-2-2b-jpn-it](https://huggingface.co/datasets/open-llm-leaderboard/results/blob/main/google/gemma-2-2b-jpn-it/results_2024-10-15T15-21-39.173019.json) | 30.82 | 54.11 | 41.43 | 0.0 | 27.52 | 37.17 | 24.67 |
+| [gemma-2-2b-jpn-it-abliterated-17](https://huggingface.co/datasets/open-llm-leaderboard/results/raw/main/ymcki/gemma-2-2b-jpn-it-abliterated-17/results_2024-10-16T07-58-03.781979.json) 16.74 | 0.0 | 29.13 | 0.0 | 25.92 | 33.73 | 11.68 |
+| gemma-2-2b-jpn-it-abliterated-17 | TBD | TBD | TBD | TBD | TBD | TBD |
+
+Indeed, it is quite dumbed down relative to the original.
+
 ## How to run this model
 
 ```py
@@ -50,7 +62,7 @@ from transformers import AutoTokenizer, AutoModelForCausalLM
 import transformers
 import torch
 
-model_id = "gemma-2-2b-jpn-it-abliterated-18"
+model_id = "gemma-2-2b-jpn-it-abliterated-17"
 dtype = torch.bfloat16
 
 tokenizer = AutoTokenizer.from_pretrained(model_id)
@@ -76,7 +88,7 @@ pip install -U "huggingface_hub[cli]"
 Then, you can target the specific file you want:
 
 ```
-huggingface-cli download ymcki/gemma-2-2b-jpn-it-abliterated-18 --include "*" --local-dir ./
+huggingface-cli download ymcki/gemma-2-2b-jpn-it-abliterated-17 --include "*" --local-dir ./
 ```
 
 ## Credits