Update README.md

README.md

@@ -1,10 +1,35 @@
 ---
 license: apache-2.0
 pipeline_tag: text-generation
-library_name: transformers
+library_name: exllamav2
+base_model:
+- ibm-granite/granite-guardian-3.2-5b
 ---
 # Granite Guardian 3.2 5B
 
+## Quants
+[4bpw h6 (main)](https://huggingface.co/cgus/granite-guardian-3.2-5b-exl2/tree/main)  
+[4.5bpw h6](https://huggingface.co/cgus/granite-guardian-3.2-5b-exl2/tree/4.5bpw-h6)  
+[5bpw h6](https://huggingface.co/cgus/granite-guardian-3.2-5b-exl2/tree/5bpw-h6)  
+[6bpw h6](https://huggingface.co/cgus/granite-guardian-3.2-5b-exl2/tree/6bpw-h6)  
+[8bpw h8](https://huggingface.co/cgus/granite-guardian-3.2-5b-exl2/tree/8bpw-h8)  
+
+## Quantization notes
+Made with Exllamav2 0.2.8 with the default dataset. Granite3 models require Exllamav2 0.2.7 or newer.  
+Exl2 models don't support native RAM offloading, so the model has to fully fit into GPU VRAM.  
+It's also required to use Nvidia RTX on Windows or Nvidia RTX/AMD ROCm on Linux.  
+
+Just in case if you downloaded the model and it answers only Yes/No, it's [intended behavior](https://github.com/ibm-granite/granite-guardian/tree/main#scope-of-use).  
+It's hardcoded in the model's Jinja2 template that can be viewed in tokenizer_config.json file.  
+By default in chat mode it evaluates if user's or assistant's message is harmful in general sense according to the model's risk definitions.  
+But it allows to choose a different predefined option, to set custom harm definitions or detect risks in RAG or function calling pipelines.  
+If you're using TabbyAPI you can either set risk_name or risk_definition via [template variables](https://github.com/theroyallab/tabbyAPI/wiki/04.-Chat-Completions#template-variables).  
+For example, you can switch to violence detection by adding: ``"template_vars": {"guardian_config": {"risk_name": "violence"}}`` to v1/chat/completions request.  
+For more information refer to Granite Guardian [documentation](https://github.com/ibm-granite/granite-guardian) and its Jinja2 template.
+
+# Original model card
+# Granite Guardian 3.2 5B
+
 ## Model Summary
 
 **Granite Guardian 3.2 5B** is a thinned down version of Granite Guardian 3.1 8B designed to detect risks in prompts and responses.
@@ -251,4 +276,4 @@ The model performance is evaluated on sample conversations taken from the [DICES
       primaryClass={cs.CL},
       url={https://arxiv.org/abs/2412.07724}, 
 }
-```
+```