File size: 1,384 Bytes
7c49df1 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
# medmekk/Qwen2.5-0.5B-Instruct.GGUF
GGUF quantized versions of [Qwen/Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct)
## Available Formats:
- `Q2_K`: Qwen2.5-0.5B-Instruct-Q2_K.gguf
- `Q3_K_S`: Qwen2.5-0.5B-Instruct-Q3_K_S.gguf
- `Q3_K_M`: Qwen2.5-0.5B-Instruct-Q3_K_M.gguf
- `Q3_K_L`: Qwen2.5-0.5B-Instruct-Q3_K_L.gguf
- `Q4_0`: Qwen2.5-0.5B-Instruct-Q4_0.gguf
- `Q4_K_S`: Qwen2.5-0.5B-Instruct-Q4_K_S.gguf
- `Q4_K_M`: Qwen2.5-0.5B-Instruct-Q4_K_M.gguf
- `Q5_0`: Qwen2.5-0.5B-Instruct-Q5_0.gguf
- `Q5_K_S`: Qwen2.5-0.5B-Instruct-Q5_K_S.gguf
- `Q5_K_M`: Qwen2.5-0.5B-Instruct-Q5_K_M.gguf
- `Q6_K`: Qwen2.5-0.5B-Instruct-Q6_K.gguf
- `Q8_0`: Qwen2.5-0.5B-Instruct-Q8_0.gguf
- `IQ3_M_IMAT`: Qwen2.5-0.5B-Instruct-IQ3_M_imat.gguf
- `IQ3_XXS_IMAT`: Qwen2.5-0.5B-Instruct-IQ3_XXS_imat.gguf
- `Q4_K_M_IMAT`: Qwen2.5-0.5B-Instruct-Q4_K_M_imat.gguf
- `Q4_K_S_IMAT`: Qwen2.5-0.5B-Instruct-Q4_K_S_imat.gguf
- `IQ4_NL_IMAT`: Qwen2.5-0.5B-Instruct-IQ4_NL_imat.gguf
- `IQ4_XS_IMAT`: Qwen2.5-0.5B-Instruct-IQ4_XS_imat.gguf
- `Q5_K_M_IMAT`: Qwen2.5-0.5B-Instruct-Q5_K_M_imat.gguf
- `Q5_K_S_IMAT`: Qwen2.5-0.5B-Instruct-Q5_K_S_imat.gguf
## Usage with llama.cpp:
```bash
# CLI:
llama-cli --hf-repo medmekk/Qwen2.5-0.5B-Instruct.GGUF --hf-file MODEL_FILE -p "Your prompt"
# Server:
llama-server --hf-repo medmekk/Qwen2.5-0.5B-Instruct.GGUF --hf-file MODEL_FILE -c 2048
```
|