File size: 6,127 Bytes
bd96a25 65f1c60 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 |
---
license: apache-2.0
base_model:
- Qwen/Qwen2.5-0.5B-Instruct
datasets:
- agentlans/common-crawl-sample
- bigcode/the-stack-smol-xl
- open-thoughts/OpenThoughts-Unverified-173k
- cognitivecomputations/dolphin-r1
tags:
- draft
- speculative-decoding
language:
- zho
- eng
- fra
- spa
- por
- deu
- ita
- rus
- jpn
- kor
- vie
- tha
- ara
---
A `0.5B` parameter draft (speculative decoding) model for use with [deepseek-ai/DeepSeek-V3-0324](https://huggingface.co/deepseek-ai/DeepSeek-V3-0324).
See [jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0) for the non-GGUF version, and a detailed explanation of how the model was created.
---
# Without `imatrix`
Link | Type
-----| ----
[DeepSeek-V3-0324-DRAFT-0.5B-BF16.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-BF16.gguf) | BF16
[DeepSeek-V3-0324-DRAFT-0.5B-F16.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-F16.gguf)| F16
[DeepSeek-V3-0324-DRAFT-0.5B-Q8_0.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-Q8_0.gguf)| Q8_0
[DeepSeek-V3-0324-DRAFT-0.5B-Q6_K.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-Q6_K.gguf)| Q6_K
[DeepSeek-V3-0324-DRAFT-0.5B-Q5_K_M.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-Q5_K_M.gguf)| Q5_K_M
[DeepSeek-V3-0324-DRAFT-0.5B-Q5_K_S.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-Q5_K_S.gguf)| Q5_K_S
[DeepSeek-V3-0324-DRAFT-0.5B-Q4_K_M.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-Q4_K_M.gguf)| Q4_K_M
[DeepSeek-V3-0324-DRAFT-0.5B-Q4_K_S.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-Q4_K_S.gguf)| Q4_K_S
[DeepSeek-V3-0324-DRAFT-0.5B-IQ4_NL.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-IQ4_NL.gguf)| IQ4_NL
[DeepSeek-V3-0324-DRAFT-0.5B-IQ4_XS.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-IQ4_XS.gguf)| IQ4_XS
[DeepSeek-V3-0324-DRAFT-0.5B-Q5_1.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-Q5_1.gguf)| Q5_1
[DeepSeek-V3-0324-DRAFT-0.5B-Q5_0.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-Q5_0.gguf)| Q5_0
[DeepSeek-V3-0324-DRAFT-0.5B-Q4_1.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-Q4_1.gguf)| Q4_1
[DeepSeek-V3-0324-DRAFT-0.5B-Q4_0.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-Q4_0.gguf)| Q4_0
# With `imatrix`
Link | Type
-----| ----
[DeepSeek-V3-0324-DRAFT-0.5B-iQ6_K.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iQ6_K.gguf)| Q6_K
[DeepSeek-V3-0324-DRAFT-0.5B-iQ5_K_M.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iQ5_K_M.gguf)| Q5_K_M
[DeepSeek-V3-0324-DRAFT-0.5B-iQ5_K_S.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iQ5_K_S.gguf)| Q5_K_S
[DeepSeek-V3-0324-DRAFT-0.5B-iQ4_K_M.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iQ4_K_M.gguf)| Q4_K_M
[DeepSeek-V3-0324-DRAFT-0.5B-iQ4_K_S.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iQ4_K_S.gguf)| Q4_K_S
[DeepSeek-V3-0324-DRAFT-0.5B-iIQ4_NL.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iIQ4_NL.gguf)| IQ4_NL
[DeepSeek-V3-0324-DRAFT-0.5B-iIQ4_XS.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iIQ4_XS.gguf)| IQ4_XS
[DeepSeek-V3-0324-DRAFT-0.5B-iQ5_1.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iQ5_1.gguf)| Q5_1
[DeepSeek-V3-0324-DRAFT-0.5B-iQ5_0.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iQ5_0.gguf)| Q5_0
[DeepSeek-V3-0324-DRAFT-0.5B-iQ4_1.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iQ4_1.gguf)| Q4_1
[DeepSeek-V3-0324-DRAFT-0.5B-iQ4_0.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iQ4_0.gguf)| Q4_0
---
See [DeepSeek-R1-DRAFT-0.5B-v1.0-GGUF](https://huggingface.co/jukofyork/DeepSeek-R1-DRAFT-0.5B-v1.0-GGUF#without-imatrix) for detailed PPL statistics and recommendations on which quant to use, etc.
I have included the [imatrix file](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/DeepSeek-V3-0324-DRAFT-0.5B-BF16.imatrix) used to generate the `Q4_0`-`Q6_K` quants, along with the [1MB sample of the fine-tuning data](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/DeepSeek-V3-0324-DRAFT-imatrix-data.txt) used to create it. |