Hugging Face's logo

jukofyork
/
DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF

GGUF
draft
speculative-decoding
conversational
Model card Files Community
1
jukofyork commited on
Commit
65f1c60
·
verified ·
1 Parent(s): 2c4f521

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +58 -0
README.md ADDED
@@ -0,0 +1,58 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ base_model:
4
+ - Qwen/Qwen2.5-0.5B-Instruct
5
+ datasets:
6
+ - agentlans/common-crawl-sample
7
+ - bigcode/the-stack-smol-xl
8
+ - open-thoughts/OpenThoughts-Unverified-173k
9
+ - cognitivecomputations/dolphin-r1
10
+ tags:
11
+ - draft
12
+ - speculative-decoding
13
+ ---
14
+
15
+ A `0.5B` parameter draft (speculative decoding) model for use with [deepseek-ai/DeepSeek-V3-0324](https://huggingface.co/deepseek-ai/DeepSeek-V3-0324).
16
+
17
+ See [jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0) for the non-GGUF version, and a detailed explanation of how the model was created.
18
+
19
+ ---
20
+
21
+ # Without `imatrix`
22
+
23
+ Link | Type
24
+ -----| ----
25
+ [DeepSeek-V3-0324-DRAFT-0.5B-BF16.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-BF16.gguf) | BF16
26
+ [DeepSeek-V3-0324-DRAFT-0.5B-F16.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-F16.gguf)| F16
27
+ [DeepSeek-V3-0324-DRAFT-0.5B-Q8_0.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-Q8_0.gguf)| Q8_0
28
+ [DeepSeek-V3-0324-DRAFT-0.5B-Q6_K.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-Q6_K.gguf)| Q6_K
29
+ [DeepSeek-V3-0324-DRAFT-0.5B-Q5_K_M.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-Q5_K_M.gguf)| Q5_K_M
30
+ [DeepSeek-V3-0324-DRAFT-0.5B-Q5_K_S.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-Q5_K_S.gguf)| Q5_K_S
31
+ [DeepSeek-V3-0324-DRAFT-0.5B-Q4_K_M.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-Q4_K_M.gguf)| Q4_K_M
32
+ [DeepSeek-V3-0324-DRAFT-0.5B-Q4_K_S.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-Q4_K_S.gguf)| Q4_K_S
33
+ [DeepSeek-V3-0324-DRAFT-0.5B-IQ4_NL.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-IQ4_NL.gguf)| IQ4_NL
34
+ [DeepSeek-V3-0324-DRAFT-0.5B-IQ4_XS.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-IQ4_XS.gguf)| IQ4_XS
35
+ [DeepSeek-V3-0324-DRAFT-0.5B-Q5_1.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-Q5_1.gguf)| Q5_1
36
+ [DeepSeek-V3-0324-DRAFT-0.5B-Q5_0.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-Q5_0.gguf)| Q5_0
37
+ [DeepSeek-V3-0324-DRAFT-0.5B-Q4_1.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-Q4_1.gguf)| Q4_1
38
+ [DeepSeek-V3-0324-DRAFT-0.5B-Q4_0.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-Q4_0.gguf)| Q4_0
39
+
40
+ # With `imatrix`
41
+
42
+ Link | Type
43
+ -----| ----
44
+ [DeepSeek-V3-0324-DRAFT-0.5B-iQ6_K.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iQ6_K.gguf)| Q6_K
45
+ [DeepSeek-V3-0324-DRAFT-0.5B-iQ5_K_M.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iQ5_K_M.gguf)| Q5_K_M
46
+ [DeepSeek-V3-0324-DRAFT-0.5B-iQ5_K_S.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iQ5_K_S.gguf)| Q5_K_S
47
+ [DeepSeek-V3-0324-DRAFT-0.5B-iQ4_K_M.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iQ4_K_M.gguf)| Q4_K_M
48
+ [DeepSeek-V3-0324-DRAFT-0.5B-iQ4_K_S.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iQ4_K_S.gguf)| Q4_K_S
49
+ [DeepSeek-V3-0324-DRAFT-0.5B-iIQ4_NL.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iIQ4_NL.gguf)| IQ4_NL
50
+ [DeepSeek-V3-0324-DRAFT-0.5B-iIQ4_XS.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iIQ4_XS.gguf)| IQ4_XS
51
+ [DeepSeek-V3-0324-DRAFT-0.5B-iQ5_1.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iQ5_1.gguf)| Q5_1
52
+ [DeepSeek-V3-0324-DRAFT-0.5B-iQ5_0.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iQ5_0.gguf)| Q5_0
53
+ [DeepSeek-V3-0324-DRAFT-0.5B-iQ4_1.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iQ4_1.gguf)| Q4_1
54
+ [DeepSeek-V3-0324-DRAFT-0.5B-iQ4_0.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iQ4_0.gguf)| Q4_0
55
+
56
+ ---
57
+
58
+ I have included the [imatrix file](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/DeepSeek-V3-0324-DRAFT-0.5B-BF16.imatrix) used to generate the `Q4_0`-`Q6_K` quants, along with the [1MB sample of the fine-tuning data](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/DeepSeek-V3-0324-DRAFT-imatrix-data.txt) used to create it.