Hugging Face's logo

jukofyork
/
DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF

GGUF
draft
speculative-decoding
conversational
Model card Files Community
1
jukofyork lbourdois commited on
Commit
bd96a25
·
verified ·
1 Parent(s): 821b93c

Improve language tag (#1)

Browse files

- Improve language tag (930ead41174f33c368f7e32b1f81d30d99e8e4df)


Co-authored-by: Loïck BOURDOIS <[email protected]>

Files changed (1) hide show
  1. README.md +75 -61
README.md CHANGED
@@ -1,62 +1,76 @@
1
- ---
2
- license: apache-2.0
3
- base_model:
4
- - Qwen/Qwen2.5-0.5B-Instruct
5
- datasets:
6
- - agentlans/common-crawl-sample
7
- - bigcode/the-stack-smol-xl
8
- - open-thoughts/OpenThoughts-Unverified-173k
9
- - cognitivecomputations/dolphin-r1
10
- tags:
11
- - draft
12
- - speculative-decoding
13
- ---
14
-
15
- ![image-3.webp](https://cdn-uploads.huggingface.co/production/uploads/65995c45539c808e84c38bf1/pqAVNCYd1BV2ljTFwO9Ab.webp)
16
-
17
- A `0.5B` parameter draft (speculative decoding) model for use with [deepseek-ai/DeepSeek-V3-0324](https://huggingface.co/deepseek-ai/DeepSeek-V3-0324).
18
-
19
- See [jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0) for the non-GGUF version, and a detailed explanation of how the model was created.
20
-
21
- ---
22
-
23
- # Without `imatrix`
24
-
25
- Link | Type
26
- -----| ----
27
- [DeepSeek-V3-0324-DRAFT-0.5B-BF16.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-BF16.gguf) | BF16
28
- [DeepSeek-V3-0324-DRAFT-0.5B-F16.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-F16.gguf)| F16
29
- [DeepSeek-V3-0324-DRAFT-0.5B-Q8_0.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-Q8_0.gguf)| Q8_0
30
- [DeepSeek-V3-0324-DRAFT-0.5B-Q6_K.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-Q6_K.gguf)| Q6_K
31
- [DeepSeek-V3-0324-DRAFT-0.5B-Q5_K_M.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-Q5_K_M.gguf)| Q5_K_M
32
- [DeepSeek-V3-0324-DRAFT-0.5B-Q5_K_S.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-Q5_K_S.gguf)| Q5_K_S
33
- [DeepSeek-V3-0324-DRAFT-0.5B-Q4_K_M.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-Q4_K_M.gguf)| Q4_K_M
34
- [DeepSeek-V3-0324-DRAFT-0.5B-Q4_K_S.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-Q4_K_S.gguf)| Q4_K_S
35
- [DeepSeek-V3-0324-DRAFT-0.5B-IQ4_NL.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-IQ4_NL.gguf)| IQ4_NL
36
- [DeepSeek-V3-0324-DRAFT-0.5B-IQ4_XS.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-IQ4_XS.gguf)| IQ4_XS
37
- [DeepSeek-V3-0324-DRAFT-0.5B-Q5_1.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-Q5_1.gguf)| Q5_1
38
- [DeepSeek-V3-0324-DRAFT-0.5B-Q5_0.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-Q5_0.gguf)| Q5_0
39
- [DeepSeek-V3-0324-DRAFT-0.5B-Q4_1.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-Q4_1.gguf)| Q4_1
40
- [DeepSeek-V3-0324-DRAFT-0.5B-Q4_0.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-Q4_0.gguf)| Q4_0
41
-
42
- # With `imatrix`
43
-
44
- Link | Type
45
- -----| ----
46
- [DeepSeek-V3-0324-DRAFT-0.5B-iQ6_K.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iQ6_K.gguf)| Q6_K
47
- [DeepSeek-V3-0324-DRAFT-0.5B-iQ5_K_M.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iQ5_K_M.gguf)| Q5_K_M
48
- [DeepSeek-V3-0324-DRAFT-0.5B-iQ5_K_S.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iQ5_K_S.gguf)| Q5_K_S
49
- [DeepSeek-V3-0324-DRAFT-0.5B-iQ4_K_M.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iQ4_K_M.gguf)| Q4_K_M
50
- [DeepSeek-V3-0324-DRAFT-0.5B-iQ4_K_S.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iQ4_K_S.gguf)| Q4_K_S
51
- [DeepSeek-V3-0324-DRAFT-0.5B-iIQ4_NL.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iIQ4_NL.gguf)| IQ4_NL
52
- [DeepSeek-V3-0324-DRAFT-0.5B-iIQ4_XS.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iIQ4_XS.gguf)| IQ4_XS
53
- [DeepSeek-V3-0324-DRAFT-0.5B-iQ5_1.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iQ5_1.gguf)| Q5_1
54
- [DeepSeek-V3-0324-DRAFT-0.5B-iQ5_0.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iQ5_0.gguf)| Q5_0
55
- [DeepSeek-V3-0324-DRAFT-0.5B-iQ4_1.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iQ4_1.gguf)| Q4_1
56
- [DeepSeek-V3-0324-DRAFT-0.5B-iQ4_0.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iQ4_0.gguf)| Q4_0
57
-
58
- ---
59
-
60
- See [DeepSeek-R1-DRAFT-0.5B-v1.0-GGUF](https://huggingface.co/jukofyork/DeepSeek-R1-DRAFT-0.5B-v1.0-GGUF#without-imatrix) for detailed PPL statistics and recommendations on which quant to use, etc.
61
-
 
 
 
 
 
 
 
 
 
 
 
 
 
 
62
  I have included the [imatrix file](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/DeepSeek-V3-0324-DRAFT-0.5B-BF16.imatrix) used to generate the `Q4_0`-`Q6_K` quants, along with the [1MB sample of the fine-tuning data](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/DeepSeek-V3-0324-DRAFT-imatrix-data.txt) used to create it.
 
1
+ ---
2
+ license: apache-2.0
3
+ base_model:
4
+ - Qwen/Qwen2.5-0.5B-Instruct
5
+ datasets:
6
+ - agentlans/common-crawl-sample
7
+ - bigcode/the-stack-smol-xl
8
+ - open-thoughts/OpenThoughts-Unverified-173k
9
+ - cognitivecomputations/dolphin-r1
10
+ tags:
11
+ - draft
12
+ - speculative-decoding
13
+ language:
14
+ - zho
15
+ - eng
16
+ - fra
17
+ - spa
18
+ - por
19
+ - deu
20
+ - ita
21
+ - rus
22
+ - jpn
23
+ - kor
24
+ - vie
25
+ - tha
26
+ - ara
27
+ ---
28
+
29
+ ![image-3.webp](https://cdn-uploads.huggingface.co/production/uploads/65995c45539c808e84c38bf1/pqAVNCYd1BV2ljTFwO9Ab.webp)
30
+
31
+ A `0.5B` parameter draft (speculative decoding) model for use with [deepseek-ai/DeepSeek-V3-0324](https://huggingface.co/deepseek-ai/DeepSeek-V3-0324).
32
+
33
+ See [jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0) for the non-GGUF version, and a detailed explanation of how the model was created.
34
+
35
+ ---
36
+
37
+ # Without `imatrix`
38
+
39
+ Link | Type
40
+ -----| ----
41
+ [DeepSeek-V3-0324-DRAFT-0.5B-BF16.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-BF16.gguf) | BF16
42
+ [DeepSeek-V3-0324-DRAFT-0.5B-F16.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-F16.gguf)| F16
43
+ [DeepSeek-V3-0324-DRAFT-0.5B-Q8_0.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-Q8_0.gguf)| Q8_0
44
+ [DeepSeek-V3-0324-DRAFT-0.5B-Q6_K.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-Q6_K.gguf)| Q6_K
45
+ [DeepSeek-V3-0324-DRAFT-0.5B-Q5_K_M.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-Q5_K_M.gguf)| Q5_K_M
46
+ [DeepSeek-V3-0324-DRAFT-0.5B-Q5_K_S.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-Q5_K_S.gguf)| Q5_K_S
47
+ [DeepSeek-V3-0324-DRAFT-0.5B-Q4_K_M.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-Q4_K_M.gguf)| Q4_K_M
48
+ [DeepSeek-V3-0324-DRAFT-0.5B-Q4_K_S.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-Q4_K_S.gguf)| Q4_K_S
49
+ [DeepSeek-V3-0324-DRAFT-0.5B-IQ4_NL.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-IQ4_NL.gguf)| IQ4_NL
50
+ [DeepSeek-V3-0324-DRAFT-0.5B-IQ4_XS.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-IQ4_XS.gguf)| IQ4_XS
51
+ [DeepSeek-V3-0324-DRAFT-0.5B-Q5_1.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-Q5_1.gguf)| Q5_1
52
+ [DeepSeek-V3-0324-DRAFT-0.5B-Q5_0.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-Q5_0.gguf)| Q5_0
53
+ [DeepSeek-V3-0324-DRAFT-0.5B-Q4_1.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-Q4_1.gguf)| Q4_1
54
+ [DeepSeek-V3-0324-DRAFT-0.5B-Q4_0.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/no-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-Q4_0.gguf)| Q4_0
55
+
56
+ # With `imatrix`
57
+
58
+ Link | Type
59
+ -----| ----
60
+ [DeepSeek-V3-0324-DRAFT-0.5B-iQ6_K.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iQ6_K.gguf)| Q6_K
61
+ [DeepSeek-V3-0324-DRAFT-0.5B-iQ5_K_M.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iQ5_K_M.gguf)| Q5_K_M
62
+ [DeepSeek-V3-0324-DRAFT-0.5B-iQ5_K_S.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iQ5_K_S.gguf)| Q5_K_S
63
+ [DeepSeek-V3-0324-DRAFT-0.5B-iQ4_K_M.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iQ4_K_M.gguf)| Q4_K_M
64
+ [DeepSeek-V3-0324-DRAFT-0.5B-iQ4_K_S.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iQ4_K_S.gguf)| Q4_K_S
65
+ [DeepSeek-V3-0324-DRAFT-0.5B-iIQ4_NL.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iIQ4_NL.gguf)| IQ4_NL
66
+ [DeepSeek-V3-0324-DRAFT-0.5B-iIQ4_XS.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iIQ4_XS.gguf)| IQ4_XS
67
+ [DeepSeek-V3-0324-DRAFT-0.5B-iQ5_1.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iQ5_1.gguf)| Q5_1
68
+ [DeepSeek-V3-0324-DRAFT-0.5B-iQ5_0.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iQ5_0.gguf)| Q5_0
69
+ [DeepSeek-V3-0324-DRAFT-0.5B-iQ4_1.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iQ4_1.gguf)| Q4_1
70
+ [DeepSeek-V3-0324-DRAFT-0.5B-iQ4_0.gguf](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/with-imatrix/DeepSeek-V3-0324-DRAFT-0.5B-iQ4_0.gguf)| Q4_0
71
+
72
+ ---
73
+
74
+ See [DeepSeek-R1-DRAFT-0.5B-v1.0-GGUF](https://huggingface.co/jukofyork/DeepSeek-R1-DRAFT-0.5B-v1.0-GGUF#without-imatrix) for detailed PPL statistics and recommendations on which quant to use, etc.
75
+
76
  I have included the [imatrix file](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/DeepSeek-V3-0324-DRAFT-0.5B-BF16.imatrix) used to generate the `Q4_0`-`Q6_K` quants, along with the [1MB sample of the fine-tuning data](https://huggingface.co/jukofyork/DeepSeek-V3-0324-DRAFT-0.5B-v1.0-GGUF/blob/main/DeepSeek-V3-0324-DRAFT-imatrix-data.txt) used to create it.