Phi-4-mini-instruct
Collection
2 items
•
Updated
Original Model : huihui-ai/Phi-4-mini-instruct-abliterated
All quants are made using the imatrix dataset.
Model | Size (GB) |
---|---|
Q2_K_S | 1.59 |
Q2_K | 1.68 |
Q3_K_M | 2.12 |
Q3_K_L | 2.25 |
Q4_K_M | 2.49 |
Q5_K_M | 2.85 |
Q6_K | 3.16 |
Q8_0 | 4.08 |
F16 | 7.68 |
CPU (AVX2) | CPU (ARM NEON) | Metal | cuBLAS | rocBLAS | SYCL | CLBlast | Vulkan | Kompute | |
---|---|---|---|---|---|---|---|---|---|
K-quants | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ 🐢5 | ✅ 🐢5 | ❌ |
I-quants | ✅ 🐢4 | ✅ 🐢4 | ✅ 🐢4 | ✅ | ✅ | Partial¹ | ❌ | ❌ | ❌ |
✅: feature works
🚫: feature does not work
❓: unknown, please contribute if you can test it youself
🐢: feature is slow
¹: IQ3_S and IQ1_S, see #5886
²: Only with -ngl 0
³: Inference is 50% slower
⁴: Slower than K-quants of comparable size
⁵: Slower than cuBLAS/rocBLAS on similar cards
⁶: Only q8_0 and iq4_nl
Base model
microsoft/Phi-4-mini-instruct