Do the Q4 quants work? On the 30b moe it says not to use them.
#3
by
Lockout
- opened
I don't want to download 125g twice :(
I don't want to download 125g twice :(
Yes it does! Make sure you use the latest llama.cpp commit.
Also we're gonna reupload all models just to be safe but should work now! We've had many users confirm with us
I can confirm Q3_K_S works perfectly. Seems like a good model, I've been using it for about 2 hours for work.