Anyone able to run this on vLLM ?

#7
by xfalcox - opened

This is using bitsandbytes 8 bit quantization that is broken on vLLM at the moment.

Any chance to release it using GTPQ or any fixes to vLLM incoming?

Your need to confirm your account before you can post a new comment.

Sign up or log in to comment