Anyone able to run this on vLLM ?

by xfalcox - opened Jul 24, 2024

Jul 24, 2024

This is using bitsandbytes 8 bit quantization that is broken on vLLM at the moment.

Any chance to release it using GTPQ or any fixes to vLLM incoming?

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Your need to confirm your account before you can post a new comment.

· Sign up or log in to comment