Remove vLLM FP8 Limitation

by simon-mo - opened 1 day ago

←

Qwen org 1 day ago

This has been fixed as of latest v0.8.5 release 🙇

about 23 hours ago

ERROR 04-29 09:46:24 [core.py:396] ValueError("type fp8e4nv not supported in this architecture. The supported fp8 dtypes are ('fp8e4b15', 'fp8e5')")

i got this when running it on an A100..does it not use the marlin kernels by default?

jklj077 changed pull request status to merged about 4 hours ago

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Your need to confirm your account before you can post a new comment.

· Sign up or log in to comment