Max-Batch-Size, max-num-sequence, and fp_cache fp8_e4m3

#11

by BenFogerty - opened Feb 14

Feb 14

What would be the guideline with setting the max batch size and max num sequences for optimal ITL and TTFT, additionally, would setting cache to e4m3 increase performance?

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Your need to confirm your account before you can post a new comment.

· Sign up or log in to comment