Thank you for your work. How should this model be used? Do you have code? How much VRAM is needed to run and how long does it take to reason once
#3
by
michaelj
- opened
Thank you for your work. How should this model be used? Do you have code? How much VRAM is needed to run and how long does it take to reason once
Hi,
Do you have code?
Oh I thought I included a gradio app, looks like I forgot. Someone seems to be building a HF space for it: Gustav0-Freind/Bagel-7B-Demo
My code is quite hacky trying to get the weights on a single GPU so Gustav0-Friend's space code is probably going to be better.
how long does it take to reason once
As in to describe an image with ?
28 seconds with RTX3090.
How much VRAM is needed to run
Here's the model loaded on my rig:
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.133.07 Driver Version: 570.133.07 CUDA Version: 12.8 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 3090 On | 00000000:01:00.0 Off | N/A |
| 53% 44C P8 39W / 390W | 10525MiB / 24576MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA GeForce RTX 3090 On | 00000000:07:00.0 Off | N/A |
| 0% 49C P8 27W / 350W | 4841MiB / 24576MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
16GB to load the model. And this is generating an image:
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 3090 On | 00000000:01:00.0 Off | N/A |
| 57% 62C P0 334W / 390W | 12015MiB / 24576MiB | 72% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA GeForce RTX 3090 On | 00000000:07:00.0 Off | N/A |
| 43% 61C P0 197W / 350W | 6901MiB / 24576MiB | 11% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
20GB