bnb-4bit-experimental
Collection
experimental 4-bit dynamic quant using unsloth, these model will be used for inference and RoPE fine-tuning.
•
16 items
•
Updated
This is a converted weight from DeepHermes-3-Llama-3-8B-Preview model in unsloth 4-bit dynamic quant using this collab notebook.
About this Conversion
This conversion uses Unsloth to load the model in 4-bit format and force-save it in the same 4-bit format.
How 4-bit Quantization Works
- The actual 4-bit quantization is handled by BitsAndBytes (bnb), which works under Torch via AutoGPTQ or BitsAndBytes.
- Unsloth acts as a wrapper, simplifying and optimizing the process for better efficiency.
This allows for reduced memory usage and faster inference while keeping the model compact.
- Downloads last month
- 4
Inference Providers
NEW
This model isn't deployed by any Inference Provider.
🙋
Ask for provider support
HF Inference deployability: The model has no library tag.
Model tree for huggingkot/DeepHermes-3-Llama-3-8B-Preview-bnb-4bit
Base model
meta-llama/Llama-3.1-8B