add model

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,183 @@
+---
+license: apache-2.0
+language:
+- en
+tags:
+- moe
+- fp8
+- vllm
+---
+
+# Mixtral-8x22B-Instruct-v0.1-FP8
+
+## Model Overview
+- **Model Architecture:** Mixtral-8x22B-Instruct-v0.1
+  - **Input:** Text
+  - **Output:** Text
+- **Model Optimizations:**
+  - **Weight quantization:** FP8
+  - **Activation quantization:** FP8
+- **Release Date:** 2/26/2025
+- **Version:** 1.0
+- **Model Developers:** Neural Magic
+
+Quantized version of [Mixtral-8x22B-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1).
+It achieves an average score of 79.5 on the [OpenLLM](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) benchmark (version 1), whereas the unquantized model achieves 79.87.
+
+### Model Optimizations
+
+This model was obtained by quantizing the weights and activations to FP8 data type, ready for inference with vLLM.
+This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%. Only the weights and activations of the linear operators within transformers blocks are quantized, except the MLP routers. 
+
+## Deployment
+
+### Use with vLLM
+
+This model can be deployed efficiently using the [vLLM](https://docs.vllm.ai/en/latest/) backend, as shown in the example below.
+
+```python
+from transformers import AutoTokenizer
+from vllm import LLM, SamplingParams
+
+max_model_len, tp_size = 4096, 4
+model_name = "neuralmagic/Mixtral-8x22B-Instruct-v0.1-FP8"
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+llm = LLM(model=model_name, tensor_parallel_size=tp_size, max_model_len=max_model_len, trust_remote_code=True)
+sampling_params = SamplingParams(temperature=0.3, max_tokens=256, stop_token_ids=[tokenizer.eos_token_id])
+
+messages_list = [
+    [{"role": "user", "content": "Who are you? Please respond in pirate speak!"}],
+]
+
+prompt_token_ids = [tokenizer.apply_chat_template(messages, add_generation_prompt=True) for messages in messages_list]
+
+outputs = llm.generate(prompt_token_ids=prompt_token_ids, sampling_params=sampling_params)
+
+generated_text = [output.outputs[0].text for output in outputs]
+print(generated_text)
+```
+
+vLLM also supports OpenAI-compatible serving. See the [documentation](https://docs.vllm.ai/en/latest/) for more details.
+
+## Creation
+
+This model was created with [llm-compressor](https://github.com/vllm-project/llm-compressor) by running the code snippet below with the following command:
+
+```bash
+python quantize.py --model_path mistralai/Mixtral-8x22B-Instruct-v0.1 --quant_path "output_dir" --calib_size 128 
+```
+
+
+```python
+import argparse
+from datasets import load_dataset
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from llmcompressor.modifiers.quantization import QuantizationModifier
+from llmcompressor.transformers import oneshot
+from llmcompressor.transformers.compression.helpers import calculate_offload_device_map
+import torch
+import os
+
+
+def main():
+    # Set up command line argument parsing
+    parser = argparse.ArgumentParser(description='Quantize a transformer model to FP8')
+    parser.add_argument('--model_id', type=str, required=True,
+                        help='The model ID from HuggingFace (e.g., "meta-llama/Meta-Llama-3-8B-Instruct")')
+    parser.add_argument('--save_path', type=str, default='.',
+                        help='Custom path to save the quantized model. If not provided, will use model_name-FP8')
+    parser.add_argument('--calib_size', type=int, default=256)
+    args = parser.parse_args()
+
+    device_map = calculate_offload_device_map(
+        args.model_id,
+        reserve_for_hessians=False,
+        num_gpus=torch.cuda.device_count(),
+        trust_remote_code=True,
+        torch_dtype=torch.bfloat16,
+    )
+
+    model = AutoModelForCausalLM.from_pretrained(
+        args.model_id, device_map=device_map, torch_dtype=torch.bfloat16, trust_remote_code=True,
+    )
+    tokenizer = AutoTokenizer.from_pretrained(args.model_id)
+
+    NUM_CALIBRATION_SAMPLES = args.calib_size
+    DATASET_ID = "garage-bAInd/Open-Platypus"
+    DATASET_SPLIT = "train"
+    ds = load_dataset(DATASET_ID, split=DATASET_SPLIT)
+    ds = ds.shuffle(seed=42).select(range(NUM_CALIBRATION_SAMPLES))
+
+    def preprocess(example):
+        concat_txt = example["instruction"] + "\n" + example["output"]
+        return {"text": concat_txt}
+
+    ds = ds.map(preprocess)
+
+    def tokenize(sample):
+        return tokenizer(
+            sample["text"],
+            padding=False,
+            truncation=False,
+            add_special_tokens=True,
+        )
+
+    ds = ds.map(tokenize, remove_columns=ds.column_names)
+
+    # Configure the quantization algorithm and scheme
+    recipe = QuantizationModifier(
+        targets="Linear", scheme="FP8", ignore=["lm_head", "re:.*block_sparse_moe.gate"]
+    )
+
+    # Apply quantization
+    oneshot(
+        model=model,
+        dataset=ds,
+        recipe=recipe,
+        num_calibration_samples=args.calib_size
+    )
+
+    save_path = os.path.join(args.save_path, args.model_id.split("/")[1] + "-FP8")
+    os.makedirs(save_path, exist_ok=True)
+
+    # Save to disk in compressed-tensors format
+    model.save_pretrained(save_path, save_compressed=True, skip_compression_stats=True)
+    tokenizer.save_pretrained(save_path)
+    print(f"Model and tokenizer saved to: {save_path}")
+
+if __name__ == "__main__":
+    main()
+```
+
+## Evaluation
+
+The model was evaluated on OpenLLM Leaderboard [V1](https://huggingface.co/spaces/open-llm-leaderboard-old/open_llm_leaderboard) using the following command:
+
+OpenLLM Leaderboard V1:
+```
+lm_eval \
+  --model vllm \
+  --model_args pretrained="neuralmagic/Mixtral-8x22B-Instruct-v0.1-FP8",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=8,gpu_memory_utilization=0.8,enable_chunked_prefill=True,trust_remote_code=True \
+  --tasks openllm \
+  --write_out \
+  --batch_size auto \
+  --output_path output_dir \
+  --show_config
+```
+
+
+### Accuracy
+
+#### OpenLLM Leaderboard V1 evaluation scores
+
+| Metric                                   | mistralai/Mixtral-8x22B-Instruct-v0.1             | neuralmagic/Mixtral-8x22B-Instruct-v0.1-FP8 |
+|-----------------------------------------|:---------------------------------:|:-------------------------------------------:|
+| ARC-Challenge (Acc-Norm, 25-shot)       |       73.29                      |                 73.29                       |
+| GSM8K (Strict-Match, 5-shot)            |         85.06                    |                 84.08                        |
+| HellaSwag (Acc-Norm, 10-shot)           |         88.94                    |                 88.85                       |
+| MMLU (Acc, 5-shot)                      |         77.77                    |                 77.62                       |
+| TruthfulQA (MC2, 0-shot)                |          68.49                   |                 68.01                       |
+| Winogrande (Acc, 5-shot)                |          85.64                   |                 85.16                       |
+| **Average Score**                       | **79.87**                        | **79.50**                                   |
+| **Recovery (%)**                            | **100.00**                       | **99.54**                                   |
+

config.json

@@ -0,0 +1,130 @@
+{
+  "_name_or_path": "mistralai/Mixtral-8x22B-Instruct-v0.1",
+  "architectures": [
+    "MixtralForCausalLM"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 6144,
+  "initializer_range": 0.02,
+  "intermediate_size": 16384,
+  "max_position_embeddings": 65536,
+  "model_type": "mixtral",
+  "num_attention_heads": 48,
+  "num_experts_per_tok": 2,
+  "num_hidden_layers": 56,
+  "num_key_value_heads": 8,
+  "num_local_experts": 8,
+  "output_router_logits": false,
+  "quantization_config": {
+    "config_groups": {
+      "group_0": {
+        "input_activations": {
+          "actorder": null,
+          "block_structure": null,
+          "dynamic": false,
+          "group_size": null,
+          "num_bits": 8,
+          "observer": "minmax",
+          "observer_kwargs": {},
+          "strategy": "tensor",
+          "symmetric": true,
+          "type": "float"
+        },
+        "output_activations": null,
+        "targets": [
+          "Linear"
+        ],
+        "weights": {
+          "actorder": null,
+          "block_structure": null,
+          "dynamic": false,
+          "group_size": null,
+          "num_bits": 8,
+          "observer": "minmax",
+          "observer_kwargs": {},
+          "strategy": "tensor",
+          "symmetric": true,
+          "type": "float"
+        }
+      }
+    },
+    "format": "float-quantized",
+    "global_compression_ratio": 1.55840454870176,
+    "ignore": [
+      "model.layers.0.block_sparse_moe.gate",
+      "model.layers.1.block_sparse_moe.gate",
+      "model.layers.2.block_sparse_moe.gate",
+      "model.layers.3.block_sparse_moe.gate",
+      "model.layers.4.block_sparse_moe.gate",
+      "model.layers.5.block_sparse_moe.gate",
+      "model.layers.6.block_sparse_moe.gate",
+      "model.layers.7.block_sparse_moe.gate",
+      "model.layers.8.block_sparse_moe.gate",
+      "model.layers.9.block_sparse_moe.gate",
+      "model.layers.10.block_sparse_moe.gate",
+      "model.layers.11.block_sparse_moe.gate",
+      "model.layers.12.block_sparse_moe.gate",
+      "model.layers.13.block_sparse_moe.gate",
+      "model.layers.14.block_sparse_moe.gate",
+      "model.layers.15.block_sparse_moe.gate",
+      "model.layers.16.block_sparse_moe.gate",
+      "model.layers.17.block_sparse_moe.gate",
+      "model.layers.18.block_sparse_moe.gate",
+      "model.layers.19.block_sparse_moe.gate",
+      "model.layers.20.block_sparse_moe.gate",
+      "model.layers.21.block_sparse_moe.gate",
+      "model.layers.22.block_sparse_moe.gate",
+      "model.layers.23.block_sparse_moe.gate",
+      "model.layers.24.block_sparse_moe.gate",
+      "model.layers.25.block_sparse_moe.gate",
+      "model.layers.26.block_sparse_moe.gate",
+      "model.layers.27.block_sparse_moe.gate",
+      "model.layers.28.block_sparse_moe.gate",
+      "model.layers.29.block_sparse_moe.gate",
+      "model.layers.30.block_sparse_moe.gate",
+      "model.layers.31.block_sparse_moe.gate",
+      "model.layers.32.block_sparse_moe.gate",
+      "model.layers.33.block_sparse_moe.gate",
+      "model.layers.34.block_sparse_moe.gate",
+      "model.layers.35.block_sparse_moe.gate",
+      "model.layers.36.block_sparse_moe.gate",
+      "model.layers.37.block_sparse_moe.gate",
+      "model.layers.38.block_sparse_moe.gate",
+      "model.layers.39.block_sparse_moe.gate",
+      "model.layers.40.block_sparse_moe.gate",
+      "model.layers.41.block_sparse_moe.gate",
+      "model.layers.42.block_sparse_moe.gate",
+      "model.layers.43.block_sparse_moe.gate",
+      "model.layers.44.block_sparse_moe.gate",
+      "model.layers.45.block_sparse_moe.gate",
+      "model.layers.46.block_sparse_moe.gate",
+      "model.layers.47.block_sparse_moe.gate",
+      "model.layers.48.block_sparse_moe.gate",
+      "model.layers.49.block_sparse_moe.gate",
+      "model.layers.50.block_sparse_moe.gate",
+      "model.layers.51.block_sparse_moe.gate",
+      "model.layers.52.block_sparse_moe.gate",
+      "model.layers.53.block_sparse_moe.gate",
+      "model.layers.54.block_sparse_moe.gate",
+      "model.layers.55.block_sparse_moe.gate",
+      "lm_head"
+    ],
+    "kv_cache_scheme": null,
+    "quant_method": "compressed-tensors",
+    "quantization_status": "compressed"
+  },
+  "rms_norm_eps": 1e-05,
+  "rope_theta": 1000000.0,
+  "router_aux_loss_coef": 0.001,
+  "router_jitter_noise": 0.0,
+  "sliding_window": null,
+  "tie_word_embeddings": false,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.49.0",
+  "use_cache": true,
+  "vocab_size": 32768
+}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "transformers_version": "4.49.0"
+}

model-00001-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8c8ffabb3909ea3bbda48822363948cf61088db105cb37ad965e7f32d01d908e
+size 4907576972

model-00002-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b69428ce35d449ab8a3e0d9b6d9f7dac8e075cffb5d30ad9e5f65550fb39f488
+size 4907602332

model-00003-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b73b298e276e0e68149ae74aa6a07c188ed60086d001510618d2e1be0ef5d87e
+size 4907602316

model-00004-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:328ed6703ed4015a27e89986396d73cc4f79a49a6268cc547846ae16736a4950
+size 4907602300

model-00005-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a087474bec9626391b7e39320ff593eefb27b00d22f27762f8d1fb8b81b59970
+size 4907602284

model-00006-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a3d020512e7950780ad13683f0bae170504a340c312e7f0de5ecf904409035bb
+size 4907602268

model-00007-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0491fa7de4c27f6e268b6a9a253e5b7846ab2834994caba026a61c1b434d491f
+size 4907602428

model-00008-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0d830f82ffed6b13c14780b9c588f3cf95b99375643c75a57eaf21a778a652d3
+size 4907602412

model-00009-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:50b2741183d1771ee6354b6855175fde84f168b77e838d83a3de56af62a9d3f6
+size 4907602396

model-00010-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b855ff894ac3db5f453b51c6facc6ac9428c7c00a5a44c3f872583829c229cd8
+size 4907602380

model-00011-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5b5437f907ab6bd46adc3330ce01495ff98a5550e339b1b2e89a6331d52f97b0
+size 4907602364

model-00012-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:05cd72e98c02f89daa80f5e19c0ceafcab2ee44f2209f2ecde5b7487279def9e
+size 4907602348

model-00013-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:02a96a662260bda5f363bfbbe399d0a6ea5fef577e2924ff4887e4669f4b9690
+size 4907602332

model-00014-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:371d13dc775fb5f4411bc0a235fe92925967db40e57abf8df35fb8deeb073378
+size 4907602316

model-00015-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:37a35c28a712251a20588dbbd6d4288aee61ffc427bd27e27fa3837bb84c51ac
+size 4907602300

model-00016-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3fdcfd911574831baeb90a7d69a70ac8f3bba2fa4f791766556d81a9e124cd4b
+size 4907602284

model-00017-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:24b369278f3c28df9558b8255115a30d26cfe76fa6abd3c3e15f2dc94a88a014
+size 4907602268

model-00018-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a20d25709381b455cabd966aa39c8f38a8466f5f29bd495fa7cff8066755ab4a
+size 4907602252

model-00019-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0a13ecec78f5c241f58e7c5f6521d5532b11baca58f9bf08d0fc8b5e0b0c41a6
+size 4907602236

model-00020-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c939243565b3bc1edfe7d6e3a90ffe97d81aac76b0fb04c6f8d051eb64929067
+size 4907602220

model-00021-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:469e12ac7c5b793be10db5176c00bb5fe590a250f227bcda0b88203c29feaa47
+size 4970418388

model-00022-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b941997e14b9918d2793cdca2c684006e587071959822c4c8203ccf3b304e3ac
+size 4995682384

model-00023-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d3c4fad8d99491f373036f1229b49b01bb4c8fcc66a590915caf42a0e7234fea
+size 4970516900

model-00024-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:225fba4dc5a9dc50c330d7c08c7f0c15c274a62b9fc71e1881348da8cc00224e
+size 4907577508

model-00025-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5ffe5c1fb55965b2e07d69ecbf8108cbf84dddaaaee0963e3bea9e270c53f89c
+size 4907602572

model-00026-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:94994d0db23b2320cb372cc81e613f06c84dd183def9feb9fbc28a82e1771f59
+size 4907602556

model-00027-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3a2d7009697ed53a1dd8a99b41a91f5b37833f90733198389c7acf8849f6821e
+size 4907602540

model-00028-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ba5735d07e09f51ce0d7882574371e383ad6e74e381e6d74c328ab1690867f83
+size 4907602524

model-00029-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:205856a91cbd5d411acc49b698a586b7ee19bacb70e9b90e846ba2386f46167b
+size 3410142452

recipe.yaml

@@ -0,0 +1,6 @@
+DEFAULT_stage:
+  DEFAULT_modifiers:
+    QuantizationModifier:
+      ignore: [lm_head, 're:.*block_sparse_moe.gate']
+      targets: [Linear]
+      scheme: FP8

special_tokens_map.json

@@ -0,0 +1,23 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:60c3fc985cbfedcb429d05994efe548bdfecd6a00226fcdc8380c36fd894a3be
+size 3671968

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:37f00374dea48658ee8f5d0f21895b9bc55cb0103939607c8185bfd1c6ca1f89
+size 587404