Spaces:

Sumkh
/

AgenticRAG

Sleeping

Sumkh commited on Feb 25

Commit

5695d95

verified ·

1 Parent(s): 19ad6b0

Update start.sh

Files changed (1) hide show

start.sh CHANGED Viewed

@@ -11,11 +11,7 @@ export USER_AGENT="vllm_huggingface_space"
 vllm serve unsloth/llama-3-70b-Instruct-bnb-4bit \
   --enable-auto-tool-choice \
   --tool-call-parser llama3_json \
-  --chat-template examples/tool_chat_template_llama3.1_json.jinja \
-  --quantization bitsandbytes \
-  --load-format bitsandbytes \
-  --gpu_memory_utilization 0.9 \
-  --enforce-eager &
 # Wait to ensure the vLLM server is fully started (adjust if needed)
 sleep 10

 vllm serve unsloth/llama-3-70b-Instruct-bnb-4bit \
   --enable-auto-tool-choice \
   --tool-call-parser llama3_json \
+  --chat-template examples/tool_chat_template_llama3.1_json.jinja &
 # Wait to ensure the vLLM server is fully started (adjust if needed)
 sleep 10