bobig/Orpheus-Jai-GGUF

Orpheus-Jai-Q4 GGUF Model

This repository contains a quantized version of the Orpheus-Porjai-Thai language model, optimized for use with llama.cpp.

Model Details

• Model Name: Orpheus-Jai-Q4
• Original Model: CMKL/Orpheus-Porjai-Thai (Assumed repository based on dataset)
• GGUF Repository: bobig/Orpheus-Jai-GGUF
• Base Model: canopylabs/orpheus-3b-0.1-pretrained
• Language: Thai (th)
• Format: GGUF (llama.cpp format)
• Quantization: Q4_K_M (Type 15) - Offers a good balance between performance and file size (~1.9 GB).
• Architecture: Llama 3 (3.3B parameters)

Description

This model is a Q4_K_M quantized GGUF version of Orpheus-Porjai-Thai, a 3.3 billion parameter Thai language model based on the Llama 3 architecture. It was fine-tuned on the CMKL/Porjai-Thai-voice-dataset-central dataset.

The GGUF format allows for efficient inference using the llama.cpp framework on various hardware.

How to Use with llama.cpp

1. Ensure you have llama.cpp built. Follow the instructions in the llama.cpp repository.

2. Download the GGUF model file: Orpheus-Jai-Q4.gguf from the Hugging Face repository.

3. Run inference:

   • Command Line:

     ./main -m path/to/Orpheus-Jai-Q4.gguf -p "สวัสดีครับ" --n-predict 100 -c 2048
     

   • Server Mode:

     ./server -m path/to/Orpheus-Jai-Q4.gguf -c 2048
     

   (Replace path/to/ with the actual path to the downloaded GGUF file. Adjust parameters like -p (prompt), --n-predict (tokens to generate), and -c (context size) as needed.)

Conversion Process

The conversion from the original Hugging Face format to Q4_K_M GGUF involved specific steps to handle the Llama 3 tokenizer correctly. For detailed technical information on the conversion process, please refer to the README_Porjai_Conversion.md file in this project.

Hugging Face Repository

This model is now available on Hugging Face:

• Repository: bobig/Orpheus-Jai-GGUF
• Direct download link: Orpheus-Jai-Q4.gguf

The model file is approximately 1.9GB in size.

License

The license of the original model should be checked. This GGUF file inherits the licensing terms of the base Orpheus-Porjai-Thai model.