PaliGemma3_FT_OCR

This model is a fine-tuned version of google/paligemma-3b-pt-448 on an unknown dataset.

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 1
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 8
• optimizer: Use adamw_hf with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 2
• num_epochs: 50

Training results

Framework versions

• PEFT 0.14.0
• Transformers 4.47.0
• Pytorch 2.2.1+cu121
• Tokenizers 0.21.0

Inference

from transformers import AutoProcessor, PaliGemmaForConditionalGeneration
from PIL import Image
import torch
import json

# Load model and processor
model_id = "google/paligemma-3b-pt-448"
peft_adapter_id = "riphunter7001x/PaliGemma3_FT_OCR"

model = PaliGemmaForConditionalGeneration.from_pretrained(model_id, device_map="auto")
processor = AutoProcessor.from_pretrained(model_id)
model.load_adapter(peft_adapter_id).eval()

TORCH_DTYPE = model.dtype
DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# Load and process image
image = Image.open("image.jpg")

prefix = "<image>extract Document data in JSON format"

inputs = processor(
    text=prefix,
    images=image,
    return_tensors="pt"
).to(TORCH_DTYPE).to(DEVICE)

prefix_length = inputs["input_ids"].shape[-1]

with torch.inference_mode():
    generation = model.generate(**inputs, max_new_tokens=512, do_sample=False)
    generation = generation[0][prefix_length:]
    decoded = processor.decode(generation, skip_special_tokens=True)
    print(json.dumps(json.loads(decoded), indent=4))

This code loads the fine-tuned PaliGemma model, processes an input image, and extracts document data in JSON format.