martin-rizzo/TinyBreaker.prototype1

TinyBreaker (prototype1)

   

Overview

TinyBreaker is a hybrid two-step model (base + refiner) designed for efficient image generation on mid-end and low-end hardware. By combining the strengths of PixArt and Photon models, it delivers high-quality images with strong prompt adherence

Key Features

• Hybrid Two-Step Architecture: Combines PixArt-Sigma as the base model with a refiner based on Photon (or any SD1.x model), both chosen for their low GPU consumption.
• Efficient Parameter Usage: The base model’s 0.6 billion parameters enable high-quality image generation with minimal computational overhead.
• Fast Performance: Produces high-quality 1536×1024 images in ~15 seconds on an NVIDIA RTX 3080 GPU, with ongoing work to cut generation times to under 10 seconds.
• High Prompt Adherence: Generates images that closely match user prompts and expectations, thanks to the robust performance of the PixArt-Sigma model and the T5 text encoder.
• Optimized Latent Space Processing: Leverages Tiny Autoencoders for efficient latent space conversion.

Usage Requirements

Currently, TinyBreaker can only be used with ComfyUI. To utilize it, you'll need to install the custom nodes specific to this model through the ComfyUI-TinyBreaker GitHub repository.

Limitations

• Text Generation: Generating legible text within images is a challenge due to PixArt's training limitations. Enhancements in this area may require extensive retraining.

• Human Anatomy in Complex Poses: While the model performs reliably with standard poses (e.g., standing, facing the camera), it struggles with anatomical accuracy in poses that require more complex or dynamic actions.

• Complex Human Interactions: The model has difficulty generating detailed scenes involving intricate interactions among people, as well as interactions between people and objects, such as collaborative tasks or dynamic object manipulation.

Note: The current "Prototype1" version of TinyBreaker utilizes PixArt-Sigma 1024 and Photon models without any additional training or fine-tuning. In the future, if I have the resources, I plan to train both models together to generate images of even greater quality

Future Directions

I am dedicated to improving TinyBreaker's performance and accessibility, especially for users with mid-range or lower-end hardware. Looking forward to future updates as I continue to expand TinyBreaker's capabilities.

Acknowledgments

• I extend my sincere thanks to the PixArt-Σ developers for their exceptional model, which has been vital to this project's development.
  PixArt-Σ GitHub Repository | PixArt-Σ Hugging Face Model | PixArt-Σ arXiv Report

• Additional thanks to Ollin Boer Bohan for the Tiny AutoEncoder models, which offer efficient latent image processing and served as the foundation for the encoding, decoding, and transcoding operations in TinyBreaker.
  Tiny AutoEncoder GitHub Repository

Resources

• TinyBreaker on CivitAI: A hub for exploring generated images, prompts, and workflows created by me and the community, showcasing the model's output quality.
• ComfyUI-TinyBreaker: Nodes and workflows for ComfyUI to experiment with the model's capabilities.
• TinyBreakerTools: Tools I'm building for the model, mainly to create the safetensors file for TinyBreaker.
• AbominableWorkflows: A predecessor of TinyBreaker. My first experiment combining PixArt-Sigma and Photon without Python code, using only standard nodes from ComfyUI.