CCC++: Optimized Color Classified Colorization with Segment Anything
  Model (SAM) Empowered Object Selective Color Harmonization
Mrityunjoy Gain Avi Deb Raha Rameswar Debnath
In this paper, we formulate the colorization problem into a multinomial
classification problem and then apply a weighted function to classes. We
propose a set of formulas to transform color values into color classes and vice
versa. To optimize the classes, we experiment with different bin sizes for
color class transformation. Observing class appearance, standard deviation, and
model parameters on various extremely large-scale real-time images in practice
we propose 532 color classes for our classification task. During training, we
propose a class-weighted function based on true class appearance in each batch
to ensure proper saturation of individual objects. We adjust the weights of the
major classes, which are more frequently observed, by lowering them, while
escalating the weights of the minor classes, which are less commonly observed.
In our class re-weight formula, we propose a hyper-parameter for finding the
optimal trade-off between the major and minor appeared classes. As we apply
regularization to enhance the stability of the minor class, occasional minor
noise may appear at the object's edges. We propose a novel object-selective
color harmonization method empowered by the Segment Anything Model (SAM) to
refine and enhance these edges. We propose two new color image evaluation
metrics, the Color Class Activation Ratio (CCAR), and the True Activation Ratio
(TAR), to quantify the richness of color components. We compare our proposed
model with state-of-the-art models using six different dataset: Place, ADE,
Celeba, COCO, Oxford 102 Flower, and ImageNet, in qualitative and quantitative
approaches. The experimental results show that our proposed model outstrips
other models in visualization, CNR and in our proposed CCAR and TAR measurement
criteria while maintaining satisfactory performance in regression (MSE, PSNR),
similarity (SSIM, LPIPS, UIUI), and generative criteria (FID).