Improving the Accuracy of Beauty Product Recommendations by Assessing
  Face Illumination Quality
Parnian Afshar Jenny Yeon Andriy Levitskyy Rahul Suresh Amin Banitalebi-Dehkordi
We focus on addressing the challenges in responsible beauty product
recommendation, particularly when it involves comparing the product's color
with a person's skin tone, such as for foundation and concealer products. To
make accurate recommendations, it is crucial to infer both the product
attributes and the product specific facial features such as skin conditions or
tone. However, while many product photos are taken under good light conditions,
face photos are taken from a wide range of conditions. The features extracted
using the photos from ill-illuminated environment can be highly misleading or
even be incompatible to be compared with the product attributes. Hence bad
illumination condition can severely degrade quality of the recommendation.
  We introduce a machine learning framework for illumination assessment which
classifies images into having either good or bad illumination condition. We
then build an automatic user guidance tool which informs a user holding their
camera if their illumination condition is good or bad. This way, the user is
provided with rapid feedback and can interactively control how the photo is
taken for their recommendation. Only a few studies are dedicated to this
problem, mostly due to the lack of dataset that is large, labeled, and diverse
both in terms of skin tones and light patterns. Lack of such dataset leads to
neglecting skin tone diversity. Therefore, We begin by constructing a diverse
synthetic dataset that simulates various skin tones and light patterns in
addition to an existing facial image dataset. Next, we train a Convolutional
Neural Network (CNN) for illumination assessment that outperforms the existing
solutions using the synthetic dataset. Finally, we analyze how the our work
improves the shade recommendation for various foundation products.