Resizing Online Photos The Right Way

8 min read

Resizing Online Photos The Right Way

Content with visual cues, or images, are always more engaging, cogent, and stirring. Correct presentation of these visual cues is as important as their presence. They cannot just loiter about.

Images come in all shapes and sizes, and their placeholders during presentation are constrained by limitations of the presenting device, and also the design and flow of surrounding content in a wider context.

Shoehorning these vital content elements into their placeholders, while preserving the aesthetics sensibilities, is an art-form into itself. More so, when it has to be done at web-scale with tens of thousands of images coming through every day and presentation constraints as varied as the images themselves. Human intervention for each and every image, for each and every screen type, is not an option.

Presentation placeholders differ from the images both in size (width and height) and aspect ratio. A simple older approach to resolve this dilemma was to resize the original image to placeholder according to maximum width or height allowable and have padding bars on top and bottom, or sides, to maintain aspect ratio. Although it looks quite ungainly and disrupts the web pages design scheme and flow, there was no better option that scaled for quite some time. This idea was actually carried over from the efforts to manage different aspect ratios of STV (Standard TV) and HDTV format videos with the different TV screen sizes. STV screen sizes are hard to comeby nowadays. The terminology in the video world for these bars is “letterboxing” and “pillarboxing”.


Images with pillar boxes and letter boxes. [Wikimedia Commons]

Apart from “letterboxing” and “pillarboxing”, there is “Windowboxing” which puts a border all around the image to fit it into the placeholder. It is not actually a border but a background image with exact same dimensions as the image placeholder over which a scaled down image is superimposed. Matching the “window box” color with the background color of the page, or text, allowed for acceptable integration.

Window boxingWindowboxing.  [Wikimedia Commons] 

For a while, “pillarboxing” and “letterboxing” were about all you could possibly do programmatically to prevent unintended squishing and squashing of images in placeholders. Today we have solutions in the view media itself like CSS (Cascading Style Sheets) support as “object-fit” property to effectively manage resizing, cropping on spill etc.  of the image on the client side. Mobile App programmatic counterparts are also provided as mobile devices have ascended to become the most dominant viewing platform by far.

Resizing can be done on the client device, but is not always feasible. Size of image still needs to be managed on the server side to reduce network load. We cannot transfer image files, often in size of a few megabytes, to view-device for resizing and cropping. Data costs can become too prohibitive for the users, and page load times can be affected adversely. We resize images on the fly before transmission depending on the viewing device.

So far we have discussed only image resizing aspects. One downside with vanilla resize-to-fit-in-placeholders is that it makes the image object too small and blurred to be discernible. Particularly when an individual, or a group in the image is the focus of interest and not the surroundings.

Resizing Resizing

Resize makes the subject too small.

 The expected solution here is to crop the image around the subject with the final image size being the required placeholder size:

ResizingCropping to preserve details.

It is evident that the subject, or the face at least, needs to be recognized first before we can proceed with cropping. The simpler alternative of cropping around the center of the image is not appropriate for all cases.

With landscapes and panoramic images, resizing and cropping on spill around the center of the image is an acceptable solution provided you don’t lose the beautiful sunset (or sunrise) altogether in the cropping exercise.

Cropping of Scenic Image 1 Cropping of Scenic Image 2

Cropping of scenic images.

The image above has been cropped around the center of the image, and not resized, to fit into the placeholder.

Unfortunately, in the news domain, panoramic images are few and far between. There is always a central subject present, be it a person, or an incident. Cropping of images, without taking into account the topic of image is hazardous and can give absolutely ludicrous results. The center of the image has to be around the subject and cannot be determined simply by the dimensions of the image. In the example below, the center of cropping should have been offset towards the top of the image, towards the face, as opposed to being on the dimensional center of the picture.

Cropping gone wrong Cropping gone wrong

Cropping gone wrong.

Here is another example of bad cropping choice where we have multiple subjects in the image. Determining the center of crop is a bit more complicated now : 

faces cropped off faces cropped off

Example of faces cropped off. 

Finer aspects of image composition have also to be accounted for. For example, there should be a correct amount of head room. Too much, or too little, of headroom has a jarring effect, as if the subject is falling off the picture, or trying to climb into it. Exhaustive discussion on aspects of composition topics like screen equilibrium, motion vectors, balance, nose room etc. can be found in books about video composition and filmmaking. Image composition rules are a subset of these.

Hereon, we’ll discuss how we have achieved face-aware automated resizing and cropping of images for presentation. There are many more aspects of feature detection that are relevant for resizing and cropping images aesthetically that will be covered in subsequent articles.          

Face-Aware Cropping:

Enablement of face-aware cropping requires face-detection, determination of center of crop from position of faces in the image, and the process of cropping itself based on the requirements of size. A brief discussion on face detection and determination of crop center follows.

Face Detection

The first step in face-aware cropping is obviously face detection. It must be stated upfront that face-detection is quite different from face-recognition. Face recognition is recognizing an identity from face image while face detection is just ascertaining the presence of face in an image and determining its positioning.

Face detection and recognition has come a long way since the 1960s when the first proof of concept that computers can actually detect faces was given by Woodrow Wilson Bledsoe. However it was not before the 80s and 90s, when computing power was adequate enough for non-trivial dimensionality reduction, that the first viable systems were made. Work done  in 1987 by Sirovich and Kirby for low dimensional representation of face images, and the Eigenfaces system for classification developed in 1991 by Turk and Pentland as an extension of the former, was a seminal breakthrough and is still relevant for machine learning systems today. In the 2010s neural networks and CNN (Convolutional Neural Networks) in particular were employed for dimensionality reduction. AlexNet, a deep CNN, reduced the error rate from around 25% to 16% on ImageNet database, and the uptick in accuracy generated quite some excitement. This result dramatically altered the direction of development away from conventional statistical approaches, for example, away from PCA (Principal Component Analysis), SVD (Singular Value Decomposition) etc. The next couple of years were spent on increasing the depth of these convolutional network architectures in an effort to improve accuracy, i.e., making them deeper still, but the returns soon hit a wall and in some cases, increasing network depth actually degraded the results. After AlexNet, the next major breakthrough came in the form of ResNet (Residual Networks). ResNet architecture finally bettered human performance on ImageNet, returning an error rate of 3.5% against human error rate of 5%.

In our use-case, we don’t particularly need face recognition. We just need to identify the regions in the image that have a face to effect error free automated cropping.  Luckily, most of the architectures in face recognition are not monolithic in the sense that various steps in face recognition like image preprocessing & binarization, face landmark detection, pose alignment, bounding box regression, cropping triplet loss or euclidean distance calculation etc. can be separated into different modules and used independently.

Our immediate concern here is face detection for which we primarily need face landmark detection and bounding box regression. We will stop just before pose alignment because cropping and subsequent steps are beyond the scope of our use. Collectively these steps determine the contour of face detection architectures.

Face detection has its own set of challenges. Initially there were different statistical models that catered to each of the different problems in face detection like variation in expressions, different orientations, occlusion or partially hidden faces, variable illumination, complex or plain backgrounds, different resolution of images, different skin colours, different scales i.e. small and big images of faces, too many faces in an image etc. Algorithms that answered a problem often broke down in presence of composite variations, or presence of concurrent variations of other kinds.  Deep learning gave rise to unconstrained face detection methodologies which took into account the presence of all of the variations at once provided the training data set incorporated these variations as such.

One of the principal techniques in object detection is universal bounding box regression technique. The training set for the network in this case is swatches of image which are “object” and “not object”. In the face detection regime these will be swatches of “face” and “not face”. The network so trained would then suggest candidate bounding boxes on an input image by classifying the convolution swatches as “face” or “not-face”. The bounding boxes would often be many for a single face and overlapping as shown below and are arrived at quickly by a shallow CNN network. These are usually referred to as candidate boxes.

Overlapping bounding boxesOverlapping bounding boxes on a face.

The bounding box regression then refines to a single bounding box out of the multiple candidate boxes with the best fit. Performance of face detection, its robustness, and its accuracy has improved significantly by combining multiple CNNs falling in two broad categories of face detection approaches – landmark detection (detection of facial landmarks like nose, eyes, lips etc.) and pose alignment (turning a tilted face image to frontal image which is more suited for face recognition).

We narrowed down on architectures that gave a probability of correct face detection of at least 90% or above. The shortlisted architectures were HR-ER, DeepIR, ScaleFace, and MTCNN. We discarded HR-ER because detection of tiny faces is not our principal concern here. DeepIR or Deep Image Targeting is more aligned to our needs as it answers to semantic resizing of images for different display sizes. We discarded DeepIR as our focus is primarily on preventing improper cropping of faces which is actually a subset of DeepIR solution. By focusing on the subset we wanted to keep our overheads to minimum. ScaleFace gives impressive results on wide variation of scales (by scale we mean size of face images, for example small face images present in a picture of a football match vs a passport face image). We settled on MTCNN as it gave acceptable real time performance and also because the expected variation of scale in our use-case was not that varied.


MTCNN or Multi-task Convolution Networks, as the name suggests, leverages the information correlation between two different categories of sub-tasks in face recognition.. These two tasks are facial detection and face alignment. One task can be auxiliary to the other which is the primary task to improve its performance and accuracy. For example, when face alignment is the primary task then face detection can act as an auxiliary task to improve its performance and vice versa.  MTCNN combines these tasks in a cascaded CNN. The CNN consists of three stages, viz. first stage that produces candidate bounding boxes, second stage that rejects non face windows or boxes, and finally the last stage that refines results and outputs facial landmark positions. In our use-case facial landmark positions are ignored. The CNN architectures in MTCNN are lightweight to ensure real-time performance. The three stages of refinement are principally Non-maximum Suppression (NMS) and bounding box regression progressing towards a more refined output with facial landmark recognition in the last step. The authors of MTCNN have named these stages as P-Net or proposal-network where multiple swatches or candidate bounding boxes are proposed, R-Net or refine-network wherein large number of swatches that are not face bounding boxes are rejected, and the O-Net or output-network which outputs the final bounding boxes with facial landmarks. There is also a pre-processing stage that creates a pyramid of resized images to take into account scale aspects.

Multi-task Convolution NetworksMulti task networks in MTCNN [4]

Once we run our image through MTCNN we have the list of bounding boxes of faces present in the image with their dimensions and position. We make use of this output in our final image-cropping pipeline.

Final Cropping

Once the faces are detected and their bounding boxes known, naive algorithms can be used to successfully crop the image without destroying its aesthetics. The center, or focal point of the image can be taken as the median center of all the bounding boxes. If there are bounding boxes of different sizes then a median can also be determined by using the bounding box size as weights.

multiple bounding boxesAn image with multiple bounding boxes [5]

Other than the median or focal point which determines the center of crop, the crop width and height is also needed to complete the cropping task. If the crop width and height are not provided then crop size is taken as the size that includes all faces in the image. If the crop size or final required image size is provided beforehand, then the provided image dimensions are first stretched, while preserving the aspect ratio, to include all the faces in the image. The image is then cropped to the thus computed dimensions, resized to the demanded dimensions, and the final image is returned completing the process.


MTCNN gives us good performance both in terms of meeting the quality requirements and also in terms of run-time performance. Runtime performance was our overarching concern as the system had to perform in real time and is demand driven. Other than performance, types of expected variations actually determine the kind of network architecture we can select. For example, face centric cropping is one of our primary requirements and it drove the selection of our architecture. For more general retargeting use cases, DeepIR might be more appropriate.


  1. Mei Wang, Weihong Deng. Deep Face Recognition: A Survey.
  2. Peiyun Hu, Deva Ramanan. Finding Tiny Faces.
  3. Jianxin Lin, Tiankuang Zhou, Zhibo Chen. DeepIR: A Deep Semantics Driven Framework for Image Retargeting. 
  4. Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li. Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks.
  5. Danai Triantafyllidou, Anastasios Tefas. A Fast Deep Convolutional Neural Network for Face Detection in Big Visual Data.  Advances in Intelligent Systems and Computing · October 2017.


Ashish Kanchan