This question is purely theoretical as I've not been able to find anyone who's done such a thing, so my question is - is it possible, and if not why?

Say you have a short video (CCTV) of a face, or number (car licence) plate, and that video is of too poor quality to discern who that person is or what the number plate says, would it be possible to combine details from several frames of video to interpolate and thus make a clearer image than can be gained from a single frame?

Surely some frames would contain detail that others lack (due to pixel boundaries, shadows etc)? Are there any examples of machine learning or CV that do this?

  • $\begingroup$ en.wikipedia.org/wiki/Super-resolution_imaging $\endgroup$ – D.W. Aug 17 '17 at 6:50
  • $\begingroup$ Thanks - it's much easier to find out information when you know what the technique (super resolution) is called! Given the linked research paper (arxiv.org/pdf/1603.08155.pdf) hopefully it won't be long until we'll see it on things like Crimewatch (a TV program in the UK) $\endgroup$ – Rich Aug 17 '17 at 8:10

It heavily depends on the setting, accuracy and goal, but shortly, yes, even considering practical application.

Theoretically of course it is possible, with various (or used together) techniques like sharpening, noise removal, scene reconstruction, frames merging, geometry reconstruction or even motion blur removal. The last one works by solving optic equation and could be used with success if you treat consecutive frames as one speeding object, applying frames as artificial motion blur, for example Practical Layered Reconstruction for Defocus and Motion Blur.
The most straightforward technique would use shifted grids (frames) and after applying constraints from "big pixels" and how they round at the discrete domain the result is in the "reconstruction domain" - all pixels represent ranges of possible chrominance / luminance. The accuracy increases with the number of frames used. Here are two problems: the additional processing is needed to estimate the repative position of frames (which in the case of static camera gets easier - the equivalent of motion capture of smart objects) and the number of frames. If these came from different cameras (angles, coordinates, intrisinc camera setting) the scene reconstruction with possibly light oblivious merge (the light condition may vary, it must be equalized).

As stated earlier it depends, the plate recognition is easier, it reduces to contours extraction of low resolution data / blurred setting or image reconstruction discarding colours (which are known and not interesting), with some degree of certainty - the numbers under noise should be matched with patterns.

For the face recognition it gets harder due to unknown pattern, possibly 3D reconstruction from frames, which are low resolution, and hopefully with full colour after processing. This is a part of active research, because many powerful algorithms yield grayscale images.

Generating perceptualy clearer images is quite established field, heavily dominated by SSIM derivative techniques (the theoretical-empirical model for images compression, that could be fed with "reconstruction domain" image and return the most plaussible one), but reconstruction beyond doubt is harder task.

Here are some good to read articles about it:
Super resolution algorithm

An Algorithm for Repairing Low-Quality Video Enhancement Techniques Based on Trained Filter

Video Superresolution Reconstruction Using Iterative Back Projection with Critical-Point Filters Based Image Matching



Your Answer

