OpenCV: Use findHomography() and warpPerspective to align a bigger image onto a smaller one
<p>My goal is to </p>

<ol>
<li>deskew a scanned image such that its text is perfectly placed on top of the text of the original image. (subtracting the images would remove the text)</li>
<li>prevent any loss of information on the deskewed image</li>
</ol>

<p>I use SURF features to feed the findHomography function. Then I use the warpPerspective function to transform the scanned image. The resulting image almost perfectly fits onto the original image.</p>

<p>However, the scanned image has content on its corners which is lost after the transformation because the text in the scanned image is smaller and has to be scaled up. </p>

<p><a href="https://i.stack.imgur.com/BpUov.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/BpUov.png" alt="Original and scanned image"></a>
<sup><i>Deskewing an image that has slightly smaller text</sup></i></p>

<p><a href="https://i.stack.imgur.com/R2u7G.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/R2u7G.png" alt="Transformed image lost content"></a></p>

<p><sup><i>Information at the borders of the image is cropped</sup></i></p>

<p>To avoid any loss of information, I convert the image to RGBA and set the borderValue parameter in warpPerspective such that any added background has transparent color. I remove the transparent pixels after the transformation again. This procedure works but seems highly inefficient.</p>

<ol>
<li>Question: I'm looking for a working code example (C++ or Python) that shows how to do this more efficiently.</li>
</ol>

<p><a href="https://i.stack.imgur.com/ViPbk.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/ViPbk.png" alt="Images aren&#39;t aligned because transformed image has different dimensions"></a>
<sup><i>Image has been deskewed and content is preserved. However, the text of the two pictures isn't on top of each other anymore</sup></i></p>

<p><a href="https://i.stack.imgur.com/xeG5l.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/xeG5l.png" alt="Text isn&#39;t aligned anymore"></a></p>

<p><sup><i>Text position is off because the warped image has a different size than what warpPerspective expected</sup></i></p>

<p>After transforming the image the problem is that the two images aren't aligned anymore because the dimensions of the transformed image are different than what the warpPerspective method expected.</p>

<ol start="2">
<li>Question: How can I realign the two images? It would be great if there was a way to do incorporate this into the previous step already. Again, a working code example would be very helpful.</li>
</ol>

<p>Here's the code that I have so far. It deskews the image while preserving its content, however, the text is not on top of the original text anymore.</p>

<pre><code>import math
import cv2
import numpy as np


class Deskewer:
    def __init__(self, hessianTreshold = 5000):
        self.__hessianThresh = hessianTreshold
        self.imgOrigGray, self.imgSkewed, self.imgSkewedGray = None, None, None

    def start(self, imgOrig, imgSkewed):
        self.imgOrigGray = cv2.cvtColor(imgOrig, cv2.COLOR_BGR2GRAY)
        self.imgSkewed = imgSkewed  # final transformation will be performed on color image
        self.imgSkewedGray = cv2.cvtColor(imgSkewed, cv2.COLOR_BGR2GRAY)  # prior calculation is faster on gray

        kp1, des1, kp2, des2 = self.__detectFeatures()
        goodMatches = self.__flannMatch(des1, des2)

        MIN_MATCH_COUNT = 10
        M = None
        if len(goodMatches) &gt; MIN_MATCH_COUNT:
            M, _ = self.__findHomography(kp1, kp2, goodMatches)
        else:
            print("Not  enough  matches are found   -   %d/%d" % (len(goodMatches), MIN_MATCH_COUNT))
            return

        return self.__deskew(M)


    def __detectFeatures(self):
        surf = cv2.xfeatures2d.SURF_create(self.__hessianThresh)
        kp1, des1 = surf.detectAndCompute(self.imgOrigGray, None)
        kp2, des2 = surf.detectAndCompute(self.imgSkewedGray, None)

        return kp1, des1, kp2, des2

    def __flannMatch(self, des1, des2):
        global matches
        FLANN_INDEX_KDTREE = 0
        index_params = dict(algorithm=FLANN_INDEX_KDTREE, trees=5)
        search_params = dict(checks=50)
        flann = cv2.FlannBasedMatcher(index_params, search_params)
        matches = flann.knnMatch(des1, des2, k=2)

        # store all the good matches as per Lowe's ratio test.
        good = []
        for m, n in matches:
            if m.distance &lt; 0.7 * n.distance:
                good.append(m)

        return good

    def __findHomography(self, kp1, kp2, goodMatches):
        src_pts = np.float32([kp1[m.queryIdx].pt for m in goodMatches
                              ]).reshape(-1, 1, 2)
        dst_pts = np.float32([kp2[m.trainIdx].pt for m in goodMatches
                              ]).reshape(-1, 1, 2)

        M, mask = cv2.findHomography(src_pts, dst_pts, cv2.RANSAC, 5.0)
        matchesMask = mask.ravel().tolist()
        i = matchesMask.index(1)

        # TODO: This is a matching point before the warpPerspective call. How can I calculate this point AFTER the call?
        print("POINTS: object(", src_pts[i][0][1], ",", src_pts[i][0][0], ") - scene(", dst_pts[i][0][1], ",", dst_pts[i][0][0], ")")

        return M, mask

    def getComponents(self, M):
        # ((translationx, translationy), rotation, (scalex, scaley), shear)
        a = M[0, 0]
        b = M[0, 1]
        c = M[0, 2]
        d = M[1, 0]
        e = M[1, 1]
        f = M[1, 2]

        p = math.sqrt(a * a + b * b)
        r = (a * e - b * d) / (p)
        q = (a * d + b * e) / (a * e - b * d)

        translation = (c, f)
        scale = (p, r)  # p = x-Axis, r = y-Axis
        shear = q
        theta = math.atan2(b, a)
        degrees = math.atan2(b, a) * 180 / math.pi

        return (translation, theta, degrees, scale, shear)

    def __deskew(self, M):
        # this info might come in handy here for calculating the dsize of warpPerspective?
        translation, theta, degrees, scale, shear = self.getComponents(M)

        # Alpha channel allows me to set unique feature to pixels that are created during warpPerspective
        imSkewedAlpha = cv2.cvtColor(self.imgSkewed, cv2.COLOR_BGR2BGRA)

        # These sizes have been randomly choosen to make sure that all the contents fit in the new canvas
        height = 5000
        width = 5000
        shift = -500

        M2 = np.array([[1, 0, shift],
                      [0, 1, shift],
                      [0, 0, 1]])
        M3 = np.dot(M, M2)

        # TODO: How can I calculate the dsize argument?
        # Newly created pixels are set to transparent
        im_out = cv2.warpPerspective(imSkewedAlpha, M3,
                                     (height, width), flags=cv2.WARP_INVERSE_MAP, borderMode=cv2.BORDER_CONSTANT, borderValue=(255, 0, 0, 0))

        # http://codereview.stackexchange.com/a/132933
        # Mask of non-black pixels (assuming image has a single channel).
        mask = im_out[:, :, 3] == 255

        # Coordinates of non-black pixels.
        coords = np.argwhere(mask)

        # Bounding box of non-black pixels.
        x0, y0 = coords.min(axis=0)
        x1, y1 = coords.max(axis=0) + 1  # slices are exclusive at the top

        # Get the contents of the bounding box.
        cropped = im_out[x0:x1, y0:y1]

        # TODO: The warped image needs to align nicely on the original image
        return cropped

origImg = cv2.imread("Letter.png")
skewedImg = cv2.imread("A4.png")
deskewed = Deskewer().start(origImg, skewedImg)
cv2.imshow("Original", origImg)
cv2.imshow("Deskewed", deskewed)
cv2.waitKey(0)
</code></pre>

<p><a href="https://i.stack.imgur.com/JH91Zt.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/JH91Zt.png" alt="Original image for testing"></a>
<a href="https://i.stack.imgur.com/RZ12Rt.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/RZ12Rt.png" alt="Skewed image for testing"></a>
<sup><i>Original and skewed image (with additional content) for testing</sup></i></p>