How can I sort contours from left to right and top to bottom?
<p>I am trying to build an character recognition program using Python. I am stuck on sorting the contours. I am using <a href="https://github.com/guard0g/HandwritingRecognition/blob/master/Handwriting%20Recognition%20Workbook.ipynb" rel="noreferrer">this page</a> as a reference. </p>

<p>I managed to find the contours using the following piece of code:</p>

<pre><code>mo_image = di_image.copy()
contour0 = cv2.findContours(mo_image.copy(),cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_SIMPLE)
contours = [cv2.approxPolyDP(cnt,3,True) for cnt in contour0[0]]
</code></pre>

<p>And added the bounding rectangles and segmented the image using this part of the code: </p>

<pre><code>maxArea = 0
rect=[]
for ctr in contours:
    maxArea = max(maxArea,cv2.contourArea(ctr))

if img == "Food.jpg":
    areaRatio = 0.05
elif img == "Plate.jpg":
    areaRatio = 0.5

for ctr in contours:    
    if cv2.contourArea(ctr) &gt; maxArea * areaRatio: 
        rect.append(cv2.boundingRect(cv2.approxPolyDP(ctr,1,True)))

symbols=[]
for i in rect:
    x = i[0]
    y = i[1]
    w = i[2]
    h = i[3]
    p1 = (x,y)
    p2 = (x+w,y+h)
    cv2.rectangle(mo_image,p1,p2,255,2)
    image = cv2.resize(mo_image[y:y+h,x:x+w],(32,32))
    symbols.append(image.reshape(1024,).astype("uint8"))

testset_data = np.array(symbols)

cv2.imshow("segmented",mo_image)
plt.subplot(2,3,6)
plt.title("Segmented")
plt.imshow(mo_image,'gray')
plt.xticks([]),plt.yticks([]);
</code></pre>

<p>However the resulting segments appear in to be in random order.
Here is the original image followed by the processed image with detected segments.</p>

<p><a href="https://i.stack.imgur.com/TCZ1H.jpg" rel="noreferrer"><img src="https://i.stack.imgur.com/TCZ1H.jpg" alt="segments"></a></p>

<p>The program then outputs each segment separately, however it is in the order: <code>4 1 9 8 7 5 3 2 0 6</code> and not <code>0 1 2 3 4 5 6 7 8 9</code>. 
Simply adding a sort operation in "rect" fixes this, but the same solution wont work for a document with multiple lines.</p>

<p>So my question is: How do I sort the contours from left to right and top to bottom?</p>