 Milne was born in Braeside of Cults, a village few miles south to Huntly, in the rural parish of Kennethmont in Aberdeenshire, Scotland.His father died when he was only six years old (1791), and his mother taught him at home.While he was still very young, he worked on a farm for a period of time before being apprenticed to a carpenter under training of Adam Sievwright.