 He was the son of William Milner (b.1662) Mayor of Leeds and his wife Mary, née Ibbetson.The second Baronet was Receiver-General of Excise and High Sheriff of Yorkshire.The third and fifth Baronets both represented York in the House of Commons, while the fourth served as High Sheriff.