 He was the eldest son of Malcolm Fleming, 3rd Lord Fleming, Lord High Chamberlain, and Johanna or Jonet Stewart, natural daughter of James IV.Malcolm Fleming, who had been taken prisoner at the battle of Solway Moss in 1542, and had been tried and acquitted of treason in 1545 for his connection with the English party, was slain at the battle of Pinkie 10 September 1547.In August 1548, the young new Lord Fleming, along with Lord Erskine, accompanied Queen Mary to France, and his mother was appointed her governess.