 Robert Gray was married on 6 September 1836 to Sophy Myddleton, the daughter of county squire Richard Wharton Myddleton of Durham and Yorkshire, and founded Diocesan College, or Bishops as it is commonly known, in Rondebosch, Cape Town, in 1849.A notable part of his life concerned his feud with Colenso about Colenso's alleged heretical opinions.Gray and his wife looked after the children of Mgolombane Sandile, including his daughter Emma, aiming to provide influence over the Gaika people.