 He was the son of James Douglas of Orchardton (1747–1821), a Glasgow merchant, and Elizabeth (née Douglas) Stevenson Douglas, who had previously been married to Captain William Stevenson, a distant cousin of his father.Among his siblings were sisters Mary Douglas (wife of William Rose Robinson) and Matilda Douglas (wife of William Maitland).His paternal grandparents were John Douglas, a Galloway farmer, and Mary (née Heron) Douglas and his maternal grandfather was William Douglas of Worcester.