 Hervey was the fourth son of Frederick Hervey, 1st Marquess of Bristol, by Elizabeth Albana Upton, daughter of Clotworthy Upton, 1st Baron Templetown.His paternal grandfather was Frederick Hervey, 4th Earl of Bristol, the Bishop of Derry.He was born at his father's London house, 6 St James's Square, on 20 August 1808.