 His older brothers were Henry Pelham, 3rd Earl of Chichester and Frederick Thomas Pelham, a rear-admiral in the Royal Navy.Pelham was educated at Westminster School and went then to Christ Church, Oxford, where he graduated with a Bachelor of Arts in 1832 and Master of Arts four years thereafter.In 1857, he received a Doctor of Divinity by the University of Oxford.