 Ashley was the son of Evelyn Ashley, second surviving son of the social reformer Anthony Ashley-Cooper, 7th Earl of Shaftesbury.His mother was Sybella Charlotte Farquhar, daughter of Sir Walter Farquhar, 3rd Baronet.William Cowper-Temple, 1st Baron Mount Temple, was his great-uncle.