 He was born the fourteenth son of Robert Shirley, 1st Earl Ferrers (and fourth son by his second wife, Selina).Robert Shirley, MP was his brother.Shirley was a noted rake, including amongst his lovers the notorious Lady Vane.