 His eldest brother, Sir Ralph, died before inheriting the title and his other two elder brothers died without issue.He married Anne of Gloucester as her second husband under special licence, as she was the widow of his brother Thomas Stafford, 3rd Earl of Stafford who had died prior to the consummation of his marriage at the age of 18.Edmund and his brothers were ward of the Gloucester family.