 Broughton did not marry, and, in 1961, a second barony was conferred on him, with the remainder to his brother, Henry Rogers Broughton.When Broughton died in August 1966 obituaries in The Times and The New York Times said he was known as "the shy peer" because he disliked publicity.Anglesey Abbey was left to the National Trust, with the stipulation that the house and collection should remain exactly as it was at his death.