 Ashley was sworn of the Privy Council in February 1924 and when the Conservatives returned to power under Baldwin in November of that year he was made Minister for Transport, an office he retained until the fall of the Baldwin administration in 1929.He left the House of Commons in 1932 and was raised to the peerage as Baron Mount Temple, of Lee in the County of Southampton, a revival of the title held by his great-uncle.Lord Mount Temple remained active within the House of Lords and was a vocal supporter of the policy of appeasement towards Nazi Germany.