 After losing the election, Ellsworth was appointed by President Eisenhower to a two-year term as chairman of the United States Civil Service Commission, serving until February 28, 1959.Ellsworth left politics and resumed his newspaper business and became a real estate broker.He retired to Albuquerque, New Mexico in the 1970s, where he lived until his death on February 7, 1986.