 Whether true or not, Hayward was often called California’s "first millionaire," and in his lifetime was frequently referred to as “the richest man in California.” Grandiose and eccentric by turns, Hayward turned even more strongly to Spiritualism in his later years, using mediums to predict business investments (with poor results).Hayward had a difficult relationship with his wife, Charity (1826-1905), whom he married in Wisconsin in the 1840s; their first son, James, was born in 1846, and would die of consumption at age 27.Divorced in 1878, the couple remarried again later that year only to separate again, living the rest of their lives apart.