 Palmer attended Wake Forest College on a golf scholarship.He left upon the death of close friend Bud Worsham (1929–1950) and enlisted in the U.S. Coast Guard, where he served for three years, 1951–1954.At the Coast Guard Training Center in Cape May, New Jersey, he built a nine-hole course and had some time to continue to hone his golf skills.