 Cardwell was born in Madison, North Carolina.His father, Richard Perrin Caldwell, died when he was an infant, and he had great difficulty in obtaining an education.As a youth, he attended public school and worked on the family farm in the summer and fall.