I think if most Americans knew what the origins of the Electoral College is, they would be disgusted.” Madison, now known as the “Father of the Constitution,” was a slave-owner in Virginia, which at the time was the most populous of the 13 states if the count included slaves, who comprised about 40 percent of its population.