 In 1797, after eight years in the House of Representatives, James Madison retired from politics.He returned with his family to Montpelier, the Madison family plantation in Orange County, Virginia.There they expanded the house and settled in.