 It was substantially reorganized in 1776 and 1777, with land swapped with both Stafford and Westmoreland Counties to form the modern boundaries.On March 16, 1751, James Madison, the fourth President of the United States, was born in Port Conway in southern King George County at Belle Grove plantation, the childhood home of his mother, Eleanor Rose "Nellie" Conway, the daughter of its owner, Francis Conway, for whom Port Conway was named.Two-time Governor of Virginia, William "Extra Billy" Smith, was born at Marengo in 1797.