 Although according to family genealogy, the first Broyhill emigrated to Halifax County, Virginia in the 18th century, Joel Broyhill's grandfather, Thomas Jefferson Broyhill (1852-1935) had been born near Moravian Falls in Wilkes County, North Carolina, the son of William Broyhill, a farmer and miller who also taught school following the Civil War.Thomas Broyhill became a carpenter and millwright, then leading citizen of the area as he established sawmills and other businesses.Joel's father Marvin Talmadge Broyhill Sr. moved his family to Hopewell, Virginia to follow an uncle who worked for the DuPont Powder Company.