 James Rouse's first wife was Elizabeth Jamieson "Libby" (née Winstead) whom he married on May 3, 1941.His daughter Robin is the mother of actor Edward Norton.His son Jim applied for conscientious objector status during the Vietnam War with his father's support.