 Nabrit III was born in Houston, Texas to James Nabrit, Jr., a prominent civil rights attorney, law professor and later President of Howard University.He grew up in Washington, D.C., where he attended segregated public schools through part of high school.He finished high school at the Mount Hermon School for Boys, now Northfield Mount Hermon, in Massachusetts.