 As bishop, Kemp invited Deacon William Levington to his diocese in 1824, and helped him establish St. James First African Episcopal Church, the third African American Episcopal church in the new country.Kemp died of injuries received in a stage coach accident near New Castle, Delaware, while returning from the consecration of Assistant Bishop Henry Onderdonk in Philadelphia.He was buried in the cemetery of Old St. Paul's Church in Baltimore.