Washington (CNN) Chairman of the powerful House oversight committee Jason Chaffetz may not finish out his term in Congress, he said Thursday, a departure that would invite jockeying for his high-profile position.