Proponents of lowering immigration levels “are just interested in coming up with reasons not to have people who are different,” said Stuart Anderson, an adjunct scholar at the conservative Cato Institute who served as counselor to the commissioner of the Immigration and Naturalization Service under Bush.