Very few patients refused to speak to him or had problems with privacy during the interview.
There were many patients who refused to speak to him.
There were not many patients who refused to speak to him.