Basic Survey Design in Software Testing

Creating, administering, and interpreting a survey is much, much trickier than you might expect. (In fact I completely mangled an argument in an earlier version of this blog entry.) Let me describe some pitfalls when you want to use the simplest type of survey that has a multiple choice format. Imagine you want to ask end users how the user interface of a new software application compares with the user interface of a current system. So, you create a survey which has instructions for respondents to select the statement which best describes the extent to which they agree or disagree to each statement from a set of 100 statements. For example, one of your statements might be, "The search feature of the user interface of the new system is easier to use than the search feature of the current system." And you give survey respondents the four choices, "Strongly Disagree", "Disagree", "Agree", "Strongly Agree" to each statement. Well, let me suggest that you may have already committed several mistakes in survey design. This is an example of a Likert scale design. The first issue is that with a Likert design you should as a rule of thumb, except in rare cases, have five responses plus an explicit n/a response. The five responses are generally some closely related form of, "Strongly Disagree", "Disagree", Neutral", "Agree", "Strongly Agree", and then slightly physically to the right of these first five responses, you should have a sixth "Not Applicable" response. You should give survey respondents a neutral option because otherwise you are forcing a positive or negative opinion when they may be neutral on a statement. And you should give respondents an explicit not-applicable choice rather than assuming or guessing that no response at all means not applicable in some way (as opposed to an invalid response because the respondent just forgot to answer). A second, almost certain mistake is that your survey has too many (100) statements. Survey respondents are going to get bored and quickly launch into an auto-complete mode just to finish your survey.

These are just a few of the dozens of issues with survey design. Analyzing the results of surveys is also very difficult. The moral of the story is that survey design is a very tricky task and you cannot simply use common sense. When I was a university professor, I taught an entire semester class on survey design; this is probably the bare minimum knowledge you need to create and interpret surveys in a software testing environment.


2 Responses to Basic Survey Design in Software Testing

  1. Unknown says:

    Is that even the question that you want to ask? How about:1. Is the new interface better than the old one? Yes/No2. What is better?3. What is worse?The point of question #1 is to get people thinking about the answers for #2 and #3. Ideally, a respondent would get stuck at #1 and spend a few minutes thinking about the answer, then be greatly relieved to see a chance to explain herself with #2 and #3. The answers to #2 go to into the bullet list on the feature sheet, and those to #3 go back to the developers. You don\’t even care about the answers to #1.

  2. James says:

    Shantirao, Yes, I agree with you. I was just using the UI design question as an example. But ultimately you probably want to know what is better or worse about a UI design, and your suggestions would get at this.

Comments are closed.