Principles of Questionnaire Construction

Question-Writing

Target the vocabulary and grammar to the population be surveyed.

• For studies within a specific organization, use the jargon used in that organization.
• Be careful to avoid language that is familiar to you, but might not be to your respondents. Avoid unnecessary abbreviations.

Avoid ambiguity, confusion, and vagueness.

• Make sure it is absolutely clear what you are asking and how you want it answered. For example, if you just ask "What is your income?" The respondent doesn't know whether you mean weekly or monthly or annual, pretax or aftertax, household or individual, this year or last year, from salary only or including dividends, interest, etc.
• Avoid indefinite words or response categories. For example, "Do you jog regularly?" What does "regularly" mean?

Avoid emotional language, prestige bias and leading questions

• Watch out for loaded words that have a history of being attached to extreme situations. For example, avoid questions like "What should be done about murderous terrorists who threaten the freedom of good citizens and the safety of our children?"
• Watch for prestige markers that cue the respondent to give the "right" answer. For example, the question "Most doctors say that cigarette smoke causes lung disease for those near a smoker. Do you agree", tends to provoke "yes" answers because people trust doctors. Similarly, "Do you support the president's policy on Zimbobutu?" provokes "yes" answers from people who have never even heard of Zimbobutu (and, in fact, I made the name up -- yet lot's of people will say "yes" to this question).
• Avoid leading questions like "You don't smoke, do you?" or "I assume you would agree that the teachers do a heroic job for our children".
• Avoid loading questions with extra adjectives and adverbs, like "Should the mayor spend even more tax money trying to keep the streets in top shape?"

Avoid double-barrelled questions

• Make each question about one and only one topic. For example, don't ask "Does your company have pension and health insurance benefits?" because if their company has only one of those benefits, it is unclear whether the respondent will say "yes" or "no".

Don't assume the respondent is an expert on themselves (unless you have no choice)

• Suppose you want to test the idea that students give better evaluations to teachers who tell a lot of jokes in class. The wrong way to investigate this is to ask "Do you rate a teacher higher if the teacher tells many jokes?" because this assumes that the student is completely conscious of everything they do and why. The right way is to ask the student two separate questions: "How would you rate the following teacher?" and "How many jokes does the teacher tell in class?" (even better is to count the jokes yourself rather than rely on the student's estimate). Then statistically correlate the answers, to see if students that have teachers that tell many jokes also tend to rate them highly.

Avoid asking questions beyond a respondent's capabilities

• People have cognitive limitations, especially when it comes to memory of past events. Asking "how did you feel about your brother when you were six years old" is probably useless.
• It is pointless to ask people about things that are not natural ways for them to think. For example, don't bother asking "How many gallons of gasoline did you buy for your car last year?".

Avoid false premises

• Asking "What is the most important thing we should do stop the economy from deteriorating any further?" assumes that the economy is deteriorating, which the respondent may not agree with. This puts the respondent in a tough spot. It would be better to rephrase as "What is the most important thing a government can do to strengthen its economy".

Avoid asking about future intentions (if you can)

• Hypothetical questions like "If a new grocery store were to open down the street, would you shop there?" are notoriously unrelated to actual future behavior.

Avoid negatives and especially double negatives

• Negatives like "Students should not be required to take a comprehensive exam to graduate" are often difficult for many respondents to process, especially if they agree with the predicate, because then they are disagreeing with not doing something, which is confusing!
• Double negatives like "It is not a good idea to not turn in homework on time" yield very unreliable data because people are unsure about whether to put a "yes" or "no" even if it is clear in their minds whether turning homework in on time is a good idea.

Question Placement

• It's a good idea to put difficult, embarassing or threatening questions towards the end of the interview when the interviewee has gotten more comfortable. This has two benefits. First, it makes them more likely to answer, and, second, if they get mad and leave, at least you've gotten most of your questions asked!
• Put related questions together to avoid giving the impression of lack of meticulousness
• Watch out for questions that influence the answers to other questions. For example:
  1. Do you think the US should let Communist newspaper reporters come in and send back to their papers the news as they see it?
  2. Do you think a Communist country should let American newspaper reporters come in and send back to their papers the news as they see it?

Filtering "Don't Know"

There are three ways of dealing with "don't know".

Standard format. No "don't know" option is presented to the respondent, but is recorded if the respondent volunteers it.

Quasi filter. A "don't know" option is included among the possible responses.

Full filter. First the respondent is asked if they have an opinion. Then, if yes, they ask the question.

Example:

Standard format. Here are some questions about other countries. Do you agree or disagree with each statement?

1. The Russian leaders are basically trying to get along with America.

Quasi filter. Here is a statement about another country. "The Russian leaders are basically trying to get along with America." Do you agree, disagree or have no opinion on that?

Full filter. Here is a statement about another country. Not everyone has an opinion on this. If you do not have an opinion, just say so. Here's the statement:

1. The Russian leaders are basically trying to get along with America.

Do you have an opinion on that? [If "YES"] Do you agree or disagree?

Results:

Open-ended versus Closed-ended Questions

An openended question is one in which you do not provide any standard answers to choose from. For example, these are all open-ended questions:

1. How old are you? ______ years.
   
   
2. What do you like best about your job?
   ____________________________________________________________
   ____________________________________________________________
   ____________________________________________________________
   ____________________________________________________________
   

A closed-ended question is one in which you provide the response categories, and the respondent just chooses one:

1. How old are you?
   (a) 12 - 15 years old
   (b) 16 - 25 years old
   (c) 26 - 35 years old
   (d) 36 - 45 years old
   (e) practically dead
   
   
2. What do you like best about your job?
   (a) The people
   (b) The diversity of skills you need to do it
   (c) The pay and/or benefits
   (d) Other: ______________________________ (write in)

There are lot of reasons for choosing one form over the other. Here are some of the issues:

Closed-Ended Questions

Watch out for overlapping response categories

This question:

1. What is your annual household income?
   a. Less than $10,000
   b. $10,000 to $25,000
   c. $25,000 to $35,000
   d. $35,000 to $50,000
   e. $50,000 to $75,000
   f. More than $75,000 

If a person's income is exactly $25,000, which category do they use?

The Ratings Format

In this format, we ask people to answer questions like this:

1. In your opinion, how liberal is your mother?
   1. Not at all liberal
   2. Somewhat liberal
   3. Very liberal
   4. She's a stinking communist!
   
   
2. Children must be allowed to make their own mistakes.
   1. Disagree strongly
   2. Disagree somewhat
   3. Neither agree nor disagree
   4. Agree somewhat
   5. Strongly agree
   

The result of a rating question is an ordinal-level variable (which is often treated as interval in data analysis).

Odd or even number of steps in the response scale?

In general, it is suggested that you use an odd number of steps in order to allow the respondent to express a middling or neutral strength of opinion. This can be problem with some respondents who refuse to express an opinion and give the middle category for all questions. However, the alternative can be even worse: respondents who have no or neutral opinion being forced to choose negative or positive and doing it randomly.

How many steps in the response scale?

Statistical reliability of the data increases sharply with the number of scale steps up to about 7 steps, then it increases more slowly, leveling off around 11. After 20 steps it decreases sharply. However, the more steps you have, the more difficult it is for the respondent, and possibly the less valid the responses because of that.

If variables are going to be combined additively, like when you create a scale or index, then the number of steps is not an issue for reliability. You can use two steps (true/false) if you like. I like to use 3-point response scales because they are quick and easy for respondents.

Direct Magnitude Scaling

Magnitude scaling is a method of obtaining ratio-scaled data from informants. The basic idea is to give respondents an anchor point, and then ask them to answer questions relative to that anchor point. For example, suppose you are interested in the severity of crimes. Start by assigning a number to one crime. For example, take "felony" and tell the respondent that the severity of a felony is 100 units. Now say: "ok, if felony is 100 points, how much is "murder"? If you think murder is 10 times as bad as a felony, then write down 1000. If its twice as bad as a felony, write down 200. Then you ask about every other crime.

Rank-Ordering

Rank ordering is a method that works well with a small number of objects, such as 10. For example, instead of rating how serious each of a set of organizational problems are, you could ask the respondent to simply sort them in order of most to least serious.

Paired-Comparisons

In this method, you present items two at a time, and ask respondent to pick which one has more of some attribute. For example, you can present organizational problems and ask which one is more serious.