A survey answer is a statement a person made, not a measurement someone took. That distinction is easy to state and hard to hold onto, because a topline turns every statement into a number, and numbers look like measurements. Researchers who study survey methods have spent decades documenting what happens in the gap between what respondents say and what is true, and the findings are consistent enough to be predictable.

The most studied of these problems has a name. Pew Research Center (2021) describes social desirability bias as the tendency of people to give inaccurate answers on sensitive subjects because they want to be liked and accepted. What makes it useful to know about is that it does not push answers around randomly. It pushes them in a direction that can be guessed in advance. Research summarized by the Center has found that respondents understate their alcohol and drug use, their tax evasion, and their racial bias, while overstating their church attendance, their charitable giving, and how likely they are to vote (Pew Research Center, 2021).

Notice the pattern. Behavior that is admired gets reported upward. Behavior that would embarrass gets reported downward. Nobody has to be lying on purpose for this to happen, which is why telling respondents to be honest does not solve it.

The effect also changes size depending on how the survey is delivered. Pew Research Center (2021) reports that social desirability bias tends to be stronger when an interviewer is present, as on a telephone or face-to-face survey, than when respondents fill out the survey themselves on paper or online. A person answering alone on a screen shades their answers less than a person answering out loud to a stranger. The question is identical. The conditions are not.

Researchers respond to this by changing how questions are built rather than by asking harder. When Pew Research Center asks whether someone voted in a past election, the question is written to make not voting easy to admit, offering the possibility that something came up and prevented it (Pew Research Center, 2021). Giving a respondent an acceptable way to report an unflattering answer is a design choice, and it changes the resulting numbers.

That leads to the second finding, which is larger than the first. The wording of a question is not a neutral container for the topic. It is part of what gets measured.

The clearest demonstrations come from experiments where the same group is split in half and each half receives a slightly different version of the same question. In a January 2003 Pew Research Center survey, respondents were asked whether they favored or opposed military action in Iraq to end Saddam Hussein’s rule. Sixty-eight percent said they favored it and twenty-five percent said they opposed it. A second version of the question added one clause, noting that the action might cost thousands of American casualties. On that version, forty-three percent favored it and forty-eight percent opposed (Pew Research Center, 2021). The topic did not change. The majority flipped.

The same effect shows up in a single word. Pew Research Center (2021) reports that respondents react differently to questions using the word welfare than to questions using the more general phrase assistance to the poor, and that experiments have repeatedly found much greater public support for expanding assistance to the poor than for expanding welfare. The policy under discussion is the same in both versions. One label carries associations the other does not, and support moves with the label rather than with the policy.

None of this means survey answers are useless. It means the item and the answer belong together. A number reported without the question that produced it is not a finding. It is a rumor with a decimal point.

Two habits follow from that. The first is to read the exact wording of the item before treating its numbers as evidence of anything, because a difference in results may be a difference in questions rather than a difference in people. The second is to ask, for any item, whether the topic is one where a respondent might want to look a certain way. That is not a reason to throw the item out. It is a reason to expect the error to run in a particular direction, and to say so when reporting what the item found.

Pew Research Center publishes the exact wording and response options for every question alongside its results, in what it calls a topline questionnaire (Pew Research Center, 2021). That practice exists because the wording is not background information. It is the evidence.