Sorting items into buckets assumes the items are stable objects, as if each one asked a fixed thing and collected a fixed response. Research on how surveys actually work complicates that assumption. The list of answer choices attached to a question is not a neutral menu. It is part of the question, and changing it changes the results.

Start with the largest version of this. A question can be closed-ended, offering a list of choices, or open-ended, letting respondents answer in their own words. Pew Research Center (2021) reports on an experiment after the 2008 presidential election in which the same question about which issue mattered most in the respondent’s vote was asked both ways. Given a list that included the economy, fifty-eight percent chose it. Asked to answer freely, only thirty-five percent said the economy on their own.

The second half of that result is the more interesting one. Among respondents who answered the open-ended version, forty-three percent gave a response that was not among the five options offered in the closed-ended version (Pew Research Center, 2021). Nearly half the group had something on their mind that the list did not contain. Those people were not counted as having that concern, because the closed-ended version gave them nowhere to put it.

This does not make closed-ended items inferior. Most survey research is closed-ended for practical reasons, and Pew Research Center (2021) notes that researchers sometimes run open-ended pilot studies first specifically to learn which answers are common, then build the list from what people actually said. The list is supposed to be built from evidence rather than guessed at. But a closed-ended item can only report on what its list allowed, and a reader who forgets that will mistake the boundaries of the list for the boundaries of what people think.

How the options are described matters as much as which ones exist. In a January 2002 Pew Research Center survey, half the sample was asked whether it was more important for the president to focus on domestic policy or foreign policy. Fifty-two percent chose domestic policy and thirty-four percent chose foreign policy. The other half received the same question with foreign policy narrowed to the war on terrorism. On that version, only thirty-three percent chose domestic policy and fifty-two percent chose the war on terrorism (Pew Research Center, 2021). Nothing about the respondents changed. One label got more specific, and the result reversed.

Even the order of the options leaves a mark. Pew Research Center (2021) reports that respondents on telephone surveys tend to pick items heard later in a list, a recency effect, while respondents filling out surveys themselves tend to pick items near the top, a primacy effect. Because of this, many Pew questions randomize the order of their response options, which does not remove the bias but spreads it evenly rather than letting it pile onto whichever option happens to sit first (Pew Research Center, 2021).

The number of options carries a limit too. Pew Research Center (2021) advises keeping the count small, around four or five, because people struggle to hold more than that in mind at once. Factual items such as religious affiliation are an exception and can use many more, since respondents are scanning a list for themselves rather than weighing choices against each other.

One more finding is worth carrying into any sorting work. There is a difference between an item that forces a single choice and an item that lets respondents check every option that applies. A 2019 Center study found that forced-choice items tend to produce more accurate responses, particularly on sensitive topics, and on the strength of that finding the Center generally avoids select-all-that-apply lists (Pew Research Center, 2021). A select-all item also breaks something simpler: its counts do not add up to the number of respondents, because one person can appear in several rows at once.

All of this bears directly on the work of sorting. An item cannot be placed in a bucket on the strength of its topic alone. The wording of the stem, the contents of the list, the specificity of the labels, and whether respondents could choose one thing or several all shape what the item is capable of showing. Two items on the same subject may belong in different buckets, or one of them may belong in no bucket at all, for reasons that live entirely in the answer choices.

Reading the exact wording is therefore not a formality performed before the real analysis. It is the step that determines whether the analysis is about people or about the questionnaire.