A survey looks simple from the outside. Someone asks a group of people a set of questions, collects the answers, and reports what the group said. The reporting is where the difficulty starts, because a pile of answers is not yet information. Turning one into the other requires several small decisions, and every one of them can be made badly.

Start with the pieces. A respondent is one person who answered. An item is one question on the survey. If forty people answer a survey with twenty questions, there are forty respondents and twenty items, and the survey has collected eight hundred separate answers. Those two words get confused constantly, including by adults who should know better, because both of them can be counted and the counts sound similar.

The first thing a researcher builds from those answers is a topline. A topline lists every item, every answer a respondent could have picked, and how many respondents picked each one. It contains no conclusions and no opinions. It is the raw shape of what the group said, and it is deliberately boring. A researcher who skips the topline and jumps straight to conclusions has no way to show anyone else where those conclusions came from.

Every item on a topline carries a number written as n. The n is how many respondents the item is based on. This matters more than it looks like it should. If a survey has forty respondents but six of them skipped a particular question, then that item rests on thirty-four people, not forty. A careful topline shows the skipped answers rather than hiding them, because a question that many people refused to answer is telling you something. Sometimes the refusal is the finding.

Now the harder problem. Almost everything a survey collects is self-report, meaning the only source for the answer is the person giving it. If a survey asks how many hours you slept last night, nobody checked. If it asks how often you argue with your brother, nobody was watching. The answer goes into the topline looking exactly like a measurement, but it is not a measurement. It is a statement.

Self-report is not worthless. For many questions there is no better instrument available, and people are often roughly right about themselves. But self-report has a specific and predictable weakness: people are inconsistent in the direction of their errors. On questions about behavior that carries any social weight, respondents shade their answers toward whatever seems acceptable. They round their exercise up and their screen time down. They do it without deciding to, which is why simply asking them to be honest does not fix it.

There is a second kind of question that looks like self-report and is not. When a survey asks a respondent to estimate what other people do, the answer is a guess about a group. Guesses about groups fail differently than statements about yourself. People tend to assume that whatever they notice most is what happens most, so a respondent who has seen one dramatic example will estimate high. Neither kind of answer is more trustworthy than the other. They are simply wrong in different directions, and knowing which kind you are holding changes how much weight it can carry.

That brings up the property that separates an interesting item from a dull one. Consider two questions asked of the same group. The first asks whether respondents have ever ridden in a car. Nearly all of them say yes. The second asks how many hours they slept last night, and the answers run from four to eleven. The second item has variability, meaning the answers spread out instead of clustering. The first item has almost none.

An item with no variability can still be true and still be useful. It just cannot explain any difference between the people in the group, because on that item there is no difference to explain. If everyone answers the same way, the item cannot tell you why one respondent does something and another does not. Researchers care about variability for this reason. It is where the differences live, and differences are what most questions are actually about.

None of this requires advanced mathematics. It requires reading a topline carefully, noticing which items rest on how many people, keeping track of whether an answer is a statement about the respondent or a guess about somebody else, and paying attention to whether the answers spread out or pile up. Those four habits do most of the work. The mistakes that follow from skipping them are not subtle ones, and they are made constantly by people who never learned that a survey answer is a claim someone made rather than a fact someone verified.