Missing data and who is left out of a count

A count describes the people, events or items that made it into the data. Anyone or anything missing from that list sits outside the total, and the number can be off in a direction you cannot see from the figure alone. Before trusting a count, ask who or what could have been included and was not, and whether the missing cases look like the ones that were counted.

Missing cases are not always random. A survey may reach people who are easy to contact, a hospital register may include only patients who came in for care, and a customer database may show only people who stayed long enough to create an account. Each gap can make a result look better, worse or simply different from the wider reality. The first skill in reading any count is noticing where the gap is.

Nonresponse: who did not answer

When a survey is sent to a group, some people will not reply. Nonresponse matters when the people who reply differ from those who do not in a way that affects the answer. People with strong opinions, free time or an interest in the topic may be overrepresented, while people who are busy, hard to reach or less engaged may be underrepresented.

To check, look for the response rate and for a description of how it was calculated. A low rate does not make a survey useless, but it raises the question of whether non-respondents would have answered differently. Look for whether the researchers compared respondents with known characteristics of the wider group, such as age or region, and whether they adjusted the results for differences. If none of this is reported, read the findings as describing the people who answered, not the whole population.

Survivorship: the cases that did not last

Survivorship bias happens when only the cases that persisted get counted. A well-known wartime analysis of returning aircraft illustrates the idea. The first instinct was to add armor where the returning planes showed the most bullet holes. The better reasoning was that the planes hit in other places had never made it back, so the areas with few holes on returning planes were the spots where a hit was most likely to bring a plane down.

The same pattern appears in business and personal finance. Lists of companies that performed well over a decade often omit those that closed or merged along the way, which makes the surviving group look stronger than the full set of starting companies. To spot it, ask whether a failing case could have appeared in the count at all. If the method made it impossible for losers to show up, the successes are probably overstated.

Records that capture only some cases

Many counts come from records created for another purpose. Emergency call logs, sales receipts, admission lists and school enrollment files each capture only a subset of what they claim to measure. A count of reported incidents depends on who chooses to report, and a count of sales may exclude returns or unpaid orders. The record is accurate about what it captured, but its total describes only that subset.

Ask what triggers an entry. Who decides to record it, and what makes that more or less likely for different people or situations? A rise in recorded cases can reflect more reporting rather than more events, especially after a campaign encouraging people to come forward. Where possible, compare the count with an independent source. If the two disagree by a large margin, the difference often reflects the missing cases, and that difference is worth explaining before drawing conclusions.

Judging how much the gap matters

A gap does not always invalidate a number. The useful question is whether the missing cases would change the answer. If they are similar to the included cases, the total may still be a fair guide. If they differ in ways that bear on the question, the total may point the wrong way. A count of customers satisfied with a service, collected only from people who completed a purchase, cannot say much about people who abandoned the purchase halfway through.

When you cannot tell, look for a sensitivity check. Imagine the missing cases had the most extreme plausible values, and see whether the conclusion still holds. If it does, the gap is less worrying. If a modest change in the missing group would reverse the story, the figure is too fragile to repeat without that caveat.

Stating what the count covers

When you pass a count on, state its scope honestly. "Among people who answered the survey" is a narrower claim than "among adults," and it is often the most accurate description available. Comparisons can still be useful when both groups were measured with the same gap, provided the gap affects them in similar ways.

A simple habit helps. Before repeating a figure, write one sentence describing what it includes and one describing what it leaves out. If you cannot write the second sentence, you have probably not yet thought about the gap. That sentence is often the most valuable part of any number you share, because it tells the next reader what the total can and cannot support.