By Dr. Lee M. Miringoff
Next time you hear a media report on a public poll, who’s ahead in an election or the approval rating of an elected official, you’re also likely to be told about the poll’s so-called margin of error. Don’t jump to any hasty conclusions about some mistake that was made in conducting the poll. There’s nothing really wrong with the margin of error. Instead, it’s an acceptable range that underscores why all polls are estimates.
If President Obama’s approval rating is reported as 46%, plus or minus 3%, that means if everyone in the population, not just 1,000 Americans, had been interviewed, the actual result would have fallen somewhere between 49% (46% plus 3%) and 43% (46% minus 3%).
The margin of error is a statistical calculation based upon the number of successfully completed interviews. It’s part and parcel of all scientifically conducted public opinion research. The more people you interview, the lower the margin of error; the fewer interviews, the range widens, and the poll results are less precise. But, it’s not an error, and it’s not some sneaky fudge factor used in polls to allow for an acceptable amount of mistakes in measuring public opinion.
Now, there’s plenty that can, and often, does go wrong in measuring public opinion. How was the sample selected, were attempts made to reach cell phone only households, were the questions appropriately worded and asked in a reasonable sequence, was the quality of the interviewing up to professional standards, were repeated attempts made to contact hard-to-reach respondents, and was the weighting of the data carried out in an expert way? These are all vital issues that affect poll accuracy. But, they have nothing to do with the margin of error.
What does this mean for the consumer of public polls? Take the case of two public polls. Poll A completes 1,000 interviews. But, the sample was not drawn well, cell phones were not contacted, question wording was shoddy, the question order badly impacted survey results, the interviewers were poorly trained, multiple callbacks were not done in an attempt to contact hard-to-reach respondents, and the weighting of the data was sloppy. The margin of error for Poll A is… plus or minus 3%.
On Poll B. In this case, the sample was selected to reflect the population, cell phone only households were included, the survey utilized excellently worded questions, administered in a well thought out order, with highly trained interviewers, who made multiple attempts to reach potential respondents, and the data was weighted with expertise. The margin of error for Poll B is… not fair looking over anyone else’s shoulder… plus or minus 3%.
So, the next time you hear a reporter cite a poll’s margin of error, think of this as not a mistake, but simply as an unappreciated statistical concept in search of better understanding.