Not even close. Last night, Donald Trump defied the polls, including ours, to win the electoral college and become President. Pollsters large and small, old and new, telephone-based and online, random sample-based and convenience sample-based all got it wrong. While the popular vote has broken for Clinton, it is a much tighter margin than anticipated by us or most anybody else. More critically, polling failed to correctly assess the state of the race in critical swing states like Michigan, Wisconsin, and Pennsylvania, where polls, including ours in the case of Pennsylvania, suggested a durable and stable lead for Clinton.
So, what happened?
When polls are wrong, there are a couple of key areas to evaluate.
- Sample bias
- Were researchers able to reach a sample that was representative of the population? Was there something about the data collection methodology, be it online or telephone-based, that systematically failed to reach a certain group?
- Response bias
- Did something about the data gathering methodology or the survey instrument systematically discourage participation or truthful responses from certain survey-takers?
- Modeling Errors
- Did pollsters and prognosticators err in fitting poll results to an accurate estimate of the population of likely voters?
Those three categories include some of the more specific challenges with polling that have been written about over the last several news cycles. Traditional, so-called gold-standard polls conducted over the phone with live-interviewers have grappled with higher costs, new technology, a growing and diversifying American population,and changing consumer behavior.
Last night the dam broke for the industry but there were cracks before that – polling misses in recent races including – the 2014 Senate reelection in Virginia, Brexit, the most recent Israeli election, and more.
On some level, one of the other failures in all this was a faith in the accuracy of polling research in media and society that had not caught up with the scale of turmoil and uncertainty within the industry itself.
We believed that our technology could help solve for some of the challenges that have emerged for pollsters over the last several years. In truth, we still do. To get there, we and others have a lot of work to do to answer the questions listed above. Over the coming weeks, we’ll attempt to do just that.