There is a belief in the Political Twittersphere, especially among Bernie supporters, that phone polls are biased towards candidates who perform better among older constituencies (primarily Biden) due to their reliance on landlines. Live phone methods do tend to undersample younger voters because they rely heavily on landlines. However, most pollsters worth their salt will account for this representation discrepancy through their weighting processes.
To get at the question of differences in polling results based on methodology, I’m going to look at some polling data provided by 538 to create separate plots for the top eight candidates looking at their poll percentages over time and broken out by methodology. The dataset consists of three methodology categories that will be removed - blank, IVR/Online, and Online/IVR (both codings indicate a poll that was done partly online and partly with phones, and therefore doesn’t help us answer this question). The dotted lines represent the date the candidate’s announced to help see if announcements coincided with any distinct polling differences.
I’m a bit surprised by these loess curves. For Biden, live phones seem to have found worse percentages prior to his announcement. Since then, they’ve been slightly more aggressive at recording his upward and downward trends, while online polls have varied a bit less.
That isn’t the case for Warren and Mayor Pete - there doesn’t seem to be a substantive difference between the two methodologies in either of their poll results.
I’m very surprised by the Bernie plot because it seems to back up a theory that I really haven’t put much store in. Since his announcement it looks like online polls have been more favorable to Bernie by 3 or 4 points, and by even more prior to his announcement. This does lend a bit of credit to the idea that traditional phone polls are not accurately compensating for their lower response rates, though it doesn’t necessarily mean the difference is due to undersampling young people. There could be an entirely different confounding variable at play (perhaps Bernie is even over-performing in online surveys).
Overall it’s hard to draw a universal conclusion from these plots. It does look like the methodology matters for Biden’s and Bernie’s poll performance, but not for Warren and Mayor Pete. I’ll be interested to see if Biden’s live phone results continue to be more volatile than his online results, and I’ll especially be interested in whether Bernie’s numbers continue to have a definitive difference in his online and live phone polls, and, ultimately, which ones prove to be more accurate.
Once the election has finished, the next step will be looking at the full dataset for each candidate, and seeing if there is a statistically significant difference between the two types of polls.