Here's a bit of background on smartphone polling and a few charts from our private surveys conducted ahead of the Holyrood (Scottish Parliamentary) election in 2021:
How we performed relative to official polls
Overall, we performed marginally worse than the official pollsters, albeit at a significantly higher cost to construct the survey panels. Both sets of polls were very accurate and within a 3% margin of error. Our polls showed less fluctuation throughout the period we conducted surveys. The official polls started further apart from ours, eventually converging as election day drew closer.
Where we did better
The Holyrood elections in 2021 were a polling coup and relatively uneventful. Few remember the majority of cases when things go as planned. The key advantage of our methodology for this election was the speed and scale in conducting surveys. We were able to identify the rising popularity of Scottish Labor ahead of the official polls.
Reducing potential sources of uncertainty
We were also able to exclusively poll younger Scottish voters (under 25), who were highly underrepresented in official panels—about 100 times more. Thus, presenting a potential source of uncertainty which we could reduce.
Lower bound limits of polling uncertainties
It is very difficult to materially improve on a 5% polling margin of error even by increasing the respondent panel size and refining models—fixing one thing introduces other sources of uncertainty that are unfamiliar. The recent introduction of educational attainments is one such example that has not resulted in a material improvement in polling accuracy. It is advisable to adjust market positions accordingly, depending on the risks priced in by the market.
When some pollsters claim to have reduced polling errors by less than 3%, as evidenced by the final tally, they were more likely lucky than clever. Even if it were possible to poll every voter beforehand. It's basic statistics. The continued uncertainty after the exit polls are announced serves as evidence.
No single method of polling is best
No single polling methodology or model is better; it very much depends on the electoral system and circumstances. That part is more art than science. Personally, I'm a big fan of pollsters who conduct extensive hours-long interviews with small groups consisting of fewer than 10 people. Often, tracking the same individuals over many years to establish a personal rapport and observe how their views may have changed over time.
The folks at Beacon Research behind the Fox News polls— we have previously worked with—conduct in-depth small panel surveys.
How are smartphone polls better
The key benefits smartphone polls are: i) identifying early emerging trends that may impact markets, and ii) reducing potential sources of uncertainty in official polling figures.
Past examples
Here are some example charts using data from various smartphone polls we conducted during January-April 2021, with panels ranging from 800 to 2000 respondents:
