Why do we care?

• Elections have consequences

• Election polling has consequences

  • Campaigns

  • Media

  • Voters

• It matters whether the polls get it right

Will the polls get it right it 2022?

To answer this question, we'll ask the following about polls:

• How do they work?

• How do we use them to forecast an election?

• How have they done in the past?

• How will they do in the future?

But first...

What do you think will happen in the 2022 Midterm Elections:

Polling

• A poll is a survey conducted on sample from a population

• The theory of polling depends on the power of random sampling

• The practice of polling tries to account and adjust for all the ways a poll can fall short of this theoretical ideal

Polling Elections

• Pollster: Who's doing the survey

• Sampling frame: A list from which the sample was drawn (e.g. a voter file)

• Sample size: How many people were surveyed

• Survey mode: How the survey was conducted

• Survey instrument: What the survey asked

• Survey weights: Adjustments to make the survey more representative of the population

• Likely voter model: A way of distinguishing (likely) voters from non-voters

• Margin of error: A range of plausible values for the true population value

Polling Elections is Hard

New York Times

Error and Bias

Source

Polling Error

Total Survey Error in election polling is a function of:

• Sampling Error

• Temporal Error

• Non-Sampling Error

Polling Error

Total Survey Error in election polling is a function of:

• Sampling Error:

  • That error that arises from sampling from a population

  • Sample Size \(\uparrow\) \(\to\) Sampling error \(\downarrow\)

  • Margins of error typically only reflect sampling error

Polling Error

Total Survey Error in election polling is a function of:

• Sampling Error:

• Temporal Error:

  • The error that comes from polling a dynamic race at specific point in time

  • Polls closer to the election \(\to\) Temporal Error \(\downarrow\)

Polling Error

Total Survey Error in election polling is a function of:

• Sampling Error:

• Temporal Error:

• Non-sampling Error:

  • Errors that arise from how a poll is implemented and analyzed
  • Coverage error: Sampling Frame \(\neq\) Population
  • Response bias: Some people are more less likely to take a poll
  • Measurement bias: Question wording, order, can influence responses
  • Processing and adjustment error: Failing to weight for key demographics
  • And more...

Election Polling: Example

12 News/Roger Williams University Poll – August 2022

• Pollster: Fleming & Associates
• Sampling frame: Probability sample of registered voters, Aug 7-10, 2022
• Sample size: 405
• Survey mode: Live caller with land lines and cell phone
• Survey Instrument: See cross tabs of the questions here Questions
• Survey weights: None that I can tell
• Likely Voter Model: Hard to say, but based on past surveys probably two-part screener:
  • Are you registered to vote?
  • How likely are you to vote in the Democratic Primary?
• Margin of Error:

$$ \begin{align} MoE &= \pm 4.9 \\ &= 1.96 *\sqrt{((p*(1-p))/405)}\\ &= 1.96 *\sqrt{((0.5*(1-0.5))/405)}\\ &= \pm 4.869659 \end{align} $$

Evaluating the Performance of a Single Poll

Forecasting Elections

• Election forecasts reflect varying combinations of:

  • Expert Opinion
  • Fundamentals
  • Polling

• Forecasts differ in the extent to which they rely on these components and how they integrate them in their final predictions

FiveThirtyEight's Approach to Forecasting

Forecasting Elections with Polls

• The preeminence of polling in modern forecasts reflects the success of Nate Silver and FiveThirtyEight in correctly predicting the 2008 (49/50 states correct) 2012 (50/50) presidential elections

  • Any one poll is likely to deviate from the true outcome

  • Averaging over multiple polls \(\to\) more accurate predictions than any one poll, provided...

  • the polls aren't systematically biased

• The present concerns over polling the failure of such approaches to predict

  • Trump's Victory in 2016

  • Strength of Trumps Support in 2020

Polling the 2016 Election:

• The polls missed bigly
  • National polls were reasonably accurate (Clinton wins Popular Vote)
  • State polls overstated Clinton's lead / understated Trump support

New York Times

How did we get it so wrong in 2016?

Some likely explanations

• Likely voter models overstated Clinton's support

• Large number of undecided voters broke decisively for Trump

• White voters without a college degree underrepresented in pre-election surveys

A full autopsy from AAPOR Image

New York Times

2018: A brief repreive?

• Polls did a better job

  • Most state polls weighted by education
  • Underestimated Democrats in House and Gubernatorial races
  • No partisan bias in Senate Races

• Forecasts correctly call:

  • Democratic House
  • Republican Senate

However...

Vox

2020: Historic Problems, Unclear Solutions

• Average polling errors for national popular vote were 4.5 percentage points -- highest in 40 years

• Polls overstated Biden's support by 3.9 points national polls (4.3 points in state polls)

• Polls overstated Democratic support in Senate and Guberatorial races by about 6 points

• Forecasts predicted Democrats would hold

  • 48-55 seats in the Senate (actual: 50 seats)
  • 225-254 seats in the House (actual: 222 seats)

2020: What Went Wrong

Unlike 2016, no clear cut explanations for what went wrong

AAPOR Report

How Will the Polls do 2022 Elections

• What are the polls saying?

• Why they might be wrong?

• Why they might be all right?

• What do we think will influence the race more broadly?

But first...

What are the Polls Saying

FiveThirtyEight

FiveThirtyEight 2022 Forecast

FiveThirtyEight

FiveThirtyEight 2022 Forecast

FiveThirtyEight

Why the polls might be wrong

• After 2016, we had a reasonable idea of what went wrong, and how to fix it (e.g. weight for education)

• Lack a similar explanation for the polling errors of 2020.

• AAPOR Report lays out three scenarios going forward. Problems from 2020:

  • Persist in 2022 and beyond

  • Are unique to Presidential Elections (Problem for 2024, but not 2022)

  • Unique to 2020/Covid/Trump (Not a problem )

Why the polls might be wrong

Why the polls might be all right

Nate Silver makes the counterargument

Should we trust the polls to get it right this time?

• Ask me November 9, 2022

• Polling is hard, and getting harder

  • Fundamental Nonresponse challenges seem real

• Pollsters are innovative, and forecasts are flexible

• What's the alternative

  • Experts?
  • Fundamentals?

• Stop filibustering!! Will the polls get it right?

How should we interpret these graphs

Finally, some predictions

• Most Likely Outcomes:

  1. Republican House, Democratic Senate (Polls are all right)
  2. Republican House and Senate (2020 Bias)
  3. Democratic House and Senate (Polls are all right/Normal Polling Error)
  4. Democratic House, Republican Senate (Polls aren't all right)

Races to watch

• Senate:

  • Pennsylvania: Fetterman (D) vs Oz (R)
  • Georgia: Warnock* (D) vs Walker (R)
  • Wisconsin: Johnson* (R) vs Barnes (D)
  • Nevada: Cortez Masto* (D) vs Laxalt (R)
  • Arizona: Kelly* (D) vs Masters (R)

• House:

  • Michigan 7th: Elissa Slotkin (D)
  • Virgina 7th: Abigail Spanberger (D)
  • Iowa 3rd: Cindy Axne (D)

• Governor:

  • Pennsylvania: Shapiro (D) vs Mastriano (R)
  • Michigan: Whitmer (D)
  • Kansas: Kelly (D)
  • Arizona: Hobbs (D) vs Lake (R)

Will it matter?

Lots of things can happen between now and November 8. Here are few things we could talk about

• The economy (CPI releases Oct 13, Employment Nov 4)

• Dobbs v. Jackson; Reproductive Rights

• Turnout (High vs low)

• Demographics (Race, Gender, Age)

• Individual Candidates

• Joe Biden (Approval, policy)

• Donald Trump (Endorsements, Legal Troubles)

• Foreign Affairs (Ukraine, Taiwan, ...)