Forum Research released its latest federal polling numbers via The National Post on Monday, showing the Liberals stopping the steady decrease that had been occurring in Forum polling since May. However, the poll has the same sampling issues I have identified before, as I will explain in detail below. But is it a sampling issue or a weighting scheme?
The poll itself, despite the Post's excited headlines about a Liberal surge, shows no significant change from Forum's last poll taken a month ago (when the Post was similarly effusive about how the summer was going well for the one-point-gaining Conservatives). The poll does show the Liberals rebounding from a steady decline in support, from 44% to 38% to 35% between May and July, however.
The Liberals were up three points to 38% and the Conservatives were down one point to 29%, while the New Democrats were unchanged at 22%. The Bloc Québécois was down one point to 6% and the Greens were steady at 4%. Another 1% said they would vote for another party, and about 4% of the entire sample seems to have been undecided (unchanged). None of these shifts appear statistically significant (assuming a good random sample, see below).
The Liberals had a four-point edge among men and an 11-point advantage among women, while they were ahead among voters between the ages of 35 and 64. Oddly enough, Forum put the Tories in front among both the oldest and youngest Canadians. They've bridged the generational gap!
Regionally, the only significant shifts occurred in British Columbia and Ontario. In B.C., the NDP was up 11 points to 37% (an unusual, but not recently unheard-of number) while the Liberals trailed at 31%. The Conservatives were down 11 points to just 22% in the province, the lowest they've been here in any poll since an April survey by Harris-Decima. The Greens were up to 10%.
In Ontario, the Liberals jumped seven points to 43%, while the Tories slipped to 34%. The New Democrats fell six points to 17% support. Of note, Forum is the only firm to have put the NDP under 19% in the province since an Innovative poll from March 2012.
Elsewhere, the Liberals led in Quebec with 38% and Atlantic Canada with 46%, while they placed second in the Prairies with 35% and third in Alberta with 18%. The Conservatives were in front in Alberta (55%) and the Prairies (42%). The Tories were at 14% in Quebec (the third consecutive poll to put them at that number) and 28% in Atlantic Canada. The NDP was at only 24% in Quebec, 23% in Atlantic Canada, 21% in the Prairies, and 20% in Alberta.
That unbalance between B.C. and the rest of the country for the New Democrats means that the party would win the plurality of its seats in that province, shifting the NDP's focus from Quebec to the west coast. The Liberals would win around 144 seats with Forum's numbers, while the Conservatives would win 118, the NDP 62, the Bloc 12, and the Greens two (only just).
If we use CROP's regional distribution for Quebec, the seat tally in that province changes to 48 for the Liberals, 12 for the Bloc Québécois, 10 for the NDP, and eight for the Conservatives. That shifts the national totals to 155 for the Liberals, 117 for the Tories, and 52 for the NDP (generally speaking, using CROP's regional distribution swings about 10 seats from the NDP to the Liberals).
Both Justin Trudeau and Thomas Mulcair showed improvement in their personal numbers, with Trudeau's approval rating up five points to 49% and Mulcair's up four points to 37%. His disapproval rating fell four points to 33%, while Trudeau's 'don't know' score dropped three points to 18%. Stephen Harper's approval rating was down to 30%, his disapproval rating up to 62%.
Harper is popular among his own supporters, with 85% approval, as is Trudeau (88% among Liberals). Mulcair has only a 58% approval rating among NDP voters, however. This carries over to the "Best Prime Minister" question, as only 46% of New Democrats actually chose Mulcair as the best option for PM, compared to 71% of Liberals who selected Trudeau and 82% of Conservatives who chose Harper. Overall, Trudeau was well ahead with 32% to 24% for Harper, 16% for Mulcair, and 9% for Elizabeth May. If we take out the "none of the aboves" and "not sures", the three leaders generally line-up with decided party support. May, however, appears to be about twice as popular as her party.
The sample
Now let's take a look at Forum's IVR sample, focusing on the distribution of respondents by age.
As you can see, the same problem that has been identified before in Forum's polling continues. The sample is way too old (61% is over the age of 55, almost double what it should be). To get the proportion of younger voters right, their sample needs to be more than tripled - magnifying any errors that are bound to creep in when the sample is so small. The sample of older voters needs to be cut in half, wasting a lot of the extra precision that is gained from over-sampling that group.
For comparison, let's take a look at the sample of Nanos's recent poll. Nanos conducted its polling with live-callers, and had detailed breakdowns included with its Bell Canada/Telus poll (we are usually not treated to such detail with their usual national polls, unfortunately).
Nanos has some of the same problems in getting younger people on the phone, as seems to be endemic to the industry, but not nearly to the same degree as Forum. Instead of having to triple the sample of young voters, Nanos has to less-than-double it. Instead of cutting the sample of older voters in half, Nanos only has to trim it by about 25%. Nanos's sample is much more representative.
To be fair, Forum has had some decent performances in recent elections. But that may be chalked up to the firm having a good turnout model. While that might be useful for our purposes, what happens when the assumptions that model is based upon turn out to be wrong? In the United States, this is what had Mitt Romney certain of victory even when the national polls showed he would lose.
But are we looking at Forum's model in these sample sizes? The proportions are remarkably similar to its national poll from May - the 35-44s decreased to 11% from 12% of the sample, and the oldest cohort increased from 33% to 34%. Is Forum showing how it weights the sample by age, or is it just an unusual coincidence? The sample sizes for household income are not the same as in the May poll, though, so it is possible that we are not looking at a weighting scheme. Unless, for instance, Forum changed how they weight by income after analyzing the results of the B.C. election.
In any case, this is something to keep in mind when looking at a poll. It should not invalidate its results, though, as Forum's sampling issues are actually quite consistent. That makes it possible to look at the trends from poll to poll. And, in that sense at least, things look to be settling in for the Liberals.