One of the hardest stories a researcher has to tell is one that the client disagrees with. They have a vested interest and research is as much art as it is science so they can contest it. It is our job to acknowledge its limitations, defend its strengths and say what we can learn from it.
We’ve just conducted a robust poll of Brighton & Hove residents on political affiliation and attitudes to legal assistance to die (for the detail people out there, it was 1,000 representative quota sample on age, gender and working status based on 2001 census data, conducted by telephone by ICM Direct in accordance with MRS code of conduct).
The political affiliation question is interesting because Caroline Lucas of the Green party is in with a good chance of taking the Labour seat of Brighton Pavilion and becoming Britain’s first Green MP. An ICM poll commissioned by the Greens in December suggested that they were ahead of Labour and Tory. Past electoral results also add weight to this. In our independent poll in February however, we asked a different question which suggested that support for the Greens was lagging behind both Labour and Tory.
The outcry from Greens that followed its appearance in the local Argus newspaper for example here, here and here are typical shoot-the-messenger protests when it comes to research:
The sample size is too small - this is counter-intuitive to most people but the key criterion determing accuracy is how representative it is (ie how respondents have been selected) not how many interviews there are. 336 is a good number for a poll and the large difference between support for each of the parties cannot be explained by sampling variance.
There is nothing for us to learn here – there is always something to learn! The key question the Greens have to ask themselves is why do only 12% of Pavilion residents say they would vote for them in a general election tomorrow compared to 26% for Labour.
Twist or ignore the interpretation – I published the full report here and made clear in the interpretation the limitations of our raw findings and the conclusion that might be drawn.
Claim that other research is right so this research must be wrong - The main factor that explains the difference between the Green party poll and the Kindle poll is the question. The Green party question told respondents first that a general election was taking place and secondly the names of 4 of the parties who would be contesting. This practice lifts mentions of lesser known parties.
Cast suspicion on the motives – independent researchers do have allegiances but what we are primarily interested in is truth and learning. Some clients may want to put a spin on research findings to serve their own interests but when they spend money they want to find out what is going on so they can invest wisely.
Lack of experience – the sample is sound, the method is sound and the questions follow standard practice.
Love it? Hate it? We want to know: Leave us a comment (20 so far)
Your research findings are interesting, but it’s highly misleading to compare them with the results of professional political pollsters.
Unlike Ipsos Mori (whose methodology you link to) you didn’t eliminate those unlikely to vote. You also tagged the political questions onto a survey about something completely unrelated which would affect the results. There was also no political weighting of the results to counter the inevitable biases of polling by telephone and so on.
Polling veterans like Anthony Wells raised a number of serious questions about your polling method http://ukpollingreport.co.uk/blog/archives/2461
This from the Ipsos Mori page you link to: “This page shows the voting intention figures of those of the public who say they are “absolutely certain to vote”". Your poll did not do that.
Comment by Cllr Jason Kitcat — February 23, 2010 @ 4:25 pm
Thanks Jason. Just to come back on some of those points:
- We didn’t eliminate unlikelys to vote because we weren’t trying to make any electoral predictions. It’s a simple measure of support.
- The political questions on our survey were asked immediately after demographics and before any other questions so that had no effect
- There are as many arguments against weighting by past voting behaviour as there are in favour.
- Anthony Wells doesn’t raise serious questions about the method, he offers a balanced view on why ICM may be a better indicator of electoral performance
Comment by Paul — February 23, 2010 @ 5:01 pm
Whats the difference between voting intention and ’support’? Surely a Poll is trying to work out what will happen in an election! What other use is there for a poll?
Anthony Wells says:
- “there were very small samples sizes”
- He talks about how it is probably wrong not to prompt for parties as this will not give an accurate result.
- He says you should have factored in likelihood to vote.
- and says “Finally, and most importantly, the poll did not have any political weighting – phone polls without political weighting of some sort will tend to grossly overestimate the support for the Labour party.
Is that enough serious questions about the method?
Comment by Allie Cannell — February 24, 2010 @ 10:53 am
And compared to the ICM poll you’ve put up very little information about the Poll!
You’ve basically only told us the results whilst ICM told us every bit of information they collected at every stage.
Comment by Allie Cannell — February 24, 2010 @ 10:57 am
Paul,
Is it possible, please, to have a full list of the questions you asked, and the data tables, as you are required to provide by the MRS code? While http://kindler.posterous.com/greens-still-have-lots-to-do-in-brighton-pavi has some relevant info, it doesn’t provide everything MRS says you should be making available.
I supervise research students (PhD research). If one of my students departed from standard methodology in they way you have done in 1) failing to weight by past vote 2) failing to factor in liklihood to vote and 3) failing to prompt by giving party names, then I would expect them to give references to appropriate peer-reviewed literature. Are you able to point to literature that justifies these aspects of your methodology?
Comment by Ben Fairweather — February 24, 2010 @ 11:40 am
We need more data. Apologies if it is here, and my navigation skills are not 100%, but life is short.
Who commissioned the poll?
What questions were asked?
You say Brighton & Hove residents were polled, but Caroline is standing in Brighton Pavilion. Can you disaggregate the BP answers?
Thanks for answering these questions.
Comment by Richard Lawson — February 25, 2010 @ 1:56 pm
@Allie Cannell, thanks for your comments. In response:
- A poll can be trying to achieve lots of things though I can understand why political parties are interested in who might win an election
- The sample sizes are robust
- Anthony makes clear that there are arguments for and against prompting
- I would have factored in likelihood to vote if I wanted to look at possible electoral performance
- Political weighting (based on what people said they voted in a previous election – of dubious accuracy) is just as likely to overestimate the support for minority parties
- Well done for getting through all ICM’s information about their poll! My report has all the relevant information needed to judge the results
Comment by Paul — February 25, 2010 @ 5:47 pm
@Ben Fairweather, thanks for your comments and your credentials.
You’re mistaken about my obligations under the MRS code. If you could point out where I haven’t complied, please let me know and I will put that right immediately.
In any event, I have posted the full data tabs here in case this is what you are after?
http://kindler.posterous.com/full-data-tables-from-bandh-political-party-v
I think the failings you refer to depend on the purpose of the research. I understand why political parties might be interested in a different approach but the analysis is clear: when you ask a representative sample of adults in Pavilion who they would be likely to vote for only 12% say Green. The question is not then about how to raise that figure to what might be a more likely indication of electoral outcome but why awareness is so low and what to do about it.
And I am mightily relieved I do not have to give reference to appropriate peer-reviewed literature!
Comment by Paul — February 25, 2010 @ 6:00 pm
@Richard Lawson, thanks for your interest and I appreciate also that life is too short!
- A poll of Brighton & Hove adults was commissioned by the Society for Old Age Rational Suicide where we were interested in analysing results by political affiliation so we added these questions in.
- The report already details the questions asked (at the bottom of each slide) and also disaggregates for each of the 3 constituencies. They are also on the data tabs (see above for link).
Comment by Paul — February 25, 2010 @ 6:11 pm
Paul,
Thanks, for the clarification and thanks for the data tables you have now made available. They sort out what I think you were obliged to do (with respect to all the results I have seen published, at least).
From what you have said about the commissioning of the poll it seems that you didn’t start from the position of being commissioned to conduct a political poll.
This makes me wonder: when did you decide to report the political data separately?
Comment by Ben Fairweather — February 25, 2010 @ 11:47 pm
Hi Paul, can I just ask you to clarify the margin of error? Jason Kitkat has said that you put it at 10%. Does that mean that a difference has to be 10% apart for it to be reliable? That would mean that the margin of error is “plus or minus 5%”, wouldn’t it? Is that what you meant?
I would expect the margin of error to be larger in your research than in the Greens’ poll – but not by much. Their base for the published poll result was 390. Yours is 205 for Pavilion, once you remove undecided, refused and won’t say. Both results were weighted, reducing the statistical reliability slightly.
So if you were to express the margin of error as a plus/minus figure, what would it be?
Comment by Tracey Hill — March 1, 2010 @ 11:53 am
Hi Tracey,
The margin of error depends on the size of the confidence level, sample size and the percentage being tested.
As you say, the sample is weighted which also affects this calculation and in pure stats terms measures of statistical significance assume a pure random sample (this is rarely done now for a range of reasons). Nevertheless, there is a convention to use this calculation when applied to a quota sample to help assess ‘accuracy’.
For a sample of 209 (the unweighted base for Pavilion when removing Would not vote, Undecided and Refused), the margin of error at the 95% confidence level for each of the parties (also removing Would not Vote, Undecided and Refused) is as follows:
Labour 41% plus or minus 7 percentage points
Conservative 26% plus or minus 6 percentage points
Green 20% plus or minus 5 percentage points
Lib Dem 9% plus or minus 4 percentage points
Hope this helps!
Comment by Paul — March 1, 2010 @ 12:34 pm
Yes it does, thanks very much.
Comment by Tracey Hill — March 2, 2010 @ 4:09 pm
Paul,
Any chance of an answer: when did you decide to report the political data separately?
Comment by Ben Fairweather — March 3, 2010 @ 3:37 pm
Ben – I’m not sure what you are after here. When the project got commissioned there was a discussion about what to cover and what to report. The questionnaire was designed, the data collected, the findings reported. That’s what tends to happen on research projects.
Comment by Paul — March 3, 2010 @ 4:18 pm
Paul, when the project was comissioned, did the discussion about what to report conclude with a decision at that stage to report the political data, or was the political data only being sought at that stage as a way of contextualising the data about attitudes to living wills etc: you do say “we were interested in analysing results by political affiliation so we added these questions in.”?
Comment by Ben Fairweather — March 4, 2010 @ 12:23 am
Ben, I’ve no idea what you are on about! Would you like to tell me what your theory is and I’ll respond directly to that?
Comment by Paul — March 4, 2010 @ 11:16 am
My theory is that you didn’t originally intend to publish party preferences: and because you didn’t intend to publish them you -very reasonably- didn’t have any reason to design the survey to include a ‘how likely are you to vote’, or a ‘how did you vote in 2005′ question.
Is that accurate?
Comment by Ben Fairweather — March 4, 2010 @ 3:33 pm
No – we intended to publish them from the time that the questionnaire was designed.
Comment by Paul — March 5, 2010 @ 11:08 am
Thanks for the answer. It leaves me more baffled than ever at your departure from standard methodology for political polling, though.
Comment by Ben Fairweather — March 5, 2010 @ 1:08 pm