The Three Common Mistakes Made in Survey Stories
May 20, 2020
KPMG The Truth about Online Consumers – 2017 Global Online Consumer Report
June 7, 2020
The Trouble with Self-Claim
As the famous quote attributed to David Ogilvy puts it: “The trouble with market research is people don’t think what they feel, they don’t say what they think, and they don’t do what they say.”
Every time this quote is repeated, I feel compelled to point out that it should be phrased as “the trouble with *bad* market research…”
Every researcher worth their salt understands System 1 and 2, post-rationalisation and social-desirability bias, and takes all of these into consideration for their work.
Unfortunately bad research is everywhere, and it’s giving all research a bad name.
Consider for example the recent headlines:
“71 per cent of respondents said that if they learned of a company’s irresponsible or deceptive business practices during the crisis, they would stop buying its products or services.”
“70% of consumers say they are more likely to buy from a brand that takes a stand on race-related issues.”
“Over 55% of Brits have made it clear that they would be prepared to pay more for a purpose-driven brand.”
These headlines are Ogilvy exemplified: survey participants giving obvious responses that bear no relationship to actual behaviour.
This sort of self-claimed and self-serving reportage is especially prevalent in publicly-reported research intended to support a sponsor’s point of view or agenda. The issue is compounded when the question itself is leading, which it often is. Many readers with or without a research background see right through this and switch off.
The good news is that it’s easy to avoid these common mistakes with your sponsored research. Here are three ways to deliver a more nuanced story, thereby standing a much better chance of cutting through and landing your message:
1. Eliminate social desirability
People naturally want to give the “right” answers to questions. The risk of over-claim is especially great when there is an obviously “right” answer to a question. Ask yourself: is it more politically correct to give a certain answer to this question? If yes, rephrase the question so a yes/no or agree/disagree response is equally probable and acceptable. Always avoid question wording that incorporates value judgements.
2. Compare groups
A comparison of cohorts is always more interesting than simply reporting a score from a single group. Whether it’s by demographic or by attitudinal or behavioural segments, contrasting differences between cohorts always makes for a richer story. And it helps to overcome the natural scepticism that the audience feels when the headline clearly supports the sponsor’s message. It’s not that useful to know that 55% of Brits say they would be prepared to pay more for a purpose-driven brand. It’s much more valuable to know which groups of Brits would be more or less likely to do.
3. Ask about actual behaviour not predicted behaviour
Ogilvy was right. Humans aren’t very good at knowing what they’ll do in the future. We are, however, reasonably capable of remembering what we’ve done in the past. Previous behaviour is a far more reliable predictor of future behaviour than asking people about future behaviour. Of course, over-claim is still a risk even when recording past behaviour, therefore it’s still essential to eliminate social desirability in the question.
There you go: three simple tips which hopefully are easy and which cost nothing to implement. If more of us followed these guidelines, perhaps we could put David Ogilvy to bed once and for all.
Stephen Yap, Intuit Research
As the famous quote attributed to David Ogilvy puts it: “The trouble with market research is people don’t think what they feel, they don’t say what they think, and they don’t do what they say.”
