The real problem with polls.

Posted by Pointman on January 30, 2014 · 9 Comments 

I see yet more opinion polls on global warming wiggling their way through the bowels of the mainstream media like tapeworms. They’re only usually interesting in terms of spotting how the alarmists have bent the questionnaire, data, statistical methods or simply misrepresented the results to achieve the required propaganda headlines. Given how often such Cook the Books or Lewpapers take a bashing from the skeptic blogosphere, the ordinary person can only conclude that the originators are either particularly inept as researchers or equally inept at cheating their way undetected to the desired result.

A few skeptics have even started running their own polls, but for reasons that I hope will become apparent, I won’t be paying them too much attention. It’s working out the target demographic the results and interpretations are being shaped for which I find more informative.

Even when the opinion polls are conducted in a professional and honest manner, things can still go seriously wrong. In the US Presidential election of 1948, all the significant opinion polls predicted that Harry Truman would be beaten by Thomas Dewey. The Chicago Daily Tribune even famously headlined the next day’s paper with “Dewey beats Truman” but of course the reverse was what actually happened. In a similar fashion, the UK General election of 1992 was consistently predicted by the opinion polls to be won by the Labour Party under Neil Kinnock but the Conservative Party, led by John Major, actually won the election. The election pundits at the BBC and Guardian had to be talked down off the ledge of their own wishful thinking.

It never ceases to amaze me that while people see a woman dressed up as a gipsy and predicting the future by staring into a crystal ball in a carnival tent as a bit of a giggle not to be taken seriously, put the equivalent of that person in a smart suit or a white lab coat, and their predictions about the future suddenly become gospel, especially when they come with numbers attached that run to three decimal places. Surprisingly, it’s those people who are clever in a lot of other ways who really fall for nonsense like that.

The subtle reasons why polls on these types of topic can be chronically inaccurate, was determined in the forties and fifties, and not by statisticians. The ground-breaking research was actually done by manufacturers trying to sell their products to consumers. Time after time, they’d run extensive market research to find out exactly what the average consumer wanted, designed the product accordingly, manufactured it in bulk – only to find that the consumer wouldn’t touch it with a barge pole.

In what seems in retrospect an obvious move but was far from it at the time, they stopped those simple market research polls which were inexplicably failing, and instead started studying the deeper question of how consumers interacted with polls. The findings, backed up by some highly innovative experimental psychology, would lead to a revolution in marketing campaigns.

The essential problem was that it was human beings filling in the polls. To understand why that is the problem, we’re obliged to take a little diversion through the enigma that resides between your ears, a little tiptoe through the tulips that border the winding path through the windmills of your mind.

Your brain is just another organ in your body, a piece of hardware, but it is the platform which runs your software, otherwise known as your mind. If you hacksaw the main processor chip of a computer in two and examine it carefully, you’ll possibly learn about junctions, connections, logic gates and such things but you’ll learn nothing useful about the software it runs. You see, your hardware stays pretty much the same but the variety of programs it runs is ever-changing from moment to moment and from person to person.

In an analogous but real sense, our knowledge of how the brain manifests that unique singularity we call a conscious mind is quite frankly abysmal. We actually don’t know the mind at all but over the last hundred years and with the development of a rudimentary psychology, we’ve made several conjectures on how it might work. Various theories come into vogue, enjoy their day in the sun and then are supplanted by more fashionable ones. The accent is still very much on that adjective fashionable.

Studying the mind is the concrete realisation of that secret nightmare of anyone who’s ever looked down a microscope to examine something; one day there might be a big brown eye looking back up at you. You can study something that has no awareness of being studied and learn a lot. Study human beings and you’re into a world which cannot be described by something so crude as equations nor does it have much in the way of reproducibility of experimental results on traits such as creativity, inspiration and matters of opinion or judgement.

The nuclear physicist Leo Szilard, when he’d retired from the blowing up the planet business, decided to study various areas of biology. One question he determined to settle once and for all was whether dolphins were as intelligent as they were rumoured to be. He thought up several ingenious behavioural experiments to test them but while the dolphins were very good at working out complex problems, the results of his experiments were rarely repeatable because the dolphins got bored. He concluded they were intelligent, not nearly as much as us but more interestingly, had attained enough of a level of consciousness to be bored.

Since anyone’s guess at how the mind works is as good as anyone else’s, I tend to go for my own home brew two-tier structure. We have a top layer that does the thinking kind of processes but is not simply some sort of organic computer. Underneath that layer we have a black box layer which the top layer cannot access directly. The black box keeps an eye on the needs of the top layer and communicates with it whenever it feels it has something to contribute. It’s the real deep thinker of the two.

It’s perhaps a trivial example but I enjoy doing cryptic crosswords just to get that sudden bolt of insight that yields the answer which I know for sure is correct. All that remains to do is for the top layer to work out exactly why it’s correct and to wonder where that insight came from, but it’s a case of thank you Ole Man black box …

The other and much more common type of communication the black box indulges in, is telling the top layer what it likes, as opposed to what’s needed and it tends to have a lot of influence over what the top layer wants. Hang onto that specific idea while I try to use it to explain motivational research, the results of which became the tenets of all marketing fifty years ago and still are. I’ll only cover those ones directly applicable to opinion polls.

For starters, you can’t trust the truism that people usually know what they want, even when they’re supposedly demanding it explicitly. If they did, then none of those products should have bombed after massive amounts of market research to find out what people wanted. Those rejected products were real world experiments and very expensive ones. You’d consulted with them, they’d told you what they wanted, you’d produced it for them but they simply wouldn’t buy it. The top layer said it wanted it but the black box simply didn’t like it.

To further complicate the situation, on any opinion poll they’re not particularly likely to tell you truthfully what they actually want or will do in the future, for a bewildering variety of conscious and subconscious reasons, which spring from both the top-level and the black box respectively.

If they’re in a group, they may be slightly intimidated by peer pressure into temporarily conforming to the majority view. They may like the person asking the questions and attempt to give answers which show themselves in a flattering light. They may dislike the person and react accordingly. There are many other factors which can skew results and simply using a bigger sample group does not necessarily mitigate the effects of such factors.

Basically, you can rely on them to fib.

An experimental example of consciously fibbing is the focus group used by a company to get solid data on people’s attitudes to personal finance. One of the questions buried away in the survey asked was if they’d ever consider taking out a personal loan. The overwhelming majority strongly condemned such a course but the financial services company doing the research through an intermediary only invited them to participate because they were already customers on its loan book.

An experimental example of unconsciously fibbing was provided under the guise of interviewing a crowd of people one at a time. To cope with the growing backlog of people waiting to be interviewed, two waiting rooms side by side were provided; one decorated in a simple modern style using warm pastel colours with comfortable chairs, and the other more expensively decorated in a formal and classical style. It was only when the seating in the modern waiting room was full, that people started to sit in the more formal and elegant room. When there was seating space available in both, new arrivals tended to pick the more modern one. A harmless question near the end of their interview asked which room they preferred. The more formal room won hands down.

Another more interesting finding is that it’s a mistake to expect people to act rationally when it comes to making decisions which are actually governed by likes and dislikes, rather than what you might have assumed to be objective criteria. The black box is making the decisions in those situations and for reasons we at times can only guess at. Again, an experimental example of such contra-intuitive behaviour was asking housewives to try out three different formulations of washing detergents.

The first came in a bright yellow box and the feedback on it was that it was too harsh and strong on delicate clothing. Indeed, there were even reports of it damaging delicate clothes. The second variety of detergent came in a dark blue box and the consensus opinion on it was it left clothes still slightly dirty. The third variety came in a box decorated in a mixture of blue and yellow. The feedback on it was that it was streets ahead of the other two, cleaning clothes thoroughly and really leaving them feeling fresh.

All three formulations of detergent were exactly the same.

You see, the fundamental change in research-led marketing using industrial psychology was switching from selling you a product on its own intrinsic merits, to selling you things like a stylish spinoff connected to it; a lifestyle associated with that product. It was really about skipping over a conversation with that top layer of your mind, and talking directly to the boss who was actually going to make the buying decision, the black box.

For example, it’s been decades since a car has been advertised on the basis of how many miles per gallon it uses, the length of service interval, the time between its statistical breakdown intervals, its top speed, its insurance category or even how much it will cost you. It’s all about showing you a person and a lifestyle you aspire to or already think is yours, and mixing that up inextricably with the car in question. Once they anneal that association in your mind, they’re half way to making a sale.

The last thing opinion polls are used for nowadays is to do research on public opinion, because no matter how sophisticated your statistical treatment of the numbers is, you just can’t trust the raw data. Rather, they’re simply used as another technique to persuade people to buy a product.

Nine out of ten felines prefer brand X cat food, so buy it for your cat. Ninety-seven percent of scientists believe in global warming, so you should too. The majority of people are going to vote for this bland suit on wheels at the next election, so you should too. They merely provide the comfortable reassurance that you’re running with the pack, you’re safely protected inside the herd, you’re marching in step, you’re in formation, that you’re not a minority.

Opinion polls have little or nothing to do with finding out your opinions. On the contrary, they’re all about reinforcing or influencing your opinion.

©Pointman

Related articles by Pointman:

The shape of things to come; Snailbats, HALsays, Scarems, LewPapers and DickPols.

Lies, damn lies and polls.

Click for a list of other articles.

Filed under Article · Tagged with Marketing, Polls, Psychology