My belated condolences. I always enjoyed reading his posts and will miss his insight and wide ranging views. May hew…
We’re going to have to do something about peer review.
Not too many years ago, there were two researchers who discovered that stomach ulcers were not the result of a stressful lifestyle, but simply an infection. The idea that ulcers were related to a high stress lifestyle was so deeply embedded in the medical establishment psyche, they couldn’t get a fair hearing, never mind getting their work published. To make that happen, one of them did an extraordinary courageous thing; he infected himself with the germs he knew would give him a stomach ulcer, got the subsequent condition unambiguously diagnosed, took the requisite medication and was cured.
Now, that’s what I call having confidence in your research.
Rather than painful decades of eating a bland diet, changing your lifestyle or getting pieces of your stomach progressively cut out, a simple course of drugs will now cure a stomach ulcer, which is why you don’t hear much about them nowadays. Thirty years down the line, even the most advanced medicine tends to look like nothing better than butchery.
That story is illustrative of some fundamental things about science. For starters, it’s never settled. It’s for that reason, if you can’t honestly examine a doubt about any theory, you’re doing belief rather than science. Most theories are superseded by a better one or are found out to be just special cases of a higher level abstraction. The Newtonian view of celestial mechanics giving way to Einstein’s work, would be the classic example.
Of course, some theories are just plain wrong to the point of being junk science. Well-known examples of the latter would be Phrenology, Lysenkoism and Eugenics, which were all essentially outbreaks of pathological science. The sobering thought is that all of them were thought of as mainstream science for decades.
Some theories are the result of bad science; the prime example of this being the Ptolemaic system, which placed the Earth at the centre of our solar system, and while it was totally wrong, it still gave some functionally accurate results, which is why it survived as mainstream thought for fifteen hundred years.
How can something that badly wrong, be believed in for a millennium and a half, by the educated people who would be the equivalent of today’s scientists?
There are two reasons. The first one is that if you’re not familiar with a particular field, the natural tendency is to assume that those who are experts in it, know their stuff. You accept their assessment of any new theories in their area. The second reason is that people who know the area well, are nearly always heavily invested in the current status quo. They’ve spent years studying a particular field and become unquestioning of its central tenets. Expecting them to concede something they’ve backed for most of their adult life was wrong, is a big ask. Anyway, by that stage, their intellectual arteries have hardened. It’s natural for them to be wary of and resist any seismic shifts in the fundamental theories underpinning it.
Given those barriers to change, how does science actually manage to keep evolving? The answer is publication of papers by researchers. The problem with this, if you’re coming up with research which is a real game changer, is the peer review system. I wrote a previous article which outlined the peer review process and touched on some of the problems associated with it, which you can find here, but if you’re unfamiliar with the process, the following paragraph from the piece summarises it.
A scientist does some research, writes up the results in the form of a paper and submits it to a scientific journal for publication. Whether it’s published or not, is an editorial decision. The editor, who obviously can’t be an expert in all branches of science, has to seek advice from other experts in the field the paper is dealing with. For each area, the editor will have two or three experts he can ask to review or referee the paper. Some journals ask the author for a list of potential reviewers. The reviewers are usually unpaid and working scientists themselves.
The peer reviewers will come back with one of four recommendations; publish it, publish after comments have been addressed, reject it outright or reject it for resubmission after rework at a later date. Obviously if the paper undermines the life’s work of the reviewers, it’ll be lucky to get the go ahead from them. That process is called gatekeeping or excluding theories that contradict ones you support. The climategate emails illustrated not only gatekeeping, but a complete subversion of the whole peer review process in climate science. The reviewers were determined that papers they disliked were not going to be published, even if that meant “redefining the whole peer review process.”
Once pre-publication peer review became common in the 1950s, it immediately pitted innovation in a particular science against the establishment of that area, and it’s a David and Goliath situation. Before considering a possible solution allowing more innovation and freedom of thought, it’s worth examining how innovation came about in the centuries before the straight jacket of the peer review process was put in place, because I think it gives a good indication of the way forward.
Quite simply, people wrote books or papers. There was nobody gatekeeping; controlling what you could and could not put out into the world. Some ideas were so bad, they sank without comment. Other people wrote their own books and papers, discussing your ideas. Beneath all that public discussion, spirited conversations between the interested parties occurred, usually by letter. As a system that encouraged active debate of new innovative theories, it worked better than peer review. Once you get past the paper dependency of the whole process, there’s a certain latter-day familiarity about the whole thing.
In areas of science subject to controversy, we’ve already begun to move over to the equivalent twenty-first century implementation of the old system using the blogosphere. Papers are published online and people discuss them, either on their own blogs or directly as comments underneath the paper. Just as in the old system, there’s a Darwinian selection process going on; loony papers sink without trace, interesting ideas get examined by a lot more minds and from a basis of different skill sets.
In fields like climate science, where airing controversial views can threaten your career or if you’re a student, who knows nonconformity will threaten your degree, it offers the protection of anonymity, both to authors and the commenters.
This isn’t my suggestion of a way forward; it’s simply my recognition of the direction we’re already heading towards in the controversial areas of science. What’s driving it is quite simple. When people are prevented from publishing in the mainstream science journals, they’ll publish on the internet. When sensible commentary on dubious science is heavily censored, that commentary moves out to the freedom of the internet. Once moved onto the internet, it won’t be coming back anytime soon, which is one of the reasons why circulations of what used to be the popular science journals, are plunging.
This change in how new ideas in science are debuted and debated is happening and it is the future, whether you’re comfortable with it or not. Novy mir.
©Pointman
Related articles by Pointman:
The decline of popular science journals.
Is climate science just a belief?
Global warming and pathological science.
Pointman's 
I always enjoy reading your articles, they are very clear and forthright, I believe the big problem is not that the Science is being moved forward, that is inevitable, but rather once an area of science has become a political band wagon, always to serve the non ethical egotism of people who obscenely over value themselves, is that the resistance, lethargy, intolerance and plain egotism of so called scientists, NGO’s, advisers, civil servants and politicians who all have a multiplicity of agendas to enhance their careers is deliberately set into default mode to protect their interests. Whilst no doubt there is a cult basis to the non scientific doctrines that are so tenaciously clung to, these are dressed up as authenticity to propagate the message, once accepted this propagander takes a lot of deconstructing, in reality it takes decades or a massive crisis which directly negatively impacts on the status quo of the proponents of the junk science for the lies to be rejected and newer more honest policies to be put in place. Speaking from someone who lives outside of the U.K. in a country with a very well established Nuclear Generation capacity I believe that, for example, the 2008 climate act of the U.K. is going to be kicking the a**e out of many supporters of the green renewables industry and ethos much sooner than any of them think AND coupled with the global crisis, I believe a much more personal crisis will overtake many people who ride this present gravy train, they will be completely submerged by the events that are about to happen. Whether this lesson will be learned by future generations is sadly, very much open to doubt.
LikeLike
You are so right about how the internet is providing a venue for dialogue that not allowed in mainstream news any more!
The results are very clear. As people become more informed and more politically “aware” they usually turn away from the mainstream feeds and you now see massive losses in the major Television programs that used to be called “news stations”. People can now see that they were nothing more than giant “propaganda machines” and are turning to the blogoshere for informed views.
Thanks for being one of those sites!
LikeLike
Stop reading my mind, it’s unnerving. Exactly my thoughts, just wanted to write something along those lines.
LikeLike
Does the name Giordano Bruno (1548 – February 17, 1600) ring a bell? Born Filippo Bruno, he was an Italian Dominican friar, philosopher, mathematician and astronomer. In general, Giordano Bruno paved the way for the cosmology of our time. His cosmological theories went beyond the Copernican model in proposing that the Sun was essentially a star. To his lasting credit, the most recent empirical discoveries in astronomy and rational speculations in cosmology (including the emerging science of exobiology) support many of his brilliant insights and fascinating intuitions. This is an appropriate legacy from a daring and profound thinker, who presented an inspiring vision which still remains relevant and significant for our modern scientific and philosophical framework.
What was Bruno’s peer review? After the Roman Inquisition found him guilty of heresy, at Rome on February 17, 1600, at the age of 52, Giordano Bruno was bound, gagged and publicly burned alive at the stake in the center of the Campo dei Fiori, not far from the Vatican, while priests chanted their litanies.
In 1600, this type of ‘peer review’ maintained the “…current scientific consensus…” Dismissing out of hand any contrary evidence that does not conform to the “…current scientific consensus…”, the OGW high priests chant their litanies, too. When the high priests of consensus control the process of peer review, the ‘heretics’ get ‘burned’.
LikeLike
I believe you are correct, in the assumption self publishing will expand, but the issue is much wider than scientific papers and I feel there are unintended consequences coming.
Musicians can now self publish as can writers.
The traditional publishing houses are being bypassed in all areas.
However the problem is then a matter of information overload. If every expert, well meaning amateur or nut job, (let alone directed misinformation from political agents) started self publishing, how do you find the pearls. We already have information overload where worthwhile writers/bloggers are being lost in a sea of mediocracy.
I enjoy reading your blog due to its considered, well argued and well written content. But there seems to be an ever increasing level of content and unfortunately I need to go to work each day, or at least go out and mow the lawns occassionaly..
Regarding the unintended consequences – a well orchestrated organisation could flood the web with opinions (eg the IPCC).
Therefore a recognised repository would make it easier to find these pearls, or do a “wiki” or “trip advisor” and allow public ratings on blogs / publishing sites.
LikeLike
A very nice article on peer review but I think there is also a lesson on motivation for scientific research and development. At the time the Australian doctors were researching a CURE for ulcers, the pharmaceutical industry was spending close to a billion dollars developing acid blockers as a TREATMENT for ulcers. These later became known as Prilosec and Nexium. The cure was a short course of existing inexpensive antibiotics. The treatment was perpetual consumption of a moderately expensive acid blocker. If you are a consumer the cure brought to the world by the Australian doctors is a wonderful discovery. However if your are big Pharma there is no money or profit in that solution. You generate much more profit from a treatment and big Pharma’s scientists had a field day mocking the Aussie doctor’s work when it was first introduced. A system like this only works in a highly regulated industry with huge barriers to entry and a government mandated monopoly (patent) that assures high prices if you get government approval.
Think of this in terms of the energy/climate debate. The green energy solution (excuse me I mean treatment) is wind, solder, geothermal and renewable fuels with a generous helping of cap in trade, feed in tarrifs, carbon taxes and subsidies. All technologies that at best supplement base load power and energy generation or drive costs higher for existing energy generation. Natural gas, which if used to it’s fullest potential in combined cycle systems with power generation close to consumption so waste low grade heat can also be put to work, can reduce co2 emissions up to 60% over remotely generated coal fired power. This is better than fossil fuel backed wind and at much lower cost. How have environmentalists reacted to this hopeful development? Mockery, derision and barriers to block this low cost, low impact fossil fuel. Nuclear and big hydroelectric get the same treatment even though they are zero emission baseload power solutions.
In green energy, like big Pharma, the motivation seems to be centered around generating a perpetual revenue stream rather than solving (curing) the underlying problem.
LikeLike
I completely agree that peer review is useless. After all, what are the journals other than glorified magazines where readers edit the stories? I think the internet will play a role in the way that new ideas will break into the establishment but in the long term neither option is satisfactory for establishing good science, where the outcome matters to society.
Climate science needs the sorts of precautionary measures established for other key sciences (medicines, chemicals, etc). Standards in procedures, documentation and data archiving; policing; penalties; review boards; testing. Basically the levels of safety you’d expect for a multi trillion dollar, life or death product.
LikeLike
Pointman
LikeLike
Call me a heretic but Maria Callas scratches my nerves. Aie. And my ears, they bleed. This is a voice: https://dl.dropbox.com/u/1828618/01%20The%20Hidden%20Face.mp3 (skip 5 minutes intro)
LikeLike
Good article PM, and nice to see someone else using the stomach ulcer research- it was my go-to example to oppose the ‘experts know best and only keep out junk science’ argumentation line.
I think however there is a bigger issue than the peer-review debacle (though as you point out, this is self-correcting with the help of the internets and the bloggosphere); and it’s pHd’s.
Or rather, it’s that once you pass your first year review it is virtually impossible to fail your pHd.
You see, it reflects badly on the organisation if a student doesn’t pass their pHd, further, they have targeted numbers of students to pass and a certain one-man-up manship with regard to how many they pass compared to say their nearest rivals (oxbridge being a prime example).
I am witnessing, in real time, a startling example of this as we speak- a shocking pHd project, with controls only ran in the FIRST set, not in any of the subsequent 6 months of work. Poor assertions, even poorer conclusions and a lack of basic statistical knowledge that frankly, scares me in a scientist. Yet they’re guaranteed a pHd, because they passed their first year review.
A review that happens when only the basic shape of your project is apparent, before any substantial work is done and you’ve had a chance to critically evaluate the work and the science. It’s getting to the point where their Supervisors (academic AND industrial, as this was a much sought after industry-placement pHd) are finishing their work fro them and helping with the write up.
Now i hear you all vry oh noes, this must surely be exposed as the travesty it is; but this is not a one off, or even that rare- this happens all the time with worrying regularity.
So, before we worry about the attitudes of the people trying to review the science, i think we need to critically look at the value and the worth of an actual pHd. Just having one is absolutely no guarantee of competence. Give me a 5 year industry scientist over a pHd graduate any day….
*discalimer- this is not to say all pHd’s are worthless or that all institutes or departments do this, but it is very widespread and there is significant pressure to get all people to pass….
LikeLike
Excellent, as usual.
LikeLike
This is going to sound made up, but it’s all true:
I know a person very well. He is honest and intelligent. He has recently completed an MSc in clever stuff where he managed to create something that was a breakthrough in his field and presented his work to an astonished crowd of professors. He was very well received.
At the University he insisted on doing all his own work, testing, making devices etc, because he knew the exchange students didn’t care about what they were doing, didn’t understand what they were doing, confused units which completely screwed results and, believe it or not, made up results to show what they thought was the right answer.
He is now doing a PhD at a different university, where he is building on his previous work. He is already considered to be highly knowledgeable in his field.
Again, he is doing all his own work, testing, measuring etc, but is frustrated because he has found fundamental problems in the peer reviewed literature, which effectively sheds doubt on around 20 years of previous research.
What does he do? Write up the failures (incomplete data, selection of data that only fits the required answer, made up data) and risk not getting his PhD or ignore the problems and produce work he knows to be wrong, but get his PhD? Would the write up documenting the failures ever get published? I doubt it, so he would just ditch his career for nothing, and this is somebody who will reach the top of his field.
His instinct is to be honest, but being honest cold make him a pariah in his field, never get a job doing something he is passionate about and possibly end the careers of many already carrying out research having gained their PhD’s, so what should he do next?
Apologies for the “NoNameProvided” moniker, Pointman, but you have my email address to verify the contents of the above
LikeLike