My belated condolences. I always enjoyed reading his posts and will miss his insight and wide ranging views. May hew…
The Seductiveness of Models.
When it comes to Global Warming (GW), great emphasis is put on what the computer models predict will happen, so let’s get some idea of what exactly a model is. We’ll do a thought experiment. This is a complex area but no maths, I promise, and I’ll pick a really really simple problem.
Imagine we’re going to build a model of the game of Snooker (or Pool) and we’ll keep it even simpler by only considering the cue ball which is hit towards one object ball. A two ball problem. We want to be able to predict where both balls will come to rest after the impact. The physics of the situation are well understood. If the cue ball hits the object ball in the centre then the object ball will roll straight on. If the cue ball hits the object ball to the left of centre, then the object ball will deflect to the right and the cue ball to the left. If the cue ball hits the object ball to the right of centre, then the reverse happens. When a ball hits the side of the table, the angle it bounces off with will be equal to the angle it came in with.
Some basic Applied Mathematics will give us the energy loss and deflection angles due to friction and the collisions. We specify the problem in minute detail for a software developer who then builds the software model as a program to be run on a computer. At the start of each run of the model, we input the positions of the balls on the table and the direction and power with which the cue ball is hit. The model predicts where the balls will come to rest with what looks like plausible locations but in order to validate the model; we build a mechanical cueing device.
We can set this to hit the cue ball with an exact amount of force and in an exact direction. The test plan is to run the model and the cueing device against each other over repeated runs. The output positions of the balls after a run will be used as the new start positions for the next run. At the end of the first run, the model has predicted where the balls actually came to rest but with a 5% error. On the second run, the error has increased to 8%. Long before we get to run ten, the model’s predictions are hopelessly inaccurate.
After some head scratching, we determine that we’ve not added to the model any information about the effect of the cue tip on energy absorption. After some research on cue tips, we amend the model and rerun the experiment. The first run error has dropped to 3% but by run ten, the predictions are still hopelessly inaccurate.
The programmer suggests that doing the calculations to 10 decimal places rather than the current 5 will improve accuracy. The software change is made and the error on the first run drops to 2.5% but by run ten, it’s hopeless. They increase the precision to 20 decimal places and the first run error drops to 2.4% but the run ten predictions are still useless.
Rather than believe the manufacturer’s numbers, they weigh the balls and find the weights are not 100% precisely as stated. They amend the model and rerun the sequence with only a marginal improvement in accuracy.
They know there’s something wrong with the model and in an effort to find out what that may be, they consult with someone who plays Snooker. He suggests their cueing machine may not be hitting the cue ball in exactly the centre, introducing ‘side’ or ‘spin’ which would throw off their predicted collision angles. They add a laser sighting device to the cueing machine and rerun. First run error drops to 2.1% but the results at run ten are still disastrous.
They add laser distance gauges to the table to determine within microns the exact positions of the balls and to ensure the table is completely flat. Again, a marginal first run improvement but the usual run ten disaster.
The suggestion is made that the coefficient of friction of the felt on the table may not be as uniform as they assumed. The measure it in grids on the table and indeed find minute variations. The model is amended to include the varying friction patches of the table and the experiment is rerun. First run error drops to 1.8% but it’s the usual disaster by run ten.
The next idea is to examine the coefficient of elasticity of all the sides of the table. Sure enough, there is variation, even if it is tiny. This new information is programmed into the model and the experiment rerun. First run error drops to 1.4% but the usual result by run ten.
The suggestion is made that slight currents of air or tiny variations in its density during the day may be having an effect. The table and cueing machine are moved to a climate controlled chamber and the experiment rerun. Minor improvements result.
The wooden cue is replaced with an iron rod which will not flex as the cue does. The model is amended to take account of that. Some improvement ensues. First run error drops to 1.34% but by run ten etc etc.
And so it goes; on and on and on and on …
I have to confess at this point that though I said I’d pick a simple problem and I did, I also know that when it comes to modelling reality, there’s no such thing as a simple problem. We understand the physics and mathematics of this problem and can come up with a first run prediction that is well within some reasonable range of error. What we cannot do, is feed the result of a run into subsequent ones because all that happens is the error margin is soon exceeded. Any error at all will soon magnify on subsequent runs.
This is exactly what climate simulators do and why they need such powerful computers; it’s not that the calculations are complex, it’s the simple fact the simulation is run hundreds or thousands of times. The output from any one run may look plausible as the input for the next run but the overall result is guaranteed to be completely wrong.
When it comes to modelling climate, we simply don’t know enough about the factors we’re including. We still don’t understand clouds and as for turbulence, come up with the math to handle that adequately and you’ll be on the receiving end of a Fields Medal and a Nobel Prize. Newton and Einstein will turn green with envy. We also have to consider that there may be major ‘unknown unknowns’ we’re simply not aware of. Until a few short years ago, the major weather phenomena whimsically called Sprites, Jets and Elves were undiscovered. What else is out there?
The people working on the Snooker problem also illustrate the biggest drawback of climate modelling; it’s not possible to validate any of the models because there’s no way to build the equivalent of the cueing machine. We’ve simply got to believe that what they’re predicting to happen in a few decades or hundreds of years is accurate. Believing them is essentially an act of faith. The models don’t even agree with each other and sometimes by quantitative amounts greater than the temperature rise they’re predicting.
Hands up anyone who thinks further development of the snooker model would have gone on if they’d not built the cueing machine which showed them how far from reality the model’s predictions were?
Weather forecasts for tomorrow are fairly accurate, for the day after that less so and for the third day; you’re looking at a little better than evens. Does that behaviour sound familiar? A whole branch of Mathematics, called rather fittingly Chaos Theory, grew out of that sort of ‘spiralling unpredictably out of control’ behaviour exhibited by one of the first attempts to simulate the weather on a computer.
The essential seductiveness of models is that the feeling grows in those working on them that by adding enough refinements to it, it will be 100% accurate. This can never happen.
The person who really knows these things now is the programmer whom I shall call Harry, for reasons that are not that particularly enigmatic. One evening, working late yet again amending the model, he decided to take a break and visit the bane of his life but also the source of most of his work; the cueing machine. He decided to run the experiment for himself.
He ran the whole thing and watched the usual disparity grow between the predicted and actual positions of the balls and then he had an idea. Forget the model; just run the cueing machine again but using the exact same starting positions for the balls. At the end of the first shot, the balls came to rest but not in precisely the same positions they had the first time. By the tenth shot, they were nowhere near where they had been the first time. Just to be sure, he reran the whole thing again with the same exact start positions. By run ten, the balls were in totally new positions.
He sat and thought for a while about what he’d just seen. Eventually he tidied up the equipment and left to go home early for a change. Harry never visited the cueing machine again.
©Pointman
Pointman's 
What a great example of the impossibility of using computers to predict anything – let alone something so complex as climate. I now know why the models will never be accurate.
Pointman you have such a gift for explaining these things in such a way that even I understand.
LikeLike
meltemian says:
January 21, 2011 at 6:31 pm
Celestial mechanics has a pretty good handle on prediction. Fortunately, for ‘action at a distance’ the number of variables which must be accounted for are somewhat less than apply in other modeling scenarios. The theory is pretty well validated too (Einstein’s relativity).
LikeLike
Great article. Do the climate models work something like this? The computer modelling all hinges on the belief that carbon is driving climate. They feed in the software determined by the amount of carbon dioxide in the atmosphere and out comes the predicted temperature… or something?
LikeLike
Pointman – I agree with Mel, your capacity for explaining the unintelligible to folks like us is much appreciated.
While the Climate is one thing, I couldn’t help drawing a comparison with Economic modeling. After all, this is what determines variable Interest Rates, predictors of inflation, Budgetary constraints or largess, the rates of Pensions and Welfare, infrastructure projects et al. You get something like a natural disaster thrown into the mix, folks thrown out of work = no income, no tax paid + disaster welfare benefits, suddenly Bob’s your Auntie.
Heard a radio interview the other day with an Actuary expounding on the ramifications of these things for the National Economy and it suddenly gets real scary. Enough to shake anyone out of their complacency.
Your example shows so clearly why there is zero certainty in ‘predicting’ anything – it’s all wing-and-a-prayer stuff.
LikeLike
Yeah, well I’m stupid, you see? How do you know what the reality is that you should be comparing the computer runs with? – Or are we talking about the disparity between computer runs describing identical shots?
LikeLike
Yes, I do see.
Pointman
LikeLike
I used to be a modeller – modelling electronic circuits which should follow the laws of physics, and telephone traffic patterns which should follow short-path network theory.
The interesting thing was that the telephone traffic patterns – number of calls between two nodes, and the mean duration of a single call, could be modelled with reasonable predictive accuracy.
Electronic circuits, where each component can be analysed individually to the “nth” degree, could not, as per your excellent example.
The difference between the two, was that each telephone call was a discrete event – with a start time, route, and finite duration within known capacity limits – they were discrete phenomena. The electronic circuits, on the other hand processed continuous signals at various frequencies and amplitudes – they were analogue phenomena.
Balls moving on a snooker table are also analogue phenomena, in a number of dimensions at once, including the loss of energy due to air resistance, etc. The same applies to the climate, only there we also have a number of analogue variables that we probably don’t even know about yet.
Computers are digital, they are very good at counting things very quickly, so they can solve digital algorithms that describe discrete events. But you can’t count something that is continuous, especially if it is continuously changing, so digital computers are the wrong tool for the job.
If you want to model analogue phenomena, you will need to invent something that we do not yet have.
“Having a super computer gives you a way of getting the wrong answer faster and with more precision”.
LikeLike
Love the clarity of this article. As you say, the tiniest variation in initial conditions can have drastic consequences “down the line” — hence the “butterfly effect”. This presumably implies the heat generated by a supercomputer used to model climate may itself have a significant effect on future weather?
LikeLike
I’m sure they’ll add it to the model …
Pointman
LikeLike
His Grace has published this re: the Report by the Met Office.
http://www.bishop-hill.net/blog/2011/1/21/a-copy-of-that-cold-winter-forecast.html
HOW much do they want for the new computer????
My money’s still on Piers and an old laptop!
LikeLike
Thankyou pointman for an excellent post
Weather forecasting is the eqivalent of casting a ping-pong ball precisely from a bridge into a moving stream and trying to predict its position some time later.
Surely it is obvious to any sane person thisis impossible, just too many unknown variables.
And yet this just what is hapening with climate models and financial models. The energy industry, the insurance industry and the pensions industry are little better but they cover their track by increasing their prices to cover losses. eg Car insurance up by 20%+ no surprise!
LikeLike
The comment by manonthemoor, about models in the insurance and pensions industries, brings another thought to mind.
There are three types of models – open, closed, and hybrid.
Models used in manufacturing, to balance machine and warehouse utilisation, for example, are closed systems because every aspect of the operation is under the control of the organisation. They work.
Models used in the insurance and pensions industries are hybrid, since they use the same fundamental processes of the closed models, with the addition of actuarial control parameters based on the known mortality tables. Such models assume that in the long run, probabilities will even out. They also assume that the world is in a steady state, or only changes very slowly. They work for most of the time, but do not, and cannot, allow for natural disasters and other unpredictable or purely random events.
Open models attempt to learn from previous experience, and use this information to extrapolate the future based on that experience. The correctness (or not) of the extrapolation is then fed back into the model as corrective parameters, and the model is then used to extrapolate a new answer. These are learning systems, and are often used in simulations. They never work! They are not supposed to work, because that is how they are designed to learn. They also require a steady state; they get confused by random events.
Now the question is: Which of these types of model best describes the ones used to predict climate change?
LikeLike
Hello Rereke and welcome. Of the three choices you presented, I’d have to say none of them are an exact fit. The nearest would be the closed model but there’s two added complications. Firstly, I’m not as convinced as the model designers that all the factors are known and understood and secondly the iterative nature. They are not one-shot first run models. The outputs from a run are used as inputs to the next.
Pointman
ps. There’s a reply to your comment at https://thepointman.wordpress.com/2010/12/21/the-msm-and-climate-alarmism/#comment-238
LikeLike
That is interesting, because I would have excluded the first model on the basis that they (should) know that there is a whole bunch of stuff that they don’t know.
They are certainly using the third type of model because they are iterating, but doing so without any real understanding that such models are almost guaranteed to give you the wrong answer – that is their function.
Perhaps they believe that they can create a hybrid that has all of the benefits of both, but none of the disadvantages of either? As they say, “Yeah, Right”.
Oh, and thank you for making me welcome – you have a nice site.
LikeLike
@Rereke. They’re certainly using the closed model and iterating but there’s no learning or annealing going on in the iteration as would happen with a neural network model. It’s as simplistic as that really.
Pointman
LikeLike
Hey P! A really excellent piece.
LikeLike
Pointman it certainly is “Hard to concentrate on the game at times …”
I worked with computers for 30 years and have now retired. I worked as a developer/analyst in a large government department. There were about 40 people in my section which handled all aspects of financial support for tertiary education in Australia. The systems involved distributing billions of dollars annually. Changes to systems involve the following. The business section asks for implementation of functionality as requested by government. The business analysts write up the specifications and the developers then implement them. This software is then tested to see if it complies with the specifications. To some this may seem like a simple enough task but it is not. Mainly the complexity occurs because of interacting rules that implement legislation over time.
The fundamental thing needed for any large computer application is definite testable rules which this has. You have this with business systems and engineering but not with the soft sciences. Any alteration has to be rigorously tested to make sure there are no side effects and that required result is obtained. With 20 people producing new functionality and 5 people testing it is not an easy task.
From this background I became interested in climate models or should I say general circulation models. I read what I could (bought some very expensive books), looked at code and sought out the details of those developing them. What I found was that these were very much research projects run by educational or government research organisations. In a sense they are low budget operations which have very expensive super computers and are outside the mainstream of the computer world. They are not developed by software engineering team in any sense and do not follow the processes I explained above. They need very powerful computers and use FORTRAN. Performance is a huge issue so what you do is switch off bug checking once it seems to work okay after that errors can be very subtle. I was looking for something like the space shuttle software development or considering the complexity of the problem something much bigger. Implementation of shuttle software initially took 400 people 5 years. Their work was openly published for all to see and obviously tested.
Digital computers just can not cope with ambiguity maybe analogue ones might but to simulate something it is imperative the interactions must understood in detail. Many types of computer model are useful but the results of the worthwhile ones are testable and that is the point your, billiards example is but GCMs are not because they try to predict an unkown future. If I was to produce the model for billiards then I would do it as follows. Get someone to play the game and record what happens. Then I would introduce adjustable parameters into the model that take effect as required. This way it is possible to hind cast and pass the test. In my opinion totally invalid but that is one of the tests used for climate models and adjustable parameters used. Another one is applied because reality can be sidestepped by comparing the models to each other, if the output matches then both must be correct. A third was proposed by the English Met office. In a hearing concerning Climate Gate the question was asked about testing. Simple the same code is used for day to day forecasts so obviously that is accurate so therefore a forecast well in the future also is. Total rubbish even tomorrows forecast can be 2 or more degrees in error.
Finally I doubt the general public understand that this is an argument about whether climate models are correct. People in the scientific world may be accepting of opinions though the filter of a computer, which lends authority. The authority of priests of climate change will have to erode before anything else for it to go away. I think it is just a perceived authority rather than any fact or logic. The way the world works unfortunately is that showing there is a same sex affair involving them would be a far more effective way to destroy their authority than proving their computer models are wrong. Even though it would have nothing to do with their validity.
LikeLike
Thanks Mike.
The trouble is that ‘numpties’ like me who know very little about computer programmes just hear “Computer Analysis” and believe the models must be right! The complexities of creating a climate forecasting system must surely be almost impossible!
LikeLike
All computer programs have more or less errors. We call them undocumented features. Human create them and human are fallible so the programs are also. Another surprising feature is that they age and become less reliable with time. This occurs because of inadequately tested bug fixs and new features.
LikeLike
Hello again Mike. I think your comment underscores some of the points I was attemting to make on this topic, especially the absence of any real testing of the models. Our Met. Office said they used the same models for both short and long term forecasts and that was good enough testing for them. After the models had made three successive years of badly wrong predictions, what did they do? They discontinued the practice of making predictions a year ahead. So much for testing then …
The topic is important because all the predictions of climate alarmists are based on what various computer models say will happen. Take these models out of the debate and all you’ve got left is Phil Jones’ statement that “there’s been no statistically significant warming in the last decade”. These models or should I call then climate simulators, are basically rubbish.
The science is toast. These people have lost credibility and I don’t see how they can ever get it back. At the end of the day, it’s always been about politics not science as we’d know it. So yes, if it takes a scandal or two to completely destroy what’s left of its integrity, I’d have no problem with that.
Pointman
LikeLike
Please don’t think I am disagreeing with you or that I think all computer models are worthless. In Australia it is blind acceptance of authority that is the problem there is a blanket acceptance of the “science”. For example our recent cyclone Yasi and the Brisbane floods have been blamed on climate change. A classical example was a recent tv show called Lateline by the ABC (like your BBC). The compere Tony Jones asked a guest Gaurnot whether he thought Yasi was caused by climate change. Answer definitely and we have not seen nothing yet. Gaurnot is the equivalent of your Stern. Advisor to our left wing labour gov and director of a gold mine in the islands. The common statement here is “the scientific evidence shows” which is used as a mantra. Gaurnot is a huge part of what is driving us to a carbon tax. I think very few of the populace here have any idea of what the scientific “evidence” is or have heard of a GCM.
I am interested in what Blackswan has to say about this. The population here will accept the carbon tax because authority says it must there is little other information to make a decision on. Maybe this will change when it starts costing the alternative proves to be a failure. The irony is even it were all true we have no way of changing anything. I we all died here achieving zero emission it would very difficult to measure it.
LikeLike
G’day MikeO
Thanks for the overview based on your experience – invaluable.
My favourite bit? “I think it is just a perceived authority rather than any fact or logic.” Now THAT sums it all up.
LikeLike
Thanks Blackswan do you think the general public here has even heard of a climate model?
LikeLike
Hi MikeO
I’m increasingly of the opinion that the majority of our population are sleepwalking through their lives, with their opinions fashioned by a hopelessly biased MSM – although some rays of light shine through from time to time.
I reckon you’d enjoy this…….. http://blogs.news.com.au/dailytelegraph/timblair/index.php/dailytelegraph/comments/cyclone_garnaut/
The comments are encouraging – pretty ‘cluey’ about Climate Fraud……. at last.
Tim Blair is just the bloke to give Garnaut a well-deserved slapping and does a fine job.
LikeLike
Anthony Watt’s blog WUWT is a finalist in the science category of the Bloggie Awards, sort of a blog Oscars. If you wish to vote for it, which I’m going to do, please visit http://wattsupwiththat.com/2011/02/06/wuwt-nominated-in-the-2011-bloggies-awards/ for instructions on how to do so.
Pointman
LikeLike
Thanks Pointman – good as done.
LikeLike
Me too. Will do.
Liked the Tim Blair link by the way Swan.
LikeLike
It all depends on your models and what you want them to do. “y = x squared” is a model. It works every time. No matter what value of ‘x’ you insert you will always get the correct value of ‘y’. Period. The trouble comes when you have an equation derived from the best understanding of physical events. No matter what you do, there are always some details that you cannot know – as Pointsman points out with reference to Snooker. And whether or not you have a model that will give you the ‘correct’ answer depends on what error you are willing to accept. Like the Railways’ version that says a train is considered to arrive on time if it arrives no more than five minues after the timetabled time. The allowable error is five minute. Only four minutes late and it is ‘on time’. Everybody is happy – well, most of them, anyway. If you change the allowable error to three minutes, a lot more trains will not arrive ‘on time’.
Pointsman’s basic point is that in trying to forecast a continuous complex event it is necessary to iterate the model using data obtained from aprevious run – which may have a minute error in it – which will almost certainly have some error in it. The Met Office can forecast weather in Melbourne and Sydney and Brisbane very well for a few days – sometimes a week ahead. This is because weather moves across Australia, in general, from west to east. So as they know what weather in Perth is like, and in the surrounding areas, they can make a pretty good prediction of weather a few days later in Adelaide and Melbourne and Sydney. As the weather moves, the error in the previous forecast can be reduced by new data. So by the time the weather reaches Melbourne or Sydney, the forecast is pretty accurate. The up to date forecast is better than the forecast given 5 days earlier because the input data is updated.
This won’t work will long range forecasting because there is no way to update the data – no time machine is available to get updated data from the future. So the errors accumulate, and the Met Office has egg on its face. In the UK it was possible to make a weather forecast for London for tomorrow by knowing what the weather in Cornwall is today. Try to get a longer range forecast right and one is thrown back on the some of model baed on pst history – if it is summer it will be warmer. If it is winter it will be colder. There has always been at least two inches of rain in Croydon in February, no more than point five inches of rain has ever been recorded in Croydon in February, the average per day was zero point two, so we expect it will rain on ten days, but it could be as few as four days, and as many as 28. Except in Leap Years that last .forecast will be 100% accurate!
Thank you, Pointsman for your article, and letting me have my little say so!
LikeLike
Reblogged this on pdx transport.
LikeLike
LikeLike
Reblogged this on 4timesayear's Blog.
LikeLike