So, was climategate a hack after all?

Posted by Pointman on July 27, 2012 · 36 Comments 

This is going to be one of those difficult articles to write, just to make it easy to read. It’s going to get technical at points, because it has to, so there’s going to be more than a few detours to explain things, but bear with me. It’s Sunday afternoon, I’m wearing sunglasses indoors, it’s a long way to the Cook County Assessor’s office, I’ve only got half a pack of smokes left, so I’m just going to put that gas pedal right to the metal and hope for the best; let’s do it.

Norfolk police are closing down their search for whoever was behind the climategate security breach. Nobody has been arrested and it’s being put down to an external hack. I’ve always felt confident it’d end that way for a number of reasons. The main reason they were doomed not to nab them is quite simply, nobody wanted them caught. Any trial would be a huge liability and embarrassment to the climate establishment. Even what they could be charged with is a highly debatable point. Was it data theft or whistleblowing in the public interest? After the whistleblower, or FOIA as they’ve come to be known, put a dead man’s hand password into the second climategate release of emails, they became untouchable. If they go anywhere near FOIA, the blood will be ankle-deep.

I’ve also always been confident it was a leak, rather than an external hack, and still think that, despite the closing statement made by the police. I said think that, rather than believe that, so I can give you my reasoning. Let’s do the high level reasons first, and then we’ll move onto the dreaded technical details.

There’s this saying; money talks but bullshit walks and it’s very true. When you add in all that cross charging of those various public and private organisations involved, the grand total spent on the whole investigation over the course of three years, appears to come down to the princely sum of 85,000 pounds sterling. Come on, that’s nothing. It wouldn’t even buy the tea bags allowance of Norfolk constabulary for the year, and those coppers are more into their coffee than their tea nowadays. With that amount of money spent on it, we’re definitely talking about going through the motions, because despite what a lot of people might like to think, cops actually know what they’re doing and they operate in a world of realities, both at a pavement and on a political level.

On this one, all the boxes have been carefully ticked. Everyone who might conceivably be of help has been roped in; the Metropolitan Police Counter Terrorism Command (SO15), the National Domestic Extremism Team (NDET), the Police Central e-Crime Unit (PCeU), that work experience kid in the office, who seems to know an awful lot about computers, and even the private sector specialists QinetiQ. Everyone got involved and even the Guardian has had its usual exclusive interview with the guy leading the investigation. Fireproof, baby, fireproof.

They always knew that nobody wanted a result, or God forbid, a trial. A complete time waster. They did their bit in the business, the guy leading the investigation will get a gold star and they’ll get back to doing some real policing. Professionals hate having their time wasted but occasionally, it’s just the loops you have to go through.

Of all the email servers, in all the world, you gotta break into mine. We all suspected the science was being rigged, but who in the hell knew the proof of it would be sitting on an email server in an obscure centre of higher education, the University of East Anglia or UEA, as it’s more commonly known. It’s not as if it’s Princetown, MIT or an Oxbridge college. For God’s sake, even the students there refer to UEA as the University of Easy Access.

Someone knew there was gold in dem thar emails. There’s no escaping that simple fact. Even if you do believe it was an external hack, there had to be an insider, who knew it was worth the big effort of a frontal assault and the risk of a prison sentence, if they were caught. The only way an insider would have known that, was because they’d seen the emails. And if you concede that point, then why on Earth wouldn’t the insider themself just copy the emails they’d seen? I’ve yet to hear credible answers to those questions from anyone, who asserts it was an external hack attack.

Another fact that came out, is that the police have no record of any similar cyber attack on any other institution in the field of climate science research. The supposed hacker hit the email jackpot with their very first try. When it comes to significant events, once is happenstance, twice is coincidence and thrice is enemy action. At the best of times, I have a real problem with happenstance, and in this context, I just don’t believe in luck like that.

All the great whites work for money and they don’t work cheap. Of course, they could have been paid by that shadowy organisation, known only as Skeptic Central in whispered conversations in the darker corners of the internet, but the science had already been ripped apart by the climate realists. Even before climategate, it was walking wounded, a dead dude staggering, a piece of green runny stuff hobbling along between two creaky crutches.

Let’s move on to the technical details. Before I launch into them, it’s important to get inside the head of a top of the food chain hacker, or what I refer to as a great white. You have to look at the world through their eyes. They’re ghosts, ghosts in your system. They live inside there and you’ll never know they’re in there. They worked hard for it, they have it, they now own it, it’s their habitat and their bitch and the last thing they’ll ever do, is something stupid that lets you know they’re there or that they ever were there, even when they decide to get out. That’s the discipline. Hunting one of them down is like tracking someone across a desert, who doesn’t leave footprints. Just keep that in mind as you read what follows, because I’m going to examine the supposed attack, purely from the footprint aspect, because I know I can keep that article to a reasonable length.

You can find the Norfolk police statement, accompanying the announcement of the closure of their enquiries here. They also issued an Operation Cabin, which is what they named the investigation, Q&As sheet which you can find here. When you read through them, you can’t help but come away with the impression that it was all the work of an unknown hacker, with no indication of anyone inside being involved. They’re short on precise details but let’s take a hard look at the few ones they give.

First off, whoever the hacker or hackers were, they were sophisticated. To quote, the investigators concluded that “the attack was highly sophisticated and was undertaken by a person or persons who were highly competent and who knew how to conceal their activity.” Sounds very much like a great white to me.

As usual with things climategate related, I’ve no IT forensics to work with, but there’s enough information to reconstruct a reasonable skeleton outline of the attack, from what few details are given. I wrote a high level description of how an external hack is done, which you can find here. If you haven’t read it, I’d recommend reading it, as it’ll give you a context to hang the following details on. It’s not possible to deduce anything about the reconnaissance or fingerprinting stages, but the break in appears to have been done on a web server called CRUWEB8. From there, the traversal stage got them to a backup server called CRUBACK3, which is where we’re told the emails, source code and documents were extracted from. The last phase, an orderly withdrawal, appears to have been a complete disaster. Not only did they leave traces but even the supposedly misleading ones they left, the clever trackers worked out were just bum steers.

We’re told there was evidence of them trying to get through password screens. I’ll admit I’m guessing here, but someone tried a few passwords like “secret” or “administrator” and the system logged it as a failed login attempt. So what? Sorry, but that happens every day to login screens on the internet. Usually, try more than three bum guesses and the system not only logs them all, but it also locks the account anyway.

Cracking passwords doesn’t work like that. You never go near a password screen. You stay away from password screens. Password screens are not good. The usual technique is that you go after the encrypted file, containing all the logon Ids and their matching passwords. Once you get a copy of that back home to Skeptic Central, you let your favourite brute force password guesser chew away on it for a few days. Feed it enough dictionaries and nine times out of ten, you’ll get the passwords for several useful accounts, which have weak passwords. A few failed login attempts prove nothing but they do indicate it wasn’t a great white chancing their arm. They don’t go anywhere near login screens, until they’ve got a logon Id and password that they know works.

Anyway, how did they know there had been failed login attempts or even the details of the security breach? The activity logs, obviously. Okay, but wouldn’t a great white just edit the logs and delete any activity they wouldn’t want anyone to know about? The slick answer to that one is that they did actually edit the logs, but the equally slick forensic people, recovered the original logs and all was revealed. How would they have done that? To answer that one, we’re gonna have to take a technical detour here, but bear with me for a moment.

When you edit a file and save it, what actually happens is a brand new file is written to the hard disk and then the original file is marked as deleted. It’s not actually deleted from the disk but the space it occupies is effectively marked as empty, so it can be used for storing new files. All those perfectly legitimate uneraser programs you can buy, which recover deleted files, simply scan across your hard disk, marking deleted files as no longer deleted. Hey presto, you’ve got them back again. Depending on whether the space marked as deleted has already been written over or not, the uneraser will recover all deleted files, in whole or in part.

Obviously, there’s no point in editing the logs, if the originals can be recovered. The way out of that problem is to write your very own program, which scans across the hard disk, writing garbage data onto the physical space occupied by files flagged as deleted. That’s a ten minute program to write and again, you can buy perfectly legitimate secure delete programs, which do exactly that.

You might think at this point, that whatever was deleted is now definitely beyond forensic recovery but unfortunately, there’s this data recovery technique, which has various names, but I’ll go with latency. It’s about a magnetic surface, like a hard disk, having ghostly imprints of old data under what’s currently written there. Think of it as a painting on a canvas that the artist has whitewashed over, because they’re going to paint something new on top of it. If you x-ray the canvas, the old image magically re-emerges. In a similar way, it’s actually possible to recover data that’s been overwritten. That could be a bit of a problem, but there’s a couple of ways around it.

The first one is to repeatedly write garbage data into the disk space occupied by deleted files. In a sense, you’re pushing the data you want to hide, so deep into the disk surface, that it becomes unreachable by any sort of latency recovery technique.

The second is more drastic and is only rarely used. You simply destroy the hard disk.

You do that by sending some instructions to the device driver of the hard disk. Again, a little diversion is necessary to explain what a device driver is. You can buy any one of hundreds of different hard disks, made by different manufacturers, and attach them to your computer. Mechanically though, each hard disk will have its own unique way of working and your operating system, say Windows or UNIX, can’t be specially written to communicate with each and every variant. Instead, interface software, which is a standardised agreed interface between the operating system and the hard disk, is written by the manufacturer themselves for each of their products. That’s a device driver. Think of it as buying any type of toaster and knowing you’ll be able to use it at home, because it has a standardised plug. The plug is analogous to a device driver. It’s the common interface.

Device drivers are very low-level programs which do detailed physical things like reading a single sector of a disk or printing a buffer of information. Hackers love them, because the people who write them rarely consider security. While the system administrator is busy welding more layers of armour-plate onto the web server, the hacker is busy reverse engineering the device driver of something like their shared printer, because that’s going to be their way in. If data can flow to and from any device attached to a system, then potentially, it’s a way in.

It depends on the physical characteristics of a particular hard drive, but there’s stuff you can do, that is guaranteed to physically destroy the thing. For example, you can spin it, and keep spinning it until it bursts into flames. It’s like gunning your car at 100 mph in first gear, eventually you’ll see the pistons and con rods exploding out of the hood or bonnet in front of you. The problem with that one, is that you might only have burnt out the motor in the thing, but the disk surface is still physically intact. What you really need to aim for, is something called a head crash.

A hard drive is very much like a record player, except it spins at thousands of revolutions per minute and the equivalent of the arm with the needle in it, is never supposed to touch the disk. If it does touch the surface, it’ll gouge lumps out of it. That’s what’s called a head crash. Once they’re into the device driver, it usually isn’t much of a problem to persuade it to do a head crash or its equivalent. If you’re being thorough, several head crashes followed by a spin it to the limit burnout will do the job quite nicely. Good luck with any data recovery after a disk grind like that.

Again, when you think about footprints left around the system, you have to ask why the great white didn’t grind every disk in the place. As soon as the material was released, it was going to be obvious where it came from. Even if they’d been careful about not leaving any clues behind, why take any chances? Just destroy all the evidence. Bring the whole temple down on your way out.

Jeez, if only the great white had known all about this stuff, they’d have gotten away with it cleanly. Strange though, even the script kiddies know something about some of this stuff, but I guess they weren’t so sophisticated after all. Perhaps they just weren’t a great white or perhaps they never even existed in the first place.

Stepping back from the whole thing, the business of leaving no footprints behind is an exercise in sheer hard work, but it’s the only thing between them and a prison sentence, so they’re always very careful with it. You’ve got a target, you’ve had the patience to determine exactly everything that they’ve got on their box, so you do the dry runs. You put exactly the same software on your very own box, turn on all possible audit logging and take what’s called an image of the hard drive. Think of it as a photocopy or a save game point of the whole machine. Next you do your run, but against your machine, and it either gets through or it fails. Every time it fails, you do a reload game, and have a crack again. They go around that loop again and again, until they get through on their own machine.

The successful dry run tells them two things; the first is that they’ve found the way in, but the second and actually more important one, is the footprint of the hack. They”ll compare the before and after images of the C drive, to see what are the differences. It’s a bit like holding the before and after photocopies together up to a strong light, just to see the differences. Every difference you see, those are the files you’re going to visit for a quick edit, to remove any traces of the run.

They’ll do a reload game yet again and repeat the run, but this time, they’ll additionally do all the edits and secure deletes of the log files, to conceal the break in. After that, they’ll run the before and after comparison again, to make sure there are no differences. Once they’ve done all that, they’ll actually launch the attack for real, and there won’t be a single keystroke deviation from the dry run. Not one.

Anything significant they do inside the system, will also be dry run in advance, just so they can spot and erase the footprints.

Selling the idea that it was a hacker is very easy. Normally sane adults, and not a few IT professionals, who should know better, fall for it because it’s kinda sexy. It’s also a very difficult story to refute, not only because you’ve no access to the forensics, but also because any explanations rapidly descend to a level of technical detail, which lose most people. Even when you’ve access to a system, which you think might have been compromised, it can be difficult to tell if it’s actually been attacked. If you’re dealing with a great white, you’ll only ever pick up the subtle signs, such as the space occupied by deleted log files being full of garbage data, and sometimes, not even that, because that in its way, is a footprint. They’re just as likely to fill it with plausible, but misleading data.

The way you break down an alibi or cover story, is by going over all the details. When I’m told climategate was the work of a “highly sophisticated” and “highly competent” hacker, but then look at some of the details, like the footprints left all over the job, I’m unconvinced, to say the least.

I’ve always considered that if climategate was a hack, then it would have to be the work of a great white. However, I’ve never encountered one, who’s made so many dumb mistakes in the cleanup phase of a run. Too many footprints. Climategate being a hack, is just a no sale for me.

©Pointman

Related articles by Pointman:

Helping to catch the climategate whistleblower.

Why Climategate was not a computer hack.

Some thoughts and some questions about the Climategate 2.0 release.

Profile of the Climategate Whistleblower.

Click for a list of other articles.

Filed under Article · Tagged with Climategate, CRU, emails, FOIA, Hacking, Leak, Security, UEA, Whistleblower