It’s not often that you’ll see the owner of a website saying this. It’s even more rare to see that person explaining what happened or going into detail about how the site got compromised and what steps were taken to rectify the situation.
However, I wished an article like this had been on the web a few days ago and, now that I know just how widespread this problem has become, I’m going to hit publish in the hope that it’ll help one of the 50,000 other WordPress users who are in the same boat.
Let me tell you a story. On Tuesday I picked up an email from Google. It stated that the “Googlebot” couldn’t crawl the site and the entire domain had a 100% loss when Google tried to access it. I did a quick check, but everything appeared fine and I could see it from various places, so I put it down to a comms blip.
That was a mistake.
Then, after getting up at 3.30 AM on Wednesday, I got home slightly early and decided to catch up on some of the stories and reviews that have been backing up. In my tired haze I lazily typed “coolsmartphone” into Google and absolutely nothing turned up. We’d effectively been delisted. This is bad.
Now, getting traffic from Twitter is one thing, but if you’re not on Google then you’re stuffed. I started looking deeper into it, but again, everything looked fine. It was only when I did a “Fetch as Google” in the Webmaster Tools (which shows how the site appears to Googlebot) that I saw a 301 page trying to redirect every single news story, review and post to some spam site…
Happily (in a way) the spam site was offline so it just errored, but it also had l several links to viagra and all sorts of other rubbish on the top of the 301. As Googlebot doesn’t follow redirects, it was picking up these links.
So, I spent all of Wednesday night, Thursday morning and Thursday night diagnosing this. I SSH’d directly into the server and did a grep (which is a find command) for anything which looked out of place in the code. Nothing really turned up.
I used the command below to speed things up – it’ll grab the site as it appears to Google, so you can do something, run this command, do another thing, run the command again etc. It’s just quicker than having to switch to the Google Webmaster Tools site..
curl -D – -A “Googlebot” site.com (for example)
I looked for “viagra” and any “href” comments in the header.php and other theme files, but everything seemed fine. Worse, a similar issue was appearing on my personal blog, so I suddenly started thinking that the entire server had been hacked.
I tried a few interim fixes, which are the usual steps any WordPress user will tell you to do – deactivate all plugins, change your theme, deleting and recreating .htaccess etc. None of these seemed to work. Even with no plugins enabled and a new theme, the links popped up at the top of the site. Weird. It must be the server, or so I thought.
Then, at around the same time, this WordPress hack started to show up on news feeds across the web. There was then news that 50,000 websites had been compromised. WordPress, for those who might not be aware, powers a lot of sites across the web, including Coolsmartphone.
After reading this I figured it wasn’t related to our “hack” as it didn’t appear to be similar and I’d removed the MailPoet plugin months ago. After a few more hours of frustration I checked it again. Unfortunately, despite MailPoet not showing in the WordPress GUI as either “installed” or “not active”, the files were still there on the server, and the hackers had used this to jump into the site.
To fix and to ensure that there’s no other hacked files (as the hack is quite nasty) I did the following last night..
1 – Backed up Coolsmartphone and the databases associated with it.
2 – Removed and reinstalled Apache.
3 – Deleted all reference to coolsmartphone.com on the server.
4 – Created a brand new coolsmartphone.com domain / virtual server setup
5 – Downloaded a fresh WordPress installation, configured it to talk to the restored database. No plugins activated.
6 – Restored ONLY the images associated with our content – no plugins or themes.
At this point all appeared well, but when I put our theme live the hack came back. So, I had to restore the theme from a backup. I’ve also altered all the file permissions which, although it means that I can’t update plugins on the fly from the GUI any longer, I’d rather have that than leaving files open to the Apache user.
This morning everything appeared fine with the restored themes, so I tried restoring one plugin. Bang. The hack came back again. I deleted the plugin, grabbed the latest new version from wordpress.com, then activated it and all was well again. They had, in effect, got into every plugin and our themes too. So, all-in-all, the hack was in ..
– The themes (all of them)
– The plugins (the majority of them)
– The original WordPress code
I replaced the themes with a freshly downloaded copy, all the plugins with freshly downloaded versions and, in step 5 above, I’d installed a fresh copy of WordPress. It’s a big task and many WordPress users will hate even disabling plugins, because it messes things up, but it’s a necessary evil.
Following all this (and I’m not going into all the detail here as there was a lot of Decode Base 64 stuff in PHP files which prevented me from finding the lines they’d altered … they’re clever). I’ve then spent time this morning re-adding us to Google, so we should be done.
If there really are some 50,000 websites dealing with this hack, then it looks like the fall-out and the text, links, malware, files or redirects deployed aren’t always the same. Hopefully, if you’re one of those webmasters searching the internet for help (as I was), you’ll find this and it’ll guide you through the mess. Short version though, take your site offline, backup your theme and uploaded images, delete your WordPress install, add a new WordPress install downloaded from their site, re-add your images, restore your theme from a known good backup and re-download the plugins direct.
There was a lot of swearing too.