As you are aware both my personal blog and the Raspbmc website went down. This was because of some ‘clever’ idiot who decided to attack me with a SYN flood attack on Port 80. This method of attack abuses the TCP protocol by flooding a network with SYN packets and ignoring the SYN/SYNACK/ACK handshaking protocol. Some suggested that Raspbmc was just very popular. I wish I could blame this as the cause of it going down. But it has been verified that the attack was malicious and was a SYN flood.
For those wondering. A SYN flood is not a skillful attack, but rather a simplistic and idiotic attack that a five year old could pull off. Conversely, it takes skill to defend against them. Initially, I tried SYN cookies, but this didn’t alleviate symptoms. As you can see here below, within a few seconds of re-enabling Port 80 on the hardware firewall, I ended up back down again.
That is the amount of SYN requests after reopening Port 80 on the hardware firewall after 30 seconds.
Thursday evening, and I couldn’t do much to fix the problem. This was because I had an exam the next day, and that takes priority. To make matters worst, my web host was having ‘email’ issues. 2AM Friday, it comes back up. I figured the guy got bored and moved on to something more challenging. Then, Friday 2PM again, the guy strikes again. That afternoon though, a nice guy at NodeDeploy, Ben, who was mirroring Raspbmc for me, steps in. The reason you are able to view this page now is because of him. By tightening the HW firewall, I configured it as such so he only has access to Port 80. We then change A records to the NodeDeploy firewall, which then reverse proxies, returning pages from my server. It’s a bit slower, but it’ll do for now. Throwing 10Gbps of SYN filtering at the problem and the DDoser is small fish.
The first thing that comes to mind when being DDosed is who may be behind it. For me, and a couple of others, it is abundantly clear. Hint: if you are going to DDos me in the future, make sure you spoof the IP of every packet. I won’t name anyone, because without 100% proof, which can’t ever be established, it would be considered slander. What I will say, is that the person who did this did not want RC3 of Raspbmc to be released, hence the impeccable timing. Another interesting thing, is that the attacker did not ICMP flood (ping), but chose to SYN flood. This allowed them to exploit a fault in Raspbmc, namely the issue that Raspbmc checks the Stm Labs server by ping, and if this is successful, attempts to contact the update system. The problem here, was that my server responded to ping, but due to SYN flood, would not serve the files. This meant the boot became stuck. Raspbmc was meant to check for uptime of my server, which it did, but only did so on ping and not HTTP. Thus the attacker has been very specific in their method of attack. Unsurprisingly, the suspected attacker was browsing SVN on Wednesday and browsed the update-system directory of the repository. The attacker was looking for bugs to exploit. This bug did not affect Crystalbuntu users, and soon enough, it will be fixed for Raspbmc users too.
Someone clearly felt threatened by the release of Raspbmc RC3 and didn’t want users to have it. Too bad. Obviously, when something like this happens, the only real thing to do is persist and hope for the best. That’s why, during down time, I continued working on Raspbmc Release Candidate 4, which will be landing shortly, I hope the attacker enjoys it
Some guys wanted me to torrent Raspbmc RC3, or asked why I didn’t already. The reason is, my friends, is that the update system dynamically fetches files from a server. The torrent however, would be static. Furthermore, despite me having a CDN for Raspbmc, there was the issue that redirection to the CDN is dependent on my server. There’s not much of a solution round this, round robin DNS has its shortcomings as well. Well done to fellows that tried to help others by hosting the installer image. Unfortunately, that wouldn’t work, as that image needs to get everything from my server, but it was kind of you to try and help others. Also thanks to anyone that offered hosting. It was generous, but unfortunately of no assistance. I’d also like to give a big hats off to:
- Bytemark hosting – they contacted me, offering help with the project. But as it’s the weekend we didn’t get far.
- NodeDeploy hosting – Ben contacted me as I mentioned and we got the firewall going. What was good though, is I don’t even have an account with these guys, yet I still got regular helpful contact from them. A lot of love for this company. If they treat non-customers like this I bet they treat their customers amazingly.
And the terrible service of 1and1. They caused me headaches because:
- They claim managing a hardware firewall is ‘my job’ but provide no access to it other than the ability to set filters. What I wanted was verification of a SYN flood.
- When trying to change the A record of Raspbmc.com, and still up until now, I am given an error message. I emailed them and got no reply.
- They have the longest TTL known to man.
- Email went down.
You get what you pay for I guess. £30.00/month for a dedicated server is not going to be special, but as a kid who’s going to be £30k in the whole after I soon go to uni, I can’t exactly splash.
So, what’s next you ask? Well. We need to wait for DNS settings to update, but after that, you should be good to go. Raspbmc and Crystalbuntu development will continue as normal and I will continue on with delivering the best XBMC experience for the AppleTV and Raspberry Pi platforms. If you flush your DNS settings, you might be able to beat the wait, but no guarantees.
Thanks for using my software guys, and sorry about the issues. I will be taking preventative measures so this can be avoided in the future. An interesting aside, the Dutch political party D66 want to legalise the kind of attack that hit me today, seeing it as a form of ‘protest’.