Network issues

Hi everyone,

As you are aware both my personal blog and the Raspbmc website went down. This was because of some ‘clever’ idiot who decided to attack me with a SYN flood attack on Port 80. This method of attack abuses the TCP protocol by flooding a network with SYN packets and ignoring the SYN/SYNACK/ACK handshaking protocol. Some suggested that Raspbmc was just very popular. I wish I could blame this as the cause of it going down. But it has been verified that the attack was malicious and was a SYN flood.

For those wondering. A SYN flood is not a skillful attack, but rather a simplistic and idiotic attack that a five year old could pull off. Conversely, it takes skill to defend against them. Initially, I tried SYN cookies, but this didn’t alleviate symptoms. As you can see here below, within a few seconds of re-enabling Port 80 on the hardware firewall, I ended up back down again.

That is the amount of SYN requests after reopening Port 80 on the hardware firewall after 30 seconds.

Thursday evening, and I couldn’t do much to fix the problem. This was because I had an exam the next day, and that takes priority. To make matters worst, my web host was having ‘email’ issues. 2AM Friday, it comes back up. I figured the guy got bored and moved on to something more challenging. Then, Friday 2PM again, the guy strikes again. That afternoon though, a nice guy at NodeDeploy, Ben, who was mirroring Raspbmc for me, steps in. The reason you are able to view this page now is because of him. By tightening the HW firewall, I configured it as such so he only has access to Port 80. We then change A records to the NodeDeploy firewall, which then reverse proxies, returning pages from my server. It’s a bit slower, but it’ll do for now. Throwing 10Gbps of SYN filtering at the problem and the DDoser is small fish.

The first thing that comes to mind when being DDosed is who may be behind it. For me, and a couple of others, it is abundantly clear. Hint: if you are going to DDos me in the future, make sure you spoof the IP of every packet. I won’t name anyone, because without 100% proof, which can’t ever be established, it would be considered slander. What I will say, is that the person who did this did not want RC3 of Raspbmc to be released, hence the impeccable timing. Another interesting thing, is that the attacker did not ICMP flood (ping), but chose to SYN flood. This allowed them to exploit a fault in Raspbmc, namely the issue that Raspbmc checks the Stm Labs server by ping, and if this is successful, attempts to contact the update system. The problem here, was that my server responded to ping, but due to SYN flood, would not serve the files. This meant the boot became stuck. Raspbmc was meant to check for uptime of my server, which it did, but only did so on ping and not HTTP. Thus the attacker has been very specific in their method of attack. Unsurprisingly, the suspected attacker was browsing SVN on Wednesday and browsed the update-system directory of the repository. The attacker was looking for bugs to exploit. This bug did not affect Crystalbuntu users, and soon enough, it will be fixed for Raspbmc users too.

Someone clearly felt threatened by the release of Raspbmc RC3 and didn’t want users to have it. Too bad. Obviously, when something like this happens, the only real thing to do is persist and hope for the best. That’s why, during down time, I continued working on Raspbmc Release Candidate 4, which will be landing shortly, I hope the attacker enjoys it :)

Some guys wanted me to torrent Raspbmc RC3, or asked why I didn’t already. The reason is, my friends, is that the update system dynamically fetches files from a server. The torrent however, would be static. Furthermore, despite me having a CDN for Raspbmc, there was the issue that redirection to the CDN is dependent on my server. There’s not much of a solution round this, round robin DNS has its shortcomings as well. Well done to fellows that tried to help others by hosting the installer image. Unfortunately, that wouldn’t work, as that image needs to get everything from my server, but it was kind of you to try and help others. Also thanks to anyone that offered hosting. It was generous, but unfortunately of no assistance. I’d also like to give a big hats off to:

  • Bytemark hosting – they contacted me, offering help with the project. But as it’s the weekend we didn’t get far.
  • NodeDeploy hosting – Ben contacted me as I mentioned and we got the firewall going. What was good though, is I don’t even have an account with these guys, yet I still got regular helpful contact from them. A lot of love for this company. If they treat non-customers like this I bet they treat their customers amazingly.

And the terrible service of 1and1. They caused me headaches because:

  • They claim managing a hardware firewall is ‘my job’ but provide no access to it other than the ability to set filters. What I wanted was verification of a SYN flood.
  • When trying to change the A record of Raspbmc.com, and still up until now, I am given an error message. I emailed them and got no reply.
  • They have the longest TTL known to man.
  • Email went down.

You get what you pay for I guess. £30.00/month for a dedicated server is not going to be special, but as a kid who’s going to be £30k in the whole after I soon go to uni, I can’t exactly splash.

So, what’s next you ask? Well. We need to wait for DNS settings to update, but after that, you should be good to go. Raspbmc and Crystalbuntu development will continue as normal and I will continue on with delivering the best XBMC experience for the AppleTV and Raspberry Pi platforms. If you flush your DNS settings, you might be able to beat the wait, but no guarantees.

Thanks for using my software guys, and sorry about the issues. I will be taking preventative measures so this can be avoided in the future. An interesting aside, the Dutch political party D66 want to legalise the kind of attack that hit me today, seeing it as a form of ‘protest’.

26 Responses to “Network issues”


  • Thanks for this post. I appreciate the elaboration on the attack. It was interesting reading.

    I’m glad you got it sorted. Damn Bastards!

    Thank you for all your hard work !

  • You should setup on github, googlecode, sourceforge. Have your update code read a mirror list from there. Or read the code/data directly from those sites. Your isolated and vulnerable right now. Consider the popularity of your projects and don’t continue to be the single point of failure. I just got interested in Crystalbuntu when you started getting DDoS’d.

    Your Partitioner program downloads the tar files every time. Have the program only download if the files we not already successfully downloaded. Please seriously make MD5 hashes available of files! Or maybe I missed them?

    Best of luck and keep up the excellent work! Much much much appreciated!

  • So why can’t we have a static image with those downloaded files already on? The Pi shouldn’t need an internet connection to install an OS, hell for now I think I’m going to get the files and host them internally with a DNS re-route, should give me some stability while I’m messing around with it so much.

  • :) I thought our land for providing domains has the longest TTL. Can you beat us with 24 hour TTL?

  • I really really support what you are doing with this Raspbmc. This is what I had in mind when I first bought my Pi in March and had it delivered on the 21st of this month. To whoever did this, they should be worried about such a great program :D keep it up! :D Good luck with school too, I’m a college student myself.

  • Have you considered CloudFlare (https://www.cloudflare.com/)? They’re pretty good at DDoS mitigation and best of all, free.

    What you’re doing with Raspbmc is really awesome. Keep up the great work!

  • That attack really sucks, and shows just how petty some people are.

    As for D66, if this is protest, then they clearly don’t understand that the freedom of expression doesn’t allow for the silencing of others, just because we don’t like them.

    Thanks for all you do!

  • Pesky idiots! Got rc3 installed at about 10 last night thank you very much. Another donation on its way later.

    Good luck with exam results. If you show the same kind of intelligence., selflessness, tenacity and dedication in your studies that you have on your projects then you have a very successful life ahead of you.

    Cheers

  • It’s the last paragraph that worries me. Legal !

  • Hi, I managed to download RC3 on friday before the DDOS attacks kicked in. I was amazed how painless the install was. I now have a fully functioning tiny media centre. I appreciate all the hard work that you and all the pi devs have put in to make this device rock. I’m sure there is an army of like minded users who feel the same way. Keep up the great coding and don’t let the haters win. Cheers

  • Hi Sam. Glad you’re back! This kind of people suck and I hope a lot of bad karma for them.
    You write that CrystalBuntu was not affected. This is not what I experienced. For me the boot either took like 10 minutes or it hung for ever in one of the steps.
    Looking forward to CrystalBuntu 2.0 as the gfx driver update in the last update has caused a lot of problems at least for me (dark image and no image if amp is switched to other source).
    Thanks for your hard work

  • hey Sam,

    don’t let this person get you down! you’ve done amazing work on this project already and it’s that hard work that makes people want to help you out:-)

    i get paid tomorrow and will make another donation, maybe if others do the same you can move to a nicer host…

    for now though, your exams are more important:-)

  • Hi Sam,

    As a Dutch citizen, I feel compelled to state that this kind of action, to legalize DDos, is one stupid mistake. It is all about gaining the attention with new elections in sight. Each political party tries to glean a few extra voters (like those script kiddies whom just earned the right to vote). Most stupid proposals are made by politicans who barely know about the internet… (e.g. Acta, three strikes, etc)

    KUTGW!

  • I honestly can’t see why everything needs to be so ‘dynamic’. Why can’t we download a single image from any number of hosts and write it to SD and finished? Fine if you need it to be updatable, but then update on the user’s request.
    Also, please make the program be happy with the SD space it has been given, since it’s not I can’t dual boot.

    • People don’t want to have to manually update. Automatic updating is why people loved Crystalbuntu. What do you mean by the last statement? Raspbmc auto resizes partitions to use the full space of your SD

      • While we’re in RC stages, we really don’t need an automatic updater. All the users are presently tech savvie and can handle their own updates. On release please at least make the “auto-updating phone home” system configurable, not everyone enjoys having their devices communicate with the outside world on a regular basis.

      • Which is what I would like it to *not* do. I want to give it, say, 4GB of my SD card (for the second partition) and want to leave the first partition at whatever it is. Then I can use the rest of my space for another partition to dual boot.

        Please let it prompt the user whether he wants to do the resizing or not, then I’ll be happy.

        • Your request is unreasonable :/

          First and foremostly, the objective is for hands-off install. Specification of partition configuration breaks that.

          Next, you say you want to give 4GB of your SD to p2 and rest unallocated. There is no way to predict this and adapt it for every card unless I did so in percentages.

          99% of users want their card to use all space. The other 1% can change partition sizes easily with GParted

  • You can try some cloud it is big scalabilty.
    You can add as many 100mbit lines as needed (And it is only for the duration of DDoS)

    My ref link:
    https://www.e24cloud.com/referal/QEeXOB59

  • Could you point to a source regarding D66? I’d be interested in reading that :)

    Sorry to hear that apparently someone could not appreciate your work. Thanks very much for persisting.

  • That’s pretty messed up! I’m glad you’re back up and running.

  • I think it is downright disgusting that ‘someone’ tries to break you distribution chain. I tried yesterday to get it to work – but to no avail. I tried today and now it worked brilliantly.
    I don’t know if you can press charges in your jurisdiction based on the logfiles, but it should be possible to chain back through ISP’s etc. – especially if the attacker didn’t spoof every packet.

  • I hope the site will soon be up for download again. Just waiting to be able to download Crystalbuntu, but downloads is still down :/ Both windows and linux fails.

Comments are currently closed.