Postmortem: Crunchyroll Taken Down In Massive DDoS Attack (2014)

Article Title: Crunchyroll Taken Down In Massive DDoS Attack 

Run Date: 12/30/2014

Honestly, I'm surprised that this one occurred more than three years ago at this point. I still remember the day like it was yesterday. The new year was upon us, folks were enjoying their last few days of the year, and much of the day felt, well... pretty average, all considered. Before this happened, the biggest news items were "The Disappearance of Nagato Yuki-chan Anime Unveils More Staff" and "Shirobako Comiket Promo Recaps 1st Season."

No, really.

(ANN's news archives for December 30, 2014)

Now, in the gaming world, we had just come off a disastrous Christmas holiday. Black-hat vandal group Lizard Squad was triumphant after a spree of attacks. In August, the group claimed responsibility for crushing League of Legends' servers. In November, they boasted about taking down popular shooter Destiny's services. 

Their reputation grew by the week, as Lizard Squad took down Sony's PSN (8/24/2014 and 12/8/2014), and Microsoft's XBox Live (12/1/2014), finally culminating in one massive attack that crippled both services on Christmas day.

Obviously, people were on edge.

So, it comes as little surprise, then, that the name was coming up in hushed whispers as people tried to figure out what had happened when Crunchyroll suddenly stopped loading up for fans across the globe. The name came up time and again, between people expressing ire at their inability to catch new episodes of Shirobako and TerraFormars, with some invokign Lizard Squad facetiously and others showing a genuine concern.

At about 4:30PM Eastern (1:30PM Pacific), Crunchyroll finally spoke on the matter with one short statement: "Hey guys, we’re under a DDoS of the same magnitude that Xbox and PSN experienced. We're working hard to fix this, thanks for your patience."

With this in mind, I felt that I was in a rather unique situation. In my day job, I'm a software engineer. I've trained in network services and BI structures, and worked part-time as a sysadmin during college. I have dozens of books and nearly 500 megs of notes on the subject, and I had been working as a Database Administrator (at the time) and coordinating operations with my company's sysadmin. 

So, really, this was my wheelhouse. 

We were looking at a story that was hugely relevant to anime fans across the globe, with developments seeming to break by the hour. At the same time, I had pretty much all of the time that I needed to deliver a real, substantive article on the matter. 

After all, what was I going to do, watch simulcasts? (Bad joke, I know!)

Aniwho, I pulled out a few of my old textbooks and notebooks, and went to work in transcribing what I could about DDoS attacks. I pulled out the "how" and "why," and the general nuts and bolts of what goes into said assaults.

Then reality hit. 

Let's be honest: while a person like me can explain what a Distributed Denial of Service attack is, the average person's eyes will glaze over the moment I start explaining things. Many know what a DDoS does (kills web services dead), but trying to understand the "why" behind it is just asking for a mountain of jargon, technical terms, and dry logic. I mean, cripes. I work in this field and I hate having to listen to these explanations!

And, really, it's tempting! It's super easy to fall back on your jargon! And when you're writing things, it's almost a given that you fall back into that way of thinking. On its surface, stating "The DDoS starved Crunchyroll's DHCP by consuming all available IP leases" makes sense! 

Because of this, you have to learn to stop yourself. You have to step back, and realize that your statements aren't going in a journal, they're not being inserted into some dry tech spec document. They're going to be read by normal, everyday people who rarely look at this world, let alone live in it. 

(In plain English, that sentence states "The attack basically cut off all air to Crunchyroll's servers. By using all available addresses, no real traffic was able to access the service.")

So I wrote, and I rewrote. Over and over again, I revised the article, until I had something that could be palatable to someone who had never thought about a computer beyond it being "the email and funny cat video machine." I pulled in diagrams and quotes from respected sources, which presented ideas like zombie units and DHCP Starvation in a way that could be immediately understood. (I still like that Cloudflare diagram, by the way!)

Using the Crunchyroll attack as a backdrop, I had basically turned the incident into a real-world case study in how web systems work, and what these attacks were actually doing to cause real harm to the internet at large. It was a fun piece, which let me really use the knowledge I had gained through years of working on software and network systems.

It was a challenge, for sure, but I can't help but smile a little when I look back on this one. It's one of those rare times where I could show off my developer cred a bit.