prokopetz

The fact that Facebook was hindered in resolving their network routing issues in part because those same routing issues made their keycard readers stop working and locked them out of their own server rooms implies that Facebook’s building-internal security systems are on the same network as their public-facing social media infrastructure, which is both terrifying and hilarious.

 prokopetz

Like, you know how funny it is that we never know why Tumblr is down because the page that reports Tumblr’s current status – and to which all outage messaging explicitly directs you – is hosted on the same server as the main site, so anything that takes out the main site also takes out the status page?

This is like that times a million.

 prokopetz

And you know what the crowning cherry atop the clowncake is?

They ended up having to break out the angle grinders and cut their way into their own server room not because there was no lock-and-key backup for the electronic security, but because the personnel who held the keys to the lock-and-key backup couldn’t be reached.

Why couldn’t they be reached?

Well, Facebook’s internal messaging app was also down (of course), and because Facebook’s company policies forbid making use of any communication method other than the app, no one had bothered to maintain a phone contact list. The whole situation could have been resolved much more quickly if anyone, anywhere had simply known the keyholder’s phone number – but not a single person did.

Like, there’s failing to implement redundancies, and then there’s wilfully obstructing your own ability to make use of redundancies that already exist, and the preponderance of evidence suggests we’re looking at the second one.

Clowns.

EDIT: Okay, upon further review, the part about the angle grinder has apparently been retracted by various media sources due to unreliable sourcing. However, the part about being unable to use the lock-and-key backup because, without the internal communication app, nobody knew the keyholder’s phone number is confirmed. Still!