/announcements/19
1:06 AM Sep 16 2021

The site is back up! You will have to log out and then log back in because [computer noises, something something sessions, cache, redis, etc.] - you'll know if you need to because your unread counter will say "Undefined unread".

As for the post mortem - basically, a freak one in a billion hardware occurrence (think like "lightning strike hit a power transformer and blew a fuse" or "cosmic ray flipped a bit at just the wrong spot") turbo fucked the VM and we needed to rebuild everything from scratch. Thankfully, we had a relatively recent backup (the 13th at 1:05 AM GMT), and were able to restore from that rather than having a full system wipe, which... I don't even know what I'd do if we had to do a full system wipe tbh.

That being said, the site should be back up and stable now - blessedly, there doesn't appear to be any long term damage to the hardware or software. We'll be doing what we can hardware-side to try and mitigate against this happening again, but, hopefully, this should never happen again to begin with.

Our sincerest apologies for the inconvenience. We hope you have a pleasant evening, and for those observing Yom Kippur, I hope you have an easy fast!