tkam Posted April 24, 2021 Report Share Posted April 24, 2021 What The Hell Happened to Head-Case?!?!? TLDR version: The datacenter that the head-case server is located in sustained a power outage on April 4. When the backup generator kicked on it caught fire and that lead to the fire suppression system (a fucking water based one!) kicking on and causing water damage to equipment. The head-case server was in the 'more affected' area of the datacenter and as such was offline until early in the morning on April 24. For further details on the incident you can see here: https://webnx.com/outage-updates/ You may be wondering why during such a long time the site wasn't simply migrated elsewhere, this is a fair question and the answer is simple. As this is a hobby site the backup process hadn't been properly verified as working in quite a while and the latest verified backup was nearly a year old. Migrating the site and loading it from that last verified backup would have meant losing a years worth of content. This was not a valid option and until I could confirm that the drives in the server were dead I wasn't willing to restore that old of a backup. From what I can tell there has been no data loss but I don't currently have a full report as to what if any hardware needed replacing to get the server back online. Due to that lack of info, I don't trust the server and I wouldn't be surprised if there's some additional downtime due to hardware issues. The Future A fresh backup at the time of this post has been taken and moved off-site. Over the next week or so a new backup process will be put in place and verified. After that is done it's very likely that the site will be moved to a new hosting provider. Further details about any move will be provided at the proper time. 10 33 1 Quote Link to comment Share on other sites More sharing options...
swt61 Posted April 24, 2021 Report Share Posted April 24, 2021 Just very happy to be back up and running! Thanks Todd! Quote Link to comment Share on other sites More sharing options...
blessingx Posted April 24, 2021 Report Share Posted April 24, 2021 Thanks for all the work, and especially time, you've put into this Todd. 1 Quote Link to comment Share on other sites More sharing options...
HiWire Posted April 24, 2021 Report Share Posted April 24, 2021 Thanks, Todd! Quote Link to comment Share on other sites More sharing options...
skullguise Posted April 24, 2021 Report Share Posted April 24, 2021 x a billion, thanks Todd! Quote Link to comment Share on other sites More sharing options...
purk Posted April 24, 2021 Report Share Posted April 24, 2021 Thank you, Todd! Quote Link to comment Share on other sites More sharing options...
morphsci Posted April 24, 2021 Report Share Posted April 24, 2021 Absolutely! Thanks Todd. Quote Link to comment Share on other sites More sharing options...
spritzer Posted April 24, 2021 Report Share Posted April 24, 2021 Thanks for the update Todd! I've received dozens and dozens of emails from some very worried people so I'm glad it is back. 2 Quote Link to comment Share on other sites More sharing options...
sbelyo Posted April 24, 2021 Report Share Posted April 24, 2021 Thanks Bro, just happy to be up and running. Who puts a water based surpression system in a data center? That's common sense 101. I mean anyone who's watched Terminator 2 should know about Halon. I don't think they use it in new installs but still. Quote Link to comment Share on other sites More sharing options...
tkam Posted April 24, 2021 Author Report Share Posted April 24, 2021 18 minutes ago, sbelyo said: Who puts a water based surpression system in a data center? That's common sense 101. Believe it or not, it's more common than you think and I know of several cases where it's done solely to meet local municipality requirements. Most people hear 'data center' and think of these super fancy expensive facilities. 95% of data centers do not match that and are built to the lowest possible cost. Building nice, enterprise grade facilities is very, very expensive. It could have been worse, OVH had an entire one of their data centers burn to the ground because they simply didn't have any fire suppression system: https://www.reuters.com/article/us-france-ovh-fire/millions-of-websites-offline-after-fire-at-french-cloud-services-firm-idUSKBN2B20NU 1 Quote Link to comment Share on other sites More sharing options...
Grahame Posted April 24, 2021 Report Share Posted April 24, 2021 1 Quote Link to comment Share on other sites More sharing options...
jamesmking Posted April 24, 2021 Report Share Posted April 24, 2021 1 hour ago, sbelyo said: Thanks Bro, just happy to be up and running. Who puts a water based surpression system in a data center? That's common sense 101. I mean anyone who's watched Terminator 2 should know about Halon. I don't think they use it in new installs but still. water cooled data centre means you can overclock the cpu(s) more.... Quote Link to comment Share on other sites More sharing options...
Torpedo Posted April 24, 2021 Report Share Posted April 24, 2021 I'm so glad we're back together that I don't care it might have been worse or better. Truth is that we're here and alive 3 Quote Link to comment Share on other sites More sharing options...
shellylh Posted April 24, 2021 Report Share Posted April 24, 2021 Thanks, Todd!!! Quote Link to comment Share on other sites More sharing options...
jp11801 Posted April 24, 2021 Report Share Posted April 24, 2021 Todd huge thanks for getting the site back up and running!! Quote Link to comment Share on other sites More sharing options...
Fitz Posted April 24, 2021 Report Share Posted April 24, 2021 Thanks Todd for undoing Brent's damage to the site. 1 8 Quote Link to comment Share on other sites More sharing options...
mikeymad Posted April 24, 2021 Report Share Posted April 24, 2021 cheers Todd. many thanks. Quote Link to comment Share on other sites More sharing options...
Voltron Posted April 24, 2021 Report Share Posted April 24, 2021 You. Are. The. Man. Thanks for persevering Todd! 1 Quote Link to comment Share on other sites More sharing options...
recstar24 Posted April 24, 2021 Report Share Posted April 24, 2021 Please send PayPal for tkam beer fund. THANK YOU 4 Quote Link to comment Share on other sites More sharing options...
luvdunhill Posted April 24, 2021 Report Share Posted April 24, 2021 Ok, who logged back into Huddler-Fi? Come clean. 1 Quote Link to comment Share on other sites More sharing options...
guzziguy Posted April 24, 2021 Report Share Posted April 24, 2021 Thanks, Todd. Quote Link to comment Share on other sites More sharing options...
Fitz Posted April 24, 2021 Report Share Posted April 24, 2021 29 minutes ago, luvdunhill said: Ok, who logged back into Huddler-Fi? Come clean. Not I. I've been good so I wouldn't punish myself with something like that. 1 Quote Link to comment Share on other sites More sharing options...
eggil Posted April 25, 2021 Report Share Posted April 25, 2021 I am so glad the site is back! I have been checking every day and tonight to my surprise it came right up when I clicked! Thank you for the hard work! 1 Quote Link to comment Share on other sites More sharing options...
Craig Sawyers Posted April 25, 2021 Report Share Posted April 25, 2021 Me too. It was a serious shock when the only thing that came up was a page from Wayback Machine from the dim and distant HC. 1 Quote Link to comment Share on other sites More sharing options...
AnakChan Posted April 25, 2021 Report Share Posted April 25, 2021 Glad to see this site back and great job in recovery. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.