What Happened to Head-Case?

[tkam]

What The Hell Happened to Head-Case?!?!?

TLDR version:  The datacenter that the head-case server is located in sustained a power outage on April 4.  When the backup generator kicked on it caught fire and that lead to the fire suppression system (a fucking water based one!) kicking on and causing water damage to equipment.  The head-case server was in the 'more affected' area of the datacenter and as such was offline until early in the morning on April 24.  For further details on the incident you can see here:  https://webnx.com/outage-updates/ 

You may be wondering why during such a long time the site wasn't simply migrated elsewhere, this is a fair question and the answer is simple.  As this is a hobby site the backup process hadn't been properly verified as working in quite a while and the latest verified backup was nearly a year old. 

Migrating the site and loading it from that last verified backup would have meant losing a years worth of content.  This was not a valid option and until I could confirm that the drives in the server were dead I wasn't willing to restore that old of a backup. 

From what I can tell there has been no data loss but I don't currently have a full report as to what if any hardware needed replacing to get the server back online.  Due to that lack of info, I don't trust the server and I wouldn't be surprised if there's some additional downtime due to hardware issues.

The Future

A fresh backup at the time of this post has been taken and moved off-site.  Over the next week or so a new backup process will be put in place and verified.  After that is done it's very likely that the site will be moved to a new hosting provider.  Further details about any move will be provided at the proper time.  
 

[swt61]

Just very happy to be back up and running!

Thanks Todd!

[blessingx]

Thanks for all the work, and especially time, you've put into this Todd. 

[HiWire]

Thanks, Todd!

[skullguise]

x a billion, thanks Todd!

[purk]

Thank you, Todd!

[morphsci]

Absolutely! Thanks Todd.

[spritzer]

Thanks for the update Todd!    I've received dozens and dozens of emails from some very worried people so I'm glad it is back. 

[sbelyo]

Thanks Bro, just happy to be up and running.

Who puts a water based surpression system in a data center?  That's common sense 101.  I mean anyone who's watched Terminator 2 should know about Halon.  I don't think they use it in new installs but still.

[tkam]

18 minutes ago, sbelyo said:

Who puts a water based surpression system in a data center?  That's common sense 101. 

Believe it or not, it's more common than you think and I know of several cases where it's done solely to meet local municipality requirements.  Most people hear 'data center' and think of these super fancy expensive facilities.  95% of data centers do not match that and are built to the lowest possible cost.  Building nice, enterprise grade facilities is very, very expensive.

It could have been worse, OVH had an entire one of their data centers burn to the ground because they simply didn't have any fire suppression system:  https://www.reuters.com/article/us-france-ovh-fire/millions-of-websites-offline-after-fire-at-french-cloud-services-firm-idUSKBN2B20NU

[Grahame]

[jamesmking]

1 hour ago, sbelyo said:

Thanks Bro, just happy to be up and running.

Who puts a water based surpression system in a data center?  That's common sense 101.  I mean anyone who's watched Terminator 2 should know about Halon.  I don't think they use it in new installs but still.

water cooled data centre means you can overclock the cpu(s) more....

[Torpedo]

I'm so glad we're back together that I don't care it might have been worse or better. Truth is that we're here and alive

[shellylh]

Thanks, Todd!!!

[jp11801]

Todd huge thanks for getting the site back up and running!!

[Fitz]

Thanks Todd for undoing Brent's damage to the site.

[mikeymad]

cheers Todd. many thanks.

[Voltron]

You. Are. The. Man. Thanks for persevering Todd!

[recstar24]

Please send PayPal for tkam beer fund. THANK YOU

[luvdunhill]

Ok, who logged back into Huddler-Fi? Come clean.

[guzziguy]

Thanks, Todd.

[Fitz]

29 minutes ago, luvdunhill said:

Ok, who logged back into Huddler-Fi? Come clean.

Not I. I've been good so I wouldn't punish myself with something like that.

[eggil]

I am so glad the site is back!

I have been checking every day and tonight to my surprise it came right up when I clicked!

Thank you for the hard work!

 

[Craig Sawyers]

Me too. It was a serious shock when the only thing that came up was a page from Wayback Machine from the dim and distant HC.

[AnakChan]

Glad to see this site back and great job in recovery.