9 replies to this topic

[Smiter]

Earlier today, our server crashed, resulting in downtime and the corruption of our database.

I've just repaired the database, so hopefully everything is running OK!

Our hosts, MediaLayer, will be carrying out a "chassis swap" for our server. This means replacing EVERYTHING except for the hard disks (which were tested and came out OK).

At 11:00 PM EST [GMT -5] on January 1st, 2011, we will be performing a full chassis swap on the server your websites are hosted on. This will include a replacement of essentially all parts in the system, except the hard disks, which are confirmed to be in complete working order based on our monitoring tools. This is in response to a recent outage on this system and is being done as a precautionary measure, as we believe the issue could be rooted in a problem with the motherboard itself. We are allocating an hour window for this maintenance and will keep you all up to date as we perform this work.

Hopefully we'll get through this OK, and there won't be the need for any more repairs!

EDIT: Just to warn everyone, I'm considering shutting down H&E just before the scheduled maintenance and re-opening it afterwards, to try and give our database the best chance of staying intact.

[Derock]

I was wondering what had happening hours ago... I thought we had experienced another crazy moment with the server.

[Smiter]

I was really worried when I woke up today, only to find that H&E had crashed! Hopefully it won't happen again for a long while at least.

I've decided to shut down H&E at 10:45PM EST (3:45AM GMT), a full 15 minutes before the maintenance, then reopen it as soon as everything comes back online. Our database corruption was caused by the sudden crash, so shutting H&E down will ensure the database isn't being used when MediaLayer shuts the server.

[Zatren6]

Thanks for the update and for taking the precautionary measures, Smiter.

[Smiter]

No problem.

We are now finally back online! The maintenance took longer than planned (as things usually do in life ), but it seems to have been a success so far. There might be some performance issues, as they're rebuilding the RAID array.

It looks like we made it through intact.

Now I'm going to catch up on my sleep. I've only had a few hours kip!

[Derock]

Yay! Thanks Smiter!

[Derock]

Now the site is back, but seriously, this one was crazy...

We kept getting this since midnight:

Fatal error: out of memory.

[Quinny52]

^ I know, I got that this morning when I tried to log in. Thought it was a problem on my end, at first. Let's hope this is the last hiccup the site has for a while.

[Gravenimage]

Whoa the site crash again I freaked out when I read that message that Derock mentioned, what's the cause of this and if there's a way to prevent it from happening again?

[Smiter]

I got the same thing when I checked H&E via my mobile phone! Everything was fine when I checked before starting my shift at work. The next time I check it, on the way home, it's all gone wrong.

It was actually a server problem that affected every single website hosted on it. I couldn't do anything about it, so I sent a ticket to MediaLayer, who have since fixed it and apologised to everyone affected:

We deeply apologize for the inconvenience. It turns out that our own email server had experienced issues last night, causing us to have a delay in being alerted to these tickets. The issue should be resolved at this point, however. We will be offering our customers credits in the form of a free upgrade or free period of service. You can contact us anytime to request this.

Hopefully this won't happen again! At least we can request a free period of service as compensation.