Heart Internet would like to apologise to those customers affected by an outage on a portion of our shared and premium hosting platforms.

At 13.00 on Tuesday, 17 January, our systems engineers identified a problem with some of our customers’ websites, caused during a routine software update to a number of core packages on our servers.

During this update, a technical fault led to some websites on some servers becoming unreachable. This was not a fault with the hardware or the data centre, but an issue with the software that affected each server differently.

As the update was a routine maintenance update as part of our commitment to continually improve website performance, it took place during normal business hours. This update was also tested on our staging platform, and had worked without any problems.

Once we restored the servers, there were further connectivity issues between the web servers, NAS drives, and the database servers. Our team attempted to automate the recovery process with scripting, to ensure a fast recovery time, but as the process continued, we realised that we could not automate the restoration process.

We wanted to ensure there was no data loss or server corruption, and our system administrators personally checked each server and website to ensure all data and content was present and running correctly. This caused the extended time delay in restoring all the websites.

We are still synchronising some of the NAS drives, which may cause intermittent problems with high load on some servers.

We are fully monitoring connectivity, software, and hardware within the data centre. We are now building into our update process the requirement that all updates will undergo staged rollouts at a slower pace, even if they have been tested successfully on staging and live platforms.

Was this article useful? Let others know

1 Star2 Stars3 Stars4 Stars5 Stars (No Ratings Yet)
Loading...

Comments

Please remember that all comments are moderated and any links you paste in your comment will remain as plain text. If your comment looks like spam it will be deleted. We're looking forward to answering your questions and hearing your comments and opinions!

Leave a reply

  • 18/01/2017

    I would never upgrade a site live and during the working day. Never. You chose to do just that at possibly one of the busiest business periods knocking out two thirds of all my clients’ sites. Good to read you’ll avoid that in the future. I’m 10+ years with you guys. I’ve not given you up yet so please don’t do this again.

     
    • Kate Bolin

      19/01/2017

      Hi Eddie,

      Thank you for remaining a loyal customer with Heart Internet.

      We regularly perform minor updates on the servers during the working day to provide better performance on your sites during peak times. Each update is tested before it goes live, and by doing it this way, we’re able to provide the best possible service as quickly as possible.

      In this instance, however, despite testing the update on our staging servers, there were still problems with the update. As we’ve said, from now on, we’ll be rolling out updates in a staggered method, thoroughly checking each server before we move to the next one.

      This will mean that it will take longer for the latest versions of software to be available, but it will also protect against any future software-related problems.

       

Comments are closed.

Drop us a line 0330 660 0255 or email sales@heartinternet.uk