Service Outage FAQ - Heart Internet Blog - Focusing on all aspects of the web

This FAQ contains any workarounds, fixes and recommendations from the Heart Internet support team for customers experiencing problems with their Heart Internet services. The FAQ will be updated as new information becomes available.

For regular service updates please visit https://www.webhostingstatus.com/

 

What caused the current service outage?

For full details, please see this blog post.

Why did you originally report the outage as a DDOS?

We came under a DDOS attack shortly before the service outage, however it later became clear that it was not the cause of the outage and so we removed this message to prevent confusion. You can read the full details of the power outage which caused the disruption here.

My Linux VPS is unavailable. What can I do?

All kvmhosts (machines that host our Virtual Private Servers) are operational and so the most likely cause is corruption in your VPS’ database or file system. You should follow the steps below in the “How can I restore my VPS” section to restore your VPS. If the steps below do not work, you will need to restore your VPS from your backups.

If you do not have backups, support can attempt to restore your VPS. We cannot guarantee that we will be able to restore your VPS to its original state via this method, and we strongly advise that you restore from backups wherever possible. Please also note that if you are on an unmanaged package it will take support a significant amount of time to complete this work as we do not regularly monitor unmanaged servers.

How can I restore my VPS?

If your VPS is unable to boot or show sites, then this machine may have suffered corruption due to the unexpected power loss. The fastest way to get this server online is to re-provision it and restore from your backups. If you have no available backups, please follow these instructions:

If this machine is a Linux machine that fails to boot and shows filesystem errors 

Please attempt to start the machine in recovery and run a filesystem check by running this command:

fsck –y /dev/sda3

If you’re unable to do this please raise a support ticket and we will carry this action out for you. Please be aware that support are currently working through a very large queue of tickets and may take some time to respond.

If this machine is a Linux VPS with MySQL database corruption the fastest way to get this server online is to re-provision it and restore from your backups.

It is possible to recover from MySQL corruption using the following process. This is not a guaranteed fix but if the databases are recoverable this process will allow that.

  1. Open the /etc/my.cnf file (MySQL’s config file)
  2. It should look similar to this (however options may vary)
    [mysqld]
     innodb_file_per_table=1
     open_files_limit=2164
     max_allowed_packet=268435456
  3. Add this line to the end:
    innodb_force_recovery = 1
  4. Try starting MySQL again.
  5. If it fails, change the line you added to my.cnf to:

    innodb_force_recovery = 2

  6. Try restarting MySQL again.
  7. If it still fails, change the line you added to my.cnf to:
    innodb_force_recovery = 3
  8. You can keep going up to 6 if needed however this should be a last resort as anything over 3 can cause data los
  9. As soon as you get MySQL started, take a full backup of your databases. MySQL is now is read only recovery mode so your sites may not be online but you should take a copy of your data as soon as possible
  10. To take full backups, run this command from the command line:
    # mysqldump --all-databases --lock-tables --compress --routines --comments --extended-insert > new_all_databases.sql
  11. log in as the root MySQL user, check the DB, exit, stop MySQL and remove the “innodb_force_recovery = […]” line from my.cnf.
  12. Start MySQL one more time. It should now be running normally
  13. You may need to repair corrupted tables using the following method:-
    mysql> USE databasename;
     mysql> REPAIR TABLE `table1` , `table2`;
  14. The above should work for you and will allow MySQL to start and run however once you do this get this started. There is more information regarding this here: https://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html

If the above method is unsuccessful, the below method may prove successful:

  1. Add the following lines to /etc/my.cnf  – this will prevent the need for a password and set the database into recovery mode:
    skip-grant-tables
    innodb_force_recovery = 3
  2. Next, attempt to start MySQL in read only recovery mode using the following commands:
    [root@vps ~]# mysql_upgrade --force
    [root@vps ~]# service mysql restart
  3. Dump your existing databases using:
    [root@vps ~]# mysqldump --all-databases --lock-tables --compress --routines --comments --extended-insert > /backups/new_all_databases.sql
  4. Stop MySQL and move the corrupt /var/lib/mysql folder:
    [root@vps ~]# service mysql stop
    [root@vps ~]# mv /var/lib/mysql /var/lib/mysql_bak
  5. Add this comment to /et/my.cnf:
    #skip-grant-tables
  6. Restart MySQL:
    [root@vps ~]# service mysqld restart

 

The above guide should correct any issues, however if the above fails to work or if you may need to restore from your backups.

If the recovery process fails, and you do not have backups please raise a support ticket and we will attempt to recover your data. Please understand however that we have an extremely large backlog of tickets and we are handling them in chronological order, it may be some time before we can arrive at and address your ticket.

I’m seeing MySQL Connection Errors on my Shared Hosting Database, what can I do?

This could have been caused by the unexpected power outage causing the last active table in your database to crash. To resolve this, log in to PHPMyAdmin and repair all database tables:

  1. Log in to PHPMyAdmin
  2. Select the affected database
  3. Select all database tables
  4. Click”Repair All”

This should allow the site to load.

How can I backup/restore my VPS from a Plesk Database Dump?

There’s an excellent guide here which may help you to restore your VPS from a Plesk Database dump.

If Nginx is non responsive on your plesk server, use the following commands to restart it:

[root@server ~]# /usr/local/psa/admin/bin/nginxmng -d
[root@server ~]# /usr/local/psa/admin/bin/nginxmng -e
[root@server ~]# service nginx restart

My Shared Hosting package is hosted on web75/web76/web77/web78 and is unavailable. What can I do?

These servers were taken offline as a result of the power outage. They are being rebuilt and we’re working to have them online as soon as possible. This is a lengthy process and support will not be able to hasten it for you. Once these servers are rebuilt and put back online, your services will be restored automatically with no need for support interaction. Updates will be posted as we receive them to https://www.webhostingstatus.com/. Thanks for your patience.

These shared hosting servers are now available and sites are displaying. If you are seeing additional errors, please contact support.

I’ve raised a support ticket about my VPS / Shared Hosting and have not received a response yet. When can I expect one?

Our support team are working as fast as they’re able to address these tickets and are prioritising tickets related to unavailable services. We understand that this is causing you frustration and we’re sorry for the delays. Our support team and SysAdmins are working 24 hours to get you back online. Please keep an eye on your email inbox for updates on your tickets. Please be aware that if you have a non-urgent issue we are unable to offer support at this time as our resource is focussed on restoring unavailable services.

Subscribe to our monthly Heart Internet newsletter, filled with the latest articles about web design, development, building your business, and exclusive offers.

Subscribe now!

Comments

Please remember that all comments are moderated and any links you paste in your comment will remain as plain text. If your comment looks like spam it will be deleted. We're looking forward to answering your questions and hearing your comments and opinions!

Leave a reply

  • Rob

    11/02/2016

    Thank you Michael,

    This worked for me. Back online. Just want to say I think Heart Internet do a great job and the support is always excellent.

    Rob

     
    • Michael Shillingford

      15/02/2016

      Thank you Rob and I’m very glad the steps above worked for you.

       
  • Anders

    12/02/2016

    Any update further to this? what if tried all this and sql still not working?

     
    • Michael Shillingford

      15/02/2016

      Hi Anders, if you’ve tried the MySQL steps above and you do not have any local backups then support may be able to restore your VPS. The success rate is not high, the process is lengthy and we cannot guarantee that we can restore the VPS to its original state. Support are working through tickets as fast as they’re able – if you haven’t raised a ticket already you can do so here: https://bit.ly/HIhelp

       
  • Michael Broadbridge

    13/02/2016

    My eco nerve shop has been offline for 3 days now and I’m losing thousands every day. I’m on 75, how much longer will it be? I have 3 full time staff to pay with nothing being able to be done because the site is down!

     
    • Michael Shillingford

      15/02/2016

      Hi Michael, sites hosted on web75-78 should all be back online now. If you’re still experiencing issues, please do contact support: https://bit.ly/HIhelp

       
  • 13/02/2016

    After 24 hours still cannot access our dedicated server at all any way.
    Tickets raised but not answered since Friday lunchtime

     
    • Michael Shillingford

      15/02/2016

      Hi Dave, apologies for the delay in getting back to your support tickets. Dedicated servers should all be back online now. If not, stand by and support will answer your ticket as soon as they’re able.

       
  • Mike Rendall

    14/02/2016

    My site is on Web75. It is showing a readonly filesystem – do you have an eta when it will be fully operational ? Thanks

     
    • Michael Shillingford

      15/02/2016

      Hi Mike, the web75-78 cluster has been rebuilt and sites hosted on these servers should be back online now. If you’re still having issues, be sure to contact support here: https://bit.ly/HIhelp

       
  • Rallport

    15/02/2016

    I’ve read on Twitter that you’re refusing to help people with unmanaged servers – their data has clearly been corrupted due to the sudden power loss. Granted, if the server is unmanaged then technically your only responsibility is to ensure it is available and powered on. Granted, restoring things from the command line isn’t the end of the world. However, as you caused the issue, surely it is yourselves who should be spending time restoring servers.

     
    • Michael Shillingford

      15/02/2016

      Hi Rallport – our VPS products are sold unmanaged. Whilst this does mean that it is the customer’s responsibility to keep backups, this is an exceptional circumstance and we are helping customers to bring their VPS’ back online. We can’t guarantee success here and the process is time consuming, but support will do everything they can and our staff are working 24/7 to offer help. Ultimately, it will be faster for customers to rebuild/restore the server themselves if they’re able to do so.

       
  • 15/02/2016

    Hi Michael,
    I’m trying to get hold of Support and Account Team for half the day, our Account is being locked out for no reason, I have made a Billing query # 1204090012 , and its not yet responded by any of the Team Members. Moreover one of the Server has gone down and we can’t even check the status, COuld you please unlock our Account and let us know what is happened?

    Thanks
    Sunil

     
    • Michael Shillingford

      15/02/2016

      Hi Sunil, just spoken to support – the account issue will be sorted for you within the hour. Apologies for the delay.

       
  • Jason

    17/02/2016

    An engineer from a competitor got our VPS to boot again and fixed the problems on the server within and hour, something you have not managed to do in 3 days. The VPS is now running but hangs every 2.5 minutes for 2.5 minutes. With all errors on the install fixed I’m not sure how in this article you can state “All kvmhosts (machines that host our Virtual Private Servers) are operational” when clearly they are suffering problems outside of our install.

     
    • Craig Cotter

      17/02/2016

      Hi Jason,

      We can only verify that the host machines are working, not the VPS on those machines. We have no similar reports across entire kvmhosts, so this suggests an issue particular to your VPS, file or MySQL corruption can result in unusual behaviour (your server may be attempting something every 2.5 minutes that is failing or attempting to access corrupted data).

       
  • James

    17/02/2016

    Was about to start this process then I noticed there is a tool in Plesk that I use on my VPS to Check/Repair MySQL databases. I had two website domains down with errors unable to connect to database. I ran the check.repair it list the error and gives options to repair. This worked for both databases that had crashed. Thought I’d let people know in case others with odin Parallels Plesk reach this page.

     

Comments are closed.

Drop us a line 0330 660 0255 or email sales@heartinternet.uk