{"id":19843,"date":"2018-11-09T14:26:46","date_gmt":"2018-11-09T14:26:46","guid":{"rendered":"https:\/\/www.heartinternet.uk\/blog\/?p=19843"},"modified":"2018-11-09T14:26:46","modified_gmt":"2018-11-09T14:26:46","slug":"status-of-heart-internet","status":"publish","type":"post","link":"https:\/\/www.heartinternet.uk\/blog\/status-of-heart-internet\/","title":{"rendered":"Update on November 5th incident"},"content":{"rendered":"

Status of Heart Internet <\/h3>\n

We\u2019ve let you down. Over the last few weeks, we\u2019ve had some challenges in keeping your sites, and potentially your customers\u2019 sites, up and running. In order to understand what happened, we want to share some background.<\/p>\n

Earlier this year, we conducted an audit on the hardware in our datacenter. We found several pieces of networking equipment that, although they still functioned properly, we decided to update so we had a more robust and reliable infrastructure in place.<\/p>\n

The plan was to start re-designing the network and updating the necessary equipment to provide customers the most reliable and fastest web hosting possible. The plan was in place, the networking was being re-designed, and things were going great. Until they weren\u2019t.<\/p>\n

November 5th<\/sup> incident<\/h3>\n

Instead of breaking down each and every incident that\u2019s happened recently, we want to focus on what happened on 5th<\/sup> November. This example is the most extreme, and is indicative of what\u2019s been happening recently.<\/p>\n

At approximately 3am, we were notified of connection issues. Within 30 minutes, we had a team working to diagnose the issue. Due to the network design and the current switches in place, this wasn\u2019t so easy.<\/p>\n

Here\u2019s why it was so difficult. Sometimes, a server will send traffic a switch can\u2019t figure out how to handle. As the traffic keeps coming, the switch becomes overwhelmed. When that happens, the switch starts acting erratically, which then cascades to other switches. This cascading effect cripples the network. There are thousands of servers, more than 100 switches as well as other hardware and networking gear that could have been causing the issue. With everything effectively down, finding the proper switch or server was like hunting for a needle in a haystack.<\/p>\n

So at 3:30am, the engineers began troubleshooting, trying to identify the root cause of the issue. Unfortunately, this isn\u2019t as quick or as easy as it should have been. All of the networking equipment was acting erratically, making it difficult to identify the problem source.<\/p>\n

At 12:44pm GMT, the team thought they had located the problematic switch. They had a switch on standby, so they changed it out, expecting to see the system start to improve. No such luck. That switch wasn\u2019t the root of the problem.<\/p>\n

They continued to work to identify the problem hardware. The team eventually identified three switches that were causing issues. Once they were identified, we were quickly able to swap out the impacted switches that brought everything back online.<\/p>\n

So \u2026 what are we doing to fix it? <\/h3>\n

Our initial plan to update our networking equipment is going to continue, just at a faster pace. We\u2019re going to do our best to minimize any downtime during the changes, and if our work is going to cause any impact to your services, we\u2019ll write to you and tell you first. When completed, the benefits will be enormous for you. In addition to a more robust network, we are building in advanced telemetry and diagnostic equipment, which will help us identify and isolate problems more quickly.<\/p>\n

Another problem \u2026 lack of communication<\/h3>\n

The other area where we fell down as a company was the lack of communication with you. In the middle of an active incident, it\u2019s always difficult to communicate effectively. As we\u2019re trying to locate the issue and work on potential resolutions, we always want to provide the most accurate information, and sometimes getting all the facts together slows us down. That being said, we can and will do better.<\/p>\n

We are looking at how to provide more timely and thorough updates. This blog is the first step. We need to engage in a dialogue about what\u2019s happening, why it\u2019s happening and what we\u2019re doing to fix it. Expect more communications like this in the future.<\/p>\n

In closing, the situation over the last few weeks has been unacceptable to us as a business and unacceptable for our customers. We apologize. We truly believe great things are ahead and the more robust and advanced network will once again provide the high level of service you have come to expect from us.<\/p>\n

Thank you,<\/p>\n

Sara Rego, Heart Internet Director<\/p>\n","protected":false},"excerpt":{"rendered":"

On Monday 5th November, we had network switch failures that caused disruption to your sites, your customers’ sites, and our sites as well. Here are the details on what happened and what we’re doing to resolve these issues.<\/p>\n","protected":false},"author":2,"featured_media":12131,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[13],"tags":[],"class_list":{"0":"post-19843","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-heart-internet"},"yoast_head":"\nUpdate on November 5th incident - Heart Internet<\/title>\n<meta name=\"description\" content=\"On Monday 5th November, we had network switch failures that caused disruption to your sites, your customers' sites, and our sites as well.\" \/>\n<meta name=\"robots\" content=\"noindex, nofollow\" \/>\n<meta property=\"og:locale\" content=\"en_GB\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Update on November 5th incident - Heart Internet\" \/>\n<meta property=\"og:description\" content=\"On Monday 5th November, we had network switch failures that caused disruption to your sites, your customers' sites, and our sites as well.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.heartinternet.uk\/blog\/status-of-heart-internet\/\" \/>\n<meta property=\"og:site_name\" content=\"Heart Internet\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/heartinternet\/\" \/>\n<meta property=\"article:published_time\" content=\"2018-11-09T14:26:46+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.heartinternet.uk\/blog\/wp-content\/uploads\/2016\/05\/emotion-wrapup.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"600\" \/>\n\t<meta property=\"og:image:height\" content=\"200\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Eliot Chambers-Ostler\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@heartinternet\" \/>\n<meta name=\"twitter:site\" content=\"@heartinternet\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Eliot Chambers-Ostler\" \/>\n\t<meta name=\"twitter:label2\" content=\"Estimated reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.heartinternet.uk\/blog\/status-of-heart-internet\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.heartinternet.uk\/blog\/status-of-heart-internet\/\"},\"author\":{\"name\":\"Eliot Chambers-Ostler\",\"@id\":\"https:\/\/heartblog.victory.digital\/#\/schema\/person\/58ed7f27cc0f3ab6e69135742a5eee28\"},\"headline\":\"Update on November 5th incident\",\"datePublished\":\"2018-11-09T14:26:46+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.heartinternet.uk\/blog\/status-of-heart-internet\/\"},\"wordCount\":711,\"commentCount\":28,\"publisher\":{\"@id\":\"https:\/\/heartblog.victory.digital\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.heartinternet.uk\/blog\/status-of-heart-internet\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.heartinternet.uk\/blog\/wp-content\/uploads\/2016\/05\/emotion-wrapup.jpg\",\"articleSection\":[\"Heart Internet\"],\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.heartinternet.uk\/blog\/status-of-heart-internet\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.heartinternet.uk\/blog\/status-of-heart-internet\/\",\"url\":\"https:\/\/www.heartinternet.uk\/blog\/status-of-heart-internet\/\",\"name\":\"Update on November 5th incident - Heart Internet\",\"isPartOf\":{\"@id\":\"https:\/\/heartblog.victory.digital\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.heartinternet.uk\/blog\/status-of-heart-internet\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.heartinternet.uk\/blog\/status-of-heart-internet\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.heartinternet.uk\/blog\/wp-content\/uploads\/2016\/05\/emotion-wrapup.jpg\",\"datePublished\":\"2018-11-09T14:26:46+00:00\",\"description\":\"On Monday 5th November, we had network switch failures that caused disruption to your sites, your customers' sites, and our sites as well.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.heartinternet.uk\/blog\/status-of-heart-internet\/#breadcrumb\"},\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.heartinternet.uk\/blog\/status-of-heart-internet\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/www.heartinternet.uk\/blog\/status-of-heart-internet\/#primaryimage\",\"url\":\"https:\/\/www.heartinternet.uk\/blog\/wp-content\/uploads\/2016\/05\/emotion-wrapup.jpg\",\"contentUrl\":\"https:\/\/www.heartinternet.uk\/blog\/wp-content\/uploads\/2016\/05\/emotion-wrapup.jpg\",\"width\":600,\"height\":200},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.heartinternet.uk\/blog\/status-of-heart-internet\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.heartinternet.uk\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Update on November 5th incident\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/heartblog.victory.digital\/#website\",\"url\":\"https:\/\/heartblog.victory.digital\/\",\"name\":\"Heart Internet\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/heartblog.victory.digital\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/heartblog.victory.digital\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-GB\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/heartblog.victory.digital\/#organization\",\"name\":\"Heart Internet\",\"url\":\"https:\/\/heartblog.victory.digital\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/heartblog.victory.digital\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.heartinternet.uk\/blog\/wp-content\/uploads\/2025\/02\/HeartInternet_Logo_Colour.webp\",\"contentUrl\":\"https:\/\/www.heartinternet.uk\/blog\/wp-content\/uploads\/2025\/02\/HeartInternet_Logo_Colour.webp\",\"width\":992,\"height\":252,\"caption\":\"Heart Internet\"},\"image\":{\"@id\":\"https:\/\/heartblog.victory.digital\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/heartinternet\/\",\"https:\/\/x.com\/heartinternet\",\"https:\/\/www.linkedin.com\/company\/heart-internet-ltd\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/heartblog.victory.digital\/#\/schema\/person\/58ed7f27cc0f3ab6e69135742a5eee28\",\"name\":\"Eliot Chambers-Ostler\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/heartblog.victory.digital\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/www.heartinternet.uk\/blog\/wp-content\/uploads\/2025\/08\/cropped-Eliot-96x96.jpg\",\"contentUrl\":\"https:\/\/www.heartinternet.uk\/blog\/wp-content\/uploads\/2025\/08\/cropped-Eliot-96x96.jpg\",\"caption\":\"Eliot Chambers-Ostler\"},\"url\":\"https:\/\/www.heartinternet.uk\/blog\/author\/eliot-chambers-ostler\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Update on November 5th incident - Heart Internet","description":"On Monday 5th November, we had network switch failures that caused disruption to your sites, your customers' sites, and our sites as well.","robots":{"index":"noindex","follow":"nofollow"},"og_locale":"en_GB","og_type":"article","og_title":"Update on November 5th incident - Heart Internet","og_description":"On Monday 5th November, we had network switch failures that caused disruption to your sites, your customers' sites, and our sites as well.","og_url":"https:\/\/www.heartinternet.uk\/blog\/status-of-heart-internet\/","og_site_name":"Heart Internet","article_publisher":"https:\/\/www.facebook.com\/heartinternet\/","article_published_time":"2018-11-09T14:26:46+00:00","og_image":[{"width":600,"height":200,"url":"https:\/\/www.heartinternet.uk\/blog\/wp-content\/uploads\/2016\/05\/emotion-wrapup.jpg","type":"image\/jpeg"}],"author":"Eliot Chambers-Ostler","twitter_card":"summary_large_image","twitter_creator":"@heartinternet","twitter_site":"@heartinternet","twitter_misc":{"Written by":"Eliot Chambers-Ostler","Estimated reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.heartinternet.uk\/blog\/status-of-heart-internet\/#article","isPartOf":{"@id":"https:\/\/www.heartinternet.uk\/blog\/status-of-heart-internet\/"},"author":{"name":"Eliot Chambers-Ostler","@id":"https:\/\/heartblog.victory.digital\/#\/schema\/person\/58ed7f27cc0f3ab6e69135742a5eee28"},"headline":"Update on November 5th incident","datePublished":"2018-11-09T14:26:46+00:00","mainEntityOfPage":{"@id":"https:\/\/www.heartinternet.uk\/blog\/status-of-heart-internet\/"},"wordCount":711,"commentCount":28,"publisher":{"@id":"https:\/\/heartblog.victory.digital\/#organization"},"image":{"@id":"https:\/\/www.heartinternet.uk\/blog\/status-of-heart-internet\/#primaryimage"},"thumbnailUrl":"https:\/\/www.heartinternet.uk\/blog\/wp-content\/uploads\/2016\/05\/emotion-wrapup.jpg","articleSection":["Heart Internet"],"inLanguage":"en-GB","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.heartinternet.uk\/blog\/status-of-heart-internet\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.heartinternet.uk\/blog\/status-of-heart-internet\/","url":"https:\/\/www.heartinternet.uk\/blog\/status-of-heart-internet\/","name":"Update on November 5th incident - Heart Internet","isPartOf":{"@id":"https:\/\/heartblog.victory.digital\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.heartinternet.uk\/blog\/status-of-heart-internet\/#primaryimage"},"image":{"@id":"https:\/\/www.heartinternet.uk\/blog\/status-of-heart-internet\/#primaryimage"},"thumbnailUrl":"https:\/\/www.heartinternet.uk\/blog\/wp-content\/uploads\/2016\/05\/emotion-wrapup.jpg","datePublished":"2018-11-09T14:26:46+00:00","description":"On Monday 5th November, we had network switch failures that caused disruption to your sites, your customers' sites, and our sites as well.","breadcrumb":{"@id":"https:\/\/www.heartinternet.uk\/blog\/status-of-heart-internet\/#breadcrumb"},"inLanguage":"en-GB","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.heartinternet.uk\/blog\/status-of-heart-internet\/"]}]},{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/www.heartinternet.uk\/blog\/status-of-heart-internet\/#primaryimage","url":"https:\/\/www.heartinternet.uk\/blog\/wp-content\/uploads\/2016\/05\/emotion-wrapup.jpg","contentUrl":"https:\/\/www.heartinternet.uk\/blog\/wp-content\/uploads\/2016\/05\/emotion-wrapup.jpg","width":600,"height":200},{"@type":"BreadcrumbList","@id":"https:\/\/www.heartinternet.uk\/blog\/status-of-heart-internet\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.heartinternet.uk\/blog\/"},{"@type":"ListItem","position":2,"name":"Update on November 5th incident"}]},{"@type":"WebSite","@id":"https:\/\/heartblog.victory.digital\/#website","url":"https:\/\/heartblog.victory.digital\/","name":"Heart Internet","description":"","publisher":{"@id":"https:\/\/heartblog.victory.digital\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/heartblog.victory.digital\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-GB"},{"@type":"Organization","@id":"https:\/\/heartblog.victory.digital\/#organization","name":"Heart Internet","url":"https:\/\/heartblog.victory.digital\/","logo":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/heartblog.victory.digital\/#\/schema\/logo\/image\/","url":"https:\/\/www.heartinternet.uk\/blog\/wp-content\/uploads\/2025\/02\/HeartInternet_Logo_Colour.webp","contentUrl":"https:\/\/www.heartinternet.uk\/blog\/wp-content\/uploads\/2025\/02\/HeartInternet_Logo_Colour.webp","width":992,"height":252,"caption":"Heart Internet"},"image":{"@id":"https:\/\/heartblog.victory.digital\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/heartinternet\/","https:\/\/x.com\/heartinternet","https:\/\/www.linkedin.com\/company\/heart-internet-ltd"]},{"@type":"Person","@id":"https:\/\/heartblog.victory.digital\/#\/schema\/person\/58ed7f27cc0f3ab6e69135742a5eee28","name":"Eliot Chambers-Ostler","image":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/heartblog.victory.digital\/#\/schema\/person\/image\/","url":"https:\/\/www.heartinternet.uk\/blog\/wp-content\/uploads\/2025\/08\/cropped-Eliot-96x96.jpg","contentUrl":"https:\/\/www.heartinternet.uk\/blog\/wp-content\/uploads\/2025\/08\/cropped-Eliot-96x96.jpg","caption":"Eliot Chambers-Ostler"},"url":"https:\/\/www.heartinternet.uk\/blog\/author\/eliot-chambers-ostler\/"}]}},"_links":{"self":[{"href":"https:\/\/www.heartinternet.uk\/blog\/wp-json\/wp\/v2\/posts\/19843","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.heartinternet.uk\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.heartinternet.uk\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.heartinternet.uk\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.heartinternet.uk\/blog\/wp-json\/wp\/v2\/comments?post=19843"}],"version-history":[{"count":0,"href":"https:\/\/www.heartinternet.uk\/blog\/wp-json\/wp\/v2\/posts\/19843\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.heartinternet.uk\/blog\/wp-json\/wp\/v2\/media\/12131"}],"wp:attachment":[{"href":"https:\/\/www.heartinternet.uk\/blog\/wp-json\/wp\/v2\/media?parent=19843"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.heartinternet.uk\/blog\/wp-json\/wp\/v2\/categories?post=19843"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.heartinternet.uk\/blog\/wp-json\/wp\/v2\/tags?post=19843"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}