Closed | Aug 01, 2024 | 16:41 GMT+01:00
We would like to provide a final update before we consider this incident closed. At approximately 22:15 on 17 July we began to receive alerts from various nodes in our core network. Our team began to troubleshoot the issue and identified a potential issue with one of our data centre interconnects. Our network is run across two data centres for redundancy, and we have multiple connections between the sites.
We contacted the network provider who confirmed they were experiencing a power related issue in their Slough facility which was the cause of the issue. This issue was causing the connection to be degraded and cause packet loss however the status of the connection continued to show as up.
Under normal circumstances if one of our data centre interconnects is down then traffic will automatically switch to the alternative connection within seconds.
To stabilize the network, we manually shutdown the degraded connection at roughly 23:20 and by 23:25 our monitoring showed all our services were operating at stable levels.
This degraded connection remained down on 18 July whilst we waited for the supplier to confirm their incident was resolved. On the evening of 18 July, we enabled this connection again to restore the automatic redundancy.
We would like to apologize for any inconvenience caused during this incident. Our network team are investigating solutions for how this could be prevented from happening in the future or to reduce the impact.
Resolved | Jul 17, 2024 | 23:47 GMT+01:00
The network issues have been resolved with the manual shut down of one of our data centre interconnects. We will continue to operate on our second interconnect until the provider of the primary interconnect gives the all clear.
Apologies for any inconvenience you may have experienced.
Monitoring | Jul 17, 2024 | 23:26 GMT+01:00
One of our network providers is experiencing an issue in their Slough location. The service they provide helps connect our two data centre locations together.
Whilst these issues are being experienced we have temporarily shut down our connection to them to restore service. We are continuing to monitor however things seems to have been stable since 22:20.
Investigating | Jul 17, 2024 | 23:09 GMT+01:00
We are currently investigating intermittent network issues affecting voice services.