Potential Call and App Issues

Closed | Mar 25, 2025 | 22:41 GMT+00:00

We would like to provide a final update for this incident before closing it down. On 13 February at around 11:00 we began to receive reports from customers that their Unity applications were disconnecting and having to be restarted. This was not affecting all customers, but we received a large enough number of reports that we began to investigate from a core network perspective. Our initial checks showed that all metrics were healthy which led to us raising a ticket with our partner Cloudflare.

Our core network is protected by Cloudflare. This protection means that instead of traffic coming directly to us it is first received by Cloudflare and analyzed in real time for security. The main benefit of this service is that it would stop a DDOS attack if one were to occur.

Cloudflare have 100’s of points of presence around the world. These locations then have a tunnel back to our core network so Cloudflare can forward on the inbound traffic. After our network checks returned healthy, we scheduled in maintenance for the evening of 13 February. This maintenance involved us removing our advertisement via Cloudflare, this meant a subset of traffic would come directly to us instead of going via Cloudflare.

After this advertisement change was made, we monitored the situation and spoke with customers who reported the problem was resolved. As we continued our investigation with Cloudflare it was established that during the period we were experiencing issues Cloudflare were experiencing issues with their location in Manchester. Cloudflare confirmed that during the day the tunnel between their Manchester location and our core network was flapping between healthy and unhealthy. This caused some traffic loss as the route taken from Cloudflare to our network switched between the Manchester tunnel and an alternative tunnel.

When the tunnel was reported as healthy traffic would pass via the Manchester tunnel. When the Manchester tunnel was reported as unhealthy there would be a brief reconvergence period as traffic was rerouted to a different tunnel. It was this reconvergence that caused the disconnection issues with Unity.

After discussing with Cloudflare we have now configured alerts so we can be proactively notified of any tunnel health issues. A subset of our traffic is still coming direct into our network and in the next few weeks we will be reenabling this traffic, so it comes to our network via Cloudflare as normal.

We would like to apologize for the inconvenience caused during this incident and the time taken to supply this closing statement. We appreciate the importance of our services to customers and strive to provide the most reliable network possible. If you have any questions regarding this incident feel free to contact our support team who will be able to direct your query to the necessary person.

Affected components

• Third Party Components : Unity
• Voice Platform