All systems operational

User VoIP Account 500 Error

Closed | Apr 08, 2025 | 09:12 GMT+01:00

A bug fix which was deployed this morning has been reverted and this issue has now been resolved. Apologies for the inconvenience caused.

Affected components

• Uboss Portal

+Show history

• Investigating | Apr 08, 2025 | 08:49 GMT+01:00

  We are currently investigating a bug in the Uboss portal which means a 500 error is displayed when selecting the VoIP Account page.
  
  This is being worked on as a top priority.

Potential Call and App Issues

Closed | Mar 25, 2025 | 22:41 GMT+00:00

We would like to provide a final update for this incident before closing it down. On 13 February at around 11:00 we began to receive reports from customers that their Unity applications were disconnecting and having to be restarted. This was not affecting all customers, but we received a large enough number of reports that we began to investigate from a core network perspective. Our initial checks showed that all metrics were healthy which led to us raising a ticket with our partner Cloudflare.

Our core network is protected by Cloudflare. This protection means that instead of traffic coming directly to us it is first received by Cloudflare and analyzed in real time for security. The main benefit of this service is that it would stop a DDOS attack if one were to occur.

Cloudflare have 100’s of points of presence around the world. These locations then have a tunnel back to our core network so Cloudflare can forward on the inbound traffic. After our network checks returned healthy, we scheduled in maintenance for the evening of 13 February. This maintenance involved us removing our advertisement via Cloudflare, this meant a subset of traffic would come directly to us instead of going via Cloudflare.

After this advertisement change was made, we monitored the situation and spoke with customers who reported the problem was resolved. As we continued our investigation with Cloudflare it was established that during the period we were experiencing issues Cloudflare were experiencing issues with their location in Manchester. Cloudflare confirmed that during the day the tunnel between their Manchester location and our core network was flapping between healthy and unhealthy. This caused some traffic loss as the route taken from Cloudflare to our network switched between the Manchester tunnel and an alternative tunnel.

When the tunnel was reported as healthy traffic would pass via the Manchester tunnel. When the Manchester tunnel was reported as unhealthy there would be a brief reconvergence period as traffic was rerouted to a different tunnel. It was this reconvergence that caused the disconnection issues with Unity.

After discussing with Cloudflare we have now configured alerts so we can be proactively notified of any tunnel health issues. A subset of our traffic is still coming direct into our network and in the next few weeks we will be reenabling this traffic, so it comes to our network via Cloudflare as normal.

We would like to apologize for the inconvenience caused during this incident and the time taken to supply this closing statement. We appreciate the importance of our services to customers and strive to provide the most reliable network possible. If you have any questions regarding this incident feel free to contact our support team who will be able to direct your query to the necessary person.

Affected components

• Third Party Components: Unity
• Voice Platform

+Show history

• Resolved | Feb 15, 2025 | 13:08 GMT+00:00

  After rerouting our traffic to avoid Cloudflare we have not had any further reports of this issue. We are continuing our investigation with Cloudflare to establish the root cause. A further update with full details will be provided before we consider this incident closed. Thank you.

• Investigating | Feb 13, 2025 | 22:54 GMT+00:00

  We are continuing to investigate this issue. You may already be aware that our core network is protected by Cloudflare Magic Transit. This means all traffic that comes into our network is analyzed in real time for improved security and protection.
  
  As a precaution we have temporarily disabled the processing by Cloudflare to rule out Cloudflare being the cause of the reported issues. We will be monitoring the situation closely tomorrow.

• Investigating | Feb 13, 2025 | 12:43 GMT+00:00

  All of our performance and monitoring metrics are suggesting the platform is operating as normal. We are continuing to investigate this issue though and appreciate your patience. We believe this issue to only be affecting a small number of customers.

• Open | Feb 13, 2025 | 11:24 GMT+00:00

  We are currently investigating reports of intermittent call quality issues and applications such as Unity disconnecting. This is being worked on as a top priority.

Broadband issues if using a DrayTek

Closed | Mar 25, 2025 | 22:09 GMT+00:00

We wanted to provide a final update on this incident now the solution to the problem is well known.

If you are using a DrayTek router for your internet connection and your circuit is dropping out frequently it is likely related to a vulnerability that is present in DrayTek routers running an older firmware.

The solution to this problem is to upgrade your DrayTek to the latest firmware available.

If you have upgraded to the latest firmware but the problem persists, it is likely that your router is an older model, and the latest firmware does not contain the required fix due to its release date. A potential workaround is to login to the management interface of the router and do the following:

- Go to “VPN and Remote Access” and then “Remote Access Control” and disable all VPN services.

- Disable remote management on the router or restrict management to specific IP addresses.

If the workaround does not resolve the issue, the final option is to replace your router.

If you are experiencing problems with your internet service and have a DrayTek router, please don’t hesitate to contact our support team who will be able to assist.

Affected components

• Broadband

+Show history

• Waiting | Mar 24, 2025 | 09:39 GMT+00:00

  We are aware of broadband issues which are affecting users with a DrayTek router.
  
  The current workaround to this problem is to login to the management interface of the router and disable all VPN services.
  
  Once VPN services have been disabled it is then recommended to upgrade your router to the latest available firmware.
  
  If you are experiencing issues with your internet this morning and have a DrayTek router please contact our support team who will be able to assist.

Intermittent silent audio

Closed | Mar 20, 2025 | 11:28 GMT+00:00

We would like to provide a final update on this incident before it is closed. Apologies for the time taken for this closing statement, we were waiting for an RFO to be sent from the supplier in question.

The majority of the calls on our platform are routed via BT. Calls between ourselves and BT are distributed between multiple nodes within the BT network for resilience and capacity management.

On 15 February at approximately 12:50 we raised a ticket with BT after we received reports of silent audio on some calls. This problem was initially investigated by the first line of BT support and was subsequently escalated to the second line support team. All affected calls were being routed via the same node in the BT network. These nodes which are operated by BT are run in pairs so there is an active and a passive node. BT confirmed that at 15:47 they performed a manual failover from the active node to the passive node which restored service fully.

BT have confirmed this issue has been raised with their technology partner and are also providing additional guidance internally so problems like these are identified sooner in the future. We would like to apologize for the inconvenience caused during this incident.

+Show history

• Resolved | Feb 15, 2025 | 16:45 GMT+00:00

  We have received a notification of the carrier that had the faulty equipment that this has been fixed.
  After testing successfully the fix, we had rolled back the network configuration and the system is working as normal.
  
  Apologies for any inconvenience these issues have caused.
  

• Investigating | Feb 15, 2025 | 12:55 GMT+00:00

  We are currently investigating an issue with one of our carriers. Calls on the platform are intermittently suffering from silent audio during a call.
  
  We are speaking with the carrier in question and as a temporary measure plan to remove that carrier for outbound calls.

Poly ZTP Issues

Closed | Jan 24, 2025 | 17:28 GMT+00:00

We have been monitoring the status of the Poly ZTP and can confirm that this incident can now be closed. Poly have confirmed that this issue was caused by their DNS provider. We would like to apologize for any inconvenience caused.

To reduce the impact of a similar issue in the future we are currently in the process of moving away from tabs on the user profile page of Uboss to instead independent pages down the left-hand menu. This means requests to external ZTP services will only be made if you are on the device page. This change will roll out over the coming months.

+Show history

• Monitoring | Jan 23, 2025 | 08:32 GMT+00:00

  Poly have informed us that they have implemented a fix and we have confirmed that factory reset Polys contact the ZTP and are redirected to our provisioning servers.
  
  We will continue to monitor and provide further updates.

• Monitoring | Jan 22, 2025 | 15:45 GMT+00:00

  Poly have confirmed that they have so far been able to partially recover the ZTP service, the API element is working so you will no longer see errors in Uboss relating to failing to update the ZTP however the element the phones use when factory reset to allow for zero touch provisioning is still down.
  
  We will continue to monitor and provide further updates.

• Monitoring | Jan 22, 2025 | 06:30 GMT+00:00

  We are no longer seeing the issue in Uboss when trying to save the profile of a user that has a Poly device and are awaiting confirmation from Poly regarding the status of their incident.

• Investigating | Jan 21, 2025 | 20:33 GMT+00:00

  Poly have confirmed that this incident remains and they are continuing to investigate the root cause. Apologies again for the inconvenience.

• Investigating | Jan 21, 2025 | 15:58 GMT+00:00

  Poly are currently investigating an issue affecting their zero touch provisioning service. User's will currently experience an error in Uboss if they are trying to save the profile of a user that has a Poly device. Apologies for the inconvenience caused. Poly are currently handling this as a major incident.

Webex mobile inbound call issue

Closed | Nov 21, 2024 | 15:48 GMT+00:00

We wanted to provide a final update before marking this incident as closed. On 13 November at around 09:00 we began to receive reports of inbound calls failing. Our team immediately began investigating and quickly discovered the issue was specific to inbound calls being presented to Webex mobile devices.

Calls presented to Webex mobile are done via push notifications from either Apple (iOS) or Google (Android). A ticket was raised with Cisco and at around 11:00 it was established that the issue was being caused by a delay in DNS responses from Cisco. DNS is used to translate human friendly names to internet addresses. As a workaround we hard coded these internet addresses on our side to bypass the DNS request, thus avoiding the chance of any delay.

This workaround remains in place until Cisco confirm they are satisfied the issue will not happen again. In addition to this we are also waiting on a patch from Cisco which will ensure any future DNS delay is handled more gracefully.

We are in continued contact with Cisco on this matter until all outstanding actions are complete. From an incident perspective we can consider this matter closed as things are stable and fully operational.

Please accept our apologies for the inconvenience caused by this incident.

Affected components

• Third Party Components: Webex

+Show history

• Monitoring | Nov 14, 2024 | 10:08 GMT+00:00

  The temporary fix which was implemented continues to work and we are continuing to monitor the situation. Cisco have confirmed that they are investigating a DNS related issue on their side which is the root cause. The incident will remain open but the status of Webex can be considered fully operational.

• Monitoring | Nov 13, 2024 | 11:13 GMT+00:00

  We have implemented a potential fix and inbound calls are now being presented to Webex mobile as usual. We will continue to monitor the situation and provide an update later today.

• Investigating | Nov 13, 2024 | 10:50 GMT+00:00

  We are continuing to investigate this issue and have been actively troubleshooting with the respective vendor. Apologies for the inconvenience caused.

• Investigating | Nov 13, 2024 | 09:43 GMT+00:00

  We are aware of an issue where inbound calls are not being received by Webex mobile. This is currently being investigated as a priority by the team.

Unity Vision 360 Reporting Issues

Closed | Nov 02, 2024 | 15:44 GMT+00:00

As of approximately 01:00 today Vision 360 has been populating data as expected. The maintenance work which we carried out this evening reversed the changes which were made on Thursday night.

We have found that the changes made on Thursday night to some of our backend servers are not compatible with the way that Vision 360 operates behind the scenes.

This should have been found in our testing after the maintenance however our plan was not comprehensive enough when testing Vision 360 specifically. All other applications were working as expected. We will modify our test process going forward to minimize the chances of an issue like this occurring again.

Again, we would like to apologize for the inconvenience caused.

+Show history

• Waiting | Nov 01, 2024 | 12:09 GMT+00:00

  We have identified an issue affecting data population in Vision 360 related to our planned maintenance work carried out yesterday evening.
  
  One of the backend servers involved in the maintenance yesterday will require an additional restart to resolve the problem.
  
  To reduce risk to the platform we are planning to perform this restart outside of working hours this evening at 23:00. During the restart we do not expect any issues but some client applications may require a restart.
  
  We would like to apologize for the inconvenience caused to any Vision 360 users. A further update will be added to this incident once the problem has been resolved.

• Investigating | Nov 01, 2024 | 08:58 GMT+00:00

  We are currently investigating an issue affecting reporting statistics in Unity Vision 360. Data for today is not currently being populated. Apologies for the inconvenience caused.

Potential schedule viewing issues

Closed | Oct 28, 2024 | 13:32 GMT+00:00

We can confirm that this incident has been resolved as of this morning. Our team identified an issue on Friday with certain schedules that showed an error when viewing them. This error was caused by a bug when calculating the duration of an event and how different time zones were handled. We would like to apologize for the inconvenience caused.

Affected components

• Uboss Portal

+Show history

• Waiting | Oct 25, 2024 | 15:44 GMT+01:00

  The main issue affecting time schedules has now been resolved. Our team are planning an additional minor release to resolve some intermittent issues with advanced time schedules which we have identified. These advanced time schedules are not as commonly used. This incident will remain open until then. Apologies for the inconvenience today.

• Resolution in progress | Oct 25, 2024 | 08:55 GMT+01:00

  We have identified an issue when viewing some schedules in Uboss after the new feature release this morning. The issue has been identified and the team are currently working on the fix. We expect the issue to be resolved in 3 hours.
  
  Apologies for the inconvenience.

Unable to change email of Webex user

Closed | Oct 08, 2024 | 11:56 GMT+01:00

We would like to confirm that this incident is now resolved. Cisco has confirmed that the issue has been rectified on their side. We have tested in Uboss and it is possible again to change the email address of a Webex user. Thank you for your patience.

Affected components

• Third Party Components: Webex

+Show history

• Investigating | Oct 07, 2024 | 16:00 GMT+01:00

  Cisco are currently investigating an issue that is making it not possible to change the email address of an existing Webex user. When changing the email address in Uboss you will receive an error. Apologies for the inconvenience caused whilst Cisco look into this.

Uboss Access

Closed | Aug 17, 2024 | 22:00 GMT+01:00

We wanted to confirm that the user updates referenced in the previous update have now been completed. We appreciate your patience whilst we made this change.

Affected components

• Uboss Portal

+Show history

• Waiting | Aug 17, 2024 | 14:02 GMT+01:00

  We wanted to provide an update about the changes to Uboss identity this morning.
  
  As part of this change if a user had a normal user (used for voice services) and also a portal user at business or site level with the same email address their Uboss login password should have remained the same as the one they have currently been using.
  
  With the roll out of this change the Uboss login pointer was set to their normal user password instead of pointing to their existing portal user password. This is not correct, and we are currently working on an update to ensure that when users login to Uboss they should still use the password they were using prior to these changes being made. This change will be released in the next few hours.
  
  We would like to clarify that this update will not affect their login to applications such as Webex or Unity.
  
  Apologies for the inconvenience this has caused. This incident will be updated once this change is complete.

• Resolved | Aug 17, 2024 | 11:41 GMT+01:00

  This is to confirm that the Uboss portal is available for all users again. We would like to apologize for the inconvenience caused during this incident. Further details about this incident will be provided next week. If you or your customers experience any issues with Uboss please contact our out of hours support team who will be happy to assist.

• Investigating | Aug 17, 2024 | 10:27 GMT+01:00

  Apologies we do not have any additional information at the moment. Our team are continuing to investigate this issue. An additional update will be provided by 11:45.

• Investigating | Aug 17, 2024 | 09:41 GMT+01:00

  We are continuing to investigate this issue and will provide the next update by 10:30.

• Investigating | Aug 17, 2024 | 08:32 GMT+01:00

  The Uboss portal is currently unavailable due to an issue we are investigating. Apologies for the inconvenience. Our team are working on it as a priority.