All systems operational

Poly ZTP Issues

Closed | Jan 24, 2025 | 17:28 GMT+00:00

We have been monitoring the status of the Poly ZTP and can confirm that this incident can now be closed. Poly have confirmed that this issue was caused by their DNS provider. We would like to apologize for any inconvenience caused.

To reduce the impact of a similar issue in the future we are currently in the process of moving away from tabs on the user profile page of Uboss to instead independent pages down the left-hand menu. This means requests to external ZTP services will only be made if you are on the device page. This change will roll out over the coming months.

+Show history

• Monitoring | Jan 23, 2025 | 08:32 GMT+00:00

  Poly have informed us that they have implemented a fix and we have confirmed that factory reset Polys contact the ZTP and are redirected to our provisioning servers.
  
  We will continue to monitor and provide further updates.

• Monitoring | Jan 22, 2025 | 15:45 GMT+00:00

  Poly have confirmed that they have so far been able to partially recover the ZTP service, the API element is working so you will no longer see errors in Uboss relating to failing to update the ZTP however the element the phones use when factory reset to allow for zero touch provisioning is still down.
  
  We will continue to monitor and provide further updates.

• Monitoring | Jan 22, 2025 | 06:30 GMT+00:00

  We are no longer seeing the issue in Uboss when trying to save the profile of a user that has a Poly device and are awaiting confirmation from Poly regarding the status of their incident.

• Investigating | Jan 21, 2025 | 20:33 GMT+00:00

  Poly have confirmed that this incident remains and they are continuing to investigate the root cause. Apologies again for the inconvenience.

• Investigating | Jan 21, 2025 | 15:58 GMT+00:00

  Poly are currently investigating an issue affecting their zero touch provisioning service. User's will currently experience an error in Uboss if they are trying to save the profile of a user that has a Poly device. Apologies for the inconvenience caused. Poly are currently handling this as a major incident.

Webex mobile inbound call issue

Closed | Nov 21, 2024 | 15:48 GMT+00:00

We wanted to provide a final update before marking this incident as closed. On 13 November at around 09:00 we began to receive reports of inbound calls failing. Our team immediately began investigating and quickly discovered the issue was specific to inbound calls being presented to Webex mobile devices.

Calls presented to Webex mobile are done via push notifications from either Apple (iOS) or Google (Android). A ticket was raised with Cisco and at around 11:00 it was established that the issue was being caused by a delay in DNS responses from Cisco. DNS is used to translate human friendly names to internet addresses. As a workaround we hard coded these internet addresses on our side to bypass the DNS request, thus avoiding the chance of any delay.

This workaround remains in place until Cisco confirm they are satisfied the issue will not happen again. In addition to this we are also waiting on a patch from Cisco which will ensure any future DNS delay is handled more gracefully.

We are in continued contact with Cisco on this matter until all outstanding actions are complete. From an incident perspective we can consider this matter closed as things are stable and fully operational.

Please accept our apologies for the inconvenience caused by this incident.

Affected components

• Third Party Components: Webex

+Show history

• Monitoring | Nov 14, 2024 | 10:08 GMT+00:00

  The temporary fix which was implemented continues to work and we are continuing to monitor the situation. Cisco have confirmed that they are investigating a DNS related issue on their side which is the root cause. The incident will remain open but the status of Webex can be considered fully operational.

• Monitoring | Nov 13, 2024 | 11:13 GMT+00:00

  We have implemented a potential fix and inbound calls are now being presented to Webex mobile as usual. We will continue to monitor the situation and provide an update later today.

• Investigating | Nov 13, 2024 | 10:50 GMT+00:00

  We are continuing to investigate this issue and have been actively troubleshooting with the respective vendor. Apologies for the inconvenience caused.

• Investigating | Nov 13, 2024 | 09:43 GMT+00:00

  We are aware of an issue where inbound calls are not being received by Webex mobile. This is currently being investigated as a priority by the team.

Unity Vision 360 Reporting Issues

Closed | Nov 02, 2024 | 15:44 GMT+00:00

As of approximately 01:00 today Vision 360 has been populating data as expected. The maintenance work which we carried out this evening reversed the changes which were made on Thursday night.

We have found that the changes made on Thursday night to some of our backend servers are not compatible with the way that Vision 360 operates behind the scenes.

This should have been found in our testing after the maintenance however our plan was not comprehensive enough when testing Vision 360 specifically. All other applications were working as expected. We will modify our test process going forward to minimize the chances of an issue like this occurring again.

Again, we would like to apologize for the inconvenience caused.

+Show history

• Waiting | Nov 01, 2024 | 12:09 GMT+00:00

  We have identified an issue affecting data population in Vision 360 related to our planned maintenance work carried out yesterday evening.
  
  One of the backend servers involved in the maintenance yesterday will require an additional restart to resolve the problem.
  
  To reduce risk to the platform we are planning to perform this restart outside of working hours this evening at 23:00. During the restart we do not expect any issues but some client applications may require a restart.
  
  We would like to apologize for the inconvenience caused to any Vision 360 users. A further update will be added to this incident once the problem has been resolved.

• Investigating | Nov 01, 2024 | 08:58 GMT+00:00

  We are currently investigating an issue affecting reporting statistics in Unity Vision 360. Data for today is not currently being populated. Apologies for the inconvenience caused.

Potential schedule viewing issues

Closed | Oct 28, 2024 | 13:32 GMT+00:00

We can confirm that this incident has been resolved as of this morning. Our team identified an issue on Friday with certain schedules that showed an error when viewing them. This error was caused by a bug when calculating the duration of an event and how different time zones were handled. We would like to apologize for the inconvenience caused.

Affected components

• Uboss Portal

+Show history

• Waiting | Oct 25, 2024 | 15:44 GMT+01:00

  The main issue affecting time schedules has now been resolved. Our team are planning an additional minor release to resolve some intermittent issues with advanced time schedules which we have identified. These advanced time schedules are not as commonly used. This incident will remain open until then. Apologies for the inconvenience today.

• Resolution in progress | Oct 25, 2024 | 08:55 GMT+01:00

  We have identified an issue when viewing some schedules in Uboss after the new feature release this morning. The issue has been identified and the team are currently working on the fix. We expect the issue to be resolved in 3 hours.
  
  Apologies for the inconvenience.

Unable to change email of Webex user

Closed | Oct 08, 2024 | 11:56 GMT+01:00

We would like to confirm that this incident is now resolved. Cisco has confirmed that the issue has been rectified on their side. We have tested in Uboss and it is possible again to change the email address of a Webex user. Thank you for your patience.

Affected components

• Third Party Components: Webex

+Show history

• Investigating | Oct 07, 2024 | 16:00 GMT+01:00

  Cisco are currently investigating an issue that is making it not possible to change the email address of an existing Webex user. When changing the email address in Uboss you will receive an error. Apologies for the inconvenience caused whilst Cisco look into this.

Uboss Access

Closed | Aug 17, 2024 | 22:00 GMT+01:00

We wanted to confirm that the user updates referenced in the previous update have now been completed. We appreciate your patience whilst we made this change.

Affected components

• Uboss Portal

+Show history

• Waiting | Aug 17, 2024 | 14:02 GMT+01:00

  We wanted to provide an update about the changes to Uboss identity this morning.
  
  As part of this change if a user had a normal user (used for voice services) and also a portal user at business or site level with the same email address their Uboss login password should have remained the same as the one they have currently been using.
  
  With the roll out of this change the Uboss login pointer was set to their normal user password instead of pointing to their existing portal user password. This is not correct, and we are currently working on an update to ensure that when users login to Uboss they should still use the password they were using prior to these changes being made. This change will be released in the next few hours.
  
  We would like to clarify that this update will not affect their login to applications such as Webex or Unity.
  
  Apologies for the inconvenience this has caused. This incident will be updated once this change is complete.

• Resolved | Aug 17, 2024 | 11:41 GMT+01:00

  This is to confirm that the Uboss portal is available for all users again. We would like to apologize for the inconvenience caused during this incident. Further details about this incident will be provided next week. If you or your customers experience any issues with Uboss please contact our out of hours support team who will be happy to assist.

• Investigating | Aug 17, 2024 | 10:27 GMT+01:00

  Apologies we do not have any additional information at the moment. Our team are continuing to investigate this issue. An additional update will be provided by 11:45.

• Investigating | Aug 17, 2024 | 09:41 GMT+01:00

  We are continuing to investigate this issue and will provide the next update by 10:30.

• Investigating | Aug 17, 2024 | 08:32 GMT+01:00

  The Uboss portal is currently unavailable due to an issue we are investigating. Apologies for the inconvenience. Our team are working on it as a priority.

Network Issues

Closed | Aug 01, 2024 | 16:41 GMT+01:00

We would like to provide a final update before we consider this incident closed. At approximately 22:15 on 17 July we began to receive alerts from various nodes in our core network. Our team began to troubleshoot the issue and identified a potential issue with one of our data centre interconnects. Our network is run across two data centres for redundancy, and we have multiple connections between the sites.

We contacted the network provider who confirmed they were experiencing a power related issue in their Slough facility which was the cause of the issue. This issue was causing the connection to be degraded and cause packet loss however the status of the connection continued to show as up.

Under normal circumstances if one of our data centre interconnects is down then traffic will automatically switch to the alternative connection within seconds.

To stabilize the network, we manually shutdown the degraded connection at roughly 23:20 and by 23:25 our monitoring showed all our services were operating at stable levels.

This degraded connection remained down on 18 July whilst we waited for the supplier to confirm their incident was resolved. On the evening of 18 July, we enabled this connection again to restore the automatic redundancy.

We would like to apologize for any inconvenience caused during this incident. Our network team are investigating solutions for how this could be prevented from happening in the future or to reduce the impact.

+Show history

• Resolved | Jul 17, 2024 | 23:47 GMT+01:00

  The network issues have been resolved with the manual shut down of one of our data centre interconnects. We will continue to operate on our second interconnect until the provider of the primary interconnect gives the all clear.
  
  Apologies for any inconvenience you may have experienced.

• Monitoring | Jul 17, 2024 | 23:26 GMT+01:00

  One of our network providers is experiencing an issue in their Slough location. The service they provide helps connect our two data centre locations together.
  
  Whilst these issues are being experienced we have temporarily shut down our connection to them to restore service. We are continuing to monitor however things seems to have been stable since 22:20.

• Investigating | Jul 17, 2024 | 23:09 GMT+01:00

  We are currently investigating intermittent network issues affecting voice services.

Intermittent Unity Login Issues

Closed | Jul 02, 2024 | 16:38 GMT+01:00

We would like to provide a final update before we consider this incident closed. On 20 June 2024 we received several reports from users across the platform that were experiencing issues when logging into the Unity software. After an initial investigation it was determined that Cloudflare was incorrectly dropping legitimate traffic.

All inbound traffic to our core network is protected by Cloudflare. This is achieved by having Cloudflare advertise the IP addresses we own to the internet on our behalf. By doing this it means all traffic is first sent to Cloudflare and analyzed before it is forwarded onto our own network. Cloudflare are setup to protect us from many common attack vectors by default.

Once we had determined that Cloudflare were the cause of the Unity issues we took the decision to modify our configuration so that Unity login traffic would come straight to our network instead of first going to Cloudflare. This configuration change consists of us modifying our Cloudflare setup, so they are no longer advertising a specific range of IP addresses on our behalf.

In the past it has been possible to make such a change during working hours, however, this time when the internet routing reconverged to its new route it resulted in a brief period of dropped traffic for some users. Going forward we will not make such a configuration change during working hours unless necessary.

We have been working with Cloudflare to ensure we are able to resolve similar issues in the future without having to modify our routing configuration. These methods would not impact service in any way.

We will reverse the routing configuration change with Cloudflare this evening (2 July 2024) to ensure our core network remains protected. We would like to apologize for any inconvenience caused during this incident and have changes in place now to ensure this doesn’t happen again.

Affected components

• Third Party Components: Unity

+Show history

• Resolved | Jun 20, 2024 | 12:07 GMT+01:00

  This incident was resolved at approximately 11:01 this morning.
  
  To resolve this issue we needed to implement a network routing change. A small numbers of users may have experienced brief connectivity issues at around 10:58. Apologies for any disruption.
  
  We will continue to monitor the situation and are working with one of our vendors regarding the requirement for the network routing change.
  
  A final update will be provided before we consider this incident closed.

• Investigating | Jun 20, 2024 | 10:35 GMT+01:00

  We are currently investigating an issue where some users are experiencing issues when logging into Unity.
  
  This issue seems to be affecting a minority of users but we are investigating urgently.

Intermittent Webex Failures

Closed | Jul 01, 2024 | 14:17 GMT+01:00

A final update was provided by Cisco on 27 June at 17:43 to confirm that their engineer team had deployed a fix and all services were stable. Apologies for any inconvenience caused by this incident.

Affected components

• Third Party Components: Webex

+Show history

• Resolved | Jun 27, 2024 | 17:18 GMT+01:00

  Cisco are reporting that this issue is now resolved and services are showing as healthy.
  
  We will continue to monitor for the next 24 hours before we proceed with closing this incident.
  
  Apologies again for any issues which may have been experienced.

• Investigating | Jun 27, 2024 | 16:21 GMT+01:00

  Cisco are currently investigating issues with the Webex service. Calling services seem to be unaffected and we are in contact with Cisco. Apologies for any inconvenience caused.
  
  https://status.webex.com/commercial/status?lang=en_US

Webex network error

Closed | Apr 17, 2024 | 12:54 GMT+01:00

Webex have confirmed that this issue has been resolved. Apologies for any confusion the error message may have caused.

Affected components

• Third Party Components: Webex

+Show history

• Investigating | Apr 17, 2024 | 09:56 GMT+01:00

  Some users may see an error in the Webex client which says the following:
  
  The network you're connected to appears to be blocking Webex
  
  It seems that the application is still working as expected. We have raised the issue with Webex and are investigating.