Exa Networks - Notice history

The incident has been resolved

Work is continuing to restore the affected service. We have sent our Engineers to the data centre in Halifax to replace some affected hardware. Following supplier maintenance work last night, there appears to have been an unusually high level of optical light on the fibre, which has resulted in damage to the SFPs at both ends. They have now been replaced, and we are starting to see affected services come back online. We will continue to monitor and update this status as soon as we have any more information. Sorry again for the inconvenience this has caused you so far today.

Our Service team worked on this throughout the day yesterday, and a fix was deployed at 8:20 pm yesterday. Following this, we have closely monitored the service and are pleased to report that we have closed this outage, as no further errors have been observed. If you are seeing any issues when using the SurfProtect PAC, please report these to the Helpdesk on 0345 145 1234, as they will be isolated incidents.

Our Services team are continuing to work on a resolution for this issue. We will provide more information as soon as it is available.

We have rolled back the maintenance from last night, and we can see the affected customers are now able to access the internet again. If you are still unable to get online, please call the Helpdesk on 0345 145 1234 and the team will be happy to assist.

Our Services engineers have identified and resolved the issue. If you are still having issues receiving email, please allow up to 60 minutes before reporting this to our Helpdesk on 0345 145 1234. This allows time for any email to be processed through our systems.

As all affected VM Fortihgate services have been restored, we will now be closing this status. We will be carrying out a post-mortem to identify the root cause to ensure we do not encounter the same issue when we next upgrade the VM Firewalls. If you are still unable to access the internet, please call our Helpdesk on 0345 145 12345 and the team will be happy to assist. Sorry again for the inconvenience this outage will have caused you this morning.

We have now rolled back the VM FortiGates, and we are seeing services come back online. We will carry out further investigations to identify what part of the upgrade was the root cause to avoid this happening again. If you are still unable to access the internet, please call our Helpdesk on 0345 145 12345 and the team will be happy to assist.

We have identified an issue with the VM Fortigate upgrade carried out yesterday evening. We are in the process of rolling back to the snapshot taken before the upgrade, and we will provide a further update once this has happened.

The service has been restored overnight, and all affected connections will now be online again. From the monitoring, we can see the circuits were back up at 23:40. We apologise for any inconvenience this may have caused you. If you are still unable to access the internet, please call our Helpdesk from 8 am on 0345 145 1234, and the team will be more than happy to help.

The engineering team have been able to confirm the fibre goes the full distance between the data centres in Sheffield and Leeds, so there is confidence that there is no fibre break. They are currently gaining access to the AQL data centre in Leeds to confirm everything is patched correctly; following this, a further update will be provided. The next update from Exa will be posted no later than 8 am.

The engineering team are still working to repair the affected fibre, and we are awaiting a comprehensive update from the suppliers. Further updates to follow as we get more information.

The engineering team arrived on site just after 7pm, and they are working to resolve the issue.

The suppliers have dispatched an engineering team to investigate the affected fibre. We will provide the next update once the engineers have carried out their investigations. We're sorry for any inconvenience that this supplier outage will be causing you.

Following a period of monitoring, we are happy that all the affected services are now stable and operational again. We will now close down this status. Thank you for your patience while we worked to resolve the outage.

The VMs have now all been rebooted, and other than the SurfProtect Real Time Alerts & Website Analytics, all other affected services are now operational again. Our Service Engineers are working to resolve the SurfProtect Real Time Alerts & Website Analytics issue, and we will provide an update on this as soon as we can. If you are still experiencing any issues, please contact our Helpdesk on 0345 145 1234, and the team will be more than happy to help.

Our Service Engineers identified and resolved the underlying issue with the VM Cluster. They are now carrying out a rolling reboot on the individual VMs within the cluster to restore the affected services. We will provide a further update once the work has been completed.

Following further investigation, our Service Engineers have identified an issue with our VM Cluster in Manchester. This cluster hosts the following services, which will also be seeing issues; Email Webspace Exa Hosted Police CyberAlarm Securus SurfProtect Captive Portal PAC Service SurfProtect Real Time Alerts & Website Analytics SurfProtect Website Classification Exa Education/Moodle We will provide updates as soon as we have them, and we are very sorry for any inconvenience this will cause you this morning.