The core services firewall has remained stable and passing traffic since 08:19 this morning.
We are scheduling an upgrade to the firewall later this evening as a precaution against a possible memory leak and repeat of this incident.
Maintenance will be posted for this activity.
Posted Jul 01, 2020 - 14:32 BST
At 07:58:43, we lost the BGP uplink to a firewall that provides various core services from our Telehouse North POP.
Our engineers were proactively notified of this incident and immediately started to investigate from 08:05.
We were not able to establish communications to the firewall out of band, and so a decision was made to restart this instance.
Within seconds of initiating the reboot command, at 08:19, connectivity was restored to the firewall, the BGP link restored, and all downstream services re-established connectivity.
We are not yet aware of why the firewall became unresponsive, and further investigations shall take place to understand this.
During the incident, duplicate resilient core services hosted from our alternate datacentre LD8 continued to operate.
At no time did this incident affect any core routing.
We apologise for any inconvenience this incident may have caused.
Posted Jul 01, 2020 - 08:40 BST
A fix has been implemented and we are monitoring the results.
Posted Jul 01, 2020 - 08:23 BST
We are currently investigating a problem with various services hosted out of our Telehouse North data centre.
This started at approximately 8am.
Our engineers are investigating.
Our core routing is operational, however if you are experiencing problems with connectivity, and your DNS is not using our DNS3 server 184.108.40.206, please ensure this is configured in your list, or use a public DNS such as Google (220.127.116.11/18.104.22.168) or Cloudflare (22.214.171.124).