Telehouse North Core Services Outage
Resolved
The core services firewall has remained stable and passing traffic since 08:19 this morning.

We are scheduling an upgrade to the firewall later this evening as a precaution against a possible memory leak and repeat of this incident.

Maintenance will be posted for this activity.
Posted Jul 01, 2020 - 14:32 BST
Update
At 07:58:43, we lost the BGP uplink to a firewall that provides various core services from our Telehouse North POP.

Our engineers were proactively notified of this incident and immediately started to investigate from 08:05.

We were not able to establish communications to the firewall out of band, and so a decision was made to restart this instance.

Within seconds of initiating the reboot command, at 08:19, connectivity was restored to the firewall, the BGP link restored, and all downstream services re-established connectivity.

We are not yet aware of why the firewall became unresponsive, and further investigations shall take place to understand this.

During the incident, duplicate resilient core services hosted from our alternate datacentre LD8 continued to operate.

At no time did this incident affect any core routing.

We apologise for any inconvenience this incident may have caused.
Posted Jul 01, 2020 - 08:40 BST
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jul 01, 2020 - 08:23 BST
Investigating
We are currently investigating a problem with various services hosted out of our Telehouse North data centre.

This started at approximately 8am.

Our engineers are investigating.

Our core routing is operational, however if you are experiencing problems with connectivity, and your DNS is not using our DNS3 server 185.102.149.13, please ensure this is configured in your list, or use a public DNS such as Google (8.8.8.8/8.8.4.4) or Cloudflare (1.1.1.1).
Posted Jul 01, 2020 - 08:19 BST
This incident affected: M12 Giganet - Core Applications (DNS01 Recursive Server, DNS02 Recursive Server, RADIUS01 Server, SMTP01 Relay Server, NTP01 Server), M12 Giganet - Data Centres & Points of Presence (Giganet Core - Telehouse North), and Contacting us/ Tools/ Portals (NetShield Networking Monitoring).