Salisbury Exchange degraded performance due to primary backhaul circuit failure

Incident Report for Giganet Status Page

Postmortem

We have now received the “reason for outage” from our backhaul carrier.

Following our carrier’s proactive network alarms at 21:35, and our trouble ticket logged with them at 21:43, their transmission network management centre raised an incident in relation to loss of signal for an optical amplifier on a London Poplar DWDM transmission site.
22:52: The carrier’s network engineer arrived at the Poplar headend site location
00:20: Giganet’s backhaul service is restored.
01:00: The carrier’s engineer completes the cleaning & scoping of multiple fibres along the fibre route. This restores all affected services.

Reason for outage

Our carrier’s network engineer confirmed that there was a site survey that was due to take place by a contractor at this Poplar Headend (amplification site).

In preparation the floor had been lifted which may have caused the fibres to become unseated.

Feedback has been provided to the contractors.

Giganet feedback

All Giganet exchange POPs shall have ‘east & west’ primary & secondary routing (as we do today at Salisbury), and be dimensioned sufficiently to take the full load in the event of a primary circuit failure. This latter aspect upgrade work is due to commence for Salisbury later this year to ensure that in the event of a primary circuit failure, the backup circuit can provide both continuity of service (as it does today), but critically at ‘usual’ performance levels. We apologise to any customers who were impacted by this incident.

Posted Apr 24, 2021 - 09:26 BST

Resolved

The service was restored at 00:20. Full service has restored.

A reason for outage shall be provided and posted here once received from our carrier.

Posted Apr 07, 2021 - 06:53 BST

Identified

The carrier is aware of a major service outage affecting multiple network providers and customers.
This is around the Poplar area in London (unconfirmed report).

Their engineer is presently on site and we await a further update.

Posted Apr 06, 2021 - 23:29 BST

Update

We have separately discovered a configuration problem with our network configuration that meant a minority of Salisbury broadband services didn't failover to the secondary uplink connection.
This affected 1 of our 8 cablelinks in Salisbury.

This problem was resolved at 22:41.
Affected customers will have had their connectivity restore from this time.
We apologise for the disruption this caused.

We are still waiting for an update from the backhaul carrier and shall chase for an update.

Posted Apr 06, 2021 - 22:49 BST

Investigating

We are currently investigating one of our backhaul circuits in the Salisbury exchange being down.
It failed at 21:19.
We have escalated the outage to the carrier and are awaiting their update on the problem.

Customer services have re-routed over our diverse backhaul circuit, however there will be degraded performance during this time.

At this time, customers whose service routes via the Salisbury exchange will be online, but performance may be degraded whilst this incident is ongoing;- reduced speeds/increased latency/increased packet loss.

We apologies for any inconvenience this causes.

Further updates shall be provided on this incident reference once we hear back from our carrier.

Posted Apr 06, 2021 - 21:58 BST

This incident affected: Giganet - Broadband and Internet (Carrier - ELITE/IGNITE (Leased line), Carrier - SuperBOLT/SuperBEAM/UltraBEAM/UltraBOLT/Legacy (Broadband)) and Giganet - Data Centres & Points of Presence (Giganet Local - Salisbury).