The carrier has now confirmed restoration of service following the fibre break repair.
We shall of course be requesting a RFO (reason for outage) and following this up on here once received.
The latest update from the carrier has explained that two different issues were occurring this morning within their network. When they restored services in Telehouse North for one of these incidents, this did not restore our link. They believed it would. This meant that they had to troubleshoot where the problem was for our circuit's outage. This then led them to discover the fibre break in the Andover area, and dispatch engineers to repair the fault.
We are incredibly frustrated that this took so long to be repaired today, and also within a week of an even more significant outage.
Please be assured that this is being followed up at the highest levels with the carrier and we are exploring options around our backhaul choice in this exchange, as well as enhancing the offering to ensure there can be no degradation of service in the event of a primary circuit failure.
We apologise for any inconvenience this caused.
Posted Nov 26, 2020 - 18:34 GMT
Update
The primary backhaul circuit has continued to remain online however we're still awaiting the all clear.
Services are fully operational
Posted Nov 26, 2020 - 18:11 GMT
Monitoring
The carrier has confirmed that the fibre break location is in Andover. In the past few moments we have seen the primary link re-establish and traffic has automatically re-routed. So we assume they have repaired it. We await the all-clear by the carrier before confirming the resolution.
Posted Nov 26, 2020 - 17:44 GMT
Update
Carrier engineers have now located the source of the fibre break with their OTDR. They are now dispatching to the site location to implement a repair.
Further updates will be posted as we learn more.
Traffic levels in Salisbury continue to be below the backup backhaul circuit's capabilities to ensure that most customers are not impacted too greatly with degraded performance.
Posted Nov 26, 2020 - 15:18 GMT
Update
The carrier has updated us to say they are still investigating the fault.
All broadband and leased line customers within Salisbury continue to remain online, connected via our backup backhaul circuit, and traffic levels are low enough to ensure that customers should notice little-to-no degradation in their service.
Posted Nov 26, 2020 - 12:26 GMT
Update
The carrier confirmed that they have re-connected a fibre patch in their head-end and they are seeing circuits restore. However we have not seen this. We have just escalated this back to the carrier and are awaiting their next update. They suggested that we'll receive an update within the next 15-20 minutes as to the status of their investigations.
All broadband and leased line customers within Salisbury continue to remain online, connected via our backup backhaul circuit, and traffic levels are low enough to ensure that customers should notice little-to-no degradation in their service.
Posted Nov 26, 2020 - 11:05 GMT
Identified
We are experiencing a repeated outage affecting our primary Salisbury backhaul circuit as of 08:30.
Currently everything has automatically failed over to our backup circuit. As such all services routing via the Salisbury exchange (broadband and leased lines) may experience at times a degradation in their service (at times there could be decreased speeds, packet loss or increased latency). We have not implemented any traffic engineering yet, as the backup circuit is sufficiently taking the load.
We have logged this with the carrier and they are aware of a MSO (major service outage) affecting their equipment within Telehouse North. There is no indication on when this problem will be solved.
They advise that an engineer was onsite in Telehouse North since 7am investigating a problem, 'have run OTDR tests', and 'on conference calls to implement a fix'. But we lost our service at 08:30.
We will update this incident as we learn more.
A high level escalation following last weekends' incident has been logged with the carrier, and a call with a Director has been scheduled. This incident shall be added to the notes. Quite clearly, this is unacceptable.
We apologise once again.
Posted Nov 26, 2020 - 09:13 GMT
This incident affected: Giganet - Broadband and Internet (Carrier - ELITE/IGNITE (Leased line), Carrier - SuperBOLT/SuperBEAM/UltraBEAM/UltraBOLT/Legacy (Broadband)) and Giganet - Data Centres & Points of Presence (Giganet Local - Salisbury).