Resolved -
City Fibre have confirmed the major incident resolved, and all mitigating steps have been taken to stablise the network.
A full reason for outage postmortem will be ascertained following the conclusion of this incident and posted here once available.
No further interruptions are expected and services have remained stable since our previous updates.
We apologise for the interruption to affected customer's City Fibre circuits over the past 24 hours.
This is clearly unprecedented and not expected behaviour.
If you have any questions, please contact support or your account manager.
Feb 17, 22:30 GMT
Monitoring -
Following our last update, City Fibre have now confirmed that they implemented some changes to their network configuration which has stablised the MPLS TE tunnels which were the cause of the loss of service and packet loss earlier.
There is still the risk of further connectivity problems, and we are awaiting the all clear from them.
We have seen no further interruptions in service since our last update.
We shall continue to monitor and provide updates.
Feb 17, 15:57 GMT
Identified -
We have observed that most/all City Fibre circuits are stable and online as of around 12:16, however all circuits remain at risk until we receive the all clear from City Fibre.
City Fibre continue to work with their networking vendor to resolve the problem and fix the root cause.
Getting more technical, the problem is in relation to some MPLS TE Tunnels within their network dropping.
City Fibre have indicated that they should be providing an update within the next hour. We will provide further updates once this information is forthcoming.
Feb 17, 13:02 GMT
Update -
We continue to see a handful of City Fibre circuits in Edinburgh down on our monitoring systems.
Other services appear online and stable at this time, however there is a risk to all City Fibre services at the moment, across the UK, but mostly in the regions previously explained.
The latest from City Fibre's Network Operations Centre (NOC) is that there are high level investigations ongoing.
They have engaged their networking vendor to identify the root cause of the network instability, and ultimately restore service ASAP.
They anticipate providing a further update within the next hour.
Feb 17, 11:27 GMT
Update -
We are continuing to chase City Fibre for updates on this case to get more detail on the problems.
They are aware of the problems and investigating.
We have seen the majority of affected circuits restore, including circuits in Leeds, Wakefield, Glasgow, Aberdeen, however we are seeing some CityFibre connections in Edinburgh currently offline.
Connections in Milton Keynes also briefly were interrupted between 09:31 - 09:34, however they quickly restored.
Circuits in other areas of the country have remained unaffected from our monitoring.
We are continuing to chase and escalate to higher levels to seek out additional information.
We suspect this is a core network/transport issue, due to the nature of this affecting multiple regions.
Feb 17, 10:09 GMT
Investigating -
We have observed that City Fibre circuits in the North of England and in certain parts of Scotland have failed this morning from approx 08:15.
This follows the successful restoration of service last night for affected customers.
We will be following this up with City Fibre as it appears there's been a repeated issue within their core network affecting certain circuits.
Customers in Scotland only seem partially affected.
All Giganet customers in Leeds and Wakefield appear to be affected with a service interruption.
Customers on City Fibre circuits in other parts of the country appear to be unaffected.
Customers with automatic failover configurations should remain unaffected, albeit potentially at reduced throughput rates; however customers with single connections will currently be experiencing a total loss of service.
We're sorry for the inconvenience this outage is causing and will be following up urgently with CityFibre to get this resolved.
Feb 17, 08:57 GMT
Update -
We are continuing to monitor for any further issues.
Feb 16, 22:01 GMT
Monitoring -
We are beginning to see services restore.
However, the carrier's engineers are currently still investigating the root cause of the outage, as there may still be some services at risk and offline until this has been completely resolved and concluded.
We continue to advise caution until the all-clear has been notified.
We apologise again for the outage, and the inconvenience this may have caused.
Feb 16, 21:23 GMT
Update -
We have chased the carrier and they are still currently investigating.
Further updates will be provided as we learn more.
We apologise for any inconvenience caused.
Feb 16, 20:09 GMT
Identified -
We are currently aware of a mass service outage (MSO) affecting some of our ELITE (leased line) services.
Apologies for the delayed notification of this on our status page. Our engineers have been investigating shortly after the outage presented, and we have escalated to the carrier. They are aware of the issue and are investigating.
Start Time: 16/02/2021 17:33
Carrier Affected: CityFibre
Areas Affected (if known): Leeds & Wakefield
Service Impact: Total loss of service for customers using CityFibre carrier services in the above areas
Further updates will be provided as we learn more.
We apologise for any inconvenience this outage causes.
Feb 16, 18:54 GMT