Sadly tonight's maintenance to revert back to the LD8 router has failed. Through troubleshooting with vendor TAC, it appears as though the replacement router chassis we've received is faulty, and the vendor has recommended a further replacement.
This has been arranged with the vendor and they are in the process of dispatching this to the data centre urgently.
As per the previous update, we have reverted the configuration to provide LD8 services via our THN site, and we can see affected services back online. The network remains 'at-risk' due to this, however we believe all customers will be online.
We do not plan any more disruptive works from now until we commence further emergency maintenance, which shall be advertised ahead of time.
This is truly exceptional, and we thank you for your patience.
Further escalations within Vendor TAC have been made.
We have re-implemented the DR procedure to fail services across to THN whilst this issue is ongoing. Therefore circuits affected should now be back online.
Next update within an hour.
Posted Apr 03, 2021 - 01:25 BST
Update
Maintenance is ongoing to implement the replacement router however this is taking longer than anticipated.
Customer connections routing via LD8 will currently be down. The DR workaround is disabled whilst we’re migrating back.
Vendor TAC is engaged and we’re working through the issue.
Further updates within the next hour.
Posted Apr 03, 2021 - 00:21 BST
In progress
Scheduled maintenance is currently in progress. We will provide updates as necessary.
We are now therefore in a position to advertise emergency maintenance to implement the new router, and migrate all services back onto the LD8 core router which have been affected.
Preparation of the new router shall take place prior to 22.00hrs, and from 22.00hrs, we shall begin the disruptive process of moving services to the LD8 router. This will cause momentary outages for all customer services routing via or within LD8. (Broadband customers will remain unaffected as these route via the alternate core site - THN ). We believe that we can minimise customer disruption to under 30mins, however we advertise a large window for contingency. Should the process to migrate services back to the LD8 router fail, we shall reimplement the DR plan as is currently configured.
We apologise for any inconvenience this may cause, however it's our priority to re-establish the core router ASAP to provide full performance and resiliency for the network.
Customers who experience any problems following this maintenance work, please contact us via our usual support methods, and in particular, please raise a ticket via our Help Centre: https://support.giga.net.uk/hc/en-gb/requests/new. We shall be monitoring our ticketing systems more closely over the bank holiday weekend for all customer requests. Naturally, any business customers with valid 24x7x365 coverage, please contact you usual 24x7 Giganet support numbers if you experience any critical support issues.