We have continued to maintain traffic away from LINX LON1 whilst LINX still engage with their vendor TAC surrounding this afternoon's network incident on their LON1 peering LAN.
Peering to LINX LON2 and LONAP, as well as upstream Tier 1 transit links, are all operating nominally.
By way of a further update and timeline of events:
LINX have informed us that the LINX LON1 network is currently stable and passing traffic since 15:40.
However with an abundance of caution, we will maintain traffic away from this peering port at the moment.
LINX report that a minority of their connected member ports experienced brief moments of instability between 11:15 - 11:28 and 11:41 - 11:57 following routine configuration changes they were making on certain devices in their network (not the same device we connect to).
Our customers will only have noticed connectivity issues at this time if your traffic was trying to reach a network on these affected devices.
However, at 15:32, following some additional debugging added to the LINX LON1 LAN following vendor TAC instructions to get to the bottom of the earlier issues, LINX saw a much larger interruption in traffic flows, and this time it affected our port as well as other ISPs and networks across the LAN.
According to LINX LON1 traffic stats (https://portal.linx.net/
), at ~15:32, traffic levels on LON1 dropped from 4.2Tb/s to 0.47Tb/s, before recovering to 3Tb/s at 16:00.
At 15:51 we took the decision to migrate traffic away from LINX LON1. Many other ISPs and networks have also taken this step as can be seen with the reduced traffic levels.
We will continue to monitor the situation and restore connectivity to LINX LON1 once we're satisfied stability has returned.