Router and Switch Upgrades in Interxion LON2 Backbone

Event Started
2022-05-10 02:00
Report Published
2022-05-07 15:38
Last Updated
2022-05-17 11:10
Event Finished
2022-05-10 05:00

Early in the morning of 10th May we will be carrying out works to upgrade software running on gunn.x.faelix.net — our peering and transit router in Interxion LON2. This is preemptive maintenance following from a previous incident that affected another router running a similar version of the routing software.

We will also be using this maintenance window to update the software on our ring switches rs145 and rs146 in Interxion LON2, also a preemptive maintenance following from an issue we identified affecting those devices.

Prior to works we will shutdown BGP sessions to upstream and wholesale carriers; to LINX peering; and to downstream customers connected to the devices being upgraded. This will minimise any potential disruption to customers.

During the period 03:00 till 06:00 (UK local time) there may be a small number of short disruptions to layer-2 and layer-3 services passing through Interxion LON2 as devices apply updates, and while routing reconverges at the start and end of the works.

Timeline (most recent first)
  • 2022-05-11
    11:10:00

    We are closing off this maintenance.

  • 2022-05-10
    03:28:00

    IPv6 errors are clearing.

  • 2022-05-10
    03:14:00

    OSPFv3 for IPv6 routing has some anomalies. We are correcting these.

  • 2022-05-10
    02:52:00

    One of the switches we planned to perform this maintenance upon in Telehouse West has gone into the failure condition we were hoping to avoid with these works. rs142 in Telehouse West stopped forwarding frames at line-rate, sending them to the CPU for processing instead, and causing high levels of packet loss for traffic passing through Telehouse West.

    We have applied the update as an emergency fix, and the switch has been rebooted to apply it.

  • 2022-05-10
    02:30:00

    We are continuing to monitor the network.

  • 2022-05-10
    02:19:00

    BGP sessions are now restoring.

  • 2022-05-10
    02:18:00

    We have now applied the updaate to rs146.

  • 2022-05-10
    02:02:00

    We are now applying the update to rs145.