Urgent Maintenance: Core Switch Update Backbone

Event Started
2019-09-18 21:52
Report Published
2019-09-20 03:00
Last Updated
2022-04-13 09:14
Event Finished
2019-09-20 05:15

We are observing some unusual behaviour on our core switches in Reynolds House and Williams House. While this is not affecting performance of any hosted customer services at the moment, we believe that this is indicative of a resource exhaustion bug with the firmware in the devices. Having studied the vendor's release notes in detail, our engineers believe that an emergency maintenance to update and reboot the affected devices will resolve this problem for the long term. However, this update must be performed across multiple switches at the same time, and so we expect a short period of disruption to connectivity during the maintenance window of 04:00-05:00 (local time) on 20th September 2019.

Timeline (most recent first)
  • 2019-09-20

    The network has stabilised following the maintenance work. Engineers are still on-site, observing the situation.

  • 2019-09-20

    After several failed attempts to upload the updated firmware, some of which caused additional switch reboots (without applying the updated firmware), all devices have now been updated to the vendor's stable version.

  • 2019-09-20

    While performing the switch firmware upgrade, the devices crashed. This was what we had been hoping to avoid by performing the maintenance. Our engineers are working to resolve this as quickly as possible.