We are observing some unusual behaviour on our core switches in Reynolds House and Williams House. While this is not affecting performance of any hosted customer services at the moment, we believe that this is indicative of a resource exhaustion bug with the firmware in the devices. Having studied the vendor's release notes in detail, our engineers believe that an emergency maintenance to update and reboot the affected devices will resolve this problem for the long term. However, this update must be performed across multiple switches at the same time, and so we expect a short period of disruption to connectivity during the maintenance window of 04:00-05:00 (local time) on 20th September 2019.
Urgent Maintenance: Core Switch Update Backbone
- Event Started
- 2019-09-18 21:52
- Report Published
- 2019-09-20 03:00
- Last Updated
- 2022-04-13 09:14
- Event Finished
- 2019-09-20 05:15
Linked Events
Timeline (most recent first)
-
2019-09-20
04:50:00The network has stabilised following the maintenance work. Engineers are still on-site, observing the situation.
-
2019-09-20
04:19:00After several failed attempts to upload the updated firmware, some of which caused additional switch reboots (without applying the updated firmware), all devices have now been updated to the vendor's stable version.
-
2019-09-20
03:59:00While performing the switch firmware upgrade, the devices crashed. This was what we had been hoping to avoid by performing the maintenance. Our engineers are working to resolve this as quickly as possible.