MA1 to THW circuit problem

Timeline (most recent first)

• 2025-02-27
  14:15:00

  We've received an RFO from the carrier. At 09:06 they also received alarms, and by 09:15 received multiple reports of loss of services. At 09:55 they suspected the issue was in Manchester. By 10:33 they had found the problem and implemented a fix. The root cause was that a change to some other infrastructure (to alleviate a separate P1 incident) caused a routing loop within access nodes in Manchester. Engineer policy has been updated.

• 2025-02-20
  09:18:00

  Carrier update:

  We had an issue with our access nodes in Manchester. We are investigating root cause and will issue a Major incident report within 5 working days however all services have now been restored.

• 2025-02-18
  11:55:00

  Supplier confirms restoration of service, but we do not yet have an RFO.

• 2025-02-18
  10:49:00

  We've called the supplier and their agent thinks it might be related to a problem they are having "on the west coast" but we are still waiting for a formal acknowledgement of the fault.

• 2025-02-18
  10:39:00

  We've just seen pings reachability alerts clear suggesting the layer-2 service might be restored. We are awaiting an update confirming what has happened.

• 2025-02-18
  09:17:00

  Ticket raised with supplier.

• 2025-02-18
  09:06:00

  Initial alerts received.