Suspected Fibre Break between Manchester and Leeds Backbone

Timeline (most recent first)

• 2022-07-27
  09:53:00

  We have published our RFO for the incident of 2022-07-25 and are closing this incident now.

• 2022-07-27
  00:00:00

  The optical engineers have made changes to the amplifier parameters. We are continuing to monitor this link.

• 2022-07-26
  22:34:00

  Delayed: optical engineers will be arriving at Williams House around 23:15 UTC.

• 2022-07-26
  22:00:00

  The optical engineers have brought forward their plans and are on way to site. ETA 22:30 UTC.

• 2022-07-26
  21:08:00

  We have been told that there are no ongoing fibre works by the underlying fibre provider, so we are currently not sure why there have been brief drops in light levels. Optical engineers are going to adjust optical amplifier thresholds at Equinix MA1 (Williams House) first thing in the morning. We will keep this circuit as a backup in the meantime.

• 2022-07-26
  20:47:00

  And has come back again. We are monitoring things, while having this link as lower preference for our north-south routing.

• 2022-07-26
  20:40:00

  We've seen the light level drop again, as the link has failed.

• 2022-07-26
  18:32:00

  Systems have been stable for the last 90 minutes. We are going to continue detailed monitoring for another hour before closing this issue and beginning the RFO write-up.

• 2022-07-26
  16:59:00

  We are monitoring the situation.

• 2022-07-26
  16:45:00

  We are bringing the link back into production.

• 2022-07-26
  16:24:00

  Optical link has been re-established. We are testing.

• 2022-07-26
  15:31:00

  Optical engineers are still at Equinix MA1 Williams House trying to establish the amplified link.

• 2022-07-26
  14:18:00

  Optical engineers are still at Equinix MA1 Williams House trying to establish the amplified link.

• 2022-07-26
  12:31:00

  The failed DWDM amplifier has been removed and the engineers on-site are beginning to test the replacement before bringing it into service. Once we've got the all-clear from them then we will begin our own tests.

• 2022-07-26
  11:32:00

  Fibre engineers have performed tests on the fibre. It appears that a DWDM amplifier has failed. They are performing some tests before replacing it with a spare unit.

• 2022-07-26
  11:03:00

  Fibre engineers have run tests from the Leeds end and found no issues. They have now arrived at the Manchester end and are investigating.

• 2022-07-26
  09:39:00

  We believe we've identified the multiple problems and worked around them. Things should start to resolve.

• 2022-07-26
  09:25:00

  Our backup link between Manchester and London is having other issues which is causing this to take longer to resolve than expected.

• 2022-07-26
  09:18:00

  The affected router refuses to commit its configuration. We have issued an emergency reboot.

• 2022-07-26
  09:10:00

  This has caused some issues with one of our routers in Manchester which is blackholing traffic. We're working on it.

• 2022-07-26
  08:57:00

  The provider suspects a problem on one of the two pairs, and are continuing to investigate.