London Ring Fibre Incident Backbone

Timeline (most recent first)

• 2025-01-17
  22:37:00

  Latest non-update from our dark fibre provider:

  We have asked our 3rd-party to confirm. We will keep you updated.

• 2025-01-17
  22:30:00

  We have chased our dark fibre provider again for an update regarding whether third party engineers accessed the ODFs at the time of the incidents.

• 2025-01-17
  20:06:00

  The latest update from our dark fibre provider confirms light levels received in Interxion are as expected.

• 2025-01-17
  18:23:00

  We have chased our dark fibre provider for an update. We are still waiting on them to report back from Interxion, and also to provide us with confirmation that nobody (including third parties) was accessing the ODFs at Telehouse West and Interxion during the times of these two incidents.

• 2025-01-17
  16:14:00

  Our dark fibre provider is going to perform intrustive testing at Interxion.

• 2025-01-17
  15:59:00

  Light levels at Telehouse West all look good.

• 2025-01-17
  12:33:00

  The fibre provider is going to perform intrusive testing on the circuit.

• 2025-01-17
  12:20:00

  We have taken the affected dark fibre path out of service.

  We have escalated this to account management at our dark fibre provider, asking for a full investigation about what is going on.

• 2025-01-17
  11:56:00

  We're waiting for an update from our dark fibre provider's NOC, because we are not expecting them to be doing any work on the circuit today.

• 2025-01-17
  11:53:00

  We've just seen light levels drop to unacceptably low levels, and the connections across the fibre drop again.

• 2025-01-17
  09:30:00

  Service continues to be stable and light levels remain good.

  We are discussing with the dark fibre provider's NOC next steps, and trying to get an understanding from them how this incident happened.

• 2025-01-17
  04:20:00

  We are seeing good light levels and normal levels of traffic. We have confirmed with our dark fibre provider's network operations centre that we believe service has been restored, and have asked for an RFO.

• 2025-01-17
  04:10:00

  Light levels are looking much improved. We're just waiting for traffic to stabilise again.

• 2025-01-17
  04:04:00

  We are about to try bringing the link back into service.

• 2025-01-16
  20:23:00

  Our fibre provider has asked that we perform tests. Our initial checks look good, but we are not prepared to even test bringing the link back into service until a quiet period. We'll check this at 4am later tonight.

• 2025-01-16
  19:49:00

  The fibre provider must be continuing to do works, because the light levels received have dropped to nothing. We've taken the circuit completely out of service while they test this properly themselves.

• 2025-01-16
  19:27:00

  The fibre provider says the issue is resolved but we are seeing -4dBm more loss on one of the two fibres in the duplex pair. We have reverted back to the fibre provider, rejecting this.

• 2025-01-16
  19:20:00

  We've seen some changes to light levels on the link that has been out of service. We've been asked to test the circuit by our fibre provider as they think they've fixed this.

• 2025-01-16
  14:48:00

  The network has been stable for almost 2 hours now. We're continuing to monitor the situation, and work with our fibre provider for our London ring to bring this back into full resilience.

• 2025-01-16
  14:33:00

  Latest update from dark fibre provider:

  We have engaged Fibre team to check the issue further. Once we receive an update will inform you accordingly.

• 2025-01-16
  13:33:00

  The fibre provider has said:

  We have engaged site owner to check the light readings. kindly confirm whether we have your approval to disconnect this circuit temporarily to take the light readings.

• 2025-01-16
  12:51:00

  We've taken the link out of service as it keeps bouncing up and down.

• 2025-01-16
  12:47:00

  The THW-IXN path is sputtering into life, unexpectedly.

• 2025-01-16
  12:07:00

  We are seeing traffic levels restore to normal.

  We are continuing to work with the dark fibre provider to fix the THW-IXN path of our fibre ring.

• 2025-01-16
  12:03:00

  CPU load on the switch in IXN has settled down.

• 2025-01-16
  12:01:00

  We've seen a large number of link bounce events on that switch at IXN, including affecting the other path between IXN and THN, such as this:

  Last Link Down Time: Jan/16/2025 12:00:54

  Last Link Up Time: Jan/16/2025 12:00:58

• 2025-01-16
  11:56:00

  Restarting the switch in THW has caused a switch in IXN to have high load. We're monitoring this to see if it returns to normal.

• 2025-01-16
  11:53:00

  We have begun initial diagnostics with the dark fibre provider.

• 2025-01-16
  11:47:00

  The locked-up switch has started responding to ping again. We are checking the situation.

• 2025-01-16
  11:45:00

  The locked-up switch is now rebooting.

• 2025-01-16
  11:38:00

  The flapping of connectivity has caused one of our switches in Telehouse West to lock up. We've forced a reboot on it, which may cause a bit more disruption, but is intended to make the problem settle down again.

• 2025-01-16
  11:36:00

  We've raised a ticket with the dark fibre provider.

• 2025-01-16
  11:26:00

  One path has restored, but the other path, between THW and IXN, remains down. We are raising this with the underlying dark fibre provider.