Core and Backbone Upgrade Backbone

Event Started
2024-04-07 00:00
Report Published
2024-03-13 13:07
Last Updated
2024-11-07 10:08
Event Finished
2024-05-15 16:07

During the month of April we will be performing a series of maintenances to establish a new, upgraded core network running on the Juniper MX204 platform. This will hugely increase the capacity of our network and ready it for the planned growth in 2024 and beyond.

Timeline (most recent first)
  • 2024-05-16
    06:47:00

    Our THW route-server configuration has been moved to the new RR-RS.

  • 2024-05-15
    21:15:00

    The second of our "OG" VyOS routers, aebi.m.faelix.net, has been deracked from Equinix MA1 Kilburn House.

  • 2024-05-15
    20:00:00

    One of the original two boxes that began our love affair and migration to VyOS in 2019 just came out: bly.w.faelix.net has been deracked from Equinix MA1 Williams House.

  • 2024-05-15
    13:50:00

    Last in, first out: fossey.a.faelix.net has been deracked from Equinix MA5.

  • 2024-05-14
    16:07:00

    Our temporary migration VLAN, connecting the old VyOS network to the new MX204 network has been removed.

  • 2024-05-14
    14:42:00

    Our alerting systems show no irregularities, and our traffic levels are unperturbed. The final step will be to power-down and de-rack the old VyOS routers, which we will carry out over the coming days.

  • 2024-05-14
    14:27:00

    Our old VyOS router, aebi.m.faelix.net, is now disconnected from the network.

  • 2024-05-14
    14:20:00

    Our old VyOS router, bly.w.faelix.net, is now disconnected from the network.

  • 2024-05-14
    10:24:00

    Our L3 switches in Equinix MA5 are now in EBGP with the MX204s.

  • 2024-05-14
    10:16:00

    Our old VyOS router, fossey.a.faelix.net, is now disconnected from the network.

  • 2024-05-14
    08:54:00

    Our old VyOS router, earhart.n.faelix.net, is now disconnected from the network.

  • 2024-05-13
    19:17:00

    Our old VyOS router, dekker.d.faelix.net, is now disconnected from the network.

  • 2024-05-13
    18:45:00

    Our old VyOS router, gunn.x.faelix.net, is now disconnected from the network

  • 2024-05-13
    17:12:00

    Our THN route-server configuration has been moved to the new RR-RS.

  • 2024-05-10
    15:32:00

    We've got one or two customers who still have BGP sessions to our old route-servers, but those are to be moved over the coming few days.

  • 2024-05-07
    23:00:00

    All EBGP sessions are off our old VyOS routers. We will gradually begin shutting these down over the coming week.

  • 2024-05-03
    09:35:00

    None of our VyOS routers in Manchester has any EBGP sessions to customers or peers any more: everything in the north has been migrated to the MX204s.

  • 2024-05-02
    08:00:00

    Aggregation routers in London are now directly sending traffic via the MX204s.

  • 2024-04-30
    06:00:00

    Aggregation routers in Manchester are now no longer part of AS41495's IGP and are directly sending traffic via the MX204s.

  • 2024-04-24
    10:44:00

    Bringing up IX route-server sessions.

  • 2024-04-24
    10:26:00

    Our second MAN-LON path has just been brought back into service after a failure at the provider.

  • 2024-04-24
    09:50:00

    Bringing up LONAP peering sessions.

  • 2024-04-24
    08:00:00

    After a frantic night of moving external peering and transit BGP sessions, the edge of AS41495 is a set of MX204s now: a huge upgrade.

  • 2024-04-23
    18:06:00

    We've moved all our peering in the north of the UK to land on the Juniper MX204s.

  • 2024-04-23
    15:50:00

    Some problems we'd been seeing on one of our old routers over the last 6-7 days had been increasing today. We've taken it out of service. It behaving this way is unrelated to our core/backbone upgrade work.

  • 2024-04-23
    03:04:00

    We've enabled BGP sessions to AS3257 on arendt.a.faelix.net (our MX204 in Equinix MA5).

  • 2024-04-22
    13:34:00

    The final pieces of the puzzle are now in place, and we'll begin migration customer sessions over to the MX204s from tomorrow.

  • 2024-04-19
    04:10:00

    We are leaving site.

  • 2024-04-19
    03:10:00

    The switch replacement is complete.

  • 2024-04-19
    02:00:00

    During evacuation of VPS hosts in Kilburn House, ahead of the switch maintenance, one of our physical virtualisation hosts crashed causing all VPSs running on it to crash as well. There is a separate incident for this.

  • 2024-04-19
    00:01:00

    During the early morning of Friday 19th April (the night of Thursday 18th), we will be replacing switches rs163.m.faelix.net and fs165.m.faelix.net in Equinix MA1 Kilburn House. This will necessitate disconnecting some wholesale carrier NNIs for a few seconds while we move the fibre cross-connects over to the new devices.

  • 2024-04-13
    02:15:00

    The switch has been added to the ring. We're now leaving site.

  • 2024-04-13
    01:48:00

    We are beginning the work adding rs147.x.faelix.net to the ring.

  • 2024-04-12
    23:59:00

    We'll be performing some work on our switches and routers at Interxion LON2 on the night of Friday 12th April, through till abut 4am. This will involve moving some subinterfaces on our peering/transit router at LON2 called gunn.x.faelix.net before installing a new switch in our London ring. As this involves breaking the ring for a short period, it will affect some services for a short while, so we will carry out that work around 04:00.

  • 2024-04-12
    02:48:00

    Linknets have been moved off the QSFP+ interfaces on gunn.x.faelix.net.

  • 2024-04-10
    13:48:00

    We've finished work at Telehouse North.

  • 2024-04-10
    12:17:00

    Our engineers are on site at Telehouse North removing some unused DACs to the existing VyOS-based routers, in preparation for the MX204s taking their place.