Suspected Crash of VPS host lu.n.faelix.net Virtual Servers

Event Started
2022-05-09 18:17
Report Published
2022-05-09 18:31
Last Updated
2022-05-16 20:16
Event Finished
2022-05-09 19:51
Timeline (most recent first)
  • 2022-05-09
    19:42:00

    All services have restored.

    We believe the root cause to be:

    • openvswitch crashing and trying to restart
    • but being unable to due to /var/log being full

    We have increased logging space allocated to all nodes in the x7 cluster, and will review why monitoring alerts were not triggered which would have advised our engineers about the space on that logical volume becoming congested.

  • 2022-05-09
    19:39:00

    VPSs have begun booting.

  • 2022-05-09
    19:33:00

    The host is now booting up.

  • 2022-05-09
    19:29:00

    OpenVSwitch on the physical host stopped, and refused to auto-restart due to disk space. We are addressing this problem.

  • 2022-05-09
    19:10:00

    The host is running, but its networking is not establishing.

  • 2022-05-09
    19:00:00

    Engineers have arrived at site.

  • 2022-05-09
    18:17:00

    We have received alerts indicating network link flaps for lu.n.faelix.net (one of our VPS hosts in Telehouse North)