Networking fault for virtual hosts on "muffy"

Service Incident

published: 2017-06-04 13:07:00
started: 2017-06-03 12:07:42
finished: 2017-06-03 12:18:00
resolved: 2017-06-03 13:00:00

Virtual Servers on suffered a networking outage for a short while. We have found that this is due to a bug in the virtual switch we use, Open vSwitch, when it encounters a crafted packet:

Jun  3 12:07:42 muffy ovs-vswitchd: ovs|00002|daemon_unix(monitor)|ERR|1 crashes: pid 1891 died, killed (Segmentation fault), core dumped, restarting

Jun  3 12:07:42 muffy ovs-vswitchd: ovs|00003|daemon_unix(monitor)|EMER|fork child died before signaling startup (killed (Segmentation fault))

This morning we have identified that this is fixed in a more recent upstream version, and so we will plan some maintenance to upgrade all our hosts.