Drive failure in fifi Incident

A drive in one of our virtual-machine hosting servers failed catastrophically at around 06:20 UTC.

Having determined that the fault required a site visit, Marek attended Reynolds House and determined that the fault was a complete failure of fifi's RAID array due to one of the drives burning up. Quite literally, the drive had caught fire!

Backups of every virtual machine on fifi had completed earlier in the night, so Marek began restoring virtual machines onto another server from that backup. All VMs have since been restored and are running again.

If you have any immediate issues which you believe are a result of your VM restarting, please contact us.

published: 2015-02-08 06:20:00
resolved: 2015-02-08 11:00:00

Timeline

2015-02-08 06:20:00
Alerts received from monitoring system.
2015-02-08 08:00:00
Attended Reynolds House and determined physical drive failure.
2015-02-08 11:00:00
Customer virtual machines restored onto other servers. Incident closed.