Power Failure at AQL DC2

Service Incident

published: 2020-05-14 12:37
started: 2020-05-14 11:37

We suspect a power outage at AQL DC2.


Timeline

2020-05-14 12:42

We have received confirmation from AQL: “We are currently aware of service disruption over the AQL network at this time. We are actively investigating this issue as a top priority and we will ensure you are updated as more information becomes available.”

2020-05-15 07:15

We have received the following update from AQL: > DC2 Salem Chapel. UPS Array 1 Salem Chapel site. > > This site contains DC2 and DC3 datacentres. The datacentres are fed from two resilient arrays of N+1 UPS’s. At this time we can identify that a software issue caused a loss of output from the entire array of UPS’s during a routine non-invasive maintenance. These UPS’s were feeding many of the “A” feeds in DC2 and a number of special rack locations in DC3 and resulted in our first power outage in 14 years of operation. > > We’re continuing to analyse the SNMP data from the UPS’s and other devices along with checking all aspects of this maintenance procedure. We will also be having a full on-site manufacturers inspection into all aspects of the units. At this time, we do not know the exact mode of failure, but we will need to carry out more invasive tests over the coming days and weeks, which we will be informing customers of in due course. This work may occur at short notice should we become more concerned about the stability of the UPS array. > > As i’m sure you’re aware, maintenance is more difficult during current conditions and we will ensure we plan the works such that they keep both our engineers and customers safe.