Power Failure at AQL DC2 Colocation and Dedicated

Event Started
2020-05-14 12:37
Report Published
2020-05-14 11:37
Last Updated
2022-04-13 09:33
Event Finished
2020-09-11 14:04

We suspect a power outage at AQL DC2.

Timeline (most recent first)
  • 2020-12-28
    10:55:00

    We are still waiting on AQL for an RFO update.

  • 2020-09-11
    14:05:00

    Over the past several months we have been in discussions with AQL regarding the power at DC2. We anticipate there may be a follow-on maintenance announced as we look to improve the resilience at this site.

  • 2020-06-15
    12:27:00

    While we have not seen any instability in the intervening time, we do not yet have satisfactory resolution to this issue.

  • 2020-05-15
    07:15:00

    We have received the following update from AQL:

    DC2 Salem Chapel. UPS Array 1 Salem Chapel site.

    This site contains DC2 and DC3 datacentres. The datacentres are fed from two resilient arrays of N+1 UPS’s. At this time we can identify that a software issue caused a loss of output from the entire array of UPS’s during a routine non-invasive maintenance. These UPS’s were feeding many of the “A” feeds in DC2 and a number of special rack locations in DC3 and resulted in our first power outage in 14 years of operation.

    We’re continuing to analyse the SNMP data from the UPS’s and other devices along with checking all aspects of this maintenance procedure. We will also be having a full on-site manufacturers inspection into all aspects of the units. At this time, we do not know the exact mode of failure, but we will need to carry out more invasive tests over the coming days and weeks, which we will be informing customers of in due course. This work may occur at short notice should we become more concerned about the stability of the UPS array.

    As i’m sure you’re aware, maintenance is more difficult during current conditions and we will ensure we plan the works such that they keep both our engineers and customers safe.

  • 2020-05-14
    12:42:00

    We have received confirmation from AQL: "We are currently aware of service disruption over the AQL network at this time. We are actively investigating this issue as a top priority and we will ensure you are updated as more information becomes available."