PDU failure in Reynolds House

Degraded Performance

published: 2019-10-28 06:05
started: 2019-10-28 02:33:00
resolved: 2019-10-29 23:30

One of our power distribution units in Reynolds House has restarted several times, including de-powering and re-powering connected devices several times. Some customers whose equipment was powered from a single feed will have lost power.


Timeline

2019-10-28 06:55

The PDU continues to show random power outlets as down each time we query its status. While we do not believe that it is actually power-cycling those ports, we do not trust it to continue behaving and so have moved all single-powered devices off that PDU and onto the other PDU in that rack.

2019-10-28 17:29

A replacement PDU has been ordered and should arrive tomorrow. Maintenance to swap out the failed unit has been provisionally arranged for 18:00-20:00 2019-10-29.

2019-10-29 19:30

Engineers have arrived on site and are about to begin work.

2019-10-29 20:46

Our engineers have removed the failing PDU and worked with the datacentre technicians to commission the replacement PDU. We are monitoring the situation before we begin to close this maintenance out.

2019-10-29 23:31

Our engineers are leaving site. This maintenance is complete and has been a success.