At 17:50 PST we were alerted by our monitoring system of a multiple server outage in our Vancouver datacenter. Upon first investigation it appears to be a power related outage. We are checking with the datacenter for further information.
At 18:01 PST technician was dispatched to the datacenter for further investigation. He will arrive within 20 minutes.
At 18:26 PST engineer is on site investigating
At 18:38 PST we have determined that the power circuit external to our rack is not providing power for unknown reason.
We are in touch with the building management on possible resolutions
At 18:55 PST Datacenter management dispatched an engineer to fix the circuit.
Meanwhile, we were able to restore power to 3 nodes: v1213, v1214 and v1216 (we moved them onto another available circuit)
At 19:32 PST circuit was restored, we will be powering on the remaining servers shortly.
At 19:54 PST remaining nodes to power on: v1215, v1218, v1219
At 19:58 PST All nodes are back online except v1215, which went into fsck upon boot. We will create a new incident for affected clients on node v1215.