Los Angeles Ecommerce/DC6 datacenter down Monday 20th June 2022 14:13:00

14:03 PST: We are observing an outage at DC6, it looks like a facility outage. We are investigating at the moment.

14:17 PST: We have confirmed that there is a major power outage in the facility, affecting both, A+B (redundant) power feeds. Engineers are working on this issue.

14:55 PST: Power restoration work is ongoing.

15:24 PST: Repairs are still ongoing.

15:55 PST: Power has been restored. We are working on bringing up our equipment.

16:34 PST: Most of our equipment is back up, we are working on restoring the remaining servers.

Currently restored nodes: all v61xx, v62xx (except v6211), v63xx, v64xx. We are working on v65xx and node v6211 (it has a hardware failure due to outage).

17:20 PST: all nodes are up, with the exception of v6211. We are migrating all clients off of this node v6211, this will take approximately 1 hour. VMs will start automatically once migrated.

19:28 PST: all services have been restored.