Partial outage @ USCA_2, USCA_3 Wednesday 30th March 2022 18:53:00

18:36 PST We are investigating an outage affecting customers on nodes v76XX (USCA_2), v77XX (USCA_3)

19:12 PST Quadranet datacenter confirmed they had a power outage. We use redundant power (A+B feeds), it is still unclear why both feeds went offline.

19:24 PST Quadranet lost power to both feeds A+B again.

19:58 PST Power has been restored again. We are bringing our equipment back online.

20:24 PST most servers have been brought back up online. We are working on the following remaining nodes: USCA_2: v7602 USCA_3: v7709, v7717, v7721, v7723, v7729, v7730

20:43 PST We managed to bring up servers 7717, 7723 and 7730, however, client VMs are not able to auto-start. We are migrating all clients off of these servers, it will take approximately 2 hours to bring all VMs hosted on these nodes back up.

We are working with Quadranet to bring up remaining nodes: USCA_2: v7602 USCA_3: v7709, v7721, v7729

21:11 PST: Node v7721 was restored.

Remaining nodes: USCA_2: v7602 USCA_3: v7709, v7729

21:56 PST Node v7602 was restored. Still working on: USCA_3: v7709, v7729

21:17 PST Node v7709 was restored.

Still working on the very last node @ USCA_3: v7729

NOTE: If your VM is on one of the following nodes: 7717, 7723 or 7730, then it will not boot until migrated. ETA is approximately 50-80 minutes for all VMs to finish migrating off of these nodes.

0:51 PST Node v7729 restored. All work has been completed.

Losing power on both primary and redundant power feeds is extremely uncommon, and therefore we will be working with Quadranet to better understand the root cause of this outage and steps they are going to be taking in order to prevent this from happening in the future.

We will be closing this issue now