12th November 2018

Virtual instances - Phoenix, Arizona Power outage - Phoenix Nap datacenter

At 20:20 PST our monitoring system reported an outage at our Phoenix location. Upon checking with the datacenter we have received a confirmation that there has been a power interruption. Datacenter technicians are already investigating.

At 20:28 PST service has been fully restored.

At 22:21 PST there has been another outage. We are investigating further.

Relevant topic on WHT: https://www.webhostingtalk.com/showthread.php?t=1739369

At 22:54 PST power to our colocation space has been restored. We will be restarting all services in the next 20 minutes.

At 22:58 PST we have restarted all our equipment and all customer's VPS will be started automatically in the next 10 minutes.

At 23:26 PST After all equipment was restarted, we are observing issues with the rebootless kernel updater. We will have to manually apply kernel upgrades and gracefully restart all VPS nodes. This will be done in the next 60 minutes - please expect a 15-20 minute downtime per VPS.

At 23:56 PST all maintenance tasks have been completed.

At 00:19 PST Phoenix Nap datacenter lost power again.

We have received the following statement from the datacenter:

Our engineers have isolated the root cause of the power issue occurring at our PHX DC. We are working to resolve this and bring all systems back online. In the process of this final work there is a possibility of additional disruptions occuring. Once all this work has been completed we will update again and let you know that all work is done.

We are awaiting for further updates from the datacenter.

At 00:21 PST power has been restored. We are bringing our equipment up again (services should be back online within 20 minutes) and waiting for further updates from the datacenter.

At 00:31 PST power was lost again.

At 00:33 PST power has been restored. We are bringing our equipment up again (services should be back online within 20 minutes) and waiting for further updates from the datacenter.

At 00:38 PST power was lost again.

At 00:39 PST power has been restored. We are bringing our equipment up again (services should be back online within 20 minutes) and waiting for further updates from the datacenter.

At 00:50 PST all services have been restored. We are awaiting for further updates from the datacenter.

At 01:29 PST we have received the following statement from the datacenter:

Our engineers have taken the steps necessary to restore full power to our DC. At this time we are seeing service up and stable and do not anticipate any further disruptions to power to devices. We will continue to monitor this issue and update with any new status.

We will be closing this issue for now and will reopen it if we observe any more issues.