Past Incidents

We will be performing a service impacting maintenance:

Window Start Time: 19-JUN-2018 13:15 PST

Window End Time: 19-JUN-2018 20:00 PST

Expected Impact: < 1 hour

Reason: Router replacement to address intermittent network degradation.

Location: DC2 The following nodes will be affected: v75XX

During the maintenance window, network connectivity will become degraded, or temporary lost.

We will be performing a service impacting maintenance:

Window Start Time: 18-JUN-2018 19:30 PST

Window End Time: 18-JUN-2018 21:00 PST

Expected Impact: < 1 hour

Reason: Router software upgrade to address intermittent network degradation.

Location: DC2 The following nodes will be affected: v75XX

During the maintenance window, network connectivity will become degraded, or temporary lost.

At 12:07 PST our monitoring system reported a failed SSD in the RAID array. We will be gracefully shutting this node down for disk replacement in a few minutes.

At 12:19 PST we have replaced the SSD disk and the node is being restarted. RAID array will rebuild in the background, and all VMs will be started automatically in the next 20 minutes.

At 11:05 PST we will be gracefully rebooting these two nodes to apply an emergency fix to the RAID arrays on these two nodes.

11:22 PST: Maintenance has been completed, all VMs on both nodes will auto restart in the next 20 minutes.

At 21:07 PST our engineers have been alerted to an outage on node v907.

At 21:38 PST a fix has been applied and all VMs will be started in the next 15 minutes.