No incidents reported
We will be performing a service impacting maintenance:
Window Start Time: 19-JUN-2018 13:15 PST
Window End Time: 19-JUN-2018 20:00 PST
Expected Impact: < 1 hour
Reason: Router replacement to address intermittent network degradation.
Location: DC2 The following nodes will be affected: v75XX
During the maintenance window, network connectivity will become degraded, or temporary lost.
Window Start Time: 18-JUN-2018 19:30 PST
Window End Time: 18-JUN-2018 21:00 PST
Reason: Router software upgrade to address intermittent network degradation.
The following nodes will be affected: v75XX
At 12:07 PST our monitoring system reported a failed SSD in the RAID array. We will be gracefully shutting this node down for disk replacement in a few minutes.
At 12:19 PST we have replaced the SSD disk and the node is being restarted. RAID array will rebuild in the background, and all VMs will be started automatically in the next 20 minutes.
At 11:05 PST we will be gracefully rebooting these two nodes to apply an emergency fix to the RAID arrays on these two nodes.
11:22 PST: Maintenance has been completed, all VMs on both nodes will auto restart in the next 20 minutes.
At 21:07 PST our engineers have been alerted to an outage on node v907.
At 21:38 PST a fix has been applied and all VMs will be started in the next 15 minutes.