19:00 PST Due to very high temperatures in Los Angeles today (42 C / 108 F) we are noticing service instability on a few nodes due to elevated temperatures.
We are coordinating with the datacenter. We may need to shut down some of our equipment in order to avoid heat damage.
Since it is already 19:00 in Los Angeles, we are hoping that the impact will be minimal as outside temperatures are already going down.
Affected nodes: v74XX
20:09 PST We have gracefully shut down the affected nodes to avoid equipment damage. HVAC team is on site investigating.
20:37 PST We have heard back from HVAC team; they will need approx. 1 hour to repair failed units. We will be starting V74XX servers one by one over the next two hours as temperature goes down.
21:22 PST We have gradually started all v74XX nodes. Node v7430 however is having issues - we are migrating all customers off of this node.
22:15 PST We have noticed instability on node v7418. We are migrating all customers from this node.
22:46 PST We have noticed instability on node v7428. We are migrating all customers from this node as well.
22:52 PST Temperatures are rising again in the datacenter. We are suspending migrations and shutting down some of the V74XX nodes at this time.
00:30 PST We have received a notification from the datacenter that AC has been fully restored. We will be starting all remaining servers shortly.
00:55 PST All equipment has been started, and remaining customers have been brought back online.
At 17:30 PST our monitoring system identified an outage on node v7229. We are investigating.
At 17:48 PST we have identified an issue with LAN adapter on this server and currently we are working on a fix.
At 18:25 PST we have deployed the fix and all VMs will start automatically in the next 20 minutes.
At 14:42 PST our Vancouver location lost network connectivity.
At 14:52 PST Our engineers got in touch with our upstream carrier and confirmed the outage is within their network. Their engineers are already working to restore connectivity.
At 15:05 PST Network connectivity was partially restored, however many of our clients are still affected.
At 16:34 PST Datacenter advised us that this should be resolved within the next 30-60 minutes.
At 17:17 PST Connectivity has been restored.
11:38 AM we have noticed an instability on node v201 and will shut it down shortly to apply a fix.
11:46 AM maintenance complete, all VMs will start automatically in the next 5 minutes.
No incidents reported