19:00 PST Due to very high temperatures in Los Angeles today (42 C / 108 F) we are noticing service instability on a few nodes due to elevated temperatures.
We are coordinating with the datacenter. We may need to shut down some of our equipment in order to avoid heat damage.
Since it is already 19:00 in Los Angeles, we are hoping that the impact will be minimal as outside temperatures are already going down.
Affected nodes: v74XX
20:09 PST We have gracefully shut down the affected nodes to avoid equipment damage. HVAC team is on site investigating.
20:37 PST We have heard back from HVAC team; they will need approx. 1 hour to repair failed units. We will be starting V74XX servers one by one over the next two hours as temperature goes down.
21:22 PST We have gradually started all v74XX nodes. Node v7430 however is having issues - we are migrating all customers off of this node.
22:15 PST We have noticed instability on node v7418. We are migrating all customers from this node.
22:46 PST We have noticed instability on node v7428. We are migrating all customers from this node as well.
22:52 PST Temperatures are rising again in the datacenter. We are suspending migrations and shutting down some of the V74XX nodes at this time.
00:30 PST We have received a notification from the datacenter that AC has been fully restored. We will be starting all remaining servers shortly.
00:55 PST All equipment has been started, and remaining customers have been brought back online.