6th November 2018

Virtual instances - Los Angeles, DC9 Network interruption - nodes v91XX

At 17:20 PST we have applied a configuration change to our top of the rack switch (in an attempt to permanently resolve connectivity issues reported earlier today). Unfortunately the configuration change brought the network offline again, and automatic configuration rollback did not restore connectivity.

We are working on resolving this as soon as possible.

At 17:31 PST connectivity has been restored and we are investigating this further.

17:41 PST: Upon further investigation we have determined that network was indeed operating properly during this configuration change, but it took a few minutes for new BGP routes to propagate (it took our monitoring service longer than usual to see the new route). This means the actual outage was most likely well under 5 minutes.

Update: we have completed all maintenance tasks. As the result, we were able to find the cause of the today's issues and permanently resolve it. Furthermore, we made changes to our backup uplink so that it will turn on automatically without any downtime.