Network outage: switch failure Tuesday 10th May 2022 02:31:00

02:23 PST We are investigating a network switch failure affecting customers located on nodes v77XX

02:42 PST From the messages in the switch console it looks like the switch ran out of RAM. Upon further checking, this is likely related to a bug in JunOS.

We are restarting the switch to bring it back online.

03:07 PST Switch is back online and connectivity has been restored. We are continuing to investigate the root cause of this outage.

03:24 PST Switch is showing more memory related issues. After further investigation it looks like this is a JunOS-related memory leak. We are applying a JunOS update.

Additionally, we are preparing a spare switch in case we need to replace.

04:13 PST Connectivity has been restored. We are continuing maintenance work to make sure everything is working as expected.

04:22 PST No more issues have been found or observed. JunOS update completely resolved this issue. We will be closing this incident.