At 01:45 PST our network monitoring system reported an intermittent network accessibility issue at our DC2 and DC3 locations affecting certain routes (this is affecting approximately 40-60% of internet routes). Upon further diagnosis we determined an issue at our upstream provider.
Upon escalation to our upstream provider we received a confirmation that their engineers are already on site and actively troubleshooting the issue.
At 02:05 PST connectivity fully restored
At 02:26 PST we observe issues again
At 02:44 PST connectivity fully restored
At 02:48 PST we have received the following final update from the upstream provider:
The outage witnessed was due to a line card failure in the penthouse edge router, while it has recovered by itself, the blade has been swapped out to prevent such outage from occurring again. We will be working with our network vendor to find out why the line card suddenly went offline.
We will continue monitoring the network but for the time being will mark this as resolved.