7th May 2018

Virtual instances - New York Partial network outage in New York - affected nodes: v21XX

At 10:03 PST our monitoring system detected a network abnormality at our New York POP affecting many customers on v21XX nodes.

Upon initial checking we were able to confirm that the issue is with our upstream provider. We have escalated the issue and waiting for an update.

At 11:52 PST Our engineers have narrowed down the issue: we announce our IPv4 subnets to our provider, however our announcements are 'flapping' to the outside world.

At 12:18 PST Together with our upstream provider we confirmed that flapping is caused on our upstream provider's side, and not on our side. Our upstream provider has escalated the issue internally.

At 12:49 PST In an attempt to restore connectivity our upstream provider moved our uplink fiber to a different linecard. However this did not resolve the issue. We are awaiting for further updates.

At 14:26 PST In order to speed up connectivity restoration we have asked our upstream provider to add an additional cross-connect between our racks, so we can route around the affected segment using our own equipment as we have spare equipment on site.

At 14:48 PST Connectivity has been restored.

Over the next few hours we will be continue working on a permanent fix, however right now there should be no issues.