Multacom USCA_4 migration
ResolvedLocation: USCA_4 (Multacom)
Affected nodes: v11xx
13:31 PST
We have received a sudden notification today that Multacom is closing down their existing facility and moving all equipment to a different location within the next 1-2 weeks.
As we have been considering to move away from Multacom due to repeated issues with HVAC system anyways, this gives us a push to complete the migration ASAP.
Sadly, since this is on a very short notice, we are not able to properly plan to execute this without downtime. On the other hand, we do not want to risk any more outages directly or indirectly related to this situation, and so we will begin moving customers today, June 30, 2026.
We will do our best to minimize the impact, however, some downtime (1-2 hours per VM) is expected.
There will be no IP change and no data loss. As always, we strongly recommend having an offsite backup.
Update 14:55 PST: We have migrated approx. 20% of all VMs.
So far we were able to avoid VM restarts on almost all transferred VMs, and vast majority of VMs experienced just a few minutes of network unavailability. We're doing our best to continue the move with absolute minimal impact.
Update 16:40 PST: We have migrated approx. 60% of all VMs.
Still the vast majority of VMs are moved without a restart, with just a few minutes of network interruption. Whenever a restart is required, it is performed gracefully (via ACPI).
Final update 17:44 PST: Migration has been completed. Less than 1% of VMs had to be restarted during the migration.
We will close the incident at this time.
Affected nodes: v11xx
13:31 PST
We have received a sudden notification today that Multacom is closing down their existing facility and moving all equipment to a different location within the next 1-2 weeks.
As we have been considering to move away from Multacom due to repeated issues with HVAC system anyways, this gives us a push to complete the migration ASAP.
Sadly, since this is on a very short notice, we are not able to properly plan to execute this without downtime. On the other hand, we do not want to risk any more outages directly or indirectly related to this situation, and so we will begin moving customers today, June 30, 2026.
We will do our best to minimize the impact, however, some downtime (1-2 hours per VM) is expected.
There will be no IP change and no data loss. As always, we strongly recommend having an offsite backup.
Update 14:55 PST: We have migrated approx. 20% of all VMs.
So far we were able to avoid VM restarts on almost all transferred VMs, and vast majority of VMs experienced just a few minutes of network unavailability. We're doing our best to continue the move with absolute minimal impact.
Update 16:40 PST: We have migrated approx. 60% of all VMs.
Still the vast majority of VMs are moved without a restart, with just a few minutes of network interruption. Whenever a restart is required, it is performed gracefully (via ACPI).
Final update 17:44 PST: Migration has been completed. Less than 1% of VMs had to be restarted during the migration.
We will close the incident at this time.