19th January 2018

Preventative emergency maintenance

On Friday, January 19 at approx. 14:20 PST we will be performing an unscheduled maintenance on the below referenced nodes in order to prevent performance degradation. Redhat advised us that we should revert CPU microcode patches applied on January 6 and instead update system BIOS to apply microcode updates - this should prevent the inevitable stability and performance issues on these nodes. Each VM will be gracefully shut down one by one for a period of approx. 30 minutes to apply the BIOS and kernel updates on the hardware nodes. After that all VMs will be started again.

17:36 PST: All updates have been completed and all VMs have been started. VMs have been offline for approximately 20-25 minutes each.

We sincerely hope this will put an end to the instability caused by the recent CPU microcode patches.

List of updated nodes:

New York 204 205 206 207 208 209 210 211

Los Angeles 426 427 430 431

Fremont 513 514 515 516 517 520 521 522 523 524

Los Angeles 755 756 757 758 759 760 761 762 763 765 766 767 768 769 770

Amsterdam 905 906 907

Los Angeles 1108 1109 1110 1111

Vancouver 1204 1205