Mission Critical Support
March Service Interruptions
We realize how critical Open LMS is to your teaching and learning efforts, especially now as many of you are working overtime to deliver even more of your learning online in response to the COVID-19 pandemic. We also know that the slowdown and outage on March 23 impacted you and caused concern at a very critical time. We want to explain what happened, how we responded, and steps we’ve taken to reduce future risk.
Beginning on March 23rd, the Open LMS team identified and responded to several regional slow-downs and temporary outages in North and Latin America. Our technical teams responded quickly and together with the teams at Amazon Web Services (AWS) restored service to normal levels. We continue to work internally and with AWS to ensure continued interruption-free service.
Prior to March 23, our team had been actively planning for increasing resource demands due to back-to-school usage and a COVID-19 related increase in the number of students relying on purely online instruction. As part of this plan, the Open LMS team had been steadily adding resources to our various data centers in order to over-provision them in advance of any huge spike in usage.
On March 23rd, it became clear that even our projected increase was outpaced by the dramatic increase in load. As you can see in Figure 1 and Figure 2, our increasing capacity maintained normal service levels, despite significantly more traffic, leading up to March 23rd. At this point, the impact on our data centers more than doubled in less than 12 hours — temporarily overwhelming our available capacity.
Our technical teams worked quickly to prevent any data loss and at the same time dramatically scaled our capacity to what was needed to return your systems to normal operating levels. We’re not yet done though — we continue to add capacity and are provisioning our data centers such that they can withstand another massive increase in traffic well beyond what we’ve already seen. If you anticipate additional dramatic increases in usage, please let us know via this short form.
We’re committed to you, your institution, your faculty and your learners. We know we need to provide excellent service to keep your business. We hope an open conversation around recent events will reinforce that trust. If you have additional questions, please reach out to your support manager, we’re here to help.
Open LMS Technical Operations & Support Team
Darmeshia Bell – System Admin
Kevin McCarthy – System Admin
Anita Hopper – North American Support Manager
Jay Hutchinson – Australia / Pacific Support Manager
Juan Paez – Latin & Central American Support Manager
Karen Underwood – Europe & Middle East Support Manager