At exactly 10:52 pm on Saturday, Labor Day weekend 2022, cyber threat actors took advantage of the holiday lull and targeted LA Unified School District, resulting in a ransomware attack on the second largest school district in the United States. The FBI would be involved; the DHS would be involved; MGT would be involved.
In this excerpt from ITS4ED, Los Angeles Unified School District Information Technology 4 Education Podcast, we take a deep dive into the recovery missions surrounding the LAUSD cybersecurity attack. (This excerpt has been edited for clarity, readability, and conciseness.)
Host: Youssef Elias, school systems manager, LAUSD
Guests:
– Soheil Katal, CIO, LAUSD
– Jack Kelanic, Infrastructure and Operations, LAUSD
– Eddard Romero, Information Technology Division, LAUSD
– Alton Kizziah, previously EVP and general manager, MGT Technology Solutions
Youssef: This cyberattack at LA Unified School District on the evening of September 3, 2022, over the Labor Day holiday weekend, caused disruptions to some of our business systems. We hear more and more of cyberattacks in the news. Let’s start with Soheil.
Can you explain what happened with the cyberattack and what the team did to minimize the disruption to instruction?
Soheil: Late Saturday night, our team observed some unusual activity in our systems. The unusual activity was the ransomware encryption, which was a failed attack on our system. As soon as alerted, our security, system administrator, and network teams engaged to prevent further breach to the rest of our IT environment.
When a breach happens, typically there are multiple stages. One of those stages is assessing and managing activity laterally within the environment. We saw movement and the team started disabling access for the bad actors’ activity.
The best course of action is to protect the district’s systems information and data and prevent any access to the data center. We “dropped the curtain” and disabled any access to the data center. It’s not a simple network task. We have 3,000 servers, multiple applications, and thousands of users.
We notified law enforcement, including the FBI, DHS, and other agencies. They deployed resources to the site on Sept 3rd, 2022 and that’s where we started doing the prevention, mitigation, and attack forensics. It was a good start but ultimately, we needed to recover from it and be ready by Tuesday morning before school starts. Prevention activities were needed, including password resets across the environment. We did the password reset for a half-million student population, 70,000 employees, contractors, as well as retirees. It was a large effort to absorb that many password-change experiences.
Youssef: Your IT team worked tirelessly – a number of people spent their days and nights non-stop for two or three days without seeing their families. Much credit also goes to our teachers and administrators who helped students reset their passwords. Great job by the entire LA Unified community to mitigate this attack.
Youssef: There are words floating around like cyberattacks, cybersecurity breach, ransomware and sometimes people use these phrases interchangeably. Can you tell us a little bit about what some of these things mean?
Eddard: The first part is cyber. Cyber is anything in virtual spaces–our computer systems and the technology we use, whether they are application systems or database systems. Cyber frames all of our technologies. The next word is attack. When attacking your cyber systems, which are your digital systems – servers, network, databases, and devices, they can prevent you from using your systems.
Cybersecurity breaches are when a person(s) gain unlawful entry into your system. They cause a breach in your environment, cross your network security thresholds, and break into your systems. The data breach is when the attacker is taking your data. They’re able to either exfiltrate on to the Internet for a sale of the data, and ransomware is part of that attack. Once they’ve taken your data, they’re pushing it out to the Internet for sale. The ransom is post-encryption, which was a failed attempt at LAUSD. So, ransomware is when your data is encrypted by the attacker(s) and they’re using that to gain financial gain from the recovery. There are two things going on: 1) encryption of your data so you’re not able to access and use it; and 2) selling of your important data on the internet.
Youssef: In layman’s terms, cyberattack is like knocking on the front door to get in, and the threats are on the rise.
Jack: Unfortunately, yes, cybersecurity threats are increasing, not just for educational institutions but every organization is potentially vulnerable. It’s not a matter of if it happens to you, but when. Now it’s a matter of planning, prevention, and knowing what you’re going to do when it happens to your organization.
During the pandemic, educational organizations, and many other companies, extended access to our IT systems because it was necessary for virtual instruction very quickly. We find ourselves defending deeper into cyberspace than perhaps we did before the pandemic.
The bad guys figured out that education institutions hold a lot of confidential data. We are managing information for students. Attackers have learned how to take advantage of some of those vulnerabilities that are now present.
This is motivated mainly by money, and we did see that double extortion-type attack in LAUSD’s case. One, they attempt to steal your data and sell that for whatever it’s worth on the dark web. And two, they attempt to apply ransomware and encryption to lock us out, and they charge a ransom for the business to get back in.
Fortunately, in LAUSD’s case, the attackers weren’t successful on either one of those fronts. They ended up with a very small amount of data. And of course, we didn’t pay the ransom.
The good news is that there are things organizations can do to mitigate the impacts of these types of attacks. Preventive planning — knowing what you’re going to do when it occurs. Some simple things are multifactor authentication and cloud backups. These are things that people use in everyday life with their smartphones. But large enterprises and sometimes education institutions have been slow to adopt this in their systems. I would end by saying that we take the responsibility very seriously for managing and protecting the information of our students, our families, and our employees. We know how important those digital identities are for them going forward. We’re going to continue to invest in and strengthen those systems to keep up with the Internet.
Youssef: There have been lots of investments going on currently in LA to add more system security. Let’s bring in Alton for this question.
Alton came to LA Unified and helped us out. Alton, you have experience with Internet recovery as a result of these cyberattacks. Tell us a little bit about recovery operations and how to bounce back from these types of attacks?
Alton: It’s tough, especially when you’re as big as LA Unified. There are so many systems, so many people, so many things that can be impacted.
At MGT, we try to make it as easy as possible. There is a repeatable cycle of things you can do. Recovery is part of that process. To give you an example, part of the process is called detection. LA Unified detected that something was going wrong. There were criminals in their environment trying to cause damage.
As soon as you detect it, you move into a phase called containment. Password resets are part of the containment process. You are basically cutting off the criminals’ accounts and they cannot communicate with their accounts within your system.
You shut down access to the system which is a very complicated thing to do as you chase the bad guys out of your environment.
As attackers were stopped in the middle of an attack, they left artifacts, programs, and tools for investigation and future prevention.
This leads to the process called eradication. This is where we go in and destroy all of the attacker’s tools and make sure the account is safe and every account is one person to one account. They’re accountable for what they’re doing in the environment.
We’re eradicating any traces of this attack and then you actually get to the part of recovery.
Recovery is where you clean up the rest of this environment. We’re going to repair any damage that was done. For example, with encryption, you’re going to restore back-up. You’re going to bring up your other systems to full access. That’s the actual recovery.
The next steps in the cycle are the lessons learned.
You fill in the gaps or vulnerabilities that existed pre-breach that caused the environment to allow for the attack. Going forward, you consistently prepare and test your systems to build resiliency and further reduce the risk of an attack happening again. You gain maturity when you apply a cyclical process:
- You are prepared.
- You have detection capability.
- You have a containment and recovery plan.
- You have the ability to eradicate harmful elements in your IT environment.
- You quickly recover organizational and business services.
- And you take the lessons that you’ve learned and reapply them continuously.
One thing to understand with cybersecurity, there is no such thing as 100% security. Be prepared because you never know when something like this could happen to you, and it never really ends. Being prepared is a maturity process that never stops, it’s a constant, continuous improvement process.
Listen to the LAUSD Episode 1 Podcast, LAUSD Response and Recovery from Cyber Attack, January 13, 2023: https://podbay.fm/p/its4ed/e/1673638943
For more information about Managed Detection and Response (MDR) engineered by MGT Technology Solutions or MGT Education Solutions, contact us here.
