The Department of Homeland Security’s Cyber Safety Review Board slams Microsoft for 2023 Exchange Intrusion that should ‘never have happened’.

On Tuesday, April 2 2023, the The Department of Homeland Security’s Cyber Safey Review Board issued a report concerning the Microsoft Online Exchange Incident that occurred in the Summer of 2023. In this report, the CSRB outines the events that lead up to the incident, failures that facilitated the incident, responses by Microsoft and affected victims, as well as the CSRB’s recommendations to Microsoft and the Cloud Service Provider community to prevent future incidents.

The DHS charged the CSRB with investigating an intrusion by the threat actor group Storm-0558 into Microsoft’s network that occurred in the Summer of 2023. The threat group known as Storm-0558 has been active since 2000, according to the report, and has been linked to the People’s Republic of China (PRC). This Nation State threat actor was also responsible for a previous intrusion into Microsoft’s systems in 2021, which the report goes on to note as relevant to the 2023 intrusion.

The report by the Cyber Safety Board pulled no punches to highlight multiple failures by Microsoft on many levels to prevent the incident. After listing these failures, the report continues:

“Individually, any one of the failings described above might be understandable. Taken together, they point to a failure of Microsoft’s organizational controls and governance, and of its corporate culture around security.”

The CSRB listed seven key failures in Microsoft’s security culture. The following 5 items were directly related to the 2023 incident:

1. The cascade of Microsoft’s avoidable errors that allowed this intrusion to succeed.
2. Microsoft’s failure to detect the compromise of its cryptographic crown jewels on its own, relying instead on a customer to reach out to identify anomalies the customer had observed
3. The Board’s assessment of security practices at other CSPs, which maintained security controls that Microsoft did not.
4. Microsoft’s failure to detect a compromise of an employee’s laptop from a recently acquired company prior to allowing it to connect to Microsoft’s corporate network in 2021.
5. Microsoft’s decision not to correct, in a timely manner, its inaccurate public statements about this incident, including a corporate statement that Microsoft believed it had determined the likely root cause of the intrusion, when in fact, it still has not; even though Microsoft acknowledged to the Board in November 2023 that its September 6, 2023 blog post about the root cause was inaccurate, it did not update that post until March 12, 2024, as the Board was concluding its review and only after the Board’s repeated questioning about Microsoft’s plans to issue a correction.

The CSRB goes on to describe how the the attack by Storm-0558 was carried out and the failings that enabled such an attack. The Storm-0558 somehow acquired a 2016 Consumer Microsoft Services Account (MSA) key that was supposed to have been retired. Microsoft normally rotates such keys to render them invalid and unusable. However, after a systems issue in 2021, Microsoft paused this rotation. Through a series of security failings, the 2016 Consumer MSA key remained active and valid. Using this stolen 2016 key, the attackers were able to forge a primary token to gain access to Microsoft’s authentication libraries. Once in Microsoft’s authentication system, they were able to issue themselves substrate tokens that allowed them to access customer tenants and read use emails directly. This attack chain is shown in the figure below, provided by the CSRB report:

Storm-0558 Attack Chain - Source - CSRB Report

The attackers leveraged their access to forge tokens to access 22 enterprise organizations and 503 personal user accounts. However, the compromise of these organizations and accounts was first noticed not by Microsoft, but rather the State department, who first noticed anomalous activity in their exchange systems on June 15 2023. On June 16 2023, the State department notified Microsoft, who then launched an internal investigation to determine the depth, severity, and remediation of the intrusion.

To date, Microsoft has been unable to say with certainty how Storm-0558 acquired the 2016 Consumer MSA key used to access Microsoft’s systems. The prevailing theory, proposed by Microsoft in a blog post and noted in the CSRB report, is that the 2016 MSA key was inadvertantly included in a crash dump due to a race condition. This crash dump was then moved from the secure authentication enviroment to a debugging environment.

In 2021, Microsoft systems were also compromised by Storm-0558 when it infected the laptop of an engineer who was acquired from a company called Affirmed Networks. Once the engineer was onboarded, his infected laptop was allowed to access Microsoft’s internal network, thereby allowing the intrusion of Storm-0558 into Microsoft’s internal network. Microsoft believes that this intrusion is related to the 2023 Online Exchange compromise. As CSRB notes, Microsoft hypotheses that the engineer had access to the debugging environment that held the crash dump mistakenly containing the 2016 MSA key. The threat actors were then able to also access this system once the engineer’s infected laptop connected with Microsoft’s internal network. After exhilarating and analysing the data obtained from the 2021 intrusion, Microsoft posits that Storm-0558 discovered the 2016 Consumer MSA key and utilized this key to begin the 2023 Online Exchange Intrusion. The timeline of these events is shown in the figure below from the CSRB report:

Timeline of the 2023 Online Exchange Intrusion - Source: CSRB Report

After determining the depth of the intrusion, Microsoft closed the door to Storm-0558 by invalidating the stolen 2016 Consumer MSA key. The threat actors subsequently lost access to the compromised accounts and Microsoft continued its efforts to remediate the security flaws that allowed the attack to occur and to notify victims.

The CSRB concludes their report by making several recommendations to Microsoft, urging them to fundamentally change their stance and attitude towards security and hold those in charge directly accountable. The Board encouraged Microsoft to get back to their roots with a security first mindset, even quoting Microsoft’s founder, Bill Gates:

So now, when we face a choice between adding features and resolving security issues, we need to choose security. —Bill Gates

The recommendations made to Microsoft included a change in the company’s security culture, prioritizing security over developmental features, encouraging senior leadership to make security a priority, and enhanced granular logging tools.

The Board also made recommendations to the Cloud Service Provider (CSP) community as a whole, recommending the following security practices:

• Review and revise logging and forensics capabilities of systems that could enable environment-level compromise.
• Engineer digital identity and credential systems in such a way that reduces the risk of complete system-level compromise.
• Robust compromise assessment and remediation processes.
• Define and adopt minimum standard audit logging.
• Implement emerging identity standards such as OAuth 2 and OpenID.
• Report all incidents involving nation-state threat actors to the U.S. government.
• Greater transparency to government agencies, customers, and stakeholders regarding unknowns.
• Prompt correction of inaccuracies in public statements.
• A commitment to disclosure of vulnerabilities via the CVE program.

The CSRB report also charged government organizations with additional reporting and documentation. The Board recommended that CISA implement an annual validation of the recommended security practices for CSPs providing services to the U.S. government. Recommendations were also made to NIST, the RMF, and JTF to update the Special Publication catalog to better account for risks to cloud-based identity systems.

Another interesting recommendation from the report included the development of ‘targeted “amber alert” style victim notification’ mechanisms that would allow CSP’s to communicate information about suspected compromises to victim account holders. The report concluded with several recommendations to the U.S. government itself, including tightening logging standards and developing better communication and reporting standards with CSP’s

While the Cyber Safety Review Board’s report criticized Microsoft strongly for their lack of security and various failings that allowed a nation-state threat actor to access their systems and compromise the email accounts of both government representatives and personal accounts, the report also made recommendations much needed changes the way Cloud Service Providers view security and how they respond to security incidents. Without massive changes to the security culture at Microsoft and the overall CSP community, sophisticated attacks such as the 2023 Microsoft Online Exchange Intrusion will continue to occur.