Last fall, Cloudflare announced it mitigated an attempted cyberattack stemming from the infamous Okta breach. But the cybersecurity vendor revealed on Thursday that this was not the case.
Cloudflare disclosed in a blog post that it had been breached by an unnamed nation-state threat actor using an access token and three service account credentials that were stolen during the Okta breach in October. Cloudflare initially detected the attacker in its self-hosted Atlassian server on Thanksgiving Day and began investigating the breach, with later assistance from CrowdStrike.
According to the blog post, the threat actor accessed Cloudflare’s internal wiki on Atlassian Confluence, its bug database on Atlassian Jira and its source code management system on Atlassian Bitbucket. Cloudflare said the operational impact of the breach was “extremely limited” and that no customer data or systems were impacted.
“Because of our access controls, firewall rules, and use of hard security keys enforced using our own Zero Trust tools, the threat actor’s ability to move laterally was limited. No services were implicated, and no changes were made to our global network systems or configuration,” Cloudflare CEO Matthew Prince, CTO John Graham-Cumming and CISO Grant Bourzikas wrote in the blog post.
The attack began on Oct. 18 and stemmed from the most recent Okta breach, in which a threat actor used stolen credentials to access a customer support case management system that contained HTTP Archive files. The threat actor used session cookies contained in those files to impersonate valid users at several Okta customers, including Cloudflare, BeyondTrust and 1Password.
Cloudflare initially believed it had prevented the attempted attack. In a blog post on Oct. 20 titled “How Cloudflare mitigated yet another Okta compromise,” the company said the threat actor used a stolen authentication token to gain access to its Okta instance. Cloudflare said its Security Incident Response Team detected the intrusion and contained the attacker.
But in Thursday’s disclosure, Cloudflare executives admitted the threat actor had moved beyond the Okta instance and gained access to its self-hosted Atlassian server.
“We’ve written about this before but, in summary, we were (for the second time) the victim of a compromise of Okta’s systems which resulted in a threat actor gaining access to a set of credentials. These credentials were meant to all be rotated,” Cloudflare executives wrote. “Unfortunately, we failed to rotate one service token and three service accounts (out of thousands) of credentials that were leaked during the Okta compromise.”
Cloudflare said the service token and service account credentials were not rotated because it was mistakenly believed they were unused. It’s unclear why they were believed to be unused.
TechTarget Editorial contacted Cloudflare for further comment, but the company had not responded at press time.
Attack timeline and “Code Red” efforts
Cloudflare said the service token was for Moveworks, an AI startup, that provided remote access to the Atlassian server. The first set of credentials were for Smartsheet, an SaaS collaboration application that had administrative access to Cloudflare’s Jira instance. The second was a Bitbucket service account that granted access to the company’s source code management system. The third was for an AWS environment used for the Cloudflare Apps marketplace.
Cloudflare emphasized that Moveworks, Smartsheet and AWS were not at fault for the breach.
After obtaining the token and service credentials on Oct. 18, the threat actor appeared to pause activity before performing reconnaissance on Cloudflare systems on Nov. 14. The threat actor gained entry to the Atlassian server the following day and began accessing a small number of Jira tickets and wiki pages.
“The threat actor accessed Jira tickets about vulnerability management, secret rotation, MFA bypass, network access, and even our response to the Okta incident itself,” Prince, Graham-Cumming and Bourzikas wrote. “The wiki searches and pages accessed suggest the threat actor was very interested in all aspects of access to our systems: password resets, remote access, configuration, our use of Salt, but they did not target customer data or customer configurations.”
The threat actor used the Smartsheet admin account to create a new Atlassian user account to maintain persistent access to the server in case the Smartsheet account was disabled. After a brief break, the threat actor returned to the Atlassian server on Nov. 22 and installed Sliver, an open-source red team framework that’s also used by attackers for command and control infrastructure.
The threat actor tried to move laterally outside of the Atlassian server and attempted to access a non-production console server in Cloudflare’s data center in São Paulo, Brazil, but those efforts failed.
However, the threat actor was able to access 120 code repositories out of a total of 11,904 repositories. 76 of those repositories were downloaded via the Atlassian Bitbucket git archive feature to the Atlassian server. Cloudflare said that although it could not confirm that the 76 repositories were exfiltrated, the company made the decision to treat them as such.
The security team detected the malicious activity the following day on Thanksgiving when the threat actor added the Smartsheet service account to an administrator group, which triggered an automated alert. Cloudflare’s security operations center began investigating and quickly disabled the Smartsheet account before later discovering and deleting the attacker-controller Atlassian account as well.
The following day, Cloudflare removed the Sliver deployment and eliminated all the threat actor’s access. The company brought in CrowdStrike on Nov. 26 to assist with incident response.
“Then, from November 27, we redirected the efforts of a large part of the Cloudflare technical staff (inside and outside the security team) to work on a single project dubbed ‘Code Red.’ The focus was strengthening, validating, and remediating any control in our environment to ensure we are secure against future intrusion and to validate that the threat actor could not gain access to our environment,” the executives wrote.
The Code Red effort included the rotation of every production credential, which included more than 5,000 individual credentials, as well as the physical segmentation of the company’s test and staging systems. Cloudflare also reimaged and rebooted every machine in its global network and conducted forensic examinations on 4,893 systems.
One notable effort under Code Red involved Cloudflare’s São Paulo data center, which was not yet in production. Even though the threat actor failed to access the console server, Cloudflare returned all equipment in the data center to its manufacturer. “The manufacturers’ forensic teams examined all of our systems to ensure that no access or persistence was gained. Nothing was found, but we replaced the hardware anyway,” Prince, Graham-Cumming and Bourzikas wrote.
In addition, engineering teams examined the 76 source code repositories, which “almost all related to how backups work, how the global network is configured and managed, how identity works at Cloudflare, remote access, and our use of Terraform and Kubernetes.” The engineering teams discovered a small number of repositories containing encrypted secrets, which Cloudflare rotated immediately.
The company’s Code Red effort ended on Jan. 5, and CrowdStrike completed its investigation on Jan. 31.
“We are confident that between our investigation and CrowdStrike’s, we fully understand the threat actor’s actions and that they were limited to the systems on which we saw their activity,” Prince, Graham-Cumming and Bourzikas wrote.
Cloudflare’s breach disclosure is the latest in a series of incidents tied to Okta. Prior to the breach of its customer support case management system, the identity and access management provider in August disclosed that several customers had been compromised via social engineering attacks that tricked victim organizations into resetting MFA factors for privileged users. Okta later confirmed that among the affected customers were Caesars Entertainment and MGM Resorts, which were hit by ransomware attacks.
In January 2022, Okta disclosed it was breached by the Lapsus$ hacking group, which is known for data extortion attacks against large enterprises. Okta revealed the attackers compromised a third-party customer support agent at Sitel and used the agent’s account to gain access to internal Okta sites and service records for about 2.5% of the customer base.
An Okta spokesperson sent the following statement to TechTarget Editorial: “This is not a new incident or disclosure on the part of Okta. On October 19th, we notified customers, shared guidance to rotate credentials, and provided indicators of compromise (IoCs) related to the October security incident. We can’t comment on our customers’ security remediations.”
Rob Wright is a longtime technology reporter who lives in the Boston area.
