Organizations are sometimes reluctant to share particulars about knowledge heart fires as a result of NDAs and PR considerations. As such, it’s typically tough sufficient to hint situations of information heart fires in any respect, until experiences to native hearth departments or newsrooms have been made, or prospects expertise important downtime and demand explanations.
This tendency to withhold detailed experiences about knowledge heart fires would possibly assist corporations defend their reputations, however it might additionally make it tougher for knowledge heart operators to acknowledge vulnerabilities, study from incidents, and implement measures to make sure the protection of their employees and prospects.
In response to an Uptime Institute blog post on knowledge heart hearth frequency – which was written after a catastrophic hearth destroyed OVHcloud’s knowledge heart in France in 2021 – there had been 11 experiences of information heart fires since its information started in 1994 – a median of 0.5 fires a 12 months.
Extra lately, the information heart requirements group said it recognized 14 “high-profile knowledge heart outages” brought on by hearth or hearth suppression techniques between 2020 and early 2023.
Whereas fires account for a comparatively small proportion of incidents that influence knowledge facilities, their potential ramifications shouldn’t be neglected. Along with the risks they pose to staff, knowledge heart fires can lead to prolonged downtime. This will probably price corporations thousands and thousands of {dollars} and result in critical inconveniences for purchasers, leading to an erosion of belief.
Whereas particulars could be scarce, DCN reviewed its archives to revisit main knowledge heart fires and energy outages over the previous decade. We additionally spoke with trade specialists to supply insights that knowledge heart employees can use to evaluate vulnerabilities and develop important security plans to guard in opposition to future incidents.
A Google knowledge heart in Iowa (Picture: Alamy)
1. Google Information Heart Hearth, Iowa
Simply earlier than midday on August eighth, 2022, a hearth broke out in a big Google knowledge heart in Council Bluffs, Iowa. The hearth, which was first reported as an “electrical incident,” was brought on by an arc flash that sparked an explosion in a substation close to the principle knowledge heart constructing.
Whereas not technically hearth, an arc flash is {an electrical} explosion that generates warmth upwards of 30,000 levels Fahrenheit, probably igniting supplies and inflicting fires. The explosion occurred whereas three employees have been accessing {an electrical} cupboard within the knowledge heart’s most important room.
The hearth injured three employees, who have been taken to a close-by hospital for remedy. It occurred on the identical day as outages of the corporate’s maps and search service, though Google stated the 2 incidents have been unrelated.
The Council Bluffs knowledge heart is one in all Google’s first amenities and one of many largest knowledge heart campuses on the earth.
2. Evocative Information Heart Hearth, New Jersey
Firefighters responded to a fire at the Evocative data center facility in Secaucus, New Jersey, on October 12, 2023. The hearth was contained to the uninterruptible energy provide (UPS) space and was rapidly extinguished. Regardless, it took its toll on the 105,000 sq.ft knowledge heart, which required a full energy shutdown. Fortuitously, nobody was injured.
Evocative, previously INAP, supplies web connectivity to many corporations within the New York metro space.
A hearth swept via OVHcloud’s SBG2 knowledge heart in Strasbourg, France, in 2021 (Picture: Alamy)
3. OVHcloud Information Heart Hearth, France
A hearth on March 10, 2021, destroyed one in all OVHcloud’s Strasbourg knowledge facilities and a part of a second one.
No OVH, firefighting, or native authorities providers workers members have been injured, the French cloud computing firm stated.
The hearth destroyed OVH’s SBG2 knowledge heart utterly and 4 rooms in SBG1, based on an incident report on the corporate’s web site. UPS was down within the SBG3 facility, and the remaining SBG4 knowledge heart had “no bodily influence.”
3. AT&T Information Heart Hearth, Texas
AT&T customers within the Dallas space misplaced web and cable providers after an “undetermined hearth” broke out in an AT&T knowledge heart in Richardson, Texas heart in Richardson, Texas, on October 15, 2018.
In response to experiences, the hearth started at a energy change and brought on prospects as much as 12 hours of downtime. The hearth didn’t trigger any accidents.
4. Fisher Plaza Information Heart, Washington
Round midday on June 2, 2009, {an electrical} hearth sparked inside Seattle’s Fisher Plaza knowledge heart, which housed the servers of widespread websites like Adhost.com, Microsoft’s Bing Journey, Verizon, and fee portal Authorize.internet.
All knowledge heart employees have been evacuated, and nobody was injured. Nevertheless, the incident resulted in $6.8 billion in damages and downtime.
An investigation performed by Energy Science Engineering, a Washington-based engineering firm, discovered that the hearth was seemingly brought on by insufficient insulation in {an electrical} duct connecting the constructing to the town energy grid.
This wasn’t the primary time Fisher Plaza had skilled energy outages and electrical fires. Only a 12 months earlier, a fireplace broke out in a garage-level electrical room, leading to the true property firm Redfin going offline for 5 hours.
5. SK Inc. C&C Hearth, South Korea
On October 15, 2021, a fireplace on the SK C&C knowledge heart in Pangyo, South Korea, affected two main tech corporations, Kakao Company and Naver Company.
Whereas Naver rapidly restored its servers, Kakao confronted extended outages, disrupting its messaging platforms, fee apps, and rideshare providers for hours.
Regardless of having a catastrophe restoration protocol, Kakao’s plan didn’t account for the facility outage throughout the hearth, delaying its restoration efforts. In response, Kakao established a ‘recurrence prevention committee’ to forestall comparable incidents sooner or later.
A CO2 hearth extinguishing system for a knowledge heart (Picture: Alamy)
Constructing Resilience: Stopping Information Heart Fires
The above examples are reminders that knowledge heart fires can get away unexpectedly and have quite a lot of causes, together with arc flashes, defective infrastructure, {hardware} failures, and human error.
Whereas there are additionally unexpected threats like pure disasters, there are a lot of situations wherein energy failures and electrical fires may need been prevented. Making certain infrastructure security is vital to mitigating the danger of information heart fires.
Sadly, bettering the protection of important infrastructure typically comes with a big price ticket. In response to Chris Brown, Uptime Institute’s chief technical operator, infrastructure security is usually compromised as a result of funds constraints.
Learn extra of the newest knowledge heart safety and danger administration information
“Restricted funds accessible and the necessity to put money into important electrical and mechanical techniques have resulted in funds needing to be pulled from different areas,” Brown stated. “Moreover, in some areas, repurposing present buildings is important as a result of an absence of house and the prevailing constructing buildings could not have been initially designed to the degrees most would take into account crucial for a knowledge heart.”
He added: “Extra issues and investments are wanted within the precise construction but additionally the compartmentalization of complementary techniques to make sure that hearth will not be allowed to propagate and likewise will not be allowed to close down your complete knowledge heart.”
Combating Fires By Compliance and Regulation
Over latest years, there was a stronger push to create stronger compliance and regulation requirements for knowledge heart infrastructure throughout the US and globally.
In December 2023, Colorado handed the Federal Information Heart Enhancement Act, which outlined minimal requirements of infrastructure resiliency within the occasion of cyber and bodily assaults, in addition to pure disasters.
In Could 2024, Maryland handed the Critical Infrastructure Streamlining Act. The UK handed an identical invoice, in January 2024, which launched new rules round reporting knowledge heart incidents and strengthened security and infrastructure necessities.
Regulation and security measures to extend infrastructure resiliency are key to mitigating knowledge heart disasters, however it’s additionally essential to keep in mind that some incidents are unavoidable. This makes it crucial to have emergency protocols that facilitate a protected and environment friendly restoration.
How rapidly knowledge facilities and consumer service could be restored typically is determined by the protocols and catastrophe restoration plans of the operator. Information heart catastrophe restoration plans embrace catastrophe restoration groups, danger evaluation, redundant infrastructure, and backup energy turbines, which defend knowledge and scale back downtime.
Key components of a knowledge heart catastrophe restoration plan. The data on this picture is reproduced with variety permission from SANS Institute.
Staying on the Ball
Catastrophe restoration plans are usually not simply paperwork and protocols. Additionally they contain robust partnerships between knowledge heart operators and their prospects. Krista Shepard, a spokesperson for Cologix, a knowledge heart developer, stated catastrophe restoration plans are usually not ‘set-it-and-forget-it’ measures, however dwelling paperwork that have to be adaptable to match a continuously evolving panorama.
“The flexibility to swiftly restore operations within the occasion of a catastrophic occasion requires knowledge backups in safe offsite places,” Shepard instructed DCN. “It additionally requires rigorous testing, drilling, and proactive collaboration to make sure the catastrophe restoration plan is carried out as seamlessly as doable to spare useful time and assets within the occasion of a catastrophic loss.
She added: “It’s essential to periodically replace and refine catastrophe restoration plans as what you are promoting and expertise evolve, and to adapt to altering environmental and climate situations.”
