Google Cloud services continue to be heavily affected at one of its West European data centers, after a flood and subsequent fire caused widespread outages at the site.
The issues appear to have begun with an unexpected leak caused by a failure in the air conditioning systems at the Global Switch data center in Paris’ Clichy commune.
Battery components affected by the water sparked a fire, which caused a major outage across the entire French cloud region.
“Water intrusion in europe-west9-a has caused a multi-cluster failure and has led to an emergency shutdown of multiple zones,” read Google Cloud’s 20:51 PDT update on April 25.
“We expect general unavailability of the europe-west9 region. There is no current ETA for recovery of operations in the europe-west9 region at this time, but it is expected to be an extended outage.”
Over 100 services were affected in the initial outage, though Google Cloud Storage (GCS), Cloud Key Management Service (KMS), Cloud Identity and Access Management (IAM), and Google Kubernetes Engine (GKE) have now fully recovered across europe-west-9.
Google Cloud Console also experienced worldwide issues, and the firm’s service dashboard appeared to link this to the Paris incident.
RELATED RESOURCE
Quantifying the public vulnerability market
An analysis of vulnerability disclosures, impact severity, and product analysis
Management tasks for those operating outside of the French cloud region are now operating normally.
On the fibre optic forum La Fibre, an administrator laid out a timeline of events using data gathered by the French Network Operators Group (FRnOG), a voluntary community of mainly technical experts who work to collect and distribute information relating to French internet services.
It stated that a “cooling system water pump problem” led to water being unexpectedly released in the early hours of the morning on April 25.
This leaked into the battery room and caused a fire that emergency services had to let burn out for some time before more could be done.
Air conditioning systems came back online just before noon, but fire teams were unable to enter the battery room and instead cooled the walls in an attempt to stem the spread of the fire.
A Global Switch LinkedIn post from 13:00 local time stated that the “Fire Brigade has been in attendance and the fire is now contained”.
“The fire response systems in the building have performed as designed and no one has been injured,” it continued.
“A number of customers have been temporarily affected and our site team is working to restore services to those customers as soon as possible.
At 17:22 local time, officials at the data center had reportedly not shut down power as the air conditioning and critical server components were unaffected.
Google launched europe-west-9, its French region, in June 2022. It has not provided an ETA for when the region will operate normally again.
Some users on the YCombinator forum post about the incident noted that the region-wide outages implied that europe-west-9 a, b, and c are all located within the Clichy site to some extent.
“Ouch. Isn't part of separate zones being protected against something, say, like a terrorist attack or a natural disaster that can take down a whole datacenter?” one user noted.
At time of writing, Google Cloud’s latest update for the incident reports that services are still impacted in europe-west9-a but that europe-west9-b and europe-west9-c have largely recovered.
Asked for more information on when the outage will be fully resolved, Google referred ITPro to its service dashboard and the statement released by Global Switch.
-
Cleo attack victim list grows as Hertz confirms customer data stolenNews Hertz has confirmed it suffered a data breach as a result of the Cleo zero-day vulnerability in late 2024, with the car rental giant warning that customer data was stolen.
-
Lateral moves in tech: Why leaders should support employee mobilityIn-depth Encouraging staff to switch roles can have long-term benefits for skills in the tech sector
-
Google shakes off tariff concerns to push on with $75 billion AI spending plans – but analysts warn rising infrastructure costs will send cloud prices sky highNews Google CEO Sundar Pichai has confirmed the company will still spend $75 billion on building out data centers despite economic concerns in the wake of US tariffs.
-
Google Cloud announces major computing boost with Ironwood chip, new hypercomputer upgradesNews Google has announced a new chip intended for AI inference, which it says is more powerful and better for AI requirements but also far more energy efficient.
-
Microsoft’s Three Mile Island deal is a big step toward matching data center energy demands, but it's not alone — AWS, Oracle, and now Google are all hot for nuclear powerNews The Three Mile Island deal comes after concerns over Microsoft’s carbon emissions surge
-
Data center carbon emissions are set to skyrocket by 2030, with hyperscalers producing 2.5 billion tons of carbon – and power hungry generative AI is the culpritNews New research shows data center carbon emissions could reach the billions of tons by the end of the decade, prompting serious environmental concerns
-
AWS, Microsoft, and Google see massive cloud opportunities in Japan - here’s whyAnalysis AWS’ bid to expand infrastructure in Japan will see it it invest over $15 billion, but it’s not the only hyperscaler that has its eye on big gains in the country
