Google has been forced to roll back an update after an event caused a major outage for its cloud customers using its load balancers.
The company's network has been suffering problems from Wednesday morning, when at around 1am Pacific time (9am GMT), the service went down. The problems continued throughout the day, with Google providing regular updates for all to see.
At first, the company explained it was looking into the problem, and suggested it could be related to an infrastructure component that was causing the service to fall over. It thought it had fixed the issue, but the problems persisted with Googe announcing: "Our previous actions did not resolve the issue. We are pursuing alternative solutions".
An hour and a half later, the company said it had identified what caused the outage and had made good progress to fix it. However, the solution involved rolling back a configuration change that happened at the time its systems went down.
When fixed, Google then had to tidy up after itself, ensuring it wouldn't happen again. Google explained at this point that although only a limited number of customers were affected, any instances that were running when the outage occurred wouldn't work. It advised customers to create a targetpool and move the affected instances to the targetpool.
"Wait for the VMs to start working in their existing load balancer configuration. Delete the new TargetPool. DO NOT delete the existing load balancer config, including the old target pool. It is not necessary to create a new ForwardingRule," it instructed.
Google's status monitor says everything's working fine now, although it took a significant amount of time for the fix to be implemented, with some instances not being fixed until Thursday morning.
Image credit: Bigstock
