Server maintenance blunder shut down 14 Toyota manufacturing plants, firm admits

Production was halted at more than a dozen Toyota manufacturing sites in Japan due to a server that ran out of disk space, the company has revealed.

Routine server maintenance on systems responsible for parts orders was conducted by staff last week, Toyota revealed.

During the maintenance process, accumulated data in a company database was routinely deleted and organized, Toyota said. However, “insufficient disk space” led to an error that caused its systems to shut down.

This incident had a knock-on effect as backup servers were running on the same system, preventing staff from performing a switchover and taking down systems completely.

“Since these servers were running on the same system, a similar failure occurred in the backup function, and a switchover could not be made. This led to the suspension of domestic plant operations,” the car manufacturer said.

The outage lasted for over 24 hours in total, and systems were restored on 29 August after data was transferred to a server with larger capacity.

Toyota reiterated that the server outages were the “true cause” of operational disruption, following speculation that the incident had been due to a cyber-attack.

“We would also like to reaffirm that the system malfunction was not caused by a cyber attack, and apologize to all parties for any concern this may have caused,” it said.

Best practice was not followed

Manlio De Benedetto, director of systems engineering at Cohesity, told ITPro that the incident underlines the sometimes precarious nature of server maintenance and the necessity of a heightened focus on data resilience among IT practitioners.

“It looks like Toyota didn’t see this coming, and if they were monitoring it, then their team has not reacted to the critical issues in the right way,” he said. “In simple terms, the best practices are to know what you’re backing up, where from and where to.”

De Benedetto emphasized that organizations should employ stringent monitoring standards and robust countermeasures to ensure that maintenance processes don’t result in down-time and disruption for customers.

“Companies need to understand exactly which IT processes are the most important in the company and how they are related,” he added.

“We advise our customers to prioritize the systems in resilience categories. The highest category reflects the most important data, systems and processes, the loss or failure of which would cause the greatest economic damage.”

In this instance, he noted that the affected Toyota servers would be classified in the “highest resilience category” due to the risk that failure would bring production to a halt and incur significant costs.

In response to the incident, Toyota said it will conduct a thorough review of its maintenance procedures and “strengthen our efforts to prevent a recurrence”.  

What’s going on in Toyota’s tech teams?

This incident marks the third serious IT-related issue for Toyota in the space of three months. 

In May, the car manufacturer revealed that data belonging to more than two million customers in Japan was left “publicly available” for a decade due to a cloud configuration error.

Around 2.15 million customers are thought to have been affected by the leak, which lasted from November 2013 to April this year. 

In a statement at the time, Toyota said that the leak was caused by a “misconfigured setting” in its cloud environment. This incident was attributed to human error. 

Just weeks later, the firm warned customers in Asia and Oceania that their data may have been “publicly accessible” in a separate incident dating from October 2016 to May 2023. 

Exposed customer data included names, addresses, phone numbers, email addresses, vehicle identification numbers, and registration details. 

Once again, Toyota attributed the leak to a cloud misconfiguration as a result of human error.