A firmware upgrade in one of Microsoft's datacentres led to an outage that left users of its Hotmail and Outlook webmail without access to the service for hours.
In a blog, Arthur de Haan, vice president for Windows Live test and service engineering, said that the outage happened after the firm attempted to update the firmware on a core part of datacentre infrastructure. The outage affected some people's access to a small part of the SkyDrive service, but primarily Hotmail.com and Outlook.com, according to Microsoft.
"This is an update that had been done successfully previously, but failed in this specific instance in an unexpected way," said de Haan. "This failure resulted in a rapid and substantial temperature spike in the datacenter."
De Haan added that the spike was significant enough before it was mitigated that it caused safeguards to come into place for a large number of servers in this part of the datacentre.
"These safeguards prevented access to mailboxes housed on these servers and also prevented any other pieces of our infrastructure to automatically failover and allow continued access," he said.
Once the safeguards kicked in on these systems, the team was alerted and began to work on restoring access. A mix of infrastructure software and human intervention was needed to bring the core infrastructure back online., according to de Haan.
"Requiring this kind of human intervention is not the norm for our services and added significant time to the restoration," said de Haan.
The majority of the impacted mailboxes were fully restored before midnight and the rest completed by 5:30 AM local time.
De Haan said that the company regretted the impact of the outage had on users.
"Now that we're through the resolution, we're also hard at work on ensuring this doesn't happen again," he said.
As reported by IT Pro, this is just the latest in a series of outages the Microsoft webmail service has suffered. At the end of February, a planned upgrade of Hotmail users to its new Outlook.com resulted in users unable to access emails when they try to sign in via its desktop and mobile sites.
Last September, Hotmail users reported problems stopping them from accessing emails.
