A firmware upgrade in one of Microsoft's datacentres led to an outage that left users of its Hotmail and Outlook webmail without access to the service for hours.
In a blog, Arthur de Haan, vice president for Windows Live test and service engineering, said that the outage happened after the firm attempted to update the firmware on a core part of datacentre infrastructure. The outage affected some people's access to a small part of the SkyDrive service, but primarily Hotmail.com and Outlook.com, according to Microsoft.
"This is an update that had been done successfully previously, but failed in this specific instance in an unexpected way," said de Haan. "This failure resulted in a rapid and substantial temperature spike in the datacenter."
De Haan added that the spike was significant enough before it was mitigated that it caused safeguards to come into place for a large number of servers in this part of the datacentre.
"These safeguards prevented access to mailboxes housed on these servers and also prevented any other pieces of our infrastructure to automatically failover and allow continued access," he said.
Once the safeguards kicked in on these systems, the team was alerted and began to work on restoring access. A mix of infrastructure software and human intervention was needed to bring the core infrastructure back online., according to de Haan.
"Requiring this kind of human intervention is not the norm for our services and added significant time to the restoration," said de Haan.
The majority of the impacted mailboxes were fully restored before midnight and the rest completed by 5:30 AM local time.
De Haan said that the company regretted the impact of the outage had on users.
"Now that we're through the resolution, we're also hard at work on ensuring this doesn't happen again," he said.
As reported by IT Pro, this is just the latest in a series of outages the Microsoft webmail service has suffered. At the end of February, a planned upgrade of Hotmail users to its new Outlook.com resulted in users unable to access emails when they try to sign in via its desktop and mobile sites.
Last September, Hotmail users reported problems stopping them from accessing emails.
-
Westcon-Comstor and Vectra AI launch brace of new channel initiativesNews Westcon-Comstor and Vectra AI have announced the launch of two new channel growth initiatives focused on the managed security service provider (MSSP) space and AWS Marketplace.
-
Third time lucky? Microsoft finally begins roll-out of controversial Recall featureNews The Windows Recall feature has been plagued by setbacks and backlash from security professionals
-
The IT Pro Podcast: Why is email still a thing?IT Pro Podcast Despite a proliferation of newer collaboration platforms, email stubbornly refuses to go away
-
Google targets phishing with full BIMI email logo authentication supportNews Gmail will tie logos to DMARC authentication
-
The most secure email services of 2023Best Email is not secure by design, but these email providers allow you to send emails with top-level security
-
SafeSend Email Security for Outlook review: A great solution for security-conscious SMBsReviews An invaluable last line of defence against Outlook email disasters that’s easy to deploy and manage
-
White House launches official investigation into use of personal email accountsNews Senate Intelligence Committee rebukes Kushner for omitting private account
-
How can IT leaders make the most of social media in business?In-depth Real world organisations explain how they avoid the pitfalls of social media
-
Outlook and Hotmail email accounts hit by spam attackNews Users were bombarded by spam emails apparently because Microsoft's spam filters weren't working properly
-
I don’t like Outlook.com. Where can I complain?Tutorials I don’t like Outlook.com. Where can I complain?
