Unplanned downtime costs the world's largest 2,000 companies a combined $400 billion per year, working out at $200 million per company on average, according to a new report by Splunk.
The report found that while many of the costs associated with downtime are well-known, such as revenue loss or regulatory fines, other ‘hidden’ costs include slower time to market or worsened brand reputations.
The Hidden Cost of Downtime study took in responses from 2,000 executives at the largest firms globally and was run in conjunction with Oxford Economics. The report assessed known costs of downtime and analyzed their most common root cause.
The largest direct cost of downtime in the report comes from lost revenue, with $49 million missed annually among the top 2000 companies and CFO respondents indicating that revenue can take up to 75 days to recover following an outage.
But hidden costs also bite in these scenarios, with stock prices found to dip as much as 9% following a major outage and 64% of respondents reporting “stagnant” developer productivity in the wake of a downtime incident. On a broader scale, across 95% of organizations innovation was found to slow in such an event.
Loss of customers following an incident can compound the financial and reputational impact of downtime, with 29% of respondents indicating they lost customers directly because of their outage.
Gary Steele, president of Go-to-Market at Cisco and GM at Splunk, described the results as “pretty startling” and called the visible costs of downtime as “the tip of the iceberg” in the opening keynote at Splunk .conf24.
Steele pointed to the importance of a strong cyber security and observability strategy to prevent downtime.
The report found human error to be the most common cause of downtime, with most organizations taking 17-18 hours on average to become aware of these incidents.
Errors in both cyber security and ITOps topped the respondents’ list for causes of downtime, with attacks with malware and phishing further down the list at the number four and six spots respectively.
Of this human error, 56% came in the form of poor cyber security and 44% in the form of application or infrastructure mistakes.
“Lot of times misconfigurations could actually facilitate cyber attacks as well, so if you have identity and access management (IAM) not configured properly, if you don't have multi-factor authentication (MFA) configured properly,” Audra Streetman, security strategist at Splunk, told ITPro.
Resting on its observability platform and growing security capabilities, Splunk said it aims to empower companies to avoid downtime through a better approach to data management and a unified approach to security observability.
The companies dodging downtime
‘Resilience leaders’, defined as companies that recover from downtime fastest, spent $12 million more on cyber security and $2.4 million more on observability per year.
These organizations averaged 28% faster time to recover from incidents with their applications or infrastructure, and as much as 84% faster to recover from ransomware attacks.
Morgan McLean, senior director of product management and OpenTelemetry co-founder, told ITPro that investment in observability can reduce downtime problems “pretty dramatically” and noted that good observability can also help firms get back on their feet sooner.
However, McLean warned against placing too much emphasis on simply onboarding an observability report to reduce downtime.
“It’s not just buying a tool, buying the tool is the prerequisite for success,” McLean said, noting that companies “you do also have to ensure that your engineering and development and release processes are also sensible.
McLean told ITPro that this is a common problem at companies, giving the example of working at Microsoft and being one of the few employees at the company who understood the full layout of its microservices architecture.
The authors also noted resilience leaders are already investing more in generative AI than their counterparts, adopting tools at four-times the rate of other respondents. Over half (51%) of resilience leaders were expanding their use of discrete generative AI tools at the time of their response, compared to just 10% of non-leaders.
In conversation with ITPro, James Hodge, GVP and chief strategic advisor EMEA at Splunk, stated that AI can act as an accelerant for a company’s existing strategy for readiness.
“What I think AI is going to do is be more efficient doing what you do,” Hodge said, adding that Splunk’s AI strategy is focused around solving specific problems that commonly plague its customers.
Hodge told ITPro that greater use of AI in observability can help simplify complex IT monitoring, such as for entity monitoring or identifying specific data entities more intelligently, all of which can help companies mitigate against downtime by identifying areas that are poorly optimized or misconfigured.
He also noted that of the $200 million downtime cost, less of this was linked directly to extortion ($8 million) than regulatory fines ($22 million) and that AI could play a role in helping firms meet their regulatory requirements.
