By now, everyone should be familiar with the sad story of Equifax's 2017 data breach. The credit-checking agency was rocked by scandal last year when it was discovered that a flaw the Apache Struts framework allowed hackers to gain access to its systems and make off with a trove of more than 145 million people's personal information.
This breach was notable in its size, but what really made it stand out was the quality of the data hackers managed to harvest. In addition to full names, dates of birth and addresses, some victims also had drivers' license numbers, credit card information and social security numbers stolen.
The breach caused international outcry, sparked the resignation of a number of key executives, spurred a number of lawsuits against the company, and prompted a large scale investigation by the US government.
With a damning title of 'How Equifax neglected cybersecurity and suffered a devastating data breach', a Senate report released last week shed light on the circumstances leading up to the breach, including details of Equifax's IT infrastructure and a number of deficiencies that were contributing factors to its eventual breach.
An embarrassment of patches
Equifax's security issues could largely be boiled down to three major failings: a sprawling and disjointed IT infrastructure, a slapdash approach to patch management, and ineffective communication between and within departments.
Equifax had long operated without a coherent corporate policy for governing the application of security patches, something that was actually addressed in 2015 by Susan Mauldin, the company's then-CSO. In preparation of her new security policy, a company-wide audit was conducted, the shocking results of which were detailed in the Senate report.
It found that a staggering 8,500-plus patches were overdue to be applied to both internal and external-facing systems. Of these, almost 90% were designed to address critical or high-severity vulnerabilities. Three-quarters of the external vulnerabilities were more than three months old, and 7% had been known about for more than a year.
The audit also revealed that patch and configuration controls were inadequate to deliver regular and timely updates, and that the company's IT team was also not proactively patching systems when updates became available, waiting instead to receive specific instruction from Equifax's Global Threats and Vulnerability Management (GTVM) division.
It transpires that the five groups within Equifax responsible for patch management operated on an 'honour system'. In practice, this meant that when the GTVM team became aware of a vulnerability, they would scan for its presence on Equifax's network. If any instances were detected, the relevant team would be notified and requested to apply a patch. If no instances were detected, no patch would be applied. However, even in those cases where a patch was mandated by the GTVM team, there were no efforts to track whether those patches had been successfully applied.
Equifax's former Countermeasures Manager was quoted as saying to Senate Commitee: "This is not an advisable approach to patching".
A February 2017 proposal, dubbed the 'Green Belt project', aimed to improve patch management. It was never officially implemented, but the security analyst behind it also observed that while the process of scanning for vulnerabilities was done on a global scale, the actual application of patches was managed at a regional level. The reason for this, according to Webb, was because the underlying infrastructure was not consistent across Equifax's global offices. He said that this discrepancy was "not important".
The 2015 audit that revealed these alarming shortcomings also made a number of key recommendations, one of which was the implementation of automated tools to manage the configuration and deployment of patches, as well as a centralised system to track their successful application. It also recommended that patches were applied proactively as soon as they became available, and that high-risk and business-critical systems were given greater priority.
These recommendations were made in October 2015, and in the 18 months between then and the eventual breach in May 2017, not one of the recommendations had been fully implemented.
What is perhaps most damning of all is that the audit found Equifax did not have a complete overview of its IT infrastructure. Recommendations that Equifax should "ensure a current and accurate accounting of all IT assets is available at all times", also appear to have been largely ignored as a full inventory of Equifax's IT was still not available by March 2017 (and efforts to conduct one are still ongoing). It was this lack of an inventory that would prove to be the company's undoing.
A patchy server
//
When a critical flaw was discovered in the Apache Struts framework on 8 March 2017, Equifax's GTVM team quickly issued an email the next day to more than 400 employees urging them to patch it immediately, pointing out its high severity and that it was currently being exploited. The team also scanned for the vulnerability repeatedly.
An employee in Spain did contact the security team to check that the two versions of Struts his team were using were not vulnerable to the exploit, and while they were subject to numerous years-old unpatched vulnerabilities, they were immune to the flaw in question.
One application that was vulnerable, however, was Equifax's online dispute portal, a critical customer-facing system. Earlier calls by the Green Belt Project for more sophisticated scanning tools had largely been ignored, and as a result the vulnerability was missed by GTVM scans. What confounded the issue further was that, because of a lack of company records, the GTVM team was unaware that the online portal was using a vulnerable version of Struts - likewise the lead developer on the portal team was unaware that his particular version was vulnerable.
Some of the blame for the eventual intrusion could potentially be laid at the feet of this one developer - after all, there's an argument to be made that one should keep up to date with the security statuses of the systems you use and maintain - but the greater failing is that of Equifax's corporate culture.
"What we've got here is a failure to communicate", to quote the Captain from Cool Hand Luke.
The GTVM's email alert regarding the vulnerability was sent out to more than 400 people, but this lead developer was not one of them - likely because Equifax's incomplete records did not include him as an application owner. His senior manager, however, did receive the alert but failed to pass it on to anyone on the relevant team.
The flaw was mentioned in two of the monthly vulnerability meetings run by the GTVM team (which the aforementioned senior manager would have seen), but attendance to these meetings was optional, including for security managers or application owners, and there were no reliable records of who actually attended.
What we do know is that "the CSO, the Senior Vice President of Product Security, the Vice President of the CTC, the Director of GTVM, and the Manager of Countermeasures did not regularly attend the monthly GTVM meetings", according to the Senate report, and that the CSO "never" learned of the security warnings issued by US-CERT.
By this point, the damage had been done; once they had access to the dispute portal, there were no security controls in place to stop the hackers from laterally traversing the network until they discovered databases containing the personal data of hundreds of millions of Equifax's customers. The credentials they used to access these databases were stored on an internal file shared between staff.
The blame game
//
Executives and managers responsible for safeguarding this information have argued that they did everything in their power to keep hackers at bay prior to the breach, and fend them off during it. The SVP of product security has said that he didn't see anything wrong with the company's security policies both during and prior to the breach, and the former countermeasures manager said that he did not think his team could have done anything differently.
The former vice president of its Cyber Threat Center also told the Senate subcommittee that one of the reasons Equifax struggled to maintain an accurate list of its IT assets was that it was difficult and time-consuming to integrate the tech stacks of the numerous companies snapped up by Equifax as part of its growth strategy.
Equifax's new CEO Mark Begor has jumped to the company's defence as well, assuring the Senate Subcommittee that the company did, in fact, take security seriously.
"The fact that Equifax did not have an impenetrable information security program and suffered a breach does not mean that the company failed to take cybersecurity seriously," he said. "Before the cyberattack, I understand that the [the company's] security program was well-funded and staffed, based on a robust set of policies, standards, and procedures, and supported by general and specialized training."
While the attacker that hit Equifax may have been sophisticated, the fact is that they needn't have been. The Senate report confirms that "the tools to exploit the March 2017 Apache Struts vulnerability were publicly available and easy to use," and the further actions the hackers took to access sensitive data once on the network were not especially complex by cybercrime standards.
Furthermore, the actions of Equifax's security teams both before and after the breach betray either inadvertent or willful ignorance of security best practices. Recommendations from the 2015 audit were never implemented, nor was a follow-up audit conducted.
Perhaps most damningly, the company waited six weeks after it learned of the breach - which affected more than 145 million people - to notify the public. In total, the hackers had access to their data for more than three and a half months before the public had any idea. It was during this delay that the former CIO of Equifax, Jun Ying, sold almost $1 million in company shares. He has subsequently plead guilty to insider trading.
Based on this report, Equifax's IT organisation appears to have fallen victim to the classic case of overly-cumbersome bureaucracy obfuscating a company's data and processes. Equifax had five different teams responsible for patch management and a system in which the successful application of critical fixes was taken on faith. Vital security updates were viewed by the CIO as a "lower level responsibility that was six levels down" from him, and top security staff neglected to attend meetings about potentially existential cyber threats.
What can businesses learn from Equifax?
Equifax's approach to security provides a reasonably complete example of what not to do, but there were three particular failings that proved to be fatal for the company.
Overly complex IT structure
One of the main reasons that Equifax's patch management was such a convoluted process was that its IT organisation was vast and sprawling, with a large number of siloed departments that each had separate but adjacent responsibilities. Reducing the complexity of your IT organisation ensures that communication is easier, and simplifies the allocation of vital tasks like patching.
For example, DevSecOps models work on the basis of integrating multiple functions into a single team, allowing that one team to successfully support an application over its entire lifecycle. This particular model may not be right for everyone, but consolidating your IT teams is a must for any business that wants to maintain adequate visibility of its IT.
Lack of regular audits
Equifax was arguably brought down due to poor visibility; the Apache flaw was left unpatched because incomplete lists of IT assets hid the vulnerability from the security team. If the company had invested in regularly auditing its IT estate, the security department could have patched it earlier, potentially avoiding the breach.
Not only does keeping an up-to-date catalogue of everything in your tech stack allow you to accurately monitor threats, it also helps prevent 'license creep' - when businesses end up footing licensing bills for software that they're not even using.
Inefficient patch management
Patches are obviously a crucial part of cyber security, but actually applying them can be tedious, time-consuming and disruptive to the business. That's possibly why Equifax chose to patch its systems reactively rather than proactively, and why it had numerous different groups responsible for it.
Using an automated patch management tool can greatly speed up the application of important fixes and take the headache out of validating them. In addition, it prevents IT from having to reactively apply patches, improving the company's overall security by automatically deploying them with minimal human interaction.
