Who decides when you need to update vulnerable software? (Equifax)
I have a trick question: Who decides when you need to update vulnerable software (presuming that if it’s unpatched it might lead to bad consequences)? In a company, is that the information technology (IT) department? The chief information officer (CIO)? A policy? The user of the computer? At home, is it the user of the computer? Perhaps the family’s “tech support” person?
Remember, it’s a trick question. What’s the answer? The answer is…
The attacker decides.
The attacker is the person who decides when you get attacked, and how. Not the computer user. Not a document. Not support. Not an executive. The attacker decides. And that means the attacker decides when you need to update your vulnerable software. If that statement makes you uncomfortable, then you need to change your thinking. This is reality.
So let’s look at Equifax, and see what we can learn from it.
Let’s start with the first revelation in 2017: A security vulnerability in Apache Struts (a widely-used software component) was fixed in March 2017, but Equifax failed to update it for two whole months, leading to the loss of sensitive information on about 143 million US consumers. The update was available for free, for two months, and it was well-known that attackers were exploiting this vulnerability in other organizations. Can we excuse Equifax? Is it “too hard” to update vulnerable software (aka “patch”) in a timely way? Is it acceptable that organizations fail to update vulnerable components when those vulnerabilities allow unauthorized access to lots of sensitive high-value data?
Nonsense. Equifax may choose to fail to update known vulnerable components. Clearly it did so! But Equifax needed to update rapidly, because the need to update was decided by the attackers, not by Equifax. In fact, two months is an absurdly long time, because again, the timeframe is determined by the attacker.
Now it’s true that if you don’t plan to rapidly update, it’s hard to update. Too bad. Nobody cares. Vulnerabilities are routinely found in software components, and have been for decades. Since it is 100% predictable that there will be vulnerabilities found in the software you use (including third-party software components you reuse), you need to plan ahead. I don’t know when it will rain, but I know it will, so I plan ahead by paying for a roof and buying umbrellas. When something is certain to happen, you need to plan for it. For example, make sure you rapidly learn about vulnerabilities in third party software you depend on, and that you have a process in place (with tools and automated testing) so that you can update and ship in minutes, not months. Days, not decades.
The Apache Struts Statement on Equifax Security Breach has some great points about how to properly handle reused software components (no matter where it’s from). The Apache Struts team notes that you should (1) understand the software you use, (2) establish a rapid update process, (3) remember that all complex software has flaws, (4) establish security layers, and (5) establish monitoring. Their statement has more details, in particular for #2 they say, “establish a process to quickly roll out a security fix release… [when reused software] needs to be updated for security reasons. Best is to think in terms of hours or a few days, not weeks or months.”
Many militaries refer to the “OODA loop”, which is the decision cycle of observe, orient, decide, and act. The idea was developed by military strategist and United States Air Force Colonel John Boyd. Boyd noted that, “In order to win, we should operate at a faster tempo or rhythm than our adversaries…”. Of course, if you want to lose, then you simply need to operate more slowly than your adversary. You need to get comfortable with this adversarial terminology, because if you’re running a computer system today, you are in an adversarial situation, and the attackers are your adversaries.
In short, you must update your software when vulnerabilities are found before attackers can exploit them (if they can be exploited). If you can’t do that, then you need to change how you manage your software so can do that. Again, the attacker decides how fast you need to react.
We’re only beginnning to learn about the Equifax disaster of 2017, but it’s clear that Equifax “security” is just one failure after another. The more we learn, the worse it gets. Here are some of the information we have so far. Equifax used the rediculous pair Username “admin”, password “admin” for a database with personal employee information. Security Now! #628 showed that Equifax recommended using Netscape Navigator in their website discussion on security, a rediculously obsolete suggestion (Netscape shut down in 2003, 14 years ago). Equifax provided customers with PINs that were simply the date and time, making the PINs predictable and thus insecure. Equifax set up a “checker” site which makes false statements: “In what is an unconscionable move by the credit report company, the checker site, hosted by Equifax product TrustID, seems to be telling people at random they may have been affected by the data breach… It’s clear Equifax’s goal isn’t to protect the consumer or bring them vital information. It’s to get you to sign up for its revenue-generating product TrustID… [and] TrustID’s Terms of Service [say] that anyone signing up for the product is barred from suing the company after.” Equifax’s credit report monitoring site was found to be vulnerable to hacking (specifically, an XSS vulnerability that was quickly found by others). Equifax failed to use its own domain name for all its sites (as is standard), making it easy for others to spoof them. Indeed, NPR reported that that ”After Massive Data Breach, Equifax Directed Customers To Fake Site”. There are now suggestions that there were break-ins even earlier which Equifax never detected. In short: The more we learn, the worse it gets.
Most obviously, Equifax failed to responsibly update a known vulnerable component in a timely way. Updating software doesn’t matter when there’s no valuable information, but in this case extremely sensitive personal data was involved. This was especially sensitive data, Equifax was using a component version with a publicly-known vulnerability, and it was known that attackers were exploiting that vulnerability. It was completely foreseeable that attackers would use this vulnerable component to extract sensitive data. In short, Equifax had a duty of care that they failed to perform. Sometimes attackers perform an unprecedented kind of sneaky attack, and get around a host of prudent defenses; that would be different. But there is no excuse for failing to promptly respond when you know that a component is vulnerable. That is negligence.
But how can you quickly update software components? Does this require magic? Not at all, it just requires accepting that this will happen and so you must be ready. This is not an unpredictable event; I may not know exactly when it will happen, but I can be certain that it will happen. Once you accept that it will happen, you can easily get ready for it. There are tools that can help you monitor when your components publicly report a vulnerability or security update, so that you quickly find out when you have a problem. Package managers let you rapidly download, review, and update a component. You need to have an automated checking system that uses a variety of static tools, automated test suites, and other dynamic tools so that you can be confident that the system (with updated component) works correctly. You need to be confident that you can ship to production immediately with acceptable risk after you’ve updated your component and run your automated checking system. If you’re not confident, then your checking system is unacceptable and needs to be fixed. You also need to quickly ship that to production (and this must be automated), because again, you have to address vulnerabilities faster than the attacker.
Of course, your risks go down much further if you think about security the whole time you’re developing software. For example, you can design your system so that a defect is (1) less likely to lead to a system vulnerability or (2) has less of an impact. When you do that, then a component vulnerability will often not lead to a system vulnerability anyway. A single vulnerability in a front-end component should not have allowed such a disastrous outcome in the first place, since this was especially sensitive data, so the Equifax design also appears to have been negligent. They also failed to detect the problem for a long time; you should be monitoring high-value systems, to help reduce the impact of a vulnerability. The failure to notice this is also hard to justify. Developing secure software is quite possible, and you don’t need to break the bank to do it. It’s impossible in the real world to be perfect, but it’s very possible to be adequately secure.
Sadly, very few software developers know how to develop secure software. So I’ve created a video that’s on YouTube that should help: “How to Develop Secure Applications: The BadgeApp Example” (by David A. Wheeler). This walks through a real-world program (BadgeApp) as an example, to show approaches for developing far more secure software. If you’re involved in software development in any way, I encourage you to take a look at that video. Your software will almost certainly look different, but if you think about security throughout development, the results will almost certainly be much better. Perfection is impossible, but you can manage your risks, that is, reduce the probability and impact of attacks. There are a wide variety of countermeasures that can often prevent attacks, and they work well when combined with monitoring and response mechanisms for the relatively few attacks that get through.
The contrast between Equifax and BadgeApp is stark. Full disclosure: I am the technical lead of the BadgeApp project… but it is clear we did a better job than Equifax. Earlier this week a vulnerability was announced in one of the components (nokogiri) that is used by the BadgeApp. This vulnerability was announced on ruby-advisory-db, a database of vulnerable Ruby gems (software library components) used to report to users about component vulnerabilities. Within two hours of that announcement the BadgeApp project had downloaded the security update, run the BadgeApp application through a variety of tools and its automated test suite (with 100% statement coverage) to make sure everything was okay, and pushed the fixed version to the production site. The BadgeApp application is a simpler program, sure, but it also manages much less sensitive data than Equifax’s systems. We should expect Equifax to do at least as well, because they handle much more sensitive data. Instead, Equifax failed to update reused components with known vulnerabilities in a timely fashion.
Remember, the attacker decides.
The attacker decides how fast you need to react, what you need to defend against, and what you need to counter. More generally, the attacker decides how much you need to do to counter attacks. You do not get to decide what the attacker will choose to do. But you can plan ahead to make your software secure.
path: /security | Current Weblog | permanent link to this entry