David A. Wheeler's Blog

Tue, 07 Apr 2015

Heartbleed found with american fuzzy lop (afl) and Address Sanitizer (ASan)

Big news in security vulnerability research: Hanno Böck found Heartbleed using american fuzzy lop (afl) and Address Sanitizer (ASan) - and in only 6 hours of execution time.

This means that software developers should seriously consider using a more-advanced fuzzer, such as american fuzzy lop (afl), along with Address Sanitizer (ASan) (an option in both the LLVM/clang and gcc compilers), whenever you write in C, C++, Objective-C, or in other circumstances that are not memory-safe. In particular, seriously consider doing this if your program is exposed to the internet or it processes data sent via the internet (practically all programs meet this criteria nowadays). I had speculated that this combination could have found Heartbleed in my essay on Heartbleed, but this confirmation is really important. Here I will summarize what’s going on (using the capitalization conventions of the various tool developers).

The american fuzzy lop (afl) program created by Michal Zalewski is a surprisingly effective fuzzer. A fuzzer is simply a tool that sends lots of semi-random inputs into a program and to detect gross problems (typically a crash). Fuzzers do not know what the exact correct answers are, but because they do not, they can try out more inputs than systems that know the exact correct answers. But afl is smarter than most fuzzers; instead of just sending random inputs, afl tracks which branches are taken in a program. Even more interestingly, afl even tracks how often different branches are taken when running a program (that is especially unusual). Then, when afl creates new inputs, it prefers to create them based on inputs that have produced different counts on at least some branches. This evolutionary approach, using both branch coverage and the number of times a branch is used, is remarkably effective. Simple dumb random fuzzers can only perform relatively shallow tests; getting any depth has required more complex approaches such as detailed descriptions of the required format (the approach used by so-called “smart” fuzzers) and/or white-box constraint solving (such as fuzzgrind or Microsoft’s SAGE). It’s not at all clear that afl eliminates the value of these other fuzzing approaches; I can see combining their approaches. However, afl is clearly getting far better results than simple dumb fuzzers that just send random values. Indeed, the afl of today is getting remarkably deep coverage for a fuzzer. For example, the post Pulling JPEGs out of thin air shows how afl was able to start with only the text “hello” (a hideously bad starting point) and still automatically figure out how to create valid JPEG files.

However, while afl is really good at creating inputs, it can only detect problems if they lead to a crash; vulnerabilities like Heartbleed do not normally cause a crash. That’s where Address Sanitizer (ASan) comes in. Address Sanitizer turns many memory access errors, including nearly all out-of-bounds accesses, double-free, and use-after-free, into a crash. ASan was originally created by Konstantin Serebryany, Derek Bruening, Alexander Potapenko, and Dmitry Vyukov. ASan is amazing all by itself, and the combination is even better. The fuzzer afl is good at creating inputs, and ASan is good at turning problems into something that afl can detect. Both are available at no cost as Free/ libre/ open source software (FLOSS), so anyone can try them out, see how they work, and even make improvements.

Normally afl can only fuzz file inputs, but Heartbleed could only be triggered by network access. This is no big deal; Hanno describes in his article how to wrap up network programs so they can be fuzzed by file fuzzers.

Sometimes afl and ASan do not work well together today on 64-bit systems. This has to do with some technical limitations involving memory use; on 64-bit systems ASan reserves (but does not use) a lot of memory. This is not necessarily a killer; in many cases you can use them together anyway (as Hanno did). More importantly, this problem is about to go away. Recently I co-authored (along with Sam Hakim) a tool we call afl-limit-memory; it uses Linux cgroups to eliminate the problem so that you can always combine afl and ASan (at least on Linux). We have already submitted the code to the afl project leader, and we hope it will become part of afl soon. So this is already a disappearing problem.

There are lots of interesting related resources. If you want to learn about fuzzing more generally, some books you might want to read are Fuzzing: Brute Force Vulnerability Discovery by Sutton, Greene, and Amini and Fuzzing for Software Security Testing and Quality Assurance (Artech House Information Security and Privacy) by Takanen, DeMott, and Miller. My class materials for secure software design and programming, #9 (analysis tools), also cover fuzzing (and are freely available). The Fuzzing Project led by Hanno is an effort to encourate the use of fuzzing to improving the state of free software security, and includes some tutorials on how to do it. The paper AddressSanitizer: A Fast Address Sanity Checker is an excellent explanation of how ASan works. My essay How to Prevent the next Heartbleed discusses many different approaches that would, or would not, have detected Heartbleed.

I do not think that fuzzers (or any dynamic technique) completely replace static analysis approaches such as source code weakness analyzers. Various tools, including dynamic tools like fuzzers and static tools like source code weakness analyzers, are valuable complements for finding vulnerabilities before the attackers do.

path: /security | Current Weblog | permanent link to this entry

Sat, 04 Apr 2015

Security presentation updates

I’ve updated my presentations on how to design and implement secure software. In particular, I’ve added much about analysis tools and formal methods. There is a lot going on in those fields, and no matter what I do I am only scratching the surface. On the other hand, if you have not been following these closely, then there’s a lot you probably don’t know about. Enjoy!

path: /security | Current Weblog | permanent link to this entry

Fri, 27 Mar 2015

Z3 is OSS!

Z3 has been released as open source software under the MIT license! This is great news. Z3 is a good satisifiability modulo theories (SMT) solver / theorem prover from Microsoft Research. An SMT solver accepts a set of constraints (such as “a<5 and a>1”) and tries to produce values that satisfy all the constraints. A satisfiability (SAT) solver does this too, but SAT solvers can only work with boolean variables; SMT solvers can handle other types, such as integers. Here is a Z3 tutorial.

SMT solvers are basically lower-level tools that have many uses for building larger capabilities, because many problems require solving logical formulas to find a solution.

I am particularly interested in the use of SMT solvers to help prove that programs do something or do not do something. Why3 is a platform that lets you write programs and their specifications, and then calls out to various provers to try to determine if the claims are true. By itself Why3 only supports its WhyML language, but Why3 can be combined with other tools to prove statements in other languages. Those include C (using Frama-C and a plug-in), Java, and Ada. People have been able to prove tiny programs for decades, but scaling up to bigger programs in practice requires a lot of automation. I think this approach of combining many different tools, with different strengths, is very promising.

The more tools that are available to Why3, the more likely it will solve problems automatically. That’s because different tools use different heuristics and focus on different issues, resulting in different ones being good at different things. There are already several good SMT solvers available as OSS, including CVC4 and alt-ergo.

Now that Microsoft has released Z3 as OSS, there is yet another strong OSS SMT solver that tools like Why3 can use directly. In short, the collection of OSS SMT solvers has just become even stronger. There’s a standard for SMT solver inputs, the SMT-LIB format, so it’s not hard to take advantage of many SMT solvers. My hope is that this will be another step in making it easier to have strong confidence in software.

path: /oss | Current Weblog | permanent link to this entry

Wed, 11 Mar 2015

Plans for HTTPS (SSL/TLS) on this site

Currently this website uses only HTTP, and does not support HTTPS. That means that users cannot trivially authenticate what they receive, and that in some cases users reveal to others what they are viewing on the site. (Technical note: HTTPS is implemented by a lower-level protocol; the current protocol versions of this protocol are named TLS, and the older ones are named SSL, but a lot of people use the term SSL to include TLS.) I would like to use HTTPS, but this website is entirely self-funded. I do have a plan, though.

My current plan is that I am waiting for Let’s encrypt to stand up and be ready. Once that gets going, I intend to use it to add support for HTTPS. I’d like to eventually only support HTTPS, since that prevents downgrade attacks, but I need to make sure that the TLS certificates and configuration works well. Also, I pay others to maintain the server; since I am not made of money, I necessarily use low-end cheap services. That will limit what I can do in terms of HTTPS configuration hardening. On the other hand, it should be better than the current situation.

The software I develop is generally available on SourceForge or GitHub, and they already provide HTTPS, so you don’t need to wait for that. Currently you have to log into SourceForge to get HTTPS, but that is expected to change, and for now just log in.

Anyway, I thought some of you might like to know that there is a plan.

path: /website | Current Weblog | permanent link to this entry

Sat, 14 Feb 2015

Learning from Disaster

Learning from Disaster is a collection of essays that examines computer security disasters, and what we can learn from those disasters. This includes Heartbleed, Shellshock, POODLE, the Apple goto fail, and Sony Pictures. If you’re interested in computer security I think you’ll find this collection interesting.

So: please enjoy Learning from Disaster.

path: /security | Current Weblog | permanent link to this entry

Tue, 06 Jan 2015

Cloud security

There seems to be a lot of confusion about security fundamentals of cloud computing (and other utility-based approaches). For example, many people erroneously think hardware virtualization is required for clouds (it is not), or that hardware virtualization and containerization are the same (they are not).

My essay Cloud Security: Virtualization, Containers, and Related Issues is my effort to counteract some of this confusion. It has a quick introduction to clouds, a contrast of various security isolation mechanisms used to implement them, and a discussion of some related issues.

So please check out (and enjoy) Cloud Security: Virtualization, Containers, and Related Issues.

path: /security | Current Weblog | permanent link to this entry

Wed, 31 Dec 2014

I hope we learn from the computer security problems of 2014

As 2014 draws to a close, I hope anyone involved with computers will resolve to learn from the legion of security problems of 2014.

We had way too many serious vulnerabilities in widely-used software revealed in 2014. In each case, there are lessons that people could learn from them. Please take a look at the lessons that can be learned from Heartbleed, Shellshock, the POODLE attack on SSLv3, and the Apple goto fail vulnerability. More generally, a lot of information is available on how develop secure software - even though most software developers still do not know how to develop secure software. Simiarly, there are a host of lessons that organizations could learn from Sony Pictures.

Will people actually learn anything? Georg Wilhelm Friedrich Hegel reportedly said that, “We learn from history that we do not learn from history”.

Yet I think there are reasons to hope. There are a lot of efforts to improve the security of Free/Libre/Open Source Software (FLOSS) that are important yet inadequately secure. The Linux Foundation (LF) Core Infrastructure Initiative (CII) was established to “fund open source projects that are in the critical path for core computing functions” to improve their security. most recent European Union (EU) budget includes €1 million for auditing free-software programs to identify and fix vulnerabilities. The US DHS HOST project is also working to improve security using open source software (OSS). The Google Application Security Patch Reward Program is also working to improve security. And to be fair, these problems were found by people who were examining the software or protocols so that the problems could be fixed - exactly what you want to happen. At an organizational level, I think Sony was unusually lax in its security posture. I am already seeing evidence that other organizations have suddenly become much more serious about security, now that they see what has been done to Sony Pictures. In short, they are finally starting to see that security problems are not theoretical; they are real.

Here’s hoping that 2015 will be known as the year where people took computer security more seriously, and as a result, software and our computer systems became much harder to attack. If that happens, that would make 2015 an awesome year.

path: /security | Current Weblog | permanent link to this entry

Sat, 20 Dec 2014

Sony Pictures, Lax Security, and Passwords

Sony Pictures, Lax Security, and Passwords is a new essay about the devastating attack on Sony Pictures. We now have new information about how Sony Pictures was attacked; from that, and public information about Sony Pictures, we can see why the attack was so devastating. Even more importantly, we can learn from it. So please, take a look at Sony Pictures, Lax Security, and Passwords.

path: /security | Current Weblog | permanent link to this entry

Sun, 23 Nov 2014

Lessons learned from Apple goto fail

The year 2014 has not been a good year for the SSL/TLS protocol. SSL/TLS is the fundamental algorithm for securing web applications. Yet every major implementation has had at least one disastrous vulnerability, including Apple (goto fail), GnuTLS, OpenSSL (Heartbleed), and Microsoft. Separately a nasty attack has been found in the underlying SSLv3 protocol (POODLE). But instead of just noting those depressing statistics, we need to figure out why those vulnerabilities happened, and change how we develop software to prevent them from happening again.

To help, I just released The Apple goto fail vulnerability: lessons learned, a paper that is similar to my previous papers that focuses on how to counter these kinds of vulnerabilities in the future. In many ways Apple goto fail vulnerability was much more embarassing compared to Heartbleed; the goto fail vulnerability was easy to detect, in a portion that was a key part of its functionality. This vulnerability was reported back in February 2014, but there does not seem to be a single place where you can find a more complete list of approaches to counter it. I also note some information that doesn’t seem to be available elsewhere.

So if you develop software - or manage people who do - take a look at The Apple goto fail vulnerability: lessons learned.

path: /security | Current Weblog | permanent link to this entry

Wed, 22 Oct 2014

Peter Miller and improving make

First, a sad note. Peter Miller died on 2014-07-27 from leukemia. He did a lot of important things, including writing the influential paper “Recursive Make Considered Harmful”. Rest in peace.

I should point out an essay I’ve written about improving make. Make is widely used, but the POSIX standard for it lacks key functions, including vital ones that Peter Miller pointed out years ago. Thankfully, progress has been made. My hope is that progress will continue to happen; I welcome help for improving the standard and implementations of make.

path: /misc | Current Weblog | permanent link to this entry

Mon, 20 Oct 2014

Open Source Software in U.S. Government

The report “Open Source Software in Government: Challenges and Opportunities” is available to the public (you can jump to the “Download full report” link at the bottom). This paper, which I co-authored, discusses key challenges and opportunities in the U.S. government application of open source software (OSS). It became publicly available only recently, even though it was finished a while back; I hope it’s been worth the wait. If you’re interested in the issues of OSS and government, I think you’ll find this report very illuminating.

path: /oss | Current Weblog | permanent link to this entry

Tue, 14 Oct 2014

POODLE attack against SSLv3

There is a new POODLE attack against SSLv3. See my page for more info.

path: /security | Current Weblog | permanent link to this entry

Mon, 13 Oct 2014


My username on Twitter is drdavidawheeler, for those on Twitter who want occasional comments on computer security, open source software, software development, and so on.

path: /website | Current Weblog | permanent link to this entry

Thu, 09 Oct 2014

A tester walks into a bar

A tester walks into a bar and orders a beer. Then he orders 999,999 beers. Orders 0 beers. Orders -1 beers. Orders a coyote. Orders a qpoijzcpx. Then he insults the bartender.

This joke (with variations) is making the rounds, but it also has a serious point. It’s a nice example of how testing should work, including software testing.

Too many of today’s so-called software “tests” only check for correct data. This has led to numerous vulnerabilities including Heartbleed and Apple’s “goto fail; goto fail;” vulnerability. The paper “The Most Dangerous Code in the World: Validating SSL Certificates in Non-Browser Software” found that a disturbingly large number of programss’ security depends on SSL certificate validation, and they are insecure because no one actually tested them with invalid certificates. They note that “most of the vulnerabilities we found should have been discovered during development with proper unit testing”.

Good software testing must include negative testing (tests with data that should be rejected) to ensure that the software protects itself against bad data. This must be part of an automated regression test suite (re-run constantly) to prevent problems from creeping in later. For example, if your programs accept numbers, don’t just test for “correct” input; test for wrong cases like too big, zero, negative or too small, and non-numbers. Testing “just around” too big and too small numbers is often helpful, too, as is testing that tries to bypass the interface. Your users won’t know how you did it, but they’ll know your program “just works” reliably.

path: /misc | Current Weblog | permanent link to this entry

Sun, 05 Oct 2014


I have posted a new paper about Shellshock. In particular, it includes a detailed timeline about shellshock, which counters a number of myths and misunderstandings. It also shows a correct way to detect if your system is vulnerable to shellshock (many postings get it wrong and only detect part of the problem).

I also briefly discuss how to detect or prevent future shellshock-like attacks. At the moment this list is short, because these kinds of vulnerabilities are known to be difficult to detect ahead of time. Still, I think it is worth trying to do this. My goal is to eventually end up with something similar to the list of countermeasures for Heartbleed-like attacks that I developed earlier.

path: /security | Current Weblog | permanent link to this entry

Tue, 19 Aug 2014

Software SOAR released!!

The Software SOAR (which I co-authored) has finally been released to the public! This document - whose full name is State-of-the-Art Resources (SOAR) for Software Vulnerability Detection, Test, and Evaluation (Institute for Defense Analyses Paper P-5061, July 2014) - is now available to everyone. It defines and describes the following overall process for selecting and using appropriate analysis tools and techniques for evaluating software for software (security) assurance. In particular, it identifies types of tools and techniques available for evaluating software, as well as the technical objectives those tools and techniques can meet. A key thing that it does is make clear that in most cases you need to use a variety of different tools if you are trying to evaluate software (e.g., to find vulnerabilities).

The easy way to get the document is via the Program Protection and System Security Engineering web page, then scroll to the bottom to look for it (it is co-authored by David A. Wheeler and Rama S. Moorthy). You can jump directly to the Main report of the software SOAR and Appendix E (Software State-of-the-Art Resources (SOAR) Matrix). You can also get the software SOAR report via IDA.

I don’t normally mention things I’ve done at work, but this is publicly available, some people have been waiting for it, and I’ve found that some people have had trouble finding it. For example, the article “Pentagon rates software assurance tools” by David Perera (Politico, 2014-08-19) is about this paper, but it does not tell people how to actually get it. I’m hoping that this announcement will give people a hand.

path: /security | Current Weblog | permanent link to this entry

Sun, 13 Jul 2014

Flawfinder version 1.28 released!

I’ve released yet another new version of flawfinder - now it’s version 1.28. Flawfinder is a simple program that examines C/C++ source code and reports on likely security flaws in the program, ranked by risk level.

This new version has some new capabilities. Common Weakness Enumeration (CWE) references are now included in most hits (this makes it easier to use in conjunction with other tools, and it also makes it easier to find general information about a weakness). The new version of flawfinder also has a new option to only produce reports that match a regular expression (e.g., you can report only hits with specific CWE values). This version also adds support for the git diff format.

This new version also has a number of bug fixes. For example, it handles files not ending in newline, and it more gracefully handles handles unbalanced double-quotes in sprintf. A bug in reporting the time executed has also been fixed.

For more information, or a copy, just go to my original flawfinder home page or the flawfinder project page on SourceForge.net. Enjoy!

path: /security | Current Weblog | permanent link to this entry

Tue, 10 Jun 2014

Interview on Application Security

A new interview of me is available: David A. Wheeler on the Current State of Application Security (by the Trusted Software Alliance) (alternate link). In this interview I discuss a variety of topics with Mark Miller, including the need for education in developing secure software, the need to consider security thoughout the lifecycle, and the impact of componentization. I warn that many people do not include security (including software assurance) when they ask for quality; while I agree in principle that security is generally part of quality, in practice you have to specifically ask for security or you won’t get it.

This interview is part of their 50 in 50 interviews series, along with Joe Jarzombek (Department of Homeland Security), Steve Lipner (Microsoft), Bruce Schneier, Jeff Williams (Aspect Security and OWASP), and many others. It was an honor and pleasure to participate, and I hope you enjoy the results.

path: /security | Current Weblog | permanent link to this entry

Wed, 21 May 2014

On Dave and Gunnar show

There is now an interview of me on the Dave and Gunnar show (episode #51). I talk mostly about How to prevent the next Heartbleed. I also talk about my FLOSS numbers database (as previously discussed) and vulnerability economics. There was even a mention of my Fully Countering Trusting Trust through Diverse Double-Compiling work.

Since the time of the interview, more information has surfaced about Heartbleed. Traditional fuzzing could not find Heartbleed, but it looks like some fuzzing variants could even if the OpenSSL code was unchanged; see the latest version for more information. If you learn more information relevant to the paper, let me know!

path: /oss | Current Weblog | permanent link to this entry

Thu, 08 May 2014

FLOSS numbers database!

If you are doing research related to Free / Libre / Open Source Software (FLOSS), then I have something that may be useful to you: the FLOSS numbers database.

My paper Why Open Source Software / Free Software (OSS/FS, FLOSS, or FOSS)? Look at the Numbers! is a big collection of quantitative studies about FLOSS. Too big, in fact. There have been a lot of quantitative studies about FLOSS over the years! A lot of people want to query this information for specific purposes, and it is hard to pull out just the parts you want from a flat document. I had thought that as FLOSS became more and more common, fewer people would want this information… but I still get requests for it.

So I am announcing the FLOSS numbers database; it provides the basic information in spreadsheet format, making it easy to query for just the parts you want. My special thanks go to Paul Rotilie, who worked to get the data converted from my document format into the spreadsheet.

If you want to discuss this database, I have set up a discussion group: Numbers about Free Libre Open Source Software. If you are doing research and need or use this kind of information, please feel free to join. If you just need a presenatation based on this, you might like my Presentation: Why Free-Libre / Open Source Software (FLOSS or OSS/FS)? Look at the Numbers!.

This database is the sort of thing that if you need it, you really need it. I am sure it is incomplete… but I am also sure that with your help, we can make it better.

path: /oss | Current Weblog | permanent link to this entry