A tester walks into a bar and orders a beer. Then he orders 999,999 beers. Orders 0 beers. Orders -1 beers. Orders a coyote. Orders a qpoijzcpx. Then he insults the bartender.
This joke (with variations) is making the rounds, but it also has a serious point. It’s a nice example of how testing should work, including software testing.
Too many of today’s so-called software “tests” only check for correct data. This has led to numerous vulnerabilities including Heartbleed and Apple’s “goto fail; goto fail;” vulnerability. The paper “The Most Dangerous Code in the World: Validating SSL Certificates in Non-Browser Software” found that a disturbingly large number of programss’ security depends on SSL certificate validation, and they are insecure because no one actually tested them with invalid certificates. They note that “most of the vulnerabilities we found should have been discovered during development with proper unit testing”.
Good software testing must include negative testing (tests with data that should be rejected) to ensure that the software protects itself against bad data. This must be part of an automated regression test suite (re-run constantly) to prevent problems from creeping in later. For example, if your programs accept numbers, don’t just test for “correct” input; test for wrong cases like too big, zero, negative or too small, and non-numbers. Testing “just around” too big and too small numbers is often helpful, too, as is testing that tries to bypass the interface. Your users won’t know how you did it, but they’ll know your program “just works” reliably.
path: /misc | Current Weblog | permanent link to this entry