It’s often said (and only half-jokingly) that if we built bridges and buildings the way we build software, we’d be living in the stone age. Sure, a lot of software may be buggy, ugly or even downright useless, but that doesn’t mean that something can’t be done to fix it. One of the ways in which a developer can produce more reliable code is by rigorous testing. However for many developers, testing goes only as far as making sure that the program actually runs and gives reasonable output with a small number of typical inputs. However, all that proves is that your program isn’t falling part at the seams. It doesn’t prove that your program will work with the majority of use cases, it doesn’t prove that the program will work in unusual situations and it certainly does not prove a complete absence of bugs.
So how do you prove that your program will work correctly at least in the majority of cases? It needs to be tested hard, rigorously, brutally, unfairly. This doesn’t mean that you should test it beyond the limits of what is feasible or reasonable. Your text-editing program doesn’t have to be able to be able to open your email or edit half the HTML files on the web at the same time (unless you’re writing an Emacs super-clone), but it does have to handle multiple large files open at the same time, color syntax properly and not choke on large copy/paste operations. Deciding what is necessary to test and what can be deemed as outside your programs operating parameters can be a bit tricky at times, but it leads into what may be the more important role of testing in programming: testing forces you to think about your program design.
Fans of unit-testing often stress that unit-tests should be written first, before any of the actual program is written. This is not a easy concept for most programmers to grasp, and even fewer actually put it into practice. How are you supposed to test something when it doesn’t even exist yet? The key idea is that you’re not so much testing your code, but rather what your code should do. To some extent, it involves taking on an outsider’s perspective: a user doesn’t care how the program is written, only if it works. But if you’re writing tests for your own code, you also start thinking about how your program is structured. The idea of unit-testing is that you should test the smallest reasonable components before stepping up to larger portions of code. So you start thinking about what are the smallest parts of your code which need independent testing. Which parts can be safely folded into others and which ones have to be broken down? It also gives an idea of the interactions betweens the parts: what sort of data needs to be pushed around, how is that data affected and is there a chance that the data will be corrupted in all the moving about.
Of course, there is a down-side to this formal, test-first methodology: if you have a rapidly evolving project which has frequent design changes and restructuring, you’re going end up with a bunch of failed tests just as frequently, some of which tests things that don’t exist any more. If this happens too often, you might find yourself spending as much time updating your tests as you are updating your code. That’s never a good thing (of course, completely restructuring your project every week is probably not good either).
Even if a test-first methodology isn’t suitable for you (I don’t do it myself) you should employ formal unit-tests as much as possible. These tests should be automated i.e. they should automatically generate inputs for your programs and check the outputs without human intervention. This might seem painfully obvious, but I’ve seen more than a few students write tests which are essentially manual verifications wrapped in code. This is doubly wasteful as not only do you not test anything which you couldn’t test easily yourself, you’ve wasted time writing code for it.
That brings us to the question of manual testing. Unit testing is certainly very handy and will probably make your code much more reliable than no testing at all, but it won’t find all the bugs and it’s almost impossible to come up with all possible test cases. The real trial of your software comes not from endless automated testing, but from actual users using it out in the field. Believe me, there is no substitute for using your programs in the real world. And that doesn’t mean just using the program yourself. As a developer, it’s very likely that you have your code’s limitations tucked away in your subconscious, causing yourself to be blind to problems which would be obvious to people who actually use it without knowing about how it’s built. Getting real people to use your program in the real world is the best possible test even though it may be more time-consuming and less clear-cut that automated unit-testing. And there are some things (like UI effectiveness) which simply can’t be auto-tested.
Ultimately the role of testing boils down to the following major points. They are not universal and they won’t cover all possible cases, but they might go a long way to making your code more robust.
- Test often and rigorously.
- Use a combination of automated testing and user testing.
- Designing and testing go together: design for effective testing, test to find flaws in the design.
- Don’t just fix the bugs your tests reveal: think about how they can affect the rest of the program and how your program will be affected by the actual fixes themselves.
- Remember that testing will probably not eliminate all possible bugs, but that doesn’t mean that you should not test, or that you should test forever.