I’ve previously advocated private testing as long as the tester was the same person who actually wrote the code under test. Over the past week I’ve been thinking more about private testing and been doing some more reading on the matter. One of the textbooks we’re using in our class is Head First Software Development where the testing is covered in quite some detail. In the book software testing is broken into three types:
- Black box testing : Testing the application as a whole from a user perspective with absolutely no knowledge of the internals. Think beta testers.
- Grey box testing: Testing the internal modules via their public interface. Think professional test teams.
- White box testing: Testing all possible code paths and branches. This would be tests written by programmers themselves.
I feel that this breakdown gets to the heart of how testing should be done in any software of reasonable size (where reasonable is anything over 15 non-trivial methods). A lot of times programmers (especially students) get the idea that testing is someone else’s job. How they get this idea is easy to see. Writing test code is not particularly fun, it can be tedious and a lot of times it can take more time to write the test than it would be to manually poke and prod the software. So if the programmers are going to spend all this time and effort writing test code, when on earth are they going to be developing the Next Big Thing?
What are tests really for?
While all the above comments are certainly valid, they all miss the point of testing. The point of testing isn’t to find obvious bugs that poking and prodding will find. Testing is for those hard-to-reach places in the code which only get executed once every thousand real-world iterations. A suite of tests makes sure that the program as a whole does what it is supposed so. As a programmer you write tests to help yourself find the nasty bugs in your code. You do the black-box testing: pick apart the internals and let the grey box tests verify the public interface. When you’re sitting in front of your monitor debugging pointer exceptions, a good set of tests and a good debugger will help you narrow down the parts of the program that are misbehaving.
What we need is a change in the way programmers (especially students) think about tests. They’re not something that lets higher management, your professor or the project lead verify that you’ve done your job. Test are meant to help YOU. Just two days ago one of my friends was writing a Sudoku solver for a class project. His code was a bit a of a mess and it seemed to him that the solver was randomly skipping certain places and leaving them blank. No matter how much I yelled at him (and yes I did literally yell at him) he refused to write even basic tests. He would rather fill his code with System.out.printlns rather than write some unit tests, all this despite the fact that he needs to turn in unit tests for his project. He was doing the same set of manual pokes and prods every time he made a change to his code. It took me a while to realize that unit tests are more than just a chore and I really hope other students take this lesson to heart.
Testing needs to evolve.
The other side of this is that tools need to evolve to support programmers and testing. In particular programming languages need to have support built-in for testing. There needs to be proper security models that allow efficient testing of private methods while keeping them private for production purposes. Neither JUnit for Java nor CppUnit for C++ allow this and I’m not sure about the technical feasability of it, but it sure would be something useful to have. I’m not a fan of IDEs, but the Eclipse environment does a good job of flagging basic errors as you type. It’s this sort of deep integration that testing needs to evolve toward. Standard hand written tests for whole methods are great, but we should also have ways to insert specific values into smaller code blocks and test the outputs. Think of a debugger mated with a test environment. A lot more programmers and students would use tests if there wasn’t boilerplate to go along with them and if they could be created and saved on the fly.
Testing is a integral part of software development. If you can’t (or won’t) write at least basic tests for your code, you shouldn’t get a job writing code. You don’t need to be proficient in the latest whizz-bang testing framework (yet), but you do need to be able to understand the basic concepts and pick up a framework in a few hours. You should understand why manually littering your code with prints is a bad idea (except in simple cases).
At the same time, we can’t pin the balme on students, programmers and the education system. Software engineering is one of the few disciplines where practitioners can seriously build their own tools. There needs to be an active effort from working programmers to create more powerful test tools that allow more direct code analysis. I can’t say more about this because I really don’t know how where the state of the art is at the moment, but I know that the tools most programmers use need to advance tremendously.
But like many things in technology, forward is the main direction of movement. As compiler and virtual machine technology improves, I’m pretty confident that testing technology will go along for the ride. I’m really looking forward to the day that I can select a block of code and run real time tests on it. What do you think? Is testing something that is worth time and effort (both in terms of creating tests and creating tools) or can we do without them?