When I first started to get really interested in computers in the late 90s, most of the computers I saw where still boring beige boxes and they had rather prominent reset buttons. This was probably good because almost all the computers I saw run Windows (either 98 or 95) and had to be restarted rather often. Since then the reset button seems to have fallen from grace. My family’s first computer in 2001 had a large power button, but the reset button was tucked sort of underneath it and doubled as the disk activity indicator. My current laptop which I’ve had for about two years now doesn’t even have a reset button. If I want to do a hard reboot I have to hold down the power button for a few seconds to turn it off and then start it again.
To be honest, I really don’t need to do a hard reboot all that much nowadays. It’s mostly when I’ve royally screwed up X configuration and nothing responds anymore. So I can’t say that I really miss it all that much and I’m not too concerned with its demise. However computers do need to be restarted every now and then, though I hope we’re getting to the point where that becomes an extremely rare necessity. But hardware isn’t the only thing that needs restarting. Software projects occassionally need a restart too.
At the recent EuroLisp Symposium there was talk of scrapping Common Lisp and starting over. The Firefox browser descends from the Mozilla which in turn originates from the Netscape code base that was open-sourced in 1998. But in the process, almost all the original Mozilla was rewritten in a better form (especially the core Gecko layout engine). The Unix concept has been reimplemented numerous times over the last decades. I’ve been working on a research project for the past year to explore applying formal grammars to describe complex patterns. In the process I’ve become the toolmaker for the group — I’ve been building a set of Python programs that the others can use to do experimentation. I’m currently working on the 5th rewrite of this toolset and each time the tools get better and I get faster at bringing things back to their previous level of usefulness (and then some). In case you’re wondering, I’d like to open source these tools in the future if the rest of the group agrees.
Hitting the reset button on a software project may seem like a hard decision to make, but I’ve come to think of it as a natural part of the software development project. Like it or not, software continually grows. Once a version goes out to the public people start finding things they don’t like or features they’d like to have. All this goes on the drawing board for the next version. As time goes by, the software pushes against the boundaries of what it was supposed to do. Layers are added and parts are written to support the new additions. In a healthy project with good developers and strong leadership (or community consensus), it’s possible to make the additions smoothly, at least for a long time. But if you add too much to a particular piece of software, the pieces gradually start to lose cohesion. Competing subsystems and inconsistent interfaces begin to emerge and it becomes harder and harder to add non-trivial functionality or even fix outstanding bugs. At some point, the codebase reaches a sort of critical mass: it’s easier to scrap the whole darn mess and rebuild it from scratch.
I took a software engineering class last semester and learned all about agile development and the iterated development model. There are some good ideas in there and it’s certainly better than waterfall models, but I think that it misses one important idea: users and there software form a feedback loop which can be essentially unbounded. Agile models help developers build better software and encourage maintainability, but there’s still a clear delivery date after which the developers role is reduced to mainly bug fixes. We don’t have a model for how to build organic software that can grow with users’ needs over the years (though I think we’re getting closer with the success of large scale open source user-centric projects like Firefox). Till we do have such a model, I think rewrites should be considered an essential part of a software project’s life-cycle.
Restarting a software project isn’t a decision to be taken lightly, but it’s sometimes the best way to go. The problem is, it takes a good amount of will power from strong leaders to make a rewrite work properly and it’s easy to mess up the process. From my personal experience of resetting a project (and from what I’ve read) perhaps the hardest part is letting go of what’s been done before. It’s tempting to just reuse components from previous generations if they worked right (even if they were stopgap hacks at the time). This is tempting because not only do developers not have to actually rewrite from scratch, it can make things much faster. A rewrite by definition means that you won’t have a fully working product for a while and that feeling can be quite frustrating especially when you know that there actually is a working version in cold-storage. If only you could just borrow a few parts, life would be much better. Of course doing that defeats the very purpose of a rewrite. I’ll admit that there are some circumstances where code reuse can work, but these situations need to be carefully thought out and the code itself needs to be scrubbed clean so that it doesn’t reintroduce any of the problems that prompted the rewrite.
As this post approaches the thousand word mark, I think I should spare a word for the users of software under rewrite (whether they be end-users or other developers). Users need to have software that works, not something that’s broken and clearly experimental. Software under rewrite shouldn’t be pushed out until the rewrite is done and the older version should be kept available. Even more important is that the rewrite should actually produce something that the users want. The new product should be sufficiently similar to the old that the learning curve is mostly negligible (at least for everyday tasks) and old bugs should be fixed. If you’re going to do a rewrite, do it properly and fix the bugs while you’re at it.
The people who are most likely to get shafted from a rewrite are developers who are using your API. If the rewrite requires substantial changes to the API it will break a lot of client software and make a lot of people very unhappy. It that seems to be the case it’s time to slam on the brakes and see if the existing API can be preserved. This isn’t as much of a problem if the API is well-designed to start with, but that’s the topic for another post. At this point I’d just like to point out one of Jamie Zawinski’s classic articles which you should read before starting any software project (including a rewrite) and at various points during the process.
I used this post as a way to organize some of the thoughts I’ve had while during my own rewrite. I’m trying very hard to keep the external interface the same, but I might have to make some sacrifices. I’m also going to be exposing a public API for the first time so that there can be multiple independent interfaces written so that’s something that will take some thought. I also have a buglist from the last version in front of me because I think that allowing bugs to propagate across generations is just lazy. I’m hoping that by the end of it I’ll have a cleaner system as well as a project that I can cleanly extend without a rewrite for a longer time. I don’t envision any more rewrites for a while to come. But then again, the only guarantee of life is death, so I’m not placing any bets. And besides, rebuilding from scratch is kinda fun too.