Writing software for non-programmers

is not very easy. As a programmer, you learn to use various pieces of software, like text-editors, compilers, debuggers, profilers, so on and so forth. All these tools are very useful, some are essential, and many of them have a learning curve, you have to invest time into learning the nuances of the particular tool if you want to use it to the maximum potential. Consequently when you are writing software that you will be used by other programmers (even if it is not a programming tool), making the software easy to use is not always the first thing that comes to mind. Let’s solve the problem and then we’ll worry about making it ‘user-friendly’. User interface gets pushed onto the backburner. The UNIX software user space is filled with various command line programs which are very effective in their own right, each with its own host of little options toggled by specifying combinations of letters and words on the command line. Not a problem for us coders, after all, text is what we live in. Not so easy for non-programmers who find the very idea of typing in commands some sort of deep black magic. Hence the rise of “graphical frontends”: point-and-click interfaces that put interaction with these text utilities in the background.

For a while i was of the idea that the GUI was for wimps. If you wanted to harness the real power inside your computer, you had to get down and dirty with the command line. However I’ve since come to realize that the real problem is not one of interface, but of thought style. I’m currently a program that (I hope) will be used by artists and architects. It is text-based and the user needs to type in configuration files. No point and click GUI (yet). I chose to take on the job of writing the front-end: the part of the program that the user would actually interact with. This meant creating a configuration language that would be easy for non-programmers to write. The language needed to describe certain elements and how those elements could be combined. My first thought was to create some sort of CSS clone for element description and something akin to Prolog for combination rules. Now, if I was writing this for programmers, I wouldn’t have given it a second thought. But I was writing software for programmers, I was writing for artists. Since then I’ve changed my language to more like Basic than any other programming language. I’ve tried to make it as close to plain English as possible, with no special symbols except for parentheses. The reason for this is simple: any configuration language, no matter how simple would mean that the artists would have to learn it’s uses. I would rather have them up and running and actually using the tool than learning my arbitrary syntax. By keeping the language as close to English as possible, I’ve tried to minimize the learning curve.

Error messages are another area where I’ve had to keep in mind that my users won’t be programmers. Compilers are notorious for having hard to read error messages. I couldn’t have my config file parser give equally esoteric error messages. Most of my parser is a simple recursive descent parser that just does simple syntactical analysis, however a part of it involves solving a system of linear equations. Once again it was very tempting to have the error messages speak in terms of the underlying logic and math, which to a programmer at least, isn’t very complicated. However, I made the conscious to have the error messages be plain English as well. The user gets a line numbers and a plain English description of what went wrong. By leaving out allusions to the math and logic, I may have made it harder for the user to fix an error in the configuration file, but that’s a risk I’m willing to take for the time being.

Programmers are instinctive tinkerers. We like playing around with things. Even when things are working right, we have a need to get in there and try to make it work faster, better, or just in a different fun way. We adore customization and love extensibility. Simply getting the job done is not enough, we want to find the most elegant, the fastest, the most efficient, the craziest way to do it. Living in our world of programmable text editors, modular IDEs and extensible browsers it’s easy to forget that often people just want software that gets the work done and stays out of their way. They don’t want unlimited power, they just want the damn thing to work.
Exercising this sort of restraint is not easy and it requires a lot of thought. I don’t want to give the user something that will overwhelm them, but at them same time, I don’t want to give them a crippled half-hearted piece of work either. Unfortunately there’s is no way to tell what a user wants or needs before you actually give them something to play with. Our software is at a tipping point: it has a small set of core features which work well, there is room to add much more, but we don’t know what will be essential and what will be extra cruft that no one ever needs. And so we are heading to meet our “clients” next week and hopefully at the end of the meeting we will have a new goal in mind.

Use your own software

Also known as “eat your own dog food”, this is the concept behind one of the most successful software engineering projects of modern times: the Windows NT kernel. The Windows NT kernel was written by a highly-talented team led by a man who is arguably one the best software engineers of all time: Dave Cutler. Dave Cutler was also the lead developer for another groundbreaking operating system: Digital’s VMS. However there was more to this project than talented developers: the whole team actually used Windows NT everyday as soon as possible. This meant that the developers exposed themselves to problems that would be encountered by the average user and could then fix those problems before actually shipping the finished product.

Let’s face it: software is buggy. We still have no clue about how to reliably write bug-free software. So we’re stuck with the situation of writing buggy software and then wrangling the bugs out of them. A lot of bugs are removed through the process of just writing a working piece of software. Automated testing also gets rid of a fair amount of bugs. However, there are some things that no amount of debugging or automated testing can get rid. Modern software systems are large and complicated and it’s hard to tell exactly how all the different parts will interact until you actually start using it all.

Even if you’re certain that your code is relatively bug-free, it’s still important to use software that you’ve written. There are a lot of things about software like look-and-feel, intuitiveness, ease of use, which can’t be determined automatically. The only way to see if your program has an elegant and smooth interface or is powerful, but clunky is to use it repeatedly. When you start using software that you’ve written on a regular basis, you start to think about how your software can be improved, what are the bottlenecks and hard-to-use features, what features are missing and what are unnecessary. This constant evaluation is a key ingredient of making better software.

Unfortunately, most of the software that is being created is made by developers who really don’t know how that piece of software is going to be used in the real world. After all, Photoshop wasn’t made my a team of artists and most users of Microsoft Word have never written a line of working code. So how do programmers go about creating software that they might never use? Enter the beta tester. Beta testers are given pre-release versions of software to evaluate and their suggestions are then folded back into the next release. The best beta testers are the very people for whom you’re writing the software in the first place. If you’re writing a software package for a specific client, then it is essential to have a continuous dialog open. Test versions should routinely be given out to get feedback and then that feedback should be incorporated into the next edition. If your software is for a mass market, then your user community will be your pool of beta testers, encourage them to give feedback and then take those opinions into account. Eating your own dog food is a good idea, but it’s not a disadvantage if you can give it to a bunch of other dogs and see what they think of it.

Throw away version 1

One of the best books on software engineering is Fred Brooks’ The Mythical Man Month. Even though it’s over thirty years old, most of the principles and ideas laid down are still applicable, some even more so now than thirty years ago. One of the ideas that I feel deserve special mention is that of the pilot system.

Like many great ideas, this one is very simple in essence. The first working system created as part of a software project is one that will be thrown away (intentionally or not), in the same way that chemical engineers will first build a pilot plant to test out their process before investing in a full scale plant. The reason for this is that it’s hard to get a grasp for what the real problems of a software system will be before some of it is actually implemented. Keeping this in mind, it’s best to start projects with the idea that version 1 will be thrown away and schedule the project accordingly.

The benefits of starting with a “throw away version 1” philosophy are many and can help the development of any software project, large or small. Knowing that the first working version doesn’t need to be perfect lets the development team quickly create a prototype to explore the different aspects of the project. It allows experiments to try out various solutions to a problem before deciding which one to use. If you’re making software for customers (as most people are) having a working model early in the project’s schedule allows you get valuable user feedback and make changes accordingly. It’s often said that software users don’t really know what they want, but it makes everyone’s life a little bit easier if there is something tangible that can be pointed to and specific aspects of it praised or criticized.

Of course, knowing that you’re going to throw away the first version doesn’t mean that this version should be made sloppily or inconsistently. The same technical rigor should be applied to the prototype as to the final delivered product. In fact, making version 1 in a half-hearted fashion defeats the entire purpose of having such a version in the first place. Problems in the design of the product will only become apparent if version 1 reflects fairly accurately what the final is supposed to be. Engineering models are made to the same specifications and requirements as what the final should be (albeit on a smaller scale) and there is no reason why software prototypes should be any different.

I’ve had the opportunity to see the benefit of this approach first hand. We recently completed version 1 of our summer research project. Though we had known that the final version would probably be very different, we worked under the assumption that version 1 would be the basis for the later version and that if we screwed up now, we would have to pay for it later. The result is that version 1 really was a proper working system, which works (from the user perspective) in a manner very close to how we want the final to work. Of course, we are still throwing away version 1. Even though it works fine, we’ve seen that there are a number of fundamental flaws to our approach. The ad-hoc parser that I wrote for our configuration language works, but is ugly and inflexible. I’ve already replaced it with a more robust and flexible recursive descent parser. The output mechanism is currently hard-coded into the rest of the system, but it should be pluggable; we now have an idea of what sort of an architecture we want. In fact, the very core of our system will probably have to be replaced or at least changed substantially, because we just realized that what we had written wasn’t really what we had set out to do. All this might sound like a disaster, and it would have been if version 1 wasn’t designed with the idea of eventually throwing it away. We used it as a test bed and learned valuable lessons from it which we’ll now apply to the creation of version 2.

Unless your project is something very trivial, it’s best to start with the idea of throwing away version 1 and learning from the lessons that it will inevitably each. There is another software engineering principle that runs complimentary to “throw away version 1” that can be best described as “eat your own dog food”, but that’s something for another post, maybe tomorrow. Till then, have fun with version 1.

Building software or growing it?

For the past week I’ve been working on my first multi-person, long term software project. As part of my summer research project we’re building an urban planning system which automatically builds sample cities from a set of design conditions. I’ve been working with two other people and my professor, and though it isn’t a large scale project like writing an operating system, it’s still quite challenging. One of the major challenges that I’ve come to face is that it’s a very hard to stick to any designs that may have been made before actual writing of code. Furthermore, in a research project such as the one I’m working on, the problems aren’t well known and we’re discovering them as we go along. This results in the problem that I’m continually rewriting parts of my code and adding parts as I go along.
In class we were always taught to plan out our programs in a engineering fashion, and then implement. I’ve tried to follow this plan-then-implement methodology, but I’ve to improvise quite a lot. So I’ve been wondering, what’s the best way to make large pieces of software: grow them or build them? Building software is what I’ve been taught in class: where we make a  plan and then follow it. Now anyone who has written a sizeable amount of code will now that it’s impossible to plan for all eventualities. That’s what debugging is all about: dealing with things that you didn’t plan for. However debugging deals with errors, not with the basic structure of the architecture. I’ve previously advocated carefully planning out algorithms before implementing, and i still to that. But when it comes to actually building large software systems, it’s important not to try to plan everything out. I’ve come to realize that it’s best to combine a moderate amount of planning with a technique of organically growing your software.
What do I mean by growing? It’s simple, start with something small, that you know works (and that you’ve tested thoroughly) and keep adding on to that. You keep adding functionality one bit at a time, getting closer to what you finally need. This method has a two fold benefit, firstly it makes sure that you have a working program before you starting adding more functionality. That lets you rollback to a working program whenever things get out of control. Secondly, it leaves you free to add functionality and deal with complications without having to worry about fitting things into a preconceived plan.
A well conceived plan is good and essential to a successful software project, however it’s good to leave room for the eventualities that will inevitably occur, Growing software and building things on top of an already working base is a way that allows creation of software in a structured manner that leaves room for dealing eventualities.

How many programming languages should I learn?

OSNews has started a series called A-Z of programming languages where they’ve been posting interviews with the creators of well-known programming languages. Till now they’ve done AWK, Ada and BASH. Of those three the only one I’ve had any experience with is BASH, and not too much of that. But considering that there are literally hundreds of programming languages (and many more dialects or implementations) which ones should one learn to be a good programmer?

I know that many programmers out there simply learn just one or two languages and then use them throughout their careers (or at least until it becomes impossible to find a job). Certainly that works, to some point at least and so the question is, do we really need to learn languages that are not the “industry standard” (i.e. whatever has the most jobs on offer). If all you’re interested in is a job, then no, you don’t. One or two languages will probably be enough. However, if you want to keep learning and keep developing as a programmer, then the answer is most certainly yes. Some programming languages are quite similar in terms of syntax and power, but some are very different and teach you think in different ways. It’s these different languages that are going to make you better as a programmer.

So we come back to our initial question: How many languages to learn and perhaps more importantly which ones? I think that there are two types of languages that are worth learning: Those that make you think differently and those that have been used to write a large amount of high quality code. Languages like Smalltalk and LISP and to some extent Java are strongly paradigm-oriented, they emphasize a specific style of programming, in the case of Smalltalk it’s object-oriented and in the case of LISP and it’s derivatives, it’s pure functional. Such languages will teach you important lessons which you can apply even when you’re using some other languages.

One of the languages that has been widely embraced by the hacker community is C. There is a incredible amount of really good code written in C, the most famous of which is probably the Linux kernel. Unless you’re a systems programmer, you probably won’t have to use C or C++ much, but you can benefit a lot from reading well written code. C and C++ can be used to write powerful code, but sometimes the power doesn’t quite justify taking the trouble of all the low-level work that you have to do. In that case, it’s good to have a general purpose high-level language lying around. I would strongly recommend Python, but you might find another language easier for everyday use. But once you have made a choice, learn it well and use it to it’s maximum.

It might be worthwhile learning a language that has powerful text processing abilities, like AWK or Perl. It might make your work easier if you know one or both. And there is an awful lot of code written in Perl for the purpose of gluing larger programs together, so it might be worthwhile to learn it. However, Perl has been falling from grace for a good few years and many people are now using Python and Ruby to the same things they used Perl for. I don’t have a concrete opinion of Perl at the moment, but I think it’s something you can put of learning until you have an actual need for it.

You should also learn Java. I personally consider Java to be a decent language, but not a good one and I probably wouldn’t use it if I had a choice. However, it is a very popular one and if you get a job as a programmer, chances are you’ll encounter a substantial amount of Java code which you have to deal with. And you won’t be much of a programmer if you can’t deal with other people’s code. So learn Java.

Knowing basic HTML or CSS is also a good idea. You might not be a full fledged web artist, but you should be able to throw together a decent web page without much trouble. Considering the growing importance of the web, learning a web programming language is becoming important. It’s not quite a necessity yet, but I think in less than 5 years it will be. I can’t recommend one now, because I have no experience, but I think that Ruby might be a good idea, because it’s a decent general purpose language as well.

I should say, that I don’t know all of the above, but I have had some experience with each of them. I think each of them have contributed to making me a better programmer, and that the more I delve into them and use them for harder problems, I will continue to improve. In conclusion, I would like to say that if you are committed enough to learn multiple languages well, you should also invest some time in learning a powerful text editor such as Vi or Emacs. Though you can certainly write great code with nothing more than Notepad, using a more powerful tool can make your job quite a bit easier (and considerably faster). Once you turn fine-tuning these editors to suit your style and habits, you won’t want to use anything else. If you’re seriously out to become the best programmer you can be, you’re going to want the best possible tools at your disposal.

I’ll be happy to here your comments on what programming languages you might recommend to anyone looking to improve their programming.