How do I write so much? I’m glad you asked is something I think all bloggers and academics should read. I think writing is  a very good way to practice thinking and spread your ideas and this article describes both why and how you should increase your writing throughput.

Media

Google Priority Inbox video While priority inbox by itself is a very useful feature for people dealing with lots of email, the video itself is a work of art. It’s really well made and I found it both entertaining and informative

Software

Google Priority Inbox is probably best thought of as a spam filter in reverse. Instead of removing spam from your inbox, it places important email in a “priority inbox” that shows up as a section on top of your main inbox. If you get lots of email a day this is for you. I suggest changing the settings so that important emails are shown in the priority inbox even after they’re read. That way it’s easier for you to find important mail after you’ve read it.

There are significant parts missing. The unit testing is very, very scant. There is almost no error handling. The previous version had a GUI which I need to port to the new API/architecture. I also want to write one more interaction mode as a proof of concept that it can support multiple, different modes. The documentation needs to be converted to HTML mode and there are some utility functions that would be helpful to have. In short, there’s a lot that needs to be done. So my question is, what version of my code is this?

I started a rewrite of this last  summer as well but never finished — a casualty classic second system effect. For a while I considered calling this version 3.0 counting the unfinished copy as 2.0. But I decided it was rather silly and so I’ve actually called it 2.0. Though it’s certainly a major major change from the last version, in some ways it’s still broken and unfinished. Is it a beta? Or a release candidate? I suppose that’s a better description. Except the additions that I want to make are more than moving it from a beta to a full release. The GUI would definitely be a point release.

In many ways the debate is purely academic and kinda pointless. As I’ve written before, software is always beta. However, releasing major and minor “versions” of software is a popular activity. In some ways it’s helpful to the user. You can tell when something’s changed significantly and when you need to upgrade. In an age where you had to physically sell software, that was a good thing to know. However, the rise of web-based software has changed that to a large extent. If you’ve been using Gmail for a while, you’ll know that it has a history of small, regular atomic improvements over time. And it’s not just Gmail, it’s most of Google’s online services. Sometimes there are major facelifts (like Google Reader a few years ago) but by and large this gradual improvement works well. Google Chrome also uses this model. Chrome is officially past version 5 now. But thanks to its built in auto update mechanism you don’t need to care (and I suspect most people don’t). Rolling releases are clearly acceptable and may just be the way software updates are going to go in the future. Of course, if you’re charging for your code you’re going to have some sort of paywall, so no, manual software updates probably won’t go away forever.

Coming back to my original question, what version did I just release? 2.0? 2.0 beta 1? 1.9.5? Honestly I don’t really care. Part of my disinterest stems from the fact that Git makes branching and merging so easy. It’s hard to care about version numbers and releases when your code is in the hands of a system that makes it so easy to spin off feature branches and then merge them back in when they’re ready. If I worked in a fully Git based team I’d just have everyone running daily merges so that everyone just automatically got the new features. In that case I wouldn’t have waited to release. The big new feature would have been pushed a week ago, the reorganization and cleanup after that and then the documentation yesterday. I’d also be sending out the later updates and addition one at a time once they were done. Everyone else uses SVN, there might still be a way to do it.

In conclusion: rolling releases are awesome. Users don’t have to worry about manually updating and automagically get new features when they’re available. Developers using a good version control system can be up-to-date with everyone else’s code. This is especially important if you’re writing developer tools (which I am): the faster and easier you can get your updates to the people making the end product the faster the end product gets developed.

PS. If you’re wondering what exactly it is I’m making, more on that later. There’s a possibility of a source release after I talk to my professor about it.

Comments on blogs are typically in the opposite order. The first comment you see will be the oldest and the most recent comment will be at the end. In some ways this makes sense. The comments are generally conversations that people are having about the post. It makes sense to have the comments in chronological order so that people can follow the conversation as it progresses and so that people read what other people have written before commenting themselves. However, this doesn’t really scale beyond a few dozen comments. There are going to be very few people who will care to follow a conversation that spans a hundred comments. The majority of people care more about expressing what they have to say than about reading in depth about the rest of the conversation. Of course this means that there is going to be repetition because these people haven’t read what’s written before.

For me there is another aspect that I think is worse than some repetition. Seeing a line of a few hundred comments I often decide not to write a comment at all. After all, my comment is going straight to the end of the line and how many people (apart from the original post author) are seriously going to take the time to go that deep to read my comment? As a writer of blog I think that is a serious issue because I don’t people staying away from from commenting because they think no one is going to read their comments.

What if blog comments were also in blog order? That way, the most recent comments are on top and readers get encouraged to write their thoughts. But this makes the repetition problem even more serious and also threatens the conversation nature of the comment system. You can’t really get in on a conversation unless you know what has been talked about before. Putting comments in blog order makes it easy to miss previous conversations and again reduces the effectiveness of comments as community builders.

Instead of making the comment the unit of organization, what if we shift focus to the conversation thread? Let’s take the most recent conversation thread and bump it to the top. Inside each thread, the individual comments are still in chronological order. This ordering has two important characteristics:

1. The newest, most active conversations rise to the top. Readers can see what others are talking about and join right in. And if there aren’t and long conversations currently going on, the latest comment is on top helping individual comments get read.
2. Each conversation is chronological encouraging readers to read through what’s been said before adding their own contribution.

The idea to emphasize the conversation over the discrete message isn’t new: it’s the main distinction that separates forums from email. It’s also the reason Gmail is so awesome: excellent support for threaded conversations. There are of course challenges to be addressed, especially in terms of UI and how to deal with threads that encapsulate divergent conversations. But these problems will only get addressed as they become more common. I think it’s about time that the web moved from the flat chronological comment system that is so popular to a richer, more useful one that plays an active role in fostering conversations and community.

Phones are designed for interruption

Phones are designed from the ground up for interruption. It all starts with a noisy ringer that will tear your mind away from whatever it is that you’re doing. That’s followed by answering the phone and dealing with whatever it is the other person is talking, which more often than not has nothing to do with whatever else you were dealing with at the moment. Even if you have a caller ID and can choose not to pick up, you still have to disconnect from the task at hand and think about who the person, why they might be calling and make the decision to pick up or not. Your concentration is practically guaranteed to be broken.

Data transfer rates are low

This might sound heartless and inhuman, but in most circumstances I’d much rather read something than listen to someone say. Given a piece of text, most people can skim over it pretty quickly and grasp the main points. You can then reread it (or just the important parts at a slower rate). But if you’re listening to someone else talk, you’re limited by their speed. If they speak too fast you can’t grasp what they’re saying and if they’re too slow it takes forever to get to the point. Not to mention the punctuations of “umm..” and “err…” that creep in. Plus actually having to write out something often forces people to rethink what it is they want to say and make it more concise.

Only one dimension

Voice communications and telephones in particular are inherently linear. You can’t skip ahead or backward when someone is talking. Most voicemail systems suck and are a horrendously bad way to see who called you or left a message. Unlike email or IM you can’t quickly glance over things to make a decision to act or put off till later. You actually have to sit down and listen to every single darn message. This isn’t really all telephone’s fault: sound is a one-dimensional medium, at least human perception of it is. People can’t listen to more than one voice at a time and make sense of it.

You have to act RIGHT NOW

I love text-based communication because it gives you a chance to think things through before actually communicating. Even if it’s just a quick email reply you at least have a few seconds to think things through before you hit “send”. But in a telephone conversation you have to act right now. If you forget to say something, you can’t just add it in a second later. You have to interrupt the other person or wait for them and remember what you wanted to say. IM also has some of these difficulties, but each party can scan the text stream on their own so you can send your message knowing that the other person will read in their time and adjust their messages accordingly.

It’s not the same as a face-to-face

Talking to a person over the phone is really not the same as talking face to face. Human communication involves a large number of non-verbal queues that get dropped when you’re limited to just voice. In contrast, written communication has developed it’s own style. It’s meant to be absorbed in a different manner using more brain power to imagine the context and surroundings of the text. If something is important enough that you need someones undivided attention: schedule a face-to-face meeting. If that’s not possible and a phone conversation is inevitable, then you make sure you set aside a particular time and stick to it.

The Disclaimer

As you might have noticed, a lot of my complaints aren’t about verbal communication, it’s about the current incarnation of the telephone. It’s possible to use a phone in a manner that mitigates these issues to a large extent. Jason Fried of 37signals has specific weekly office hours where people can call in and ask him anything they want. This eliminates the uncertainty and interrupting potential of phones. Google Voice offers a really nice modern voicemail system. even if the text transcription is somewhat buggy, at least you know who called and when and don’t have to listen through the whole queue. I would personally like things if I could plug my phone line into my computer and then just interface through some good software. Till such a day comes (or VoIP rules the world) I’ll stick to Google Voice.

Writing daily: quantity over quality

I already write a fair amount, mostly in the form of blog posts and email. But I’m also prone to slacking off terribly. I’ve gone for a week at a time without writing anything substantial. Writing isn’t a day job for me, but it is something I enjoy, something I value and something I want to improve on. So I’m going to try a bit every day.

I’ve thought about doing this at various points in the past, but I’ve always agonized about the process. I would like to sit down at any computer and just write for a few minutes. But I could never decide how exactly to do it without having writing scattered all over the place. And I always knew in the back of my head that I needed to start down for an hour or so to actually write something of value.

I’ve always been a fan of quality over quantity, but for once I’m going to give it a rest. I’m going to write everyday in the hopes that the much increased throughput will produce a greater number of good works in the long run and it will also develop my writing skills (especially in terms of avoiding writer’s block and being able to switch into writing mode at the drop of a hat). When I have an extended period of time (an hour at least) I’ll write techie articles for this blog and when I have shorter snippets I’ll just dump them into documents on Google Docs.

Reading: everywhere, anytime

While I like to write, I like to read too. Unfortunately I don’t often have the time to sit down and read for a few hours at a time. On the other hand I have short bursts of time every now and then (5-10 minutes) and instead of just sitting right or looking at funny videos of cats, I want to spend that time reading. I’ve already read one book on my iPod Touch using little snippets of time here and there. Though I don’t think I’ll want to do that with all forms of literature, I can certainly do it for short pieces. I’m considering getting the Instapaper Pro app (which lets you save stuff you want to read) and offers some features like text extraction and font customization that I think will come in pretty handy.

Using both brain hemispheres

I’m going to be graduating in just over a year with two degrees: computer engineering. So yes, my left brain is going to be very well exercised. But I want my right brain to get some training too. In retrospect it might have been a good idea to pick up a studio art major, but I like what I have know.

In order to exercise my right hemisphere I’ve taken to looking at art and design. I don’t really study anything formally (though I among going to Italy over summer to study Renaissance Art) but I do observe and absorb. In particular I’ve been looking at data presentation and web design. I plan on spending some time building “blogazine“-like content on my website, probably centered about poetry and stories I’ve written before. I might even dabble in some hand-drawing (which I haven’t seriously done in years). Of course everything I do will be free for everyone to see and reuse.

Measuring my time usage

I often have days where I feel like I did a lot and didn’t really waste time, but didn’t quite accomplish much. I tried to apply the principle of “what you measure improves” by tracing all my time usage for a day. It turned out to be rather clumsy because I wanted a system where I could write things quickly and still get fairly good analytics on how I spent my time. Unfortunately paper is great for recording, but it sucks for analytics and most time tracking solutions I found were too heavy and expensive.

A few days ago I stumbled across a new webapp called Freckle which seems to hit the sweet spot between features and usability. All you do is enter a time (or use their timer bookmarklet), what project it was for and a bunch of tags and it gives you a set of fairly decent analytics. You have to pay for it and I just started a free month long trial. If I find that it actually works well, that I use it and that I’m getting more stuff done, then it’s a keeper and I’ll gladly fork over the $12 a month and wish them well.

Agile daily productivity

The agile development methodology eschews large complicated schedules and project plans in favor of smaller chunks of work, quicker feedback and review and greater flexibility. I’ve been an applying a similar system to my own daily workloads and it seems to be working, but I’ll be enforcing it better. Being a college student it makes absolutely no sense for me to have long schedules because every day brings new challenges (homework, tests, projects, random coffee drinking sessions) and any long-term plan would be shattered in a day. Instead I’m using a dual system: due dates to make sure I’m on track with my long term goals and shorter lists of daily and weekly tasks that need to be done. I’ll try to set aside large blocks of time for things like homework sets and fill in shorter blocks with reading and writing. I’m also consider doing weekly reviews but I’m not sure how much of a value that will provide to me right now.

In 30 days my free trial of Freckle will run out and that’s also when I’ll sit down, take a deep breath and see if all this actually worked or not. If it doesn’t work, I’ll try to see what it failed and see if I can fix it. Even if it didn’t work, I’m sure there will be places to tweak and improve. And though I’m tired from having written this (and from everything else I’ve done and need to) I feel pretty excited for this month.

For starters, technology, especially computer technology has permeated every aspect of our lives. And along with the computer has come the network. Within the next decade mobile broadband will become ubiquitous (at least in urban areas) meaning that we will always be connected to the full knowledge and collective intelligence of the internet. As a direct result we are all gradually becoming cyborgs: our machines, especially in the form of mobile network connected devices are becoming an inseparable part of us. Sure, we may not be jacking in with our brains as a part of the regular morning routine, but connecting to the global network of computers (and hence indirectly to everyone else using those computers) is already a routine occurrence which we don’t give a second thought.

A recent Wired article talks about how average chess players combined with the right machine assistance can beat out better human players as well as other players with better software. The key is in the human’s ability to make the most of their machine assistants: figuring out which machine results to accept, which to reject and how to ask the right questions. Our currently technology is in exactly the same position. The talent of the person using a computer or the computational power of the machine is less important than being able to combine the two properly.

Leaving chess aside, there are more practical areas where this combination of man and machine is producing great payoffs. Successful blogger and author Tim Ferriss makes no secret of the fact that he uses analytics extensively to fine-tune how his website operates and is viewed in order to maximize his earnings. In earlier days, Paul Graham created effectively the world’s first web application, Viaweb and successfully beat out better funded competitors by placing powerful tools (Common Lisp) in the hands of experienced users (himself and his team).

People my age and younger have never lived in a world when we couldn’t connect with people across the globe at the click of a mouse. All that has ever stood before us and the vast stores of information on the Internet has been a single text box with a button titled some variation of “Search”. We’re cyborgs in the sense that the use of our machines is natural and reflexive, requiring little explicit mental bandwidth. Who needs a port in the back of the skull when you have a copy of Google Hacks tucked into your brain?

Of course, not all cyborgs are made equal. Even among people my age there are both those who revel in technology and its gifts and those who would prefer to keep it at arm’s lengths. And I’m not talking about the difference between computer science graduate students and theater majors. I’m talking about the people who are content to use the Microsoft Word’s default font and paragraph spacing and those who spent hours tinkering with their websites to get things looking just right. I’m talking about the people who tweet a dozen times a day and those who log in to Facebook once a week. I’m talking about those who have three different emails and those who pull all their email into Gmail. I’m talking about… you get the point.

On the flip side there’s a careful balance between using technology to achieve a further goal (Tim Ferriss’ website tweaks) and technology for technology’s sake (the hours spent tweaking the CSS on a blog only your mum reads). The Wired article says that there is a difference between people who use technology productively and hence feel smarter and more focused and the people who seem lost and intimated by online life. I would add a third category: those who feel smarter, but really aren’t better than the baseline. Cyborgization may be becoming ubiquitous, but that doesn’t mean that it’s easy.

The growing cyborgization of our society is also the reason why I’m excited about the second coming of tablet computers: the iPad and whatever Chrome-based offering Google throws its weight behind. Take a few minutes to check out the new guided tours of the iPad and you might get a hint of what I feel. The interface is completely different from how we use computers today and I think that’s a great idea. Let’s face it: most people today don’t really need a real computer. They need basically two devices: a internet connection device and some sort of glorified typewriter/calculator for writing reports and spreadsheets. Of course the iPad doesn’t excite those of us who type hundreds of words a minute or write code for a living. That’s because we’ve already crossed the line of cyborgization: we know (or are at least trying to find out) what we can do with our machines. The iPad is for the people on the other side, those who couldn’t care less about how many cores or how much RAM they have. It’s for people who are more than willing to trade their freedom (and their wallets) for a computing experience that they can relate to better and easier. It’s for the mum who wants to snuggle up in bed with her kid and Winnie the Pooh. It’s for the people who still consider reading a newspaper in the morning a holy rite. It’s for the people who have by and large been on the outskirts of the computer technology revolutions of the last few decades. It’s for a new generation of cyborgs who stop thinking of their machines as computers and rather view them as constant, unobtrusive, electronic companions.

With some luck, my children will be growing up in a world where they are surrounded from birth by the warm embrace of the internet. For them, actually sitting down in front of a computer will be quaint and outmoded in the same way we don’t go to a landline phone to talk to someone anymore. And it will be devices like the iPad connecting remotely to powerful servers running recommendation engines and personalized search databases that will be their first connection to the world of computation. As Pranav Mistry says, people don’t really care about computation, they care about knowledge and information. We’ve been able to bring people closer to information by erasing it’s physicality and making everything available remotely. Our children will be getting that information without the burden of thinking about a browser or keyboard or URLs. For them, all sorts of data will be all around them accessible at the tap of a touchscreen (or hopefully without requiring even that).

Here’s looking forward to the Age of Cyborgs, of which we are the heralds and first citizens. We live in exciting times.

who’s done some really interesting data-driven art pieces. Yesterday I went down to Philadelphia to attend a one-day course taught by Edward Tufte: an Emeritus Professor at Stanford and author of four great books on data analysis, presentation and visualization. He’s also a great observer of user interfaces and presentations and a sworn enemy of PowerPoint, Excel and the “lowest common denominator” school of design. The course is a bit pricey ($200 for students and $380 for everyone else) but you get all four of his books and you get to attend one of the best presentations you’ve attended. Here are some of the things that I learned (to learn everything, you really need to take the course).

If you’re going to a course taught by a famous person, get there early

I was under the impression that the course would be a fairly small affair, say 50 to 100 people. Blame it on going to a liberal arts school with very small class sizes.  There turned out to be more like 400 people there and I happened to arrive just as the thing was getting started. I ended up getting a seat right at the back, though I quickly made it up to the middle. Edward Tufte is a great presenter and the way he conducts the course means that it doesn’t matter very much where you sit, but it’s nice to actually see the man as he talks. Also take something to write with.

Keep your ears, eyes and mind open

The course is presented in a rather informal way. It’s not disorganized, but there’s no simple outline either. He says a lot of simple but important things, and if you try to write down everything, you’ll be writing a lot and not really taking in what’s being talked about. Write down the important things but also pay attention. Open the books and look at the examples he points at. There’s a lot you can learn by just following along. Since you’ll have all the books you can pop them open later and refresh your memory later.

Leave your preconceptions at the door

The course teaches you a lot of things about how to think about and visualize data that go contrary to what the popular opinions. You’re not going to get much out of the course unless you go in open to give other ideas a try. Also don’t worry immediately about how you are going to apply what you learned to your problem. This will probably prevent you from getting the most out of the general principles taught. Think about and absorb the principles first and then think about the specifics. If you’re someone who keeps an eye on the internet, especially in the web and interface design worlds, you’ll also find some of his advice conflicting with what you read online (such as emphasizing content over design). Do be your own judge, but make sure you’ll judging on the basis of actual merits as opposed to hearsay and group-think.

You’ll need to deal with your problems yourself

This course isn’t about giving out prepackaged solutions. Like many high-level thinkers, Tufte is more concerned about identifying the overarching principles and then applying them, rather than focusing myopically on niche issues. He will give you some solutions (especially on regarding preparing and giving presentations) with some specifics (like how to use paper and avoid Powerpoint) but they’re templates that certainly need to be filled in with information specialized to the task. I also think that it’s important that you think about your own problem and bring your sense of creativity to the issue (without which you’ll be cloning someone else’s stuff).

Read, read, read and think. A lot.

Edward Tufte is a very well read and very intelligent man. He draws on examples from people all in all sort of fields through history (from Euclid to Feynman people you’ve probably never heard of). It’s not expected that you know everything about all the things he shows you (if you did you wouldn’t be going to the course). But if you want to understand things the way he does and come up with new ideas of your own, you’re going to want to keep reading about the things he refers to. It’s also important to keep exploring other things and actively playing around with and implementing the things you learn. And that means going out there and actually giving presentations and creating graphics based on what you’ve learned.

I’m at the stage where I can understand most of the principles that he’s talked about, but I’m not sure about how to apply them to the problems I have at hand. Some of the issues he talked about are similar to ones that I’ve had myself (and some I haven’t encountered at all). I love all the great historical examples he used and I intend to read up on them more. What I need to do now is to look harder at my own problems with Tufte’s examples as a guideline. As he said, it’s generally a good idea to take a strong model and copy it for your own purposes.

It’s occurred to me more than once that I really don’t know many great programmers. I go to a small school and most of my fellow students are about the same level as I am (or lower). Actually I don’t really know what level they are on because I don’t get to work with them as much as I would have to in order to find out. There is only one person whom I can say for certain is a better programmer than I am and I’m currently doing a project with him. My professors are good computer scientists, but I have no idea how good at programming they are.

But from what I’ve read (and what I can attest to from personal experience, to some extent) achieving excellence in any field (especially programming) requires a lot of dedication and hard work — about 10,000 hours of it. I’ve thought about whether or not it really is worth it to invest so much time in one activity, even if you really love it. I love spinning code, but I don’t want to be sitting in front of a bright screen all the time. Watching the dancers the other night, I got to see what dedicated practice can bring. They put on a great performance and everyone enjoyed it (including myself) and creating something beautiful and wondrous is a worthy cause. But I’m also interested in getting to know about what they dancers themselves think about their work. I’d love to know their feelings and emotions as they dance and have people cheer and whether they think it’s fair compensation for their commitment. And I don’t mean compensation in only the monetary sense, though that is important too.

Admittedly there is nothing in the programming world that is quite the same as a great dance performance. Our victories are more personal and what people see (and sometimes applaud us for) is often a small sliver of everything that we do. But that’s fine by me. When I solve a hard problem after a long time (my personal record is 3 hours hunting a pointer bug) or make something that I think is really cool (a recursive-descent parser for a little language), I think I feel some of the elation, satisfaction and relief that I think the dancers would have felt too. Yes, it does feel really good. As each year goes by I get better at doing what I love doing. But I rarely ever think about all the practice and experience that has gone into making me capable of whatever it is I am doing. Not too long ago, I would never have imagined myself capable of writing a UNIX shell or designing a programming language, but know I am doing both those things and it feels almost natural.

Ok, that last sentence was a lie. It doesn’t feel natural, but it feels like it’s just outside the range of being natural. When I’m doing things like that, I’m on the edge of incompetence. It was hard and it was painful, but now that I know I can do it, I feel much better. In some ways, I wish such chances came more often (I think the education system for computer-related studies needs to be revamped significantly, but that’s another matter) and I know that each such experience leaves me just a little bit better. Do flamenco dancers feel the same way? Maybe. It would be interesting to find out.

Excellence is a rather strange thing in that it’s hard to achieve and the return on investment on its pursuit can be very little until you get to a certain tipping point. And then there all the people who seem to be trying really hard without getting anywhere. I’m not surprised that many people choose not to put in the investments that it takes to be excellent. As a girl I liked once told me, there are a lot of people leading average lives who are very happy about it. I guess that’s true. I’m not clear about where I stand on excellence myself. I do want to be really good at what I want to do and I fully understand that it won’t be easy. But I also don’t want to give up everything on the quest for excellence. “No sacrifice, no victory” sounds very noble and all, but there’s a tinge of recklessness that I really don’t like.

At this point, the word “balance” might seem appropriate. But that’s bullshit too. I don’t think people who are great at something got there by seeking balance. The better option is breaking the rules, or at least fracturing them. The prime example is 37signals. They’re a small company, with little VC funding who don’t give away their products for free and still make millions of dollars. And they didn’t do it by working round the clock either. They broke “rules” like working 80 hours a week and making free products and other such things. But they also knew what rules to break. They didn’t break rules about being thrifty or having a solid business plan. They might not be the paragon of excellence and they’re certainly arrogant, but they’re doing well so far.

As someone seeking excellence myself, I’m trying to bend my own set of rules. I bend rules by taking courses out of sequence, doing independent studies where I can write lots of code and meet interesting people and actively trying to talk to people I admire. I need to put in 10,000 hours, so I build my life to provide opportunities to do just that. I really wish that there were a lot more people doing the same.

Happy hacking.

Ars Longa, Vita Brevis

As I’ve pondered before, life is short and it takes a fair amount of dedicated effort and time to come up with something beautiful and useful. With the rise of the Internet and instantaneous communications, we’re becoming a culture that is very much used to continuous streams of small information packets. The essay is becoming a holdover from the old days when having long periods of times to do nothing but sit and read was common. However, there are a number of really good essayists alive today, and a lot of them are on the Internet. There’s Paul Graham, whose essays are practically the stuff of legend for programmers. There is also Steve Yegge who seems to have retired, but left behind a fairly large collection of essay-length material (including an article on why you should write a blog). Outside the Internet there is Warren Buffet who has written long detailed letters to shareholders for the last 32 years each of which is an education in and of itself (and I can’t help but wonder how many shareholders actually read through them all).

I don’t think I’m making a mistake when I say that the essay is still alive and well today, albeit in somewhat modified forms. But the fact remains that putting out something of such length and depth takes up a lot of time and energy (not to mention the countless hours that go into accumulating the knowledge and organizing the thoughts that must flow into such a work). In many ways, writing an essay is similar to a software project. There is planning and preparation that must happen upfront, but nothing is really for certain until you sit down and start writing. Writing a good essay that other people will want to read and tell their friends about is no easier than writing good software that others will want to use.

Blog meet Essay

The blog and the essay are fundamentally different things. A blog is a magazine compared to an essay’s book. The blog as a format is great for some things: without easy blogging I probably wouldn’t be writing at all. But the rise of blogs (and accompanying software) has left the long form essay in the dark. You could simply write long articles and put them on your blog like Steve Yegge. But reverse chronological ordering really isn’t the best format for a collection of essays. For small numbers, a simple list of titles, maybe with a blurb is probably the best. Once you get to a larger number (Paul Graham for example), a simple list doesn’t cut it any more.

There is also the actual writing experience. Whenever you write a longer piece over the course of many days, you start to go back and visit the old parts. Part of it is for editing, but you also want to read what you’ve read before so that you know you’re keeping your essay coherent. Blog software doesn’t easily let you do this. I know WordPress stores revisions, but there doesn’t seem to be an easy, upfront way to see diffs of different versions against each other. I suppose a wiki could be better as an essay platform. Dokuwiki has excellent visual diff function and Writeboard also lets you compare versions.

Perhaps we do need some sort of specialized software for writing essays. Something that puts drafting editing at the center as opposed to at the edges. Personally I’ve been using Emacs with Git to get some of the same result, but I would really like to see a webapp that can do something like that. After all, there isn’t much use in writing an essay if no one is going to read it (and how better to get people to read it than to put it out on the Internet).

I, Essayist

Even though there may be no quick-and-easy publishing solution like WordPress for essays, writing an essay is far less dependent on tech tools than most other things today. Like I said before, Emacs and Git do a fairly good job together. I would like to be able to put all my drafts online with some sort of commenting system so that people can see the evolution of my essays, but I’ll settle for just being able to show a final product.

Separate from showing the essay is the mental exercise of actually sitting down and writing the essay (and then revising and editing). That’s something that I’ll have to get back into the habit of doing and will probably take time. Subject matter is also an issue, but a good starting point would be to simply expand on the themes that I cover in this blog, while making sure that people who read my blog can read my essays without getting bored (and vice versa). Expect my first essay to be on essays, sometime in the next few weeks.

What led me to flesh out this idea was Martin Fowler’s observation that the three most popular version control systems: Subversion, Git and Mercurial are all completely open source. With just a little bit of thought, I realized that there’s a company that does exactly what I propose: Apple. Apple’s dev tools like XCode and the Objective-C compilers and libraries are open source are free for everyone to download and use. The core of OS X is the open source Darwin kernel. But layered on top of this is a closed and really high quality system of interfaces and applications that make for a very appealing user experience.

First let me say that what I’m about to say isn’t a commentary on the pay-for-support model that Red Hat and Canonical use. I think that’s a perfectly legitimate model that has it’s advantages for consumers, companies and open source hackers. But I want to propose an alternative that focuses on making money from selling the software itself.

Opening Doors

Why does this make sense? Leaving aside any moral imperatives, one of the biggest reasons for open source is that you can tinker and fix things that are broken. If you see something that is not right, you can easily pop open the hood and dig around in the internals. Now, the only way that you’ll know if something is broken is if you either use it yourself or have someone tell you that it’s broken. However, in my experience it’s much easier to fix something when you see it broken for yourself. It also gives you a better idea of what the fix should be like. It’s also more likely that you’ll change a program for the better if you’re using it day in and day out. Eating your own dog food has its advantages. If you’re a programmer, you’re most likely to find bugs and missing features in tools that you use every day. So it makes sense for you as a developer to use open source tools so that there’s an easy way for you to fix or change things that in turn will help your own developer experience.

If you’re a corporation (or organization) like Apple or Microsoft then it makes sense to open source your developer tools as well. For a corporation, the benefit to open sourcing a product is that you get feedback both in terms of ideas, comments and bug reports and also in terms of real working code. What you end up with is a positive feedback loop: you make tools and release them to developers for free who then help you make even better tools. I love the lesson that 37signals has taught the world in terms of tools and basic infrastructure: Ruby on Rails is part of their basic infrastructure, but they’ve decided to open source. As DHH claims, it’s hard to make money off of basic infrastructure (unless maybe if you’re Microsoft with a huge already installed base). Better to release it into the world and benefit from the improvements that other people make to it.

Making Money

So that’s all nice and dandy for developers, but what about the poor corporation that wants to make a decent buck? The basic idea is that the number of people who aren’t developers far outweighs the number of people who are. Also the number of people who just want something that works far outweighs the number of people who want something that they can tinker with ad infinitum. Even most of us hacker types occasionally break down and just want something that we can shut off our brains and use (for me it’s Excel in my engineering labs and Photoshop for image manipulation). People will pay for convenience, quality and polish. That should be the motivating factor behind businesses who want to sell software.

To make money from selling software you first need to hire good developers and give them good tools that they customize to their needs. Fortunately, you already have great free dev tools. Now pay these people well and give them a concrete goal so that they put real polish into the products, especially in the areas that open source hackers might tend to neglect. Since you’re going to be charging money, you can afford to put some money into hiring professional designers and UI experts and make sure that your app really looks good as well as works good (something that is still sorely lacking in a lot of open source software). Of course you need other people on team too: at least a fair number of testers who are the same people who will be using your for-pay product day in, day out. You end up with a product that has been crafted by motivated developers using great tools with feedback from your target user base. Admittedly you can still end up with a crappy product that no one wants to pay for, but that doesn’t mean the idea itself doesn’t work.

Though I have no real experience making products like that, the cases in front of us are pretty clear to see. Apple and Adobe both make lots of money selling high quality consumer software and I personally consider Microsoft Office to still be a better user experience that OpenOffice.org (especially with the ribbon interface).

One of the questions I haven’t fully resolved is whether your for-pay app should be open sourced. I am tempted to say no because I can’t think of a business model where you can open source your product (code and design elements) and still expect people to pay a decent price for it. If someone with more experience than me can come up with a working model, do let me know. However, one compromise that might be fair is to open up your app a few years later once your company has a newer product. Under this model, Apple would open source all their code prior to OS X. I’m less familiar with Adobe, but I think prior to the Creative Studio editions would be a good time point. Microsoft might consider releasing the pre-NT Windows code and maybe Office prior to 2000 or 2003. That way, your business still makes money today while making a community contribution in the future. It also helps users who can get their hands on a copy of a program if they need to open a file from 10 years ago that the current version no longer supports. It frees up the company to go ahead while worrying a little less about backward compatibility. It also keeps companies on their toes because they know that any market advantage they have will be effectively erased in a few years.

The Plan Isn’t Perfect

No plan survives contact with the enemy

– General Field Marshall Helmuth von Moltke

And this plan will definitely have to be modified if and when a company adopts it. One immediate problem I can see is that inevitably there will be calls to open source all the company’s product. I don’t have a suitable response to that apart from saying that would probably cause the business plan to collapse. Also if your product’s core algorithms don’t vary much from one generation to the next, open sourcing older versions might be equivalent to shooting yourself in the foot (though if you’re expecting a handful of algorithms to bankroll your company till the heat death of the Universe, you’re probably doing it wrong).

Again, I don’t have any real world business experience (though I would like to, someday). I also don’t know any company that has a policy like this at its core. If you’re interested in starting a company like this (or know of one already), do drop me a line. I’d really like to play a part in making a world where people can get paid for writing their code and then releasing it for other people to use and improve. A business that makes money but still open sources its code (even if its a few years late) would be a great step in that direction.