Starting Common Lisp

I’ve been wanting to learn a Lisp for quite some time now. And by learn, I mean “be able to write a sizeable program in”. While I consider myself a language buff, the truth is that I only have real experience in a few languages: Java, C++, Python and a smattering of assembler. In any of those I would be able to write a medium size program without taking too much time or learning too much on the job. I have about 5 weeks of summer left and for 3.5 of those weeks, I have practically nothing to do. So I’ve decided to buckle down and learn some Lisp.

Choosing a Lisp

Before I got down to actually learning Lisp, I had to make a choice as to which dialect of the language I should actually use. I’ve used a bit of Scheme before (mostly as a result of getting halfway through SICP) and though I liked it, public opinion was that Common Lisp was more suited for real world development. And since I did plan on doing real world development, Common Lisp seemed the better choice. There was also Clojure which I could have considered. I really didn’t consider it because Common Lisp was good enough and it had both excellent tools and good books. I should also mention Paul Graham’s Arc language which is currently under heavy development. Arc development is going somewhat slow (but with good reason) and if Arc was ready, I would definitely have considered it. but I really wanted something that would give me a good environment today and Common Lisp gave me what I wanted

Getting Lisp (and tools)

Languages like Python, Perl and Ruby all have a single ‘reference implementation’ and some have alternate implementations (like Jython, JRuby). Common Lisp is an ANSI standard with a number of standards-compliant implementation from various vendors. There are a number of good open source Common Lisp implementations (like GNU CLISP and Steel Bank Common Lisp) as well as proprietary ones like Allegro and LispWorks. I’ve decided to use the Steel Bank Common Lisp. It’s a branch off the older Carnegie Mellon Common Lisp. It runs on a bunch of platforms and compiles down to high performance native code. The native compiler (which I’ve heard good things about) was the main reason for choosing it, plus I think the name is cool.

If you’re serious about developing in Lisp on a Linux box (as I am), your IDE of choice is almost certainly Emacs with SLIME mode. SLIME stands for Superior Lisp Interaction Mode for Emacs and it really deserves it’s name. Not only does it provide solid editing features like highlighting and indentation, it also integrates very well with whatever Lisp runtime you’re using. It allows you to dynamically evaluate definitions and expressions and send them to a Lisp process where you can interact with your code just like you would with any modern dynamic language. SLIME also provides excellent debugging tools and they’ve already proved quite useful, even though I’ve only scratched the surface.

Finding Reading Material

The internet is a great place to find information, but when it comes to learning something seriously, I really like to have a proper book, or at least a well organized tutorial. Luckily for Common Lisp there are at least two great free books. The introductory level book is Practical Common Lisp, which I’m currently using. It’s a great book for people who already have a good idea of programming and want to learn Lisp. I like to think of it as a sort of ‘Dive into Python’ but for Lisp. I’ll put up a more thorough review once I progress a bit more.

The second book I’m going to start is On Lisp by Paul Graham. It’s for people who already have a good idea of Lisp programming and want to learn more advanced techniques. I read the first chapter (which is a rehash of basic Lisp) but it makes more sense to get used to Lisp first before I try for advanced stuff.

Practice makes perfect

Computer science has a lot of theory behind it, but when it comes to something like learning a new language or framework, the only way to get good is to practice. I’m not at the point where I can write large scale programs in Lisp yet and till then I need to practice. I’ve found a really great web page that has 99 problems to be solved in Lisp which vary in difficulty and cover a large range of the things you can do in Lisp. I’m at number 20 at the moment and trying to keep a good pace.

Future plans

I eventually want to write  some server-end software in Lisp, I’m contemplating some sort of content management system. I’d like to get a start on it before summer ends, but right now I’m more concerned about just learning the ropes. I hope to work my way through Practical Common Lisp by the end of next week and start On Lisp after that. If you guys know of any other good resources or small sized Lisp projects I could get involved in, do let me know. And I’ll have more updates on my Lisp adventures soon.

The balance of the blog

I love to write. If I didn’t I wouldn’t maintain this blog or the write the multipage emails that I ocassionally send to friends. In terms of this blog, I would really like to post regularly (2-3 times a week). I get somewhat irritated when the RSS feeds I subscribe to go for a week or more at a time without an update. I also hate  it when I see posts which are quite clearly ‘filler’: little content or style, posted just for the sake of posting. I try hard not to post filler material myself, but it’s sometimes hard reconciling that with the desire to post regularly.

Good writing (and any other form of creative endeavor) needs time. A draft has to be created and then beaten into shape and polished. Sometimes the first draft needs to be scrapped or sends off tangents that need to be dealt with before the original can be completed. Of course, some of this can be mitigated by planning ahead but that’s still a time-consuming part of the process and doesn’t always result in time savings later. Sometimes ideas strike at the least convenient moment. Or if you’re using your writing as a thought exploratory vehicle like I do, then you’ll sometimes realize that you’ve been barking up the wrong tree all along and need to do a complete about face.

The process of writing a blog post

My more recent posts have eached weighed in at about 1000 words, give or take a hundred. I think that’s a good size for a post. That’s about 10 minutes of reading time and roughly an hour or so of writing time. Admittedly, I’m not a very fast typist (though my problem is more accuracy than speed), but a lot of the time is spend more in thinking about how to structure what I want to say next than in actually hitting the keys. That hour is for the first draft and though things don’t change very much from the draft to the final published version, there is still some amount of editing that goes on, as well as insertion of links and images if needed. All this takes in about half an hour of extra time. So I can put out a post in about 1.5 hours. Now that doesn’t sound like a very long time and it isn’t. Even with the other things I do, I could probably still write a post a day on average.

But I obviously don’t do that. So what’s going on? It would be easy to just put it down to the old quality-vs-quantity argument, but that glosses over a lot of pretty important details. For starters, almost everything I write is based on stuff that I’ve seen or experienced first hand. That means that if I don’t see or experience anything interesting, I can’t really write about anything interesting. The good news is that thanks to the internet, finding a constant stream of interesting things isn’t very hard at all. However, after all the consumption of interesting stuff is done, I still need to actually process everything that I’ve read and extract some coherent thoughts from them. It’s this processing that is the most interesting part of this process. Part of this happens while I’m doing other things, thoughts take root and germinate. But a lot of it happens while actually writing.

Let’s take this post as an example. The seed for this post comes from the fact earlier this week I had posted 3 articles in a row but I was a bit stuck about what to post in the later half of the week. Since I wanted to post something before the weekend, I decided to just write about my dilemma and see if it gave rise to any interesting ideas. I had started out thinking that I’d just write about the merits of quantity versus quality, but it’s becoming a deeper investigation of my writing process. I just erased a paragraph that was supposed to describe my writing process because though I thought I was describing the right process, I realized that I actually work very differently. I’m doing most of my thinking as I go along and putting it down on the screen as fast as I can, to be proofread and cleaned up later. I’m also doing some high level editing as I go along (like the mentioned deletion) in order to make sure that the post maintains some coherence while tying together the multiples ideas that I’m having.

Interestingly, one consequence of the think-as-I-write method is that post titles often become incorrect at some point between the starting and ending of a post. I started this post with the title Blog Musings: Quality or Quanity, but I think something along the lines of The Blogging Process might be more correct (though that sounds somewhat unwieldy).

Striking a balance

Coming back to the original question, how can I publish good length, content-rich posts about 2-3 times a week? The first part of the answer is time management. The process I’ve described depends on having enough time to read, think and write. Making time for all that isn’t easy, especially when school is in session and there are more immediate concerns. At that the same time, it’s important to keep my brain well stocked with processable material. It’s possible to just sit down with idea of writing a post and in a few hours do all the reading, processing, writing and editing required to create a final product. But it doesn’t always work so smoothly and even when it does, I don’t always enjoy it. It’s like fighting against a deadline and makes me feel like I’m working a 9-5 job. That’s definitely not something I want my writing to feel like. I’d much rather let the things I’ve read about soak in a for a while and do some background processing before I put finger to keyboard. Over the summer, I’ve figured out that the best way to do that is to read in the morning, just before I start work and then do my writing in the evenings.

The final piece is editing. It’s important to maintain a good quality of writing, especially since exploratory writing can often stray from the original topic. In some cases I’ve had to break up posts so that I don’t have multiple important threads running in parallel. That’s good because not only does it keep each post focussed, it also gives you more of them. A win-win solution to the original question.

In conclusion, the three key points that I’ve found important in keeping balance are:

  1. Making time for reading, thinking and writing
  2. Leaving time in between reading and writing so that unexpected ideas have a chance to come up
  3. Editing with an eye towards coherence and consistency.

I should point out that this isn’t an exhaustive list and might not work for everyone. It’s only a start towards keeping myself in check while holding myself up to proper standards. The actual writing itself is something that is quite separate from gathering all the relevant information and following a schedule. That’s something that can only be improved by a lot of practice. Writing this post has provided me some interesting insights into my own process and I hope you discover some interesting things about your own process as well. Please share your own ideas and methods in the comments.

Writing in the 21st century

The last post looked at some questions about how reading text has changed with the rise of computers and internet. There’s been a great increase in both the amount and type of reading matter, all of which is actually written by someone at some point.

I think that the major contributing factor in the increase in how much people write is the fact that it is now so easy to get your writing out to other people. Using a keyboard, you can put down over a hundred words per minute with some practice, far more than you can write with pen and paper. Once you have something written down, you can then send that to large audiences very easily, either by private communication (email or IM) or publicly as part of a mailing list/forum or a blog or website. That’s not the end of the story. Once you have something ‘out there’, people can start writing about what you’ve written. The internet has allowed large groups of people to take part in massive conversations which in their turn generate even more text and spawn other conversations.

Given so much opportunity and incentive for writing (and publishing), there are two main questions that are worth answering: how do you actually go about writing and what is it that you actually write.

How do you write in the computer age?

With all the choices of how to get your message out into the world, you have to actually choose which one to use. Not too long ago, simple static webpages were pretty much the only way to communicate on a large scale across the net. Today, with the explosion of such things as microblogging, wikis, social networks and the such, a simple web page is probably the last option you’ll consider. I personally think that’s mostly a good thing. I’d rather be thinking about content and writing style than about HTML and CSS (though I do enjoy making the ocassional web page layout).

While some of these tools (blogs and wikis) are conducive to long-form writing, many of them aren’t, no matter how much users might try. Most of my writing is done on this blog and in the form of emails (though I have a few other projects up my sleeve). I would find it very hard to do the same sort of writing I can do here if I were restricted to (for example) Facebook’s notes. Part of the reason is that WordPress offers a more feature-righ experience, but also that a blog is designed to make content presentation the prime objective and discussion or feedback secondary (even with comments). I know some bloggers use their blogs as discussion tools as well, but I feel that a forum or mailing list offers a better environment for that than a blog post.

You could apply the same argument to wikis as well. Wikipedia in particular makes discussion a background process. But some wikis such as the very useful EmacsWiki and the originial wiki at c2 make discussion an important part of the wiki. However, both of them still have strong seed articles and the discussions provide supplementary information and allow disagreements to be placed on record.

Even when it comes to private communication, there are choices. You have email of course, as well as instant messaging. But you also the have various social networks and forums each with their ad-hoc email-like messaging system (and maybe basic chat). A private Twitter account or blog can also be used to let people close to you know about things in your life if you’re not comfortable with laying bare to the world. I’m personally just an email guy, but with multiple accounts to keep things separate. I also use IM, but only for close friends.

While it’s certainly great to have all these cool toys to play with (many of which are generally helpful), an equally important question is what do I write and who gets to read it?

What do you write in the computer age?

This answer depends a lot on what sort of person you are. How much of your life and thoughts do you want to share? I’m personally comfortable with sharing a lot of my life. I have this blog and I Twitter. I do it because I want people to know who I am and I’d like there to be a public record of my thoughts, ideas and transformations over time for people in the future who might be interested (and might learn something). Many people don’t think the same way as I do, and that’s fine. I agree completely with Steve Yegge regarding the reasons you should blog, though I think any other medium is fine.

That being said, I certainly don’t write about everything I think about. If I did, I wouldn’t have time to do anything else. I have a few criteria for what I spend my time on. I put in writing what I think would be most interesting for other people to read. I also use writing as a thinking tool and so I write about whatever problem I’m facing to see if I can write my way to a solution. I don’t write much about my personal life because I think it’s not very interesting to most people and I’d get bored writing about it. Once again, other people probably have different criteria and reasons and that’s fine too, as long as you’ve given the issue serious thought. Above all, you should make sure that you write what you want to. Stephen King’s advice has been helpful for me: “you must not come lightly to blank page”.

While you should absolutely place yourself first when thinking about what to write, you should also give some thought to who’s going to read what you’ve written. Thanks to the internet, the chances of other people seeing what you’ve written have greatly increased, especially if they’re looking for it. Computers may be more secure than a paper diary, but for a determined enough seeker, no electronic hurdle is too much. Public encryption technology is pretty good quality, but its not bulletproof. On a less sinister note, there are chances that you yourself might accidentally send out something that you don’t want others to read (remember those great jokes about hitting the “reply-to-all” button). What if you’re afraid that you might accidentally put out something that will cause you great distress later? I found the answer a few weeks ago while reading Ender’s Shadow (which is a really good book that you should read). It goes something like this:

He would never, never commit his real thoughts to a readable form. He keeps his own counsel. Always. You will never find a document written by him that is not meant to be read.

Sister Carlotta, about Bean

Pretty simple, huh? Don’t put in writing something you wouldn’t want people to read. This doesn’t just hold for embarassing stuff, it’s also for stuff that you may think is ‘copyrighted’ or something along those lines. Unless you want people to actually read your stuff, don’t write it. Its ok to want to make money off you’re writing, but understand that not everyone who reads your stuff will pay and it’s best that you accept that and not let it stop you. Newspapers are having real trouble getting to grips with this fact about the computer age, but you can be smarter than that.

In conclusion

Writing in this age is different because computer technology has made distribution very easy. This basic fact is what you have to keep in mind while writing or thinking about writing. Publishing has never been easier, but that puts the onus on you as the writer to decide carefully what you want to publish and make the most of the wonders of computer technology.

Reading in the computer age

Once upon a time, computers used to be used for computation. To some extent they still are. But for the millions of people with computers on their desks and in their bags and pockets, computers are more regularly used for communication. And no matter how much we might like video conferencing or VoIP, much of that communication is textual. People read and people write. A lot. I don’t have any definite statistics, but I think it’s fairly safe to say that the average person today writes and reads a lot more than they did ever before. While there has certainly been an increase in literacy levels over time due to a number of socio-economic reasons, technology has played no small part in the growth of human informational output (and input). Though there’s lots of ways to produce and distribute information, text is still the easiest and most popular and so it’s something I want to focus on.

World Literacy levels (from wikipedia)

World Literacy levels (from wikipedia)

With the rise of blogs and dead-easy online publishing tools, it’s become easier than ever for the common man to take part in an existing conversation or to start his own. Even more recently, microblogging tools like Twitter let you take textual snapshots of your life and start mini-conversations around them. Furthermore, social network sights let you create conversations around pretty much anything that you can find online (and a lot of what you find offline). FriendFeed and Facebook in particular do good jobs of encouraging conversations. While it’s become really easy to write and publish your thoughts (which I think is a very good thing), the first problem that creates is:

Who’s going to read all the stuff that’s out there?

Certainly not us. Let’s face it, there is far more text out there than any human could ever hope to read and comprehend in a reasonable amount of time. Being the tool-building primates that we are, the solution that we’ve started to come up with isn’t surprising: we have machines to do it for us. Search engines and the many recommendation tools are all essentially reading machines that go through all the text on the web and produce a very small subset that our limited human brains to handle. Sometimes the machines don’t quite do the job, so we need some human intervention a la Digg and Reddit.

But the question is still far from fully answered. Alex Payne of Twitter takes a long, hard look at the future of feed readers and comes to the conclusion that “feed readers as we’ve known them are dying” but neither algorithmic processing nor human-social-network filtering works well enough to be a viable replacement. I tend to agree and I don’t see a solution myself.

I prefer a judicious use of both computational and human filtering. I rely on Reddit and Hacker News to provide fresh, new information sources, and then use Google Reader or Diigo to collect, index and manage my readings. However, it’s not ideal and requires more manual work than I’m comfortable with. We will certainly need better semantic technology to make sense of the growing amount of data, thoughts and opinions out there. At the same time, it’s not just a question of finding and organizing information, but also of remembering it when we need to. This brings up Question 2.

How do I remember everything I’ve read?

I read a lot. I read books for school and pleasure. I read scholarly articles for research. I read blogs to get a continual stream of interesting ideas and information. I read articles that float to the top of social news sites. I read Twitter and Facebook to keep track of what my friends are doing (and sometimes to read things they’ve read and written). Even by a conservative estimate, I would say I read an average of 5000 words a day, probably more on some days. I’d be lying if I said I remembered even a fraction of this. I can remember perhaps the name of the article, maybe where I read it, a few keywords and phrases here and there. This isn’t just a case of bad memory, (though I’ll admit my memory’s not as good as it could be), part of it is a conscious decision not to hold in my mental RAM stuff I don’t need at the moment (or in the very near future).

It would be great if I could remember even detailed summaries of what I’d written, but in the absence of really good memory and recall, outsourcing my memory is the next best step. And it’s not just me. In the old days you’d have tons of bookmarks saved in your browser or maybe even save the HTML files to disk. I did a bit of both. Today I use social bookmarking in the form of Diigo and Zotero for managing journal articles PDFs. For any form of memory system, full search is a must. Loose organization is also essential. Dynamic, multiply redundant systems such as tags and keywords are superior to static filesystem-like organization. It would be nice if this tagging were smart and automatic, but even manual tagging is very useful. Highlighting and inserting notes is also helpful if used right and in proper amounts.

The main point of all these tools is to simply to make sure that you find what you’re looking for with minimal time and effort. In some ways, there’s a redistribution of effort to save-time from find-time in the form of tagging and the like. Some tools like DevonThink go a step further and automatically extract relationships between documents. Being able to highlight sections and add notes becomes useful if you’re reading articles as part of a research project. These tools aren’t perfect and certainly won’t do your thinking for you, but they are a step in the right direction.

Such tools only make sense if there’s really something you want to do with the information that you read. Thankfully there are a lot of things you could with it. You could (and probably do) share it with other people. You can also join the conversation surrounding any piece of text, either by commenting on it (if it supports comments) or posting the article to an aggregator like Reddit and commenting there. You could also use it as the base seed for your own independent piece. Personally I view all my reading as input that will ultimately get processed into some form of writing. In many cases it gets translated into blog posts, comments or email, but in many others its still waiting for the right time.

Writing in the 21st century deserves a blog post all its own. Since I’m already over a thousand words, my observations on writing are getting pushed to the next post. I want to explore the issues related to writing in a bit more depth because I feel they are a bit more nuanced (and as a writer myself, I have a serious stake in writing in the computer age). Stay tuned.

Is code reuse a lie?

I took my first real programming class in high school. Someone up in the higher echelons of the Indian education system decided that we should all start learning Java even though it’s a perfectly terrible language for beginners. Anyways, here we are, some twenty-odd high school kids being indoctrinated in the many wonderful benefits of object-oriented programming despite Java not really being an honest-to-goodness OO language. One the things we memorized (and later spilled onto our exam papers) is that OO enables this really good, absolutely awesome thing called ‘code reuse’. I don’t remember anyone actually defining what code reuse is. I can’t really blame them because it seems rather self-explanatory, doesn’t it? You use old code instead of rewriting stuff over again. Until recently, I didn’t realize all the nuances that came with the idea of code reuse.

I had always thought that code reuse meant being able to lift code unchanged from one project and drop it into another. And I couldn’t think of any case over the last few years where I’d actually done something like that. With what experience I had with OO programming, it didn’t seem to me that reuse in that manner fell out naturally from using OO. In fact, it seemed quite the opposite. In each project, my classes were fairly specialized to the task at hand and I’d only be reusing them if I was doing a rewrite of the project along the same basic lines. It struck me that code reuse might just be yet another one of the myths that software engineering has picked up in its short, but interesting lifetime. Or as tycho garen tweeted to me : “cake is a lie. code reuse? just a little fib”.

However I didn’t want to base any broad generalizations like this on only my limited personal experience. I first tried to see if there were any published statistics regarding code reuse across large corporate projects. I found some academic papers regarding how code reuse could be encouraged, but nothing by the way of real hard data. I next turned to Arch Linux community contains a fair number of programmers. This turned out to be a really good as the resulting discussion brought up a number of points I hadn’t thought about regarding the different forms of code reuse.

1. Code Reuse between projects

This was the type of reuse that I had in mind. It seems that this sort of reuse is mostly a myth and understandably so. In most cases you want to get a project as simply as you can (though simplicity can be measured in different ways). That in turn means that your code will most likely be designed to solve the problem at hand as opposed to being suited for reuse. At the same time I think that if you have a group of similar projects, you can be careful to make sure you can use code between them. However the cance you’re going to be working on suffiiciently similar projects at the same time is pretty slim

2. Code reuse through libraries

Even though copy-and-paste reuse between projects may be a myth, there’s still a lot of code being resued out there. If it weren’t our field wouldn’t get very far. A lot of this reuse is done via libraries: collections of code that solve a particular problem and are designed to be used as part of larger programs. No matter what sort of programming, it’s almost inevitable that you’ve used some sort of third party library. This sort of reuse has been around a lot longer than object-orientation and I personally think that the appearance of OO hasn’t made library use more common or even easier. The ease with which code can be distributed and reused is very dependent on the support a particular language has for it which is orthogonal to how much it supports object-orientation (or any other paradigm for that matter).

3. Code reuse through class hierarchies

I’ve come to realize that this is probably what is actually meant by code reuse in OO languages. You don’t reuse classes between projects, but rather structure your class hierarchy so that you minimize the amount of code you have to write and try to eliminate copy-paste code duplication (which can be a nightmare to maintain). A proper design means that you can isolate common functionality and place them in superclasses leaving subclasses to deal with the variation. The code in the superclass gets reused without having been actually placed in the subclasses. Trent Josephsen gave a more detalied answer in the Arch Linux thread:

Imagine three classes A, B, C that share an interface.  The status() method is the same for all three, but the serialize() method is different in each class.  Without code reuse, you would copy identical code from A.status to B.status and C.status, which could be a problem if the code in A has a bug in it which becomes a serious problem in B or C.  With OOP-style code reuse, you derive A, B, and C from an abstract class Z which contains an implementation of status() and a declaration of serialize().  Then each child “re-uses” the code in Z without having to write it in each class.

Furthermore, imagine that you derive class AA from A, and you need to modify the serialize() method ever so slightly.  If most of the code is identical, you may be able to call A.serialize() from AA.serialize(), hence “re-using” the code in A.serialize() without copying it into AA.  You might argue this is a redefinition of “reuse”, but I think it is the generally accepted definition in OOP.

It’s possible to take this view on code reuse and implement it even if you’re not working in an object oriented languages. Functional languages with higher order functions allow just this. If you have a bunch of functions which do almost the same thing with a small amount of variation you can isolate the common part into its own higher-order function and pass in smaller functions as arguments (which each implement the different behavior).

So it turns that code reuse isn’t a lie after all, but it didn’t mean what I thought it did. Another comment that came out of the discussion was “Code reuse is a choice” which I think is a very appropriate way of putting it. You can have code reuse in a number of different ways: Using libraries, proper class or function organization or even writing your code as a set of reusable libraries tied together with some application specific code. There really is more than one way to do it and it’s up to the programmer to decide which path to take.