Shell scripting: the silent hero

In 5+ years of using Linux I’ve never really taken the time to learn shell scripting. A combination of using mostly graphical tools, Emacs and other scripting languages means that I’ve rarely found myself in a position where I’ve had to depend on a shell script. In fact the most I’ve done is writing short aliases for longer commands I use on a regular basis. When I needed to do automation I generally fell back on Python (and more recently Ruby). I prefer using a small, uniform set of tools so I tried to do as much as I could within a bunch of Python scripts.

I’ve been doing some data collection work for my thesis and normally I would have been perfectly fine using Ruby and Rake for automation. Rake is Ruby task runner, similar to good old Make but with all the power of Ruby behind it. But I needed to run the same set of tasks under different Ruby VMs (specifically MRI and JRuby). So some of the automation needed to be outside of Ruby. I currently have a shell script that uses RVM to set the Ruby VM, runs a set of Rake tasks, changes over to another VM and repeats the Rake tasks. Each VM sends output to a different file. In between each iteration of Rake tasks, the shell script drops in the iteration number into the output file for that VM. I could have had this happen inside Ruby by passing the iteration number to the Rake task, but it’s easier to put it in from the shell script.

I’m using the BASH shell, mostly because it’s what comes default on Linux boxes, but I think what I’m going to say will be true of other shells like zsh as well. Shell scripting is a powerful tool for lots of reasons. Firstly the scripting language itself is pretty full featured. It may not be as clean and “batteries included” as Python or Ruby, but there’s enough there to get a lot done. Variables, conditionals, loops and functions are all there and the syntax is pretty relaxed compared to some languages (especially when it comes to interpolating strings or commands with variables). BASH also plays to the strengths of the Unix philosophy. It works well with the small, powerful UNIX utility programs and the idea that “everything is a file”. Being able to directly run programs and then pipe the results around is an extremely flexible way to get things done. IO redirection is also very powerful, especially if you’re working with files (like I am).

Perhaps the best thing about Bash scripting (that has been inherited by Perl, Python, Ruby and the rest) is that you can get stuff done with very little knowledge of shell scripting. It’s just as easy to pick up new tricks and tools as you go along. Personally, though I had only used aliases for years I was able to move up to functions, loops and conditionals in just a few hours. I wouldn’t say I’m a proficient shell scripter, but I could certainly hold my own against some ugly automation task at this point. Admittedly having a few years experience in similar languages helped as I knew what to look for.

I’m very glad I got a chance to add shell scripting to my personal arsenal of programming tools. For me, I’m not going to sit down with a book to study shell scripting in any formal way. I’ll keep my current knowledge tucked away in my brain for future reference and pulling it out (maybe adding to it) when I need to. It’s definitely a learn-as-you-go skill for me, but one that I’m glad to have.

Sunday Selection 2011-04-09

Around the Internet

How to get a real education in College I’m going to be finishing classes in about a month. On the whole, I’ve had a good college experience and certainly learned a lot. However, there are a lot of things which are clearer in retrospect and I do wish I had more things of an organizational nature. If you’re interested in an unusual college experience to set the tone for a life of entrepreneurship, Scott Adams has some really good pointers.

C is a desert island language I like the C programming language. It’s small, tight language; austere would be a good description. It puts you in close touch with the machine and expects you know what you’re doing. It can be rather liberating if you’re used to programming in Java or some other managed, batteries-included language. This is the introduction of a long series of discussions and commentary surrounding C and its relationship to other languages.

Tron:Legacy Computer Art If you’ve seen the movie you’re familiar with the large numbers of dazzling special effects as well as the scenes of programmers interacting with computers (which are actually believable for once). Here’s a behind-the-scenes guide of how all those beautiful images were created.

From the Library

Outliers: The Story of Success I’ve had this book for about a year and I know read the first half, but I couldn’t remember if I actually finished it. So over the last few days I sat down and just read it cover to cover. Malcolm Gladwell is a really good writer and it’s a pleasure to follow his explanations and trains of thought. He provides some really compelling arguments for why success is a combination of personal character, upbringing, opportunities and cultural context. It is a bit disheartening to find out that there probably isn’t much by the way of what we consider “true genius” but it’s also uplifting to learn that hard work and dedication can have amazing payoffs — “no one who can rise before dawn 360 days a year fails to make his family rich”.


Simplenote I’m a fan on hacker-oriented, infinitely customizable tools (like Emacs) but sometimes you just want to reach for the closest tool at hand. Simplenote is a text-based simple note taking service. It’s available as a webapp as well as native apps for various platforms. Furthermore there are a growing number of third-party text editors and lists managers that are using Simplenote as a backend. I personally use the iOS app and the third-party nvALT application on my mac.

Breaking ranks

As the amount of reading I do on a daily basis has increased I’ve found some really good writers writing on really important and interesting topics. One of these people is Mandy Brown — she’s a veteran of the publishing interesting and has her hand in many pies including Typekit, A List Apart and A Book Apart. She has a very insightful (and thoughtfully curated) blog entitled A Working Library where she writes about libraries, reading, writing and how they interact with each other and society. Her latest post is about how the way we read our news is “breaking ranks” with the way it gets produced and distributed.

I don’t consider myself much of a news junkie (though a lot of the current tech articles and blogs I read daily could be considered news). I don’t have very strong opinions about the way the news conglomerates are trying to adapt nowadays (though paywalls do leave a bad taste in the mouth). However, I do agree with how Mandy identified the current situation as “breaking ranks” and why that’s really important. I believe that the most important things happen when this sort of rank-breaking takes place — when an idea or product starts moving in a direction that takes it away from what we consider its natural surroundings.

Case in point is the iPad (which I’m still agonizing over buying, by the way). I see the iPad as indicative of the way people use information breaking ranks with the way people use computers. The form factor, the app store, the interaction model everything is sharply different from what came before it and yet is more in-tune with what’s important — letting people use and interact with data and information without technology getting in the way. It’s unconventional, slightly alien and a fair number of people wish it would just go away.

Even on a personal level, progress is made when ranks get broken. Lately the way I need to work in order to get stuff done has come into conflict with the general environment I want to work in. I want to work in the sunny, spacious and generally aesthetically pleasing college library. But the library is generally filled with people and as a programmer and writer I work best in solitude so that I can concentrate without distractions. The way I want to work is breaking ranks with the way I need to work. The solution in this case is to go to the library in the morning — when it’s sunniest and yet there are few people. I can find a nice quiet spot and get work done. I carry my Chrome Netbook with Ubuntu to do my writing and some light hacking (more on that in a later post). In the afternoons and evenings I retreat to my room for music without headphones and my desk Linux machine to get to more heavy duty hacking. It’s been working out pretty well so far.

Progress and improvement, whether it’s personal or large-scale social and technical, is a combination of both slow, gradual improvements and larger quantum leaps. When situations get to breaking points small tweaks and improvements won’t do. You can’t drag print media to the Internet by just digitizing content. You can’t get a sizable increase in your productivity if you stick to your old habits and routines. When the breaking of ranks starts, you have to take equally ambitious measures to ensure that the breaking is for the better and that what comes out of the process is more than what went in.

Sunday Selection 2011-04-03

Happy April everyone! I hope you all had a fun April Fools and that you took any jokes at your expense in good spirit. Laughter is the best medicine and all that. Without further ado, here’s this weeks Selection.

Around the Internet

Why I Chose Typekit Businesses, business models and the psychology and ethics behind it all continue to interest me. This is one designer’s description of why he chooses Typekit over the other web-based type delivery services. There aren’t any long charts or big numbers, it’s more personal and honest.

The Holy Trinity In the process of making plans for actually going to graduate school, I’ve been spending some thinking about what I want to research and what motivations and goals are. Apart from the technical things I’m interested in, I’m starting to believe that what we need more than ever is a “philosophy of computation” — ideas and concepts that define computation and our relationship to it at a higher level. Robert Harper’s recent blog post is a milestone on that journey.

This Hack was Not Planned Another gem from the man, the legend, the hacker _why the luck stiff. Not matter how much we talk about agile processes and software development methodologies, sometimes we just need to sit down and churn out a neat hack. This one is for the knife-edge hacker in all of us.

From the Bookshelf

Rework When I read and reviewed this book almost exactly a year I was perhaps less than charitable. I stick by my point that it is largely the best material pulled from their blog, but after a year I’m seeing it from the eyes of someone who hasn’t recently been drinking the 37signals kool-aid non-stop. There are powerful and useful ideas distilled into a very potent form. If you’re looking to start a business (or even just a new project) but are unsure how set yourself apart from the Jones’ this book should give you some really good ideas.

Software My reading has gone up a lot in the last few months and I’ve been making a conscious effort to track everything I read. Since most of my reading is online, I’ve been using an excellent bookmarking service called Pinboard. It’s not free and it’s not overflowing with social features, but it stores and organizes your bookmarks and does it well. If you’re someone who reads a lot online and you want to keep track of what you’re reading, the $9.29 signup fee is a small fee to pay. (The price goes up based on the number of people who sign up, so hurry. It was a bit over $6 when I joined.)

Shun the non-believers

From the ever-insightful Seth Godin:

There will always be someone telling you that you’re not hip enough, famous enough, edgy enough or whatever enough. That’s their agenda. What’s yours?

Shun the non-believers.

And it’s not just for expert chefs. It’s for hackers, artists, writers, even students. Anyone who makes things, does things and wants to make a mark on the world. And that means all of us.