Amazon’s Digital Wonderland

A few weeks ago I found myself in Seattle, WA. Contrary to popular belief, it was a rather bright and sunny few days (if somewhat chilly). Here’s an obligatory picture of the Sky Needle.

Sky Needle

Anyways, on the first day there I fought a mostly losing battle against travel-induced tiredness (I was up at 4:30 in the morning) and walked around downtown for a while, somewhat zombie-like. I spent the most of the next day in one of Amazon’s new buildings attending their first ever PhD Symposium. I got to meet Amazon employees like Swami Sivasubramanian, one of the creators of Amazon’s Dynamo database, as well as fellow graduate students like Rahul Potharaju. The day was full of interesting presentations and the breaks in between were packed with lots of cool conversations. I presented my current project, Merlin(excuse the visuals) and got some good feedback. All-in-all it was a great day, I had a wonderful time and I hope Amazon keeps having more of these research Symposia.

But that’s not what this post is about. Personally, I think of Amazon as a retailer first and a technology company second. In fact, I’ve even written a post about their exemplary customer service. Even though I’ve known about EC2 for years and have used both S3 and Glacier as personal backup, the idea of Amazon as a technology company has always been at the back of my mind. In fact, it was only while attending the symposium that I really thought about the full weight of Amazon as a technology services company.

After coming home I looked up the keynote from Amazon’s recent Re:Invent conference. The keynote shows off some of their more interesting recent technology (including new EC2 instances) as well as client technologies built on top of it (including companies like Netflix and Vimeo). I also stumbled across Dave Winer’s post on Amazon’s support of static JavaScript applications and why that’s so interesting and important.

The more I think about it, the more I like Amazon. They make incredible technology, employ lots of really smart people and have a refreshingly honest and direct business model in an industry dominated by advertising and harvesting user data. From computation, to storage, to scalable DNS, Amazon offers a suite of services that’s just about stunning in its breadth. Though I’ve had little use for their services personally (apart from Glacier for backup), I can see myself extensively using their systems and technology if I was building any of type of scalable, distributed service.

Even as I write this, I’m trying to come up with excuses for trying out more of their technology. What would I build? I honestly don’t know. But looking at the range of Amazon technologies and thinking about the possibilities reminds me of the feelings I got when I first started programming and learning about computers.

In many ways, the world has changed since I started writing code about 12 years ago. I had a lot of fun writing LOGO and BASIC programs and then hacking together little Perl scripts. Today I find myself wondering what the loosely coupled services and technologies offered by Amazon and other cloud computing services enable. I wonder if the new programmers of today, still learning on primarily single-threaded, single-box computing platforms, should be encouraged to move on to the brave new world of instantly accessible, practically unlimited computing power. I wonder what we’ll achieve if we were to take distributed, connected computation as the starting point, rather than the state of the art.

As an ending note, let’s think about Microsoft. It’s become standard to talk about Google as today’s Microsoft, but I’m starting to wonder if that title doesn’t rightfully belong to Amazon. I’m not talking about monopolistic activities or questionable business practices, but rather their similarities in making computing more popular. Microsoft’s goal (ostensibly) was to put a computer in every household. Amazon, for its part, has commoditized high-powered computing and distributed systems and made them available to people with modest budgets. I suppose the more things change, the more they stay the same.

Nothing left to lose

Today’s been a productive day. Lot of code written, good conversation and a lot of reading done. I caught up with my backlog of stuff I had to read around the internet and even let myself carry on reading links. Out of everything I read, there are two things I read that I feel worth writing.

The first was about Microsoft and the recent Kin debacle. It’s a good article with lots of remarks from people inside Microsoft and it’s definitely worth a read if you’re interested in technology or business at all. The article paints a pretty grim picture of where Microsoft is going. MS is definitely not my favorite technology company (and I’ve been Windows free for a while now) but it still hurts a little to see a once-great company go so tragically wrong. As I was reading that article I kept thinking that Microsoft really needed a cold, hard reboot. A complete restructuring where they would identify their core strengths (Windows and Office), unify their various disparate projects (Mobile/Kin, all their various web efforts) and bring more developers onto their side (C#, F# and the rest of .NET). Is any of this going to happen? I don’t know. I hope so, but my gut says no, sadly.

However, when I read this next article (about weak AI applied to cars and massive data sets) one line stuck out that pretty much summed up what I thought about Microsoft and its current situation. The line was this: It turns out that, innovation, like freedom, “is just another word for having nothing left to lose.” Very fitting, yes?

I think what everyone is feeling that Microsoft is simply far too big and unwieldy to make the drastic changes that are really necessary for it to stay in the game against the likes of Apple and Google. Restructuring a large company to fend off faster moving rivals is never an easy thing to do and it’s even harder when:

  1. You’re company is violently divided politically into jealous corporate fiefdoms
  2. Everyone in the world seems to know about how bad it is

The feeling that I think is prevalent is that even though Microsoft desperately needs to make an about turn in a lot of different areas, they’re not going to. Not yet, not unless more heads have rolled and not until they’ve lost a lot more, both in terms of interesting products and shareholder value. In other words, Microsoft will have to be forced into a situation where they have nothing left to lose before they start to really make the changes that they need to.

And that is sad. Sure, I’ve bashed Microsoft before and I’m certainly not a fan of Windows, but I too want them to get their act together and become a formidable software company again. I have good friends working at Microsoft who have had good experiences and I’ve heard good things about C# and .NET (language geek that I am). If nothing else, diversity is good and the more sources of interesting technology we have, the better. So I wish Microsoft the best of luck and really, really hope that the Kin (and Courier before it) are what finally kick Microsoft into action and push themselves to get back to the top of the game. Will that actually happen? We’ll see.

Combining free and proprietary software

I was reading Martin Fowler’s unscientific agglomeration of opinion on version control tools yesterday when I came up with an idea about finding a compromise between open and closed source tools. Thinking about it, I realize that I’ve come close to getting this idea a number of times before, but never really quite reached the tipping point. Anyways, if you’re in a hurry, the basic idea is this: make developer tools (compilers, IDEs, version control tools etc.) completely free and open source. Also make underlying architecture free and open source as well (like the Linux kernel or Ruby on Rails). But you also want to make some money from software, right? To do that, anything that is not developer focused stays closed source and is charged for.

What led me to flesh out this idea was Martin Fowler’s observation that the three most popular version control systems: Subversion, Git and Mercurial are all completely open source. With just a little bit of thought, I realized that there’s a company that does exactly what I propose: Apple. Apple’s dev tools like XCode and the Objective-C compilers and libraries are open source are free for everyone to download and use. The core of OS X is the open source Darwin kernel. But layered on top of this is a closed and really high quality system of interfaces and applications that make for a very appealing user experience.

First let me say that what I’m about to say isn’t a commentary on the pay-for-support model that Red Hat and Canonical use. I think that’s a perfectly legitimate model that has it’s advantages for consumers, companies and open source hackers. But I want to propose an alternative that focuses on making money from selling the software itself.

Opening Doors

Why does this make sense? Leaving aside any moral imperatives, one of the biggest reasons for open source is that you can tinker and fix things that are broken. If you see something that is not right, you can easily pop open the hood and dig around in the internals. Now, the only way that you’ll know if something is broken is if you either use it yourself or have someone tell you that it’s broken. However, in my experience it’s much easier to fix something when you see it broken for yourself. It also gives you a better idea of what the fix should be like. It’s also more likely that you’ll change a program for the better if you’re using it day in and day out. Eating your own dog food has its advantages. If you’re a programmer, you’re most likely to find bugs and missing features in tools that you use every day. So it makes sense for you as a developer to use open source tools so that there’s an easy way for you to fix or change things that in turn will help your own developer experience.

If you’re a corporation (or organization) like Apple or Microsoft then it makes sense to open source your developer tools as well. For a corporation, the benefit to open sourcing a product is that you get feedback both in terms of ideas, comments and bug reports and also in terms of real working code. What you end up with is a positive feedback loop: you make tools and release them to developers for free who then help you make even better tools. I love the lesson that 37signals has taught the world in terms of tools and basic infrastructure: Ruby on Rails is part of their basic infrastructure, but they’ve decided to open source. As DHH claims, it’s hard to make money off of basic infrastructure (unless maybe if you’re Microsoft with a huge already installed base). Better to release it into the world and benefit from the improvements that other people make to it.

Making Money

So that’s all nice and dandy for developers, but what about the poor corporation that wants to make a decent buck? The basic idea is that the number of people who aren’t developers far outweighs the number of people who are. Also the number of people who just want something that works far outweighs the number of people who want something that they can tinker with ad infinitum. Even most of us hacker types occasionally break down and just want something that we can shut off our brains and use (for me it’s Excel in my engineering labs and Photoshop for image manipulation). People will pay for convenience, quality and polish. That should be the motivating factor behind businesses who want to sell software.

To make money from selling software you first need to hire good developers and give them good tools that they customize to their needs. Fortunately, you already have great free dev tools. Now pay these people well and give them a concrete goal so that they put real polish into the products, especially in the areas that open source hackers might tend to neglect. Since you’re going to be charging money, you can afford to put some money into hiring professional designers and UI experts and make sure that your app really looks good as well as works good (something that is still sorely lacking in a lot of open source software). Of course you need other people on team too: at least a fair number of testers who are the same people who will be using your for-pay product day in, day out. You end up with a product that has been crafted by motivated developers using great tools with feedback from your target user base. Admittedly you can still end up with a crappy product that no one wants to pay for, but that doesn’t mean the idea itself doesn’t work.

Though I have no real experience making products like that, the cases in front of us are pretty clear to see. Apple and Adobe both make lots of money selling high quality consumer software and I personally consider Microsoft Office to still be a better user experience that OpenOffice.org (especially with the ribbon interface).

One of the questions I haven’t fully resolved is whether your for-pay app should be open sourced. I am tempted to say no because I can’t think of a business model where you can open source your product (code and design elements) and still expect people to pay a decent price for it. If someone with more experience than me can come up with a working model, do let me know. However, one compromise that might be fair is to open up your app a few years later once your company has a newer product. Under this model, Apple would open source all their code prior to OS X. I’m less familiar with Adobe, but I think prior to the Creative Studio editions would be a good time point. Microsoft might consider releasing the pre-NT Windows code and maybe Office prior to 2000 or 2003. That way, your business still makes money today while making a community contribution in the future. It also helps users who can get their hands on a copy of a program if they need to open a file from 10 years ago that the current version no longer supports. It frees up the company to go ahead while worrying a little less about backward compatibility. It also keeps companies on their toes because they know that any market advantage they have will be effectively erased in a few years.

The Plan Isn’t Perfect

No plan survives contact with the enemy

– General Field Marshall Helmuth von Moltke

And this plan will definitely have to be modified if and when a company adopts it. One immediate problem I can see is that inevitably there will be calls to open source all the company’s product. I don’t have a suitable response to that apart from saying that would probably cause the business plan to collapse. Also if your product’s core algorithms don’t vary much from one generation to the next, open sourcing older versions might be equivalent to shooting yourself in the foot (though if you’re expecting a handful of algorithms to bankroll your company till the heat death of the Universe, you’re probably doing it wrong).

Again, I don’t have any real world business experience (though I would like to, someday). I also don’t know any company that has a policy like this at its core. If you’re interested in starting a company like this (or know of one already), do drop me a line. I’d really like to play a part in making a world where people can get paid for writing their code and then releasing it for other people to use and improve. A business that makes money but still open sources its code (even if its a few years late) would be a great step in that direction.

The Chrome Wars have begun

I come to work this morning and the intertubes are shaking with Google’s latest announcement: the coming of it’s Linux-based, web-oriented operating system for netbooks: Google Chrome OS. You’ve probably already read a lot of the other posts about the Chrome OS and know something about how it works. It’s an operating system at the core, but more importantly its a platform tuned to running web apps. It’s a clear signal from Google to pretty much every other operating system maker out there, including Microsoft and Apple, but also the Linux distribution providers like Red Hat and Canonical. The message is simple and clear; move over OS makers, the browser is the new application platform.

Google Chrome as the operating system for web apps

Google Chrome as the operating system for web apps

Though the reactions from around the web are mostly positive, there are some articles that are raising real issues. ZDNet Australia criticizes Chrome OS on the grounds that it will further fragment the Linux community (who will be contributing the kernel of the new OS) and a better solution would have been to join with Ubuntu which already has pushed Linux to new heights. A prediction from The Next Web makes the claim that Chrome OS will be “the beginning of the end for Ubuntu & co” and the real battle will be between Apple and Google, leaving everyone else in the dust. There’s also concern about the fact that Google already has a operating system for the web: Android, even though it’s only for mobile devices (though it has been ported to x86). Personally, I feel that these criticisms and fears surrounding Chrome contain the more interesting food for thought.

Microsoft isnt beat yet (image from Engadget)

Microsoft isn't beat yet (image from Engadget)

Chrome OS is undoubtedly going to be interesting, both in terms of technology and in terms of the market forces that it will affect. Also certain is that Google is more clearly than ever taking a swipe at Microsoft. Even though Google may have become the most powerful player in  the web sphere, the desktop operating system stronghold was undoubtedly held by Microsoft. Even many of Google’s own applications (including Chrome) target Windows as the primary platform. Microsoft is still a force to be reckoned with. Windows 7 is shaping up well and they have a few tricks up their sleeve, including a new browser project: Gazelle and even a cloud-centered operating system called Midori in the works. They also have a powerful research wing which does some really interesting work and a very big budget (which is enough for them to sit things out for a few years while they make a better product). Whether or not they will actually do so is still questionable, but lets not write them off just yet.

And there is Apple. The last few years have seen Apple’s gradual re-rise to stardom starting with the beautiful new OS X and continuing today with it’s dominance of the online music store arena and the strength of the iPhone platform. Not many people seem interested in pitting Google against Apple, especially since Apple has stayed out of the mainstream operating system and netbook markets. However, when it comes to the internet, Apple has a considerable stake. The iPhone is as much a portable internet device as it is a phone. And though it has carefully stayed away from the low-cost netbook market, it’s unlikely that they’ll sit by while Google plays its hand in the portable computer market.

Apple may the best suited to withstand Google (from The Next Web)

Apple may the best suited to withstand Google (from The Next Web)

However Apple’s strength in the current situation probably stems directly from the closed, proprietary nature of it’s technology. Apple has a reputation for both creating and support great desktop apps. Good design has always been a hallmark of software running on a Mac and most web apps are still far for matching the polish that Apple has  to offer. The user experience offered by the complete OS X operating system by virtue of the way it can tie together information across different apps is still something that web applications (even suites like Google Apps) have not matched to a large extent. I agree with the Next Web post that Apple probably has the most chance of retaining its user base as Google begins it’s foray into the operating system arena. With the iPhone they’ve shown that they’re still capable of market-shaking innovation and that will probably help them survive the coming OS wars.

One more important player in this market is Linux. Thanks in no small measure to Canonical, desktop Linux has gained some ground in the last few years. However, it’s still holding a very small piece of the desktop market. It’s a valid concern that Google’s entry into the market might eat into the Linux market share. Though it’s certainly possible, I’m not quite sure if this will come to pass. A lot will depend on how easy it is to get things working on Chrome OS besides the browser and web apps. What new webapps have to offer will also be influential. I personally have never been very hopeful of Linux’s position on the consumer desktop. It’s great for hacker-types like me, but I’m still not fully convinced if I would recommend it for everybody. In my opinion, most Linux desktop apps still lack some amount of external polish. That being said, I wouldn’t recommend Vista either. I do think that OS X is the best OS for most users. I don’t see Chrome as contributing to the ‘fragmentation’ of the Linux distribution scene because I expect it to be very different from traditional distros, but in this case, only time will tell.

So what can we expect in the months to come before and after Chrome OS hits the markets? Undoubtedly Google’s announcement will cause the other big players in the field to sit up and take notice. I think this move might consider other companies, especially Microsoft to push out web-centric products sooner than they otherwise would have. Google is clearly looking to shake things up in the near future and it would be folly not to plan to do something about it. However, it’s also worth keeping in mind that Chrome OS is still some time away and there is a lot of work to be done — Chromium works on Linux, but only just.

yodaBefore we make and declarations about drastic change in the OS market, it would be prudent to wait and watch and see what Chrome OS actually looks like when it releases. There is also the fact that Google will have to get people to actually use it and that may be easier said than done (considering the fact that most netbooks run Windows XP). Of course, as the iPhone has shown, there is room in the market for a sleek new product if it is made right. I will be interested in seeing how Chrome OS turns out, but I certainly won’t be giving up my Linux laptop or my Mac Mini anytime soon. I wish Chrome OS luck and hope to see some good ideas being implemented. As Yoda would say, ‘Begun the Chrome Wars have’, I’m not ready to pick sides just yet.

Using Latex for college papers

A few months ago I posted about how I was tired of Microsoft Word and was looking for an alternative. I had decided on Latex as a substitute because it was a plain-text format that easily translated to PDF. It also has great tools including Mactex for OS X and the Auctex package for Emacs. This semester in college has been the most writing intensive so far. I have an Engineering Professionalism and Ethics course that has regular memos, reading summaries and four longer papers (and planning sheets for most of the above). My digital circuits course also requires regular lab memos. I’m glad to say that I’ve been using Latex full time for all my writing.

During the past few months using Latex full-time, I’ve learned a few lessons which might be useful for anyone starting to use it for full time writing. Here’s a quick guide to get you on your feet with Latex for daily college life.

1. Get Latex.

First order of business: head over to the Latex website and find yourself a package that you like. If you’re on a Mac I highly recommend the Mactex distribution. It’s somewhat heavy, but certainly worth it as it will have you all set to get going. For Linux, it’s best to find a package for your distribution, but if you’re an Emacs user, the Auctex package will come in handy.

2. Find a good tutorial

I’ll admit, Latex can be a bit overwhelming for beginners. So it’s important to find a good tutorial. The official documentation page is a good starting point. I also found Getting to Grips with Latex to be a very good jump start guide. Once you start feeling comfortable, be sure to use Google to find tips and tricks to achieve what you want. With its long history, chances are someone has solved the problem you are facing. On the same note, you probably want to just learn as you go along instead of reading one of the tutorials chapter by chapter.

3. Build some templates

Latex can be a bit of a pain for quick documents, but it can become very effective once you start using it for larger projects. If you’re writing a lot of memos or a number of papers with the same general format, it would be best to spend some time and create a Latex format that you’re comfortable with. I’m currently using two such templates: One for memos which is set up with To:/From: fields and all the usual trappings of a memo, and another for longer papers set up with proper spacing, title page, table of contents, bibliographies, the works. Making these templates might take some work, especially if you need custom packages like I do, but believe me, they’re worth the trouble later.

4. Keep focused on the content

Coming from the WYSIWYG world of Word, it’s very tempting to continually typeset your work to see what it looks like. Here’s a suggestion: don’t. Latex is designed to handle the looks for you so that you can focus on content. Take advantage of the fact that you don’t have to worry about how your paper looks and what it says at the same time. I find it helpful to do at least two passes on my paper. One is where I just write, without seeing what it looks like. I focus 100% on content. I care about nothing besides getting my point across. I don’t even bother about page limits at this point. Once I’ve put down everything that I need saying, I start pruning. I do a typeset to check basic things like length and to make sure that I don’t have huge solid blocks of text. I’ll then start shortening (or lengthening) to get to the limit and then alter paragraph and section boundaries to make sure that the text flows properly. However, none of this done at the cost of the quality of my writing.

5. Look over the result

One of the consequences of using a system like Latex instead of Word is that it harder to edit on the fly. So make sure that the result (probably in PDF form) is what you want it to be, (typos, formatting errors etc). I generally go through everything two to three times to make sure that everything is how it should be.

Like many powerful tools, Latex will take you some time to get used to. But once you get used to the powerful and clean efficiency that it offers, you won’t want to go back to WYSIWYG again.