Tag Archives: emacs

Software to keep your PDFs and papers organized

Posted on by

My summer research job involves a fair amount of reading. It also involves searching for papers on databases such as the ACM’s Digital Library. The only problem is that once you get to more than a few papers, it becomes really hard to figure out which paper is which. The papers often have long titles, so you really don’t want the filenames to be the paper titles. There’s also a lot of metadata associated with each paper (title, authors, where and when it was published and so on). It quickly became apparent to me that I really needed a better way to organize all the papers I was downloading and reading. The problem became even worse when I decided to start making detailed outlines and notes for the more important papers, because now I had to have someway to connect each paper to its notes.

I looked at a number of really good tools out there before making a choice. The very first program I remember looking at was Papers for the Mac. Papers is a really slick application. Not only can you collect and organize your papers, you can also read them right in Papers, take notes and send a copy to fellow researchers. It also lets you search popular databases and download papers without needing to step out into a browser. If I was a full time Mac user, then I would almost certainly be using Papers. But since most of my more scholarly is done on my Linux laptop, it’s not really an option. It’s priced at a very reasonable $42 with a 40% discount for undergraduate students. Not a bad deal at all.

The Papers interface (from the website)

The Papers interface (from the website)

I happen to do most of my scholarly work on my linux laptop and so I needed something that worked well on Linux. There is a program called gPapers for Linux which is similar to Papers. I didn’t actually check this out myself. From the screenshots it seems like a good tool, but development on it seems to have stopped for a while and I wasn’t sure if it was complete enough for daily use. My next choice was the Firefox plugin Zotero. Since it lives right in your browser, it makes it very easy to collect papers that you read on the Internet. If you get a Zotero account you can even sync your papers and notes between multiple machines. Once again, Zotero is a great tool. Not only can you add any web page to it’s library, you can also attach notes to any item in your library. The most awesome feature is that for PDFs it will automatically retrieve bibliographic information from Google Scholar. You can also export bibliographic information in a variety of formats. Zotero makes a lot of things very easy. To be honest, I haven’t entirely ruled it out yet. Perhaps the only real reason that I’m not using it right now is that the interface seems a bit cramped on my not-too-big laptop screen.

Zotero in Firefox (from the documentation)

Zotero in Firefox (from the documentation)

After trying all these out I took a look at one online reference management tool: CiteUlike. I was very interested in this as it would make sense for whenever I worked from library computers. It’s a really good resource and works very well with popular databases in terms of automatically gathering metadata. The only problem I have is that it doesn’t work if I point it directly to the PDF of the article. There is an option to store the PDF online, but this requires an upload from the local computer and not from an URL. I can understand why this is needed because many of the journals require subscriptions, but it is a bit of hassle. I’m not using it now because I really don’t need an online service at the moment. I like keeping all my papers organized locally. However, if I find myself moving about a lot once school starts again, I might seriously consider using CiteUlike more regularly. There are similar online services such as Connotea, but I haven’t tried them out myself.

So what do I use? There were a number of factors that affected my choice. Firstly, I wanted to have copies of the PDFs on my disk. I needed to know where they were and they had to be named according to some common scheme so that if I had to switch tools I could do so easily. Secondly, I had to be able to easily extract reference information as a BibTex file. I use Latex to write my own papers, so there was no compromising on that. Thirdly, there had to be a good way to edit and view notes for each PDF. Zotero was certainly the tool that came closest to meeting my needs (except perhaps for Papers). In fact, in the process of writing this review I’ve been sorely tempted to actually use it full time. And I would too, if weren’t for one almost unrelated piece of software: Org-mode for Emacs.

Org-mode is a package for Emacs that turns it into a powerful note-taking and organizational tool. It offers some really good features including very easy-to-use (and smart) show/hide allowing you to concentrate on parts of your notes. It also allows to attach tags to parts of your org files and then search and re-organize based on them. You can insert links to URLs or other files (which can be opened in Emacs or in other programs). Over the past few days I invested a few hours to make a homegrown solution based around Org-mode and Emacs.

I combine some low-tech organization with some simple scripting to make my system work. Here’s how: each project for which I need to do research has a papers and a notes directory. The name of any file that goes into either directory is of the format <publication><year>-<first author> with the appropriate extension. I also have a BibTex file that contains bibliographic information. The key for each entry is in the same format as the filename. I then have a quick Python script that matches each PDF with it’s notes file and bibliographic data and combines it into an easy to read org file with appropriate links. Since I have Emacs open all the time, whenever I need to look up a paper all I need to do is open up the org file and I get organized links to both the paper and its notes. I also attach tags to the papers’ names to make locating them easier. Here’s a screenshot of what an example org file looks like after I’ve added some tags:

Listing of papers with metadata and links in Org-mode

Listing of papers with metadata and links in Org-mode

I’m currently using this system on a project with about 15 papers and growing. It works well so far and I’m really comfortable in it. The only part of it that I don’t like is having to manually get the BibTex data. Since gathering metadata is a feature that Zotero has, I’m considering using Zotero as the ‘front-end’ of my system i.e. I use to Zotero to download and store the PDFs and their data, but I then use my Emacs-based system as the actual interface. This is something I still have to explore, but I think it could work. I should also note that I use Delicious to quickly bookmark interesting articles that pop up in searches, before going back and doing a preliminary quick scan prior to actually downloading the PDF. As usual I would love to hear any comments or suggestions you might have.

Too many formats

Posted on by

For most of last week, I’ve been evaluating various options for starting a public, personal wiki. I’ve looked at a number of solutions, both large-scale by wiki providers and homegrown using open source tools. I still haven’t made my decision and to tell the truth I am getting a little frustrated at this point. However, if there’s anything that I learned, it’s that there are way too many formats out there.

I’m not talking about large-scale industry standard formats like HD-DVD vs Blu-Ray or something on the level of paper vs digital. I’m talking about much simpler things like the variety of publishing formats and ad-hoc text formats floating around in cyberspace. The lingua franca of the internet is still HTML, at least if you want a simple website as opposed to something running on a content management system. However, if you’ve done any sort of web development, then you’ll know that building websites from scratch using HTML is not fun. It’s XML after all, and no one should have to write XML by hand.

Even without straight HTML, there’s still a ton of formats to choose form. If you want others to read something you’ve written, what do you choose? You could use a word processor format like .doc or the newer OOXML (.docx) or if you’re more of an open source fan, OpenDocument might be your thing. But it’s a bit harder to spread an office file like that. You can’t just drop it onto a webpage unless you convert it to HTML first. You could email to people, but that doesn’t scale and most people might not care enough to actually open it. Same arguments go for PDFs.

If you really  want a lot of people to read what you’ve written, you want to provide in a form that’s accessible via a plain browser, hopefully without plugins. You could use something like Scribd‘s iPaper, but I think that’s more useful if you have a complicated PDF that you want to show without people having to actually download and open the PDF. It’s a bit overkill for normal text. But we alread decided that writing HTML is bad and so like the hackers we are, we’re going to find a way around it.

You could go the CMS-route and use something like WordPress (for blogs) or MediaWiki (for wikis) or Drupal (for something more versatile). These have the advantage of having a pretty WYSIWYG interface and a whole list of administrative features. But they require you to have your own server or web space or find a free host like WordPress.com. But if you’re someone like me, you want a simpler solution that’s more under your direct control and that you can add on to later as your needs increase. The good news and bad news is that there are a bunch of simpler, human-readable (and writeable) markup formats that can be translated to HTML with fairly good results.

The good news is that these formats are simple and the HTML conversion tools are mostly open source and of good quality. The bad news is that there are a multitude of such markups, all of which are mutually incompatible. The ones that seem most popular seem to be Markdown, Textile and reStructured Text. All of them have their strengths and weaknesses and I don’t really like any of them. If you can pick one and use it right, you can have a good experience. But you can just as easily push against the boundaries of what they offer and end up being frustrated by their limitations.

The above are all markups that are meant to be translated to HTML at some point, but can be read by people directly. Though they do allow some form of structuring (in the form of HTML headings) and allow inline HTML too, they’re not really all that good for highly structured text, like when you’re trying to make outlines for scholarly papers. There are other text-based tools to do that and my personal favorite is Org-mode for Emacs. It’s a package for Emacs that turns it into a powerful outlining, note-taking and organization tool. It lets you create an outline as a series of headings and subheadings (nested many levels deep) along with plain text and normal lists. The different levels can then be hidden arbitrarily letting you take a bird’s eye view or just focus on one part. Many people use it for GTD or some other productivity system. I prefer Google Docs and Tasks for that, but Org-mode is a note taking tool unparalleled in it’s simplicity and ease of use.

Org-mode uses a simple, custom text format to actually store any notes you make. It’s also human readable, so you can easily copy/paste it into an email and share with others. But without actually using Org-mode, it’s hard to exploit the format to it’s full potential. It becomes even less ideal when it’s exported to some other format like HTML. The org-mode concept of headers don’t neatly map onto HTML headings. Org-mode encourages header nesting which looks terrible in HTML unless you carefully lay out a CSS stylesheet for it. You can use the headers more judiciously preferring to use plain lists, but that defeats the purpose of Org-mode to some extent. It seems to me that you can’t have the bost of best worlds.

At this point any self-respecting hacker will point out that I could just stop whining and start writing converters from one format to another. After all, they all convert to HTML, it can’t be that hard to convert between them, right? It’s probably not and the Pandoc system does it to some extent, but the problem isn’t with the formats themselves as much as with the tools that work on them. HTML is good for publishing normal documents, but if you have something with many levels of nesting such as an Org-mode document, you really need something that doesn’t show all the information at once. Unfortunately there isn’t an easy way to do this without resorting to JavaScript or something similar. HTML in it’s raw form is still a very static format, presentation wise and it doesn’t always scale well to complex sets of information.

I’m going to take a break for a moment and think about what an ideal system would be like. It would be based on a simple text format that was explicit about what things meant, so you didn’t need a reference. One of my complaints about Markdown is that anything indenting 4 spaces gets treated as a block of unformatted HTML placed withing <pre><code> tags. There’s no way you would guess this by looking at the plain text and it also makes list nesting and using indentation to present your text very awkward. But I digress. While you could write this text in a plain editor, the preferred way would be in an editor that supported folding and searching and all the other editing niceties. The editor would actually double as the presentation so that there would be any export or rendering step. And it will probably be on the web. The editor will be in a browser and the actual data will be on a remote server. However, unlike many web services today, export and import to a local form will be a core strength. This way you can pull the bare text out of the editor and send to others who don’t use the same tools as you without having to go through some silly registration/sign-up step.

I started this post thinking it would be about the markup, but now at the end its turned out to be about the tools as well. HTML is a great medium because it’s so easy to render and produce. However, our data needs are going beyond what a simple document-oriented format can easily supported. I don’t think Org-mode or the like will ever become the de facto standard. But the fact that a lot of very smart people choose it over mode ‘modern’ Web 2.0 stuff is a testament to the power of simple, easily editable formats. I hope we could standardize around a dual markup system that had a simple human readable form for quick writing and a more snazzy display form. I doubt that’s going to happen any time soon, so till then I’m looking for the perfect set of text tools to store my data and ultimately show it to the world. All this ties in directly to my efforts to having a wiki that’s easy to create, easy to back up and good to look at. More on that later.

Is Bespin the 21st century text editor?

Posted on by

Integrated Development Environments are one of the primary tools of our trade. But personally I think an IDE can only be a good product if it has a robust text editing component. The king of text editors is Emacs. No offense to Vi users, but the Emacs + Elisp combo is unbeatable in my opinion, for a number of different reasons, programmability being the main one. Of course Emacs does have its problems. Steve Yegge discusses these problems quite thoroughly in his post on XEmacs. What’s interesting is that he claims that if Emacs doesn’t get its act together and fix its problems, it won’t be long before a browser sprouts enough editing capabilities to eclipse Emacs. I think that time is close at hand.

Enter Bespin. It’s a Mozilla project that is still under heavy development, but there’s a public demo already available. Bespin is an online text-editing environment written entirely in open source JavaScript. It’s provides syntaz highlighting and some project management and looks really slick by making use of the HTML5 canvas element. The best way to understand what Bespin can do would be to watch their introductory video:

It may not be immediately obvious what all the fuss is about if you’re used to using a modern IDE like Eclipse. After all, it’s just a text editor, right? Yes, it is just a text editor and that’s where the beauty of it lies. Firstly, t’s programmable in the Emacs tradition. That means that you are allowed and encouraged to write your own bits of JavaScript to adapt it to work the way you want to. Secondly, it’s sports an open server API which means that you could potentially use Bespin as an editor-like interface to anything you want. This has already led to Bespin running on top of the Eclipse IDE. Eclipse acts as the server and Bespin acts as the editor interface to it. Here’s a screenshot of Bespin+Eclipse in action showing Eclipse communicating errors in the Java code that Bespin is editing.

Keep in mind that Bespin is still in a very experimental stage. However, the example above is clear evidence that Bespin is going to be a force to reckon with in the IDE sphere soon enough. I think that bringing a client-server model to IDEs is going to offer a significant productivity improvement to programmers especially as developers and companies have to wrangle increasingly complex, heavy weight codebases.

Development machines need to be real monsters nowadays to efficiently handle the demands of the software being developed. Using Bespin, developers could invest in industrial strength servers which serve as both code repositories and build farms. The developer’s machine then act simply as terminals or can run local lightweight server instances for the Bespin frontends to connect to. This means that developers can afford having cheaper machines themselves, but don’t need to wait for long compile times as they’re being outsourced to the powerful servers. Code is also centrally stored meaning that it can be pulled up a moment’s notice and worked on collaboratively. This could mean more efficient workflows for software companies.

But the biggest win would be for the developers themselves. Here is how I can see myself using Bespin: I’d have a machine sitting in a safe, well connected location running a Bespin server. This would be my main development machine so I’d trick it out with as much RAM and processor speed as I could afford. The server itself might be some form of headless Eclipse or something equivalent for whatever language I happen to be progamming in. The cool thing is that since there is an implementation agnostic server API, I could even write my own server that lashes together compilers and build tools for whatever language I happen to be using and displays results via Bespin. The server would also use a version control system like Git seamlessly as part of the storage infrastructure. I could then connect to this server either from any modern browser meaning that I wouldn’t have to drag a machine with me to write some code. Anywhere hacking would be a boon for open source developers and students. I can also imagine a Github-like service that hosts both files and their development tools so that developers can avoid even running their own Bespin servers.

So is the age of always-available, online IDEs upon us? I think not quite yet. It’s still untested territory to a large degree. Bespin itself is coming along quite nicely and is implementing some very interesting ideas. But a lot is going to depend on how well the Bespin servers work. Headless Eclipse is fine for Java, but there needs to be similarly capable systems evolving for other languages and platforms as well. The API does a good job of keeping things minimal and implementation agnostic, but it also puts the onus on server developers to make quality systems targeting their respective platforms. I’m looking forward to some good work coming out of the Bespin and related projects in the near future. I’m even seriously considering learning some JavaScript so that I can better understand how the whole thing works. For the time being though I’ll stick to Emacs, though I’ll certainly keep an eye on the Bespin mailing list.

IDEs, education and the UNIX philosophy

Posted on by

I’ve never really been a big fan of IDEs, though I can appreciate how they can help speed up the Edit-Compile-Test cycle. For almost two years now I’ve been trying to use a text editor + command line strategy. However, at the moment I’m in a position where I need to start using IDEs again. In particular my software engineering class uses KDevelop and over the summer I’ll be working on a research project developing an Eclipse plugin, so I need to get familiar with Eclipse before then.

I must say that I was initially put off by the idea of having to use Kdevelop for C++ development in my software engineering class. I was really hoping to be able to use Emacs, GCC and make full time, adding to my current level of Emacs and BASH knowledge. But I have been using KDevelop for about 2 weeks now (somewhat grudgingly I must admit), there are a number of things that KDevelop makes rather smooth. Creating an SVN repository for your project and committing to it is rather well integrated with the interface. At the same time, you can’t directly checkout out a working copy and start working on it as a project: you need to check it out manually to a directory, import the directory as a project and then edit the project options to have it be version controlled. Not exactly a very smooth workflow. KDevelop’s integration with the Qmake program to automatically create Makefiles is also a time-saver. Makefiles are files which contain instructions (used by a program called make) to build large projects containing lots of files. However it is very tedious to write these Makefiles by hand and KDevelop mostly takes that trouble away. Though I’m only just starting to explore KDevelop, I think I’ll enjoy using it.

However, on a larger scale, I’m still in two minds on the use of IDEs. I still firmly believe that IDEs should not be used as primarily teaching tools, certainly not in a course like Software Engineering, which is for people who are fairly sure that they will be computer science students. I learned about Subversion before using Kdevelop. I can understand and appreciate how KDevelop helps speed things up and more importantly when things go wrong or the interface doesn’t quite work the way I want it to, I can easily drop into a shell and fix things. However I know much less about Makefiles, since I’ve never really used them. I know enough about it to understand that there is a significant amount going on under the hood which is being kept hidden from me. If something went wrong I’d be helpless to fix it. It’s not a feeling I’m comfortable with. I really wish that we had been told about Makefiles and how they work and then been told about Qmake and how the integration with KDevelop speeds things up. At this point I really hope we learn about what goes on under the hood and do so soon.

One of the reasons that I am uncomfortable with IDEs is that I’m a strong believer in the UNIX philosophy. This school of thought as applied to programs can be stated thus:

Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.

I have some doubts about the text streams part, but I firmly believe in the first two segments. Once again coming back to education, it is important that students learn the UNIX philosophy and learn it well. They may not necessarily be writing completely separate programs for every little thing, but the need to preserve modularity in software is generally acknowledged. By learning how programming actually depends on lots of different tools each with their own functions, but with good interfaces, students can better understand how to write modular software. The IDE encourages the idea that powerful software comes in monolithic chunks. Even if the idea is modular and depends on other programs under the hood, that fact is not always obvious to the student. It is much easier to understand the power of modularity when your tools are clearly modular in nature. Consider Emacs: it is extremely modular and extremely extensible by virtue of it’s embedded Lisp interpreter. When you want to give Emacs the powers of an IDE you string together a number of Elisp packages. More importantly you are actively encouraged to write your own Elisp code to bend Emacs to your own. This flexibility is on a completely different level even when compared to modular IDEs like Eclipse.

I suppose that many of my gripes with IDEs would disappear if they really were integrated. But they’re not. I’m using a bunch of tools for my Software Engineering course: Umbrello for UML, Doxygen for code documentation extraction and Mantis for bug tracking. The only integration here is between Doxygen and KDevelop. This means that designing, implementing and then documenting a feature will mean using several different tools. Umbrello can generate code templates from my UML diagrams, but it can’t update the diagrams as I change the code. That means that when I need to write utility classes or methods, I have to put them in code and then put them in the diagrams again manually. By using an external bug tracker I need to remember to check off on the buglist when I fix something. In a truly integrated environment, the bug tracker would be part of the IDE. As a developer I could then add information to the bug report, such a link to a specific part of the code where the bug exists. Anyone looking at my code later would be able to see that there was a bug associated with that part of the code which had been fixed (or was still active). Of course, this would mean that version control would have to be really tightly integrated so that all the changes ever made could be pulled up and compared at a moment’s notice. If I need to play the juggle-the-program to do my work, I would much rather use distinct programs and tie them together with Elisp or BASH script. In such a case a common text interface would be very useful. UNIX philosophy wins again (to some extent).

Of course, I am picking on the tools I’m using and that may not be a representative sample. I’m also aware that KDevelop isn’t exactly the gold standard in IDEs. At this point some open source enthusiasts will be pulling out the do-it-yourself card: if you don’t like the state of KDevelop, write some code to fix it. An understandable argument, and normally I’m all for it. Do it yourself is a lesson well worth learning if you’re in the technology field. But in this case, I feel it is somewhat besides the point. Like I said, I’d rather just use Emacs. That’s not to say however that I won’t ever be swayed. Right now, I would willingly use an IDE if it unified the tools that I’d be using and actually sped things up.

Ultimately it comes down to the simple fact that as someone who intends to write software as a livelihood (or as part of a livelihood at any rate), I insist on using the best tools for the job. Right now that tool is Emacs + GCC (and all the supporting tools like make). If an IDE does come along that offers me the same customizability and raw power that Emacs, I would not hesitate to give it a fair chance to prove itself. This is not to say that Emacs doesn’t have some problems, but it’s better than the competition for my use. On the same note, I also wish very much that my fellow students would learn to understand what really makes a tool powerful. However considering that one of my classmates commented that he doesn’t like the terminal because of how much he has to type, I don’t think that will happen anytime soon. Education in computer science deserves a good few posts all to itself, and it has it’s own very large set of problems, but if you actually are reading this post, I think you already have enough knowledge to know when your tools don’t quite cut it. Just make sure you act on that knowledge.