My summer research job involves a fair amount of reading. It also involves searching for papers on databases such as the ACM’s Digital Library. The only problem is that once you get to more than a few papers, it becomes really hard to figure out which paper is which. The papers often have long titles, so you really don’t want the filenames to be the paper titles. There’s also a lot of metadata associated with each paper (title, authors, where and when it was published and so on). It quickly became apparent to me that I really needed a better way to organize all the papers I was downloading and reading. The problem became even worse when I decided to start making detailed outlines and notes for the more important papers, because now I had to have someway to connect each paper to its notes.
I looked at a number of really good tools out there before making a choice. The very first program I remember looking at was Papers for the Mac. Papers is a really slick application. Not only can you collect and organize your papers, you can also read them right in Papers, take notes and send a copy to fellow researchers. It also lets you search popular databases and download papers without needing to step out into a browser. If I was a full time Mac user, then I would almost certainly be using Papers. But since most of my more scholarly is done on my Linux laptop, it’s not really an option. It’s priced at a very reasonable $42 with a 40% discount for undergraduate students. Not a bad deal at all.
The Papers interface (from the website)
I happen to do most of my scholarly work on my linux laptop and so I needed something that worked well on Linux. There is a program called gPapers for Linux which is similar to Papers. I didn’t actually check this out myself. From the screenshots it seems like a good tool, but development on it seems to have stopped for a while and I wasn’t sure if it was complete enough for daily use. My next choice was the Firefox plugin Zotero. Since it lives right in your browser, it makes it very easy to collect papers that you read on the Internet. If you get a Zotero account you can even sync your papers and notes between multiple machines. Once again, Zotero is a great tool. Not only can you add any web page to it’s library, you can also attach notes to any item in your library. The most awesome feature is that for PDFs it will automatically retrieve bibliographic information from Google Scholar. You can also export bibliographic information in a variety of formats. Zotero makes a lot of things very easy. To be honest, I haven’t entirely ruled it out yet. Perhaps the only real reason that I’m not using it right now is that the interface seems a bit cramped on my not-too-big laptop screen.
Zotero in Firefox (from the documentation)
After trying all these out I took a look at one online reference management tool: CiteUlike. I was very interested in this as it would make sense for whenever I worked from library computers. It’s a really good resource and works very well with popular databases in terms of automatically gathering metadata. The only problem I have is that it doesn’t work if I point it directly to the PDF of the article. There is an option to store the PDF online, but this requires an upload from the local computer and not from an URL. I can understand why this is needed because many of the journals require subscriptions, but it is a bit of hassle. I’m not using it now because I really don’t need an online service at the moment. I like keeping all my papers organized locally. However, if I find myself moving about a lot once school starts again, I might seriously consider using CiteUlike more regularly. There are similar online services such as Connotea, but I haven’t tried them out myself.
So what do I use? There were a number of factors that affected my choice. Firstly, I wanted to have copies of the PDFs on my disk. I needed to know where they were and they had to be named according to some common scheme so that if I had to switch tools I could do so easily. Secondly, I had to be able to easily extract reference information as a BibTex file. I use Latex to write my own papers, so there was no compromising on that. Thirdly, there had to be a good way to edit and view notes for each PDF. Zotero was certainly the tool that came closest to meeting my needs (except perhaps for Papers). In fact, in the process of writing this review I’ve been sorely tempted to actually use it full time. And I would too, if weren’t for one almost unrelated piece of software: Org-mode for Emacs.
Org-mode is a package for Emacs that turns it into a powerful note-taking and organizational tool. It offers some really good features including very easy-to-use (and smart) show/hide allowing you to concentrate on parts of your notes. It also allows to attach tags to parts of your org files and then search and re-organize based on them. You can insert links to URLs or other files (which can be opened in Emacs or in other programs). Over the past few days I invested a few hours to make a homegrown solution based around Org-mode and Emacs.
I combine some low-tech organization with some simple scripting to make my system work. Here’s how: each project for which I need to do research has a papers and a notes directory. The name of any file that goes into either directory is of the format <publication><year>-<first author> with the appropriate extension. I also have a BibTex file that contains bibliographic information. The key for each entry is in the same format as the filename. I then have a quick Python script that matches each PDF with it’s notes file and bibliographic data and combines it into an easy to read org file with appropriate links. Since I have Emacs open all the time, whenever I need to look up a paper all I need to do is open up the org file and I get organized links to both the paper and its notes. I also attach tags to the papers’ names to make locating them easier. Here’s a screenshot of what an example org file looks like after I’ve added some tags:
Listing of papers with metadata and links in Org-mode
I’m currently using this system on a project with about 15 papers and growing. It works well so far and I’m really comfortable in it. The only part of it that I don’t like is having to manually get the BibTex data. Since gathering metadata is a feature that Zotero has, I’m considering using Zotero as the ‘front-end’ of my system i.e. I use to Zotero to download and store the PDFs and their data, but I then use my Emacs-based system as the actual interface. This is something I still have to explore, but I think it could work. I should also note that I use Delicious to quickly bookmark interesting articles that pop up in searches, before going back and doing a preliminary quick scan prior to actually downloading the PDF. As usual I would love to hear any comments or suggestions you might have.
Hey… For my thesis project in astrophysics I also have to organize a lot of literature. Since the outcome of this organization process has to be in html format, I really want to use org-mode, also in combination with some scripts…
Would it be possible, that you can send (or link here) your python script?
Interesting. I just started to use a similar setup: Zotero is the central database for all my papers, books etc. and annotations go into orgmode files. The “bridge” between Zotero and Emacs[1] is a BibTeX file that contains all Zotero items and is created with a perl script written by Jason Friedman[2] that exports the whole Zotero database using mozrepl[3]. As I don’t like the default BibTeX key generated by Zotero I use the “CallNumber” field to store my BibTeX keys and a slightly modified BibTeX exporter for Zotero.
To make inserting new references easy I also turn on RefTex minor mode for my orgfile[4] so I can press C-c [ to insert a new reference. So I do not use any kind of script to mess with my orgfiles. The only thing that is missing in my current setup is the link back to the pdf document (if present): The link in the org files point to the BibTeX database so I cannot open the document without switching back to Zotero, search the item and open it there. *Probably* that issue could be solved by a small script that adds an additional link to the pdf itself.
Compared with your setup I do not use an extra file for notes and bibliography information but rather link to the BibTeX entry I annotate directly in the headline (as the screenshot[5] may or may not illustrate).
[1] LaTeX as well as Orgmode
[2] http://www.personal.psu.edu/jxf35/zoterotobibtex.html
[3] http://wiki.github.com/bard/mozrepl
[4] http://www.mfasold.net/blog/2009/02/using-emacs-org-mode-to-draft-papers/#comment-569
[5] http://s7b.directupload.net/images/user/090628/nqwweuf2.png
Hi,
A bit off topic, but I’m very interested in setting up a minimalistic emacs environment to be used in conjunction with zotero, org-mode and fireforg.
From what I can see from
[5] http://s7b.directupload.net/images/user/090628/nqwweuf2.png
You somehow have network info and cpu info available on screen.
Might I ask to describe a bit your environment?
I’d be interested in testing a similar setup for myself.
Thanks a lot!
Erwin
His WM is obviously DWM
Erwin, I don’t know where that screenshot is from, it’s not my setup. Personally, I use the Awesome window manager with Emacs for text editing. I also use GNU Screen with the XFCE terminal as my terminal setup. There is a line at the bottom of Screen which has a clock and how many terminals I have open. I don’t really use a global system line of any sort because I don’t really care about how much CPU I’m using and my laptop is normally always plugged in.
@Knusper: Close, but I don’t use dwm but the Awesome window manager, too.
@Erwin: My environment is quite similar to Shrutarshi Basu’s. I use
the Awesome window manager on a Debian GNU/Linux system and the
network info, cpu, load etc. you noticed are simple widgets that run
in a widget box on the bottom of the screen. So it is not Screen[2]
that displays this information.
When I log into my graphical environment I start Emacs as a daemon and
spawn a graphical client for text editing. As you can see on the top
left corner of the screenshot I have two fixed “Tag” (i.e. virtual
desktops): one for text editing (“emacs”), one for browsing
(“www”). Awesome is configured in a way that firefox is always startet
on the www-tag and emacs on the emacs-tag, so I can switch between
these two applications with the cursor keys (while holding down the
“Windows”-key).
Depart from the integration of Zotero and Orgmode mentioned above I
make use of the org-protocol extensions and have a bookmarklet in one
of my firefox bars that triggers a org-remember action on the emacs
side and let me file a remember note with link and desription of a
website I find worth remembering.[3] Up to now I just took a quick
look on fireforg so I cannot tell anything about it.
The most delicious part of using window managers like dwm, Awesome,
XMonad etc. (=so called “tiling” window managers) is that they are
designed to be controlled by keystrokes so my usage of the mouse is
limited to browsing and working with Zotero.
[1] http://awesome.naquadah.org
[2] http://www.gnu.org/software/screen/
[3] What are normally websites that contain information on a topic I
deal with in orgmode.
@Knusper: I’ll be putting my script (and others that I’ve written) on github soon
@dmj: I get most of my papers from sources that have BibTeX info readily available so I just spend a few seconds copy pasting into an Emacs buffer and changing the key to my own format. if I start having to deal with papers that don’t come with their BibTeX I’ll seriously consider Zotero and the script you linked to. Thanks for that.
If you want PDFs, exportable BibTeX, and annotation, you should give Mendeley a try. You just feed it a directory of PDFs and it extracts the citation info for you. The new version, 0.9, also has a built-in PDF viewer that allows you to annotate the papers. It’s cross-platform desktop software, so it’ll work on PC, Mac, and your Linux box.
That’s an ingenious bit of scripting you’ve done, to be sure, but perhaps Mendeley could handle your reference management needs?
FYI – I’m working with the guys at Mendeley, so if you have any comments or suggestions about how to make it work better for you, I can pass them along to the developers.
I have tried mendeley. Good piece of software, give it a try. Found, however:
– that it is not intuitive (if possible) how to export a couple of papers into a bib file;
– mendeley categories don’t match those of bib;
– you can’t change the bib description manually;
Thank you for writing such an excellent review! I do like your org-mode method, but I don’t like the idea of manually entering BibTeX information either.
Since zotero can store attachments with each record, I wonder if it would be feasible to keep a .org notes file associated with each entry. Ideally zotero searches would also look in these notes, and when you go to read a document, you could also open the .org attachment, and zotero would launch an emacs session.
Does zotero keep these attachments as plain files somewhere, or are they placed inside its sqlite database?
I always prefer plain text and files instead of a database (I heard that zotero uses SQLite). Easy to back up, and I can sync them to Dropbox so I can access them anywhere. I also prefer BibTeX because I use LaTeX most of the time. Here is my setup:
– I use Jabref to manage a BibTeX file which contains all the papers/books references. Jabref is a very nice free software. It is written in Java (thus cross platform) and has full support for BibTeX. It can generate keys automatically (you choose the format). It can automatically search for files in a certain folder that are associated to a BibTeX entry, given that the files’ names start with the BibTeX key. Its import is not very good, so I often end up searching for BibTeX in my web browser and copy it manually to Jabref. Then I download the PDF/PS file and put it in a certain folder, rename it to the BibTeX key, then let Jabref attach it to the entry.
– I write my notes/reviews/comments of a paper in an ORG file in Emacs, save it with the same name to the same folder, and let Jabref attach it to the entry.
– On Mac OS, I use Skim to view and annotate PDF files, then save the notes in a file of the same name to the same folder, and (you guess) let Jabref attach it to the entry.
When I view an entry in Jabref, I can easily access the document and my notes/reviews by double-clicking on the files. One feature of Org-mode that I really like is links. When I write notes in Org-mode, if I need to refer to an entry in my BibTeX database, I just open the BibTeX file in Emacs, browse to the entry, press C-C l to save the location, then go back to my Org note and press C-c C-l to insert the link. Opening the link in the note will direct me to the corresponding entry in my BibTeX file. The link is dynamic: it can always find the correct entry even if the BibTeX file has changed, provided that the entry is still there. Very convenient.