Creating extensible programs is hard

As part of my research work I’m building a program that needs to have a pluggable visualizer component. I would like users to be able to create and use their own visualization components so that the main output from the program can be viewed in a variety of ways: simple images, 3D graphics, maybe even sounds. My program is in Python, but I would rather node limit my users to having to write Python visualization code. Ideally, they should be able to user any language or toolkit that they prefer. The easiest way to do this (as far as I can see) is to have the visualizers to be completely separate standalone programs. The original program would save its output as a simple text file and the visualizer would then be responsible for reading the file and performing its actions. However at the same time I would like to make it easy to write Python visualizers and that wouldn’t have to supply their own file reading operations. These goals means that I need to design and implement a stable framework that can handle all this extensibility.

I only really started working on this last night, but in that time I’ve realized that this is harder than it sounds. Here are some the things that I’ve figured that I have to do:

  1. Determine whether the visualizer is a Python module or a standalone program
  2. If it’s a standalone program, save the output as a file and and then call the program with the output file as a parameter
  3. If it’s a Python module, there should be a class (or a number of classes) correspond to visualizers.
  4. The visualizer objects should have a clean API that the main program should be able to use.

While none of these tasks are impossible or require advanced computer science knowledge, they do require a considerable amount of care and planning. Firstly, the mechanism to detect whether the visualizer is a Python module should be robust and accept user input, which means that there has to be error checking and recovery. There also needs to be a stable interface between the main program and modules that are loaded.  There should be clear communication between the parts but also the modules should not be able to interfere with the main program.

Allowing third party (or even second party) code to run inside your framework is not something to be considered lightly. Malicious, or even sloppily written code can have very dangerous effects on your own code. In my case, I’ll be directly translating my programs output to API calls on the Python visualizer objects. Calling non-existent methods would throw exceptions and at the very least I need to make sure these exceptions are caught. I also need to make decisions regarding just how much information the visualizers should get. Luckily for me, I won’t be allowing the modules to be sending back any information, so that makes my job easier.

Writing an extensible program like the one I am now is an interesting experience. I’ve been interested in software engineering for a quite a while now and though I’ve written large programs before, this is the first time that I’ve made one specifically geared towards extensibility. Extensibility brings to the forefront a number of issues that other types of development can sweep under the carpet. Modularity, security, having a clean API, interface design, everything is a necessity for making a properly extensible system. Furthermore, having an extensible system means that you are never quite sure what is going to happen or how your software will be used. This being the first time that I’m making such a system, I’m going to be very careful. I’m putting more time into the design phase because I don’t want to do a rewrite partway through the project. Let’s hope that this experience proves to be a good one.

One thought on “Creating extensible programs is hard

  1. One technique that I’ve found to be very flexible for this sort of thing is to define a network interface and have a client-server architecture. This helps in several ways. One is that you can write pluggins in any language. Another is that while you might assume everything running on one machine initially later you can split the tasks (client and server) to two or more machines very easily.

    For instance, I once needed to be able to control a digital camera from multiple computers. I wrote a camera server in C++ and then the client in VB (unfortunately) which allowed a system in which any computer in the office could access any camera very easily. This wasn’t even a requirement of the project, we were simply required to have two computers be able to use one camera, but by going client-server we ended up with an n*m ability that gave us increased flexibility.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s