Breaking out

So like a lot of people, I've dabbled in Lisp programming. Well, "dabble" might not be the right word. I've put significant time and effort into learning both Common Lisp and Scheme; while I've not truly mastered either, I've achieved a certain level of competence.

But the problem with both languages is, how do you take a program written in a language designed to be interactive and package it to be generally useful? It's great that I can start a session with some Common Lisp implementation or another and write some cool code: but in the end, it's not something anyone else is likely to use.

I got around that in Scheme by using Gambit, which allows one to compile Scheme code into C that is then compiled to a native binary. That makes legitimate command-line applications in Scheme that can be feasibly useful to more than just one's self.

But I wanted to try another tack, and decided to try using Lisp as a web-application server. I've tried installing any number of Lisp libraries and frameworks before, with mixed success. After battling for what seemed eternities with ASDF, I'd always give up.

But in some downtime at work and my evenings this week, I finally wrote a simple proof-of-concept AJAX application. It's running on my MacBook Pro, and seems to be working just fine. So I thought it would be fun to try and explain how I got all the parts to work together, in case someone else out there is having the same issues that had almost made me resign Lisp to the "cool but useless" shelf.

It might also be a worthwhile exercise to document this, in case I ever have to start again from scratch. I did rely heavily on "Lisp for the Web" by Adam Petersen (April 2008) in addition to the documentation for each library, framework, and package.

First the application stack. I'm running my little AJAX application on the following frameworks and libraries:

1. Mac OS X 10.5


2. Postgresql 8.1


3. Clozure Common Lisp (used to be OpenMCL)


4. Hunchentoot webserver


5. qooxdoo

So my database is PostgreSQL, and I'm using Common Lisp to retrieve that information, package it, and send it to the web browser. qooxdoo handles the browser-end.

Inside Lisp, I'm running several libraries/frameworks/systems to make this chain work. The most prominent are:

1. Postmodern


2. CL-JSON


3. Hunchentoot

And of course I'm doing all this work in Emacs (Aquamacs) and SLIME.

I found one of my toughest problems was getting Lisp libraries and frameworks installed. I've had a lot of trouble with this over the last several years, and I think I finally figured out what the problem is. It's not that I have some oddball Lisp configuration (although OS X + CCL does have some problems), it's that I wasn't paying attention to dependencies. ASDF does some primitive dependency management, but I think I'd been putting a lot more confidence in it than I ought to have. And I found that Lisp libraries generally have poor documentation for their dependencies. So it took me a while to track all this down, but I got there eventually.

One note is, I latched onto CL-PPCRE quite some time ago, and it's been part of my standard Lisp configuration since. So it's possible it is actually a dependency for some of what follows, and I didn't realize it.

The first task is Postmodern. My install steps for Postmodern looked something like this:

(asdf:operate 'asdf:load-op :split-sequence)
(asdf:operate 'asdf:load-op :usocket)
(asdf:operate 'asdf:load-op :md5)
(asdf:operate 'asdf:load-op :closer-mop)
(asdf:operate 'asdf:load-op :ieee-floats)
(asdf:operate 'asdf:load-op :trivial-utf-8)
(asdf:operate 'asdf:load-op :bordeaux-threads)
(asdf:operate 'asdf:load-op :postmodern)

So Postmodern had 7 dependencies I had to apply. After I loaded those libraries (in that order), Postmodern started working.

I actually tried to use CLSQL, but was unable to get it to load due to some UFFI errors. I finally gave up and used Postmodern, which I think is a fine product. The main advantage of CLSQL is wider (not just PostgreSQL) support. If anyone has CLSQL running on OS X plus Clozure, I'd be interested to know how you got it working.

CL-JSON was a lot easier:

(asdf:operate 'asdf:load-op :cl-json)

No dependencies, it just installed lickety-split. Of course, its ease of install was offset by its non-existent documentation. I followed the manual's advice and read the unit tests to figure out how to use it.

After Postmodern and CL-JSON came Hunchentoot. Hunchentoot took some scrounging to get right, but I finally got it going.

(asdf:operate 'asdf:load-op :md5)
(asdf:operate 'asdf:load-op :cl-base64)
(asdf:operate 'asdf:load-op :rfc2388)
(asdf:operate 'asdf:load-op :cl-fad)
(asdf:operate 'asdf:load-op :chunga)
(asdf:operate 'asdf:load-op :url-rewrite)
(asdf:operate 'asdf:load-op :cl-who)
(asdf:operate 'asdf:load-op :babel)
(asdf:operate 'asdf:load-op :alexandria)
(asdf:operate 'asdf:load-op :trivial-features)
(asdf:operate 'asdf:load-op :cffi)
(asdf:operate 'asdf:load-op :cl+ssl)
(asdf:operate 'asdf:load-op :hunchentoot)
(asdf:operate 'asdf:load-op :hunchentoot-test)

13 dependencies! And I'm sure CL-PPCRE is another, but it didn't get listed, as it's in my standard configuration.

In the process, I learned some things about how to organize my Lisp files, and I'm going to share my discoveries with you. My Lisp code is on my MacBook under "/Applications/Lisp". This directory contains two or three Common Lisp implementations, two or three Scheme implementations, and various supporting files and directories. I gathered all the Lisp libraries and frameworks I've downloaded, unzipped them, and put them into a single directory under there called "lib". So that directory now looks something like this:

PeevTop:~ mark$ ls -lh /Applications/Lisp/ | colrm 1 46

 MzScheme v352
 PLT Scheme v4.1
 arc2
 asdf
 bigloo
 bigloo3.1a
 ccl
 clisp-2.41
 lib
 sbcl -> sbcl-1.0.19
 sbcl-1.0.19
 scmxlate
 slatex
 slime
 snow-generic
 snow-site
 tex2page

Notice there is a directory under there called "asdf". It contains a directory called "registry", which is full of symlinks to the ASD files of all the libraries under "lib":

PeevTop:~ mark$ ls -lh /Applications/Lisp/asdf/registry/ | colrm 1 45 

 alexandria-tests.asd -> /Applications/Lisp/lib/alexandria/_darcs/pristine/alexandria-tests.asd
 alexandria.asd -> /Applications/Lisp/lib/alexandria/_darcs/pristine/alexandria.asd
 asdf-install.asd -> /Applications/Lisp/lib/asdf-install/asdf-install/asdf-install.asd
 babel-streams.asd -> /Applications/Lisp/lib/babel_0.3.0/babel-streams.asd
 babel-tests.asd -> /Applications/Lisp/lib/babel_0.3.0/babel-tests.asd
 babel.asd -> /Applications/Lisp/lib/babel_0.3.0/babel.asd
 bordeaux-threads.asd -> /Applications/Lisp/lib/bordeaux-threads/bordeaux-threads.asd
 cffi-examples.asd -> /Applications/Lisp/lib/cffi_0.10.3/cffi-examples.asd
 cffi-grovel.asd -> /Applications/Lisp/lib/cffi_0.10.3/cffi-grovel.asd
 cffi-tests.asd -> /Applications/Lisp/lib/cffi_0.10.3/cffi-tests.asd
 cffi-uffi-compat.asd -> /Applications/Lisp/lib/cffi_0.10.3/cffi-uffi-compat.asd
 cffi.asd -> /Applications/Lisp/lib/cffi_0.10.3/cffi.asd
 chunga.asd -> /Applications/Lisp/lib/chunga-0.4.3/chunga.asd
 cl+ssl.asd -> /Applications/Lisp/lib/cl+ssl-2008-11-04/cl+ssl.asd
 cl-base64.asd -> /Applications/Lisp/lib/cl-base64-3.3.2/cl-base64.asd
 cl-fad.asd -> /Applications/Lisp/lib/cl-fad-0.6.2/cl-fad.asd
 cl-json.asd -> /Applications/Lisp/lib/cl-json/_darcs/pristine/cl-json.asd
 cl-postgres.asd -> /Applications/Lisp/lib/postmodern-1.01/cl-postgres.asd
 cl-ppcre-test.asd -> /Applications/Lisp/lib/cl-ppcre-1.3.0/cl-ppcre-test.asd
 cl-ppcre.asd -> /Applications/Lisp/lib/cl-ppcre-1.3.0/cl-ppcre.asd
 cl-who.asd -> /Applications/Lisp/lib/cl-who-0.11.1/cl-who.asd
 closer-mop.asd -> /Applications/Lisp/lib/closer-mop_0.55/closer-mop.asd
 clsql-aodbc.asd -> /Applications/Lisp/lib/clsql-4.0.3/clsql-aodbc.asd
 clsql-db2.asd -> /Applications/Lisp/lib/clsql-4.0.3/clsql-db2.asd
 clsql-mysql.asd -> /Applications/Lisp/lib/clsql-4.0.3/clsql-mysql.asd
 clsql-odbc.asd -> /Applications/Lisp/lib/clsql-4.0.3/clsql-odbc.asd
 clsql-oracle.asd -> /Applications/Lisp/lib/clsql-4.0.3/clsql-oracle.asd
 clsql-postgresql-socket.asd -> /Applications/Lisp/lib/clsql-4.0.3/clsql-postgresql-socket.asd
 clsql-postgresql.asd -> /Applications/Lisp/lib/clsql-4.0.3/clsql-postgresql.asd
 clsql-sqlite.asd -> /Applications/Lisp/lib/clsql-4.0.3/clsql-sqlite.asd
 clsql-sqlite3.asd -> /Applications/Lisp/lib/clsql-4.0.3/clsql-sqlite3.asd
 clsql-tests.asd -> /Applications/Lisp/lib/clsql-4.0.3/clsql-tests.asd
 clsql-uffi.asd -> /Applications/Lisp/lib/clsql-4.0.3/clsql-uffi.asd
 clsql.asd -> /Applications/Lisp/lib/clsql-4.0.3/clsql.asd
 dcm.asd -> /Applications/Lisp/lib/elephant/_darcs/pristine/src/contrib/rread/dcm/dcm.asd
 flexi-streams.asd -> /Applications/Lisp/lib/flexi-streams-1.0.5/flexi-streams.asd
 gzip-stream.asd -> /Applications/Lisp/lib/gzip-stream/_darcs/pristine/gzip-stream.asd
 hunchentoot-test.asd -> /Applications/Lisp/lib/hunchentoot-0.15.7/hunchentoot-test.asd
 hunchentoot.asd -> /Applications/Lisp/lib/hunchentoot-0.15.7/hunchentoot.asd
 ieee-floats.asd -> /Applications/Lisp/lib/ieee-floats/ieee-floats.asd
 lml-tests.asd -> /Applications/Lisp/lib/lml-2.5.7/lml-tests.asd
 lml.asd -> /Applications/Lisp/lib/lml-2.5.7/lml.asd
 md5.asd -> /Applications/Lisp/lib/md5-1.8.5/md5.asd
 parenscript.asd -> /Applications/Lisp/lib/parenscript-20071104/parenscript.asd
...

So now all the ASDF packages I have on my system can be installed from "/Applications/Lisp/asdf/registry/".

Similarly, the packages I've written are in a registry under "~/Documents/Code/Lisp/asdf-registry". So my Lisp init file contains this:

#+:asdf (pushnew "/Applications/Lisp/asdf/registry/" 
   asdf:*central-registry* 
   :test #'equal)
#+:asdf (pushnew "/Users/mark/Documents/Code/Lisp/asdf-registry/" 
   asdf:*central-registry* 
   :test #'equal)

And now all the Lisp software that's either a dependency or tool I've gotten online or software I've written myself can be loaded into the Lisp environment without having to mess around and find it. This little piece of organization helped tremendously in getting this system working.

So once all this was installed, I was suddenly able to fire up Hunchentoot and serve out "pages". I used the default test pages for a while, then decided to get down to business.