Wednesday, February 25, 2009

BibTeX convertor

I've been thinking about using my LibraryThing account with BibTeX. The problem is, LibraryThing doesn't export to BibTeX format. So I wrote a kludgy convertor: this takes a CSV export file from LibraryThing and generates BibTeX entries from it. I'm sure it could be a zillion times better, but it seems to work pretty well under SBCL. It requires CL-PPCRE.

I apologize for the formatting, I'll fix it if I can figure it out.

;;; Attempt to actually process LibraryThing export CSV file
;;; Mark Peever
;;; 2009-02-16

;;; Requires:
;;; cl-ppcre

(defparameter library-thing-encoding :ISO-8859-1)

;; The CSV fields may contain commas
(defparameter line-pattern (concatenate 'STRING
"^\"(.+?)\"," ; title
"\"(.+?)\"," ; author-simple
"\"(.+?)\"," ; author
"([^,]*)," ; date
"([^,]*)," ; lcc
"([^,]*)," ; ddc
"([^,]*)," ; isbn
"\"(.+?)\"," ; publication
"([^,]*)," ; rating
"([^,]*)," ; review
"\"(.+?)\"," ; entry-date
"([^,]*)," ; copies

;; Structure to hold a book entry
(defstruct book

(defun create-bibtex-file (export-file bibtex-file)
"Create a BibTex file from a LibraryThing export file."
(write-data bibtex-file
(mapcar #'bibtex-entry (parse-export-file export-file))))

(defun load-data (export-file)
"Load a LibraryThing export file into a list of CSV lines. This is almost entirely a wrapper for 'fare-csv:read-csv-file."
(read-text-file export-file :format library-thing-encoding))

(defun write-data (bibtex-file entries)
"Write a list of BibTeX entries to a file."
(with-open-file (out-stream bibtex-file
:direction :output
:if-exists :append
:if-does-not-exist :create
:external-format :utf-8)
(loop for entry in entries
do (write-line entry out-stream)
finally (probe-file bibtex-file))))

(defun read-text-file (filename &key (format :utf-8))
"Read a text file into a list of lines. The EXTERNAL-FORMAT is passed as a keyword argument."
(with-open-file (istream filename
:direction :input
:external-format format)
(loop for line = (read-line istream nil)
while (not (null line))
collect line)))

(defun line->book (csv-line)
"Generate a BOOK object from a CSV line from the export file."
(cl-ppcre:register-groups-bind (title
(line-pattern csv-line)
(make-book :title title
:author-simple author-simple
:author author
:date date
:lcc lcc
:ddc ddc
:isbn isbn
:publication-data publication-data
:rating rating
:review review
:entry-date (remove-if-not #'safe-char? entry-date)
:copies copies)))

(defun parse-export-file (filename)
"Return a list of BOOKS parsed from the output file."
(remove-if-not #'book-p
(mapcar #'line->book (load-data filename))))

(defun safe-char? (char)
"Check whether CHAR is safe for use in a BibTex cite key"
'(#\. #\, #\; #\% #\" #\' #\) #\( #\# #\Space #\Tab #\NewLine))))

(defun cite-key (book)
"Create a cite-key for a book."
(concatenate 'string
(remove-if-not #'safe-char? (book-author book))
(remove-if-not #'safe-char? (book-title book))
(remove-if-not #'safe-char? (book-date book))))

(defun bibtex-entry (book)
"Generate a BIBTEX entry from a BOOK object."
(if (not (book-p book))
(concatenate 'STRING
(format nil "@book {~A,~%" (cite-key book))
(format nil "title={~A},~%" (book-title book))
(format nil "author=~S,~%" (book-author book))
(format nil "publisher=~S,~%"
(book-publication-data book))
(format nil "date={~A} ~%" (book-date book))

Tuesday, February 24, 2009

emacs and vi

There is an age-old debate (er, war) between Unix people who use vi and those who use emacs. I use them both daily, so it's a debate I understand. I can see both sides to this one.

I think I got a glimmer of insight into the debate this morning, and thought I'd share it. Emacs and vi represent very different software philosophies: vi is from the Unix school of thought, emacs is from the Lisp school.

The Unix school of thought is generally described by the pithy saying "Do one thing and do it well." Each utility or application on a Unix system is small and very limited in its scope. The power of Unix comes from combining and mixing these utilities to create larger single-use "applications". I am convinced that the two "commands" that most characterize Unix are ``|'' and ``tee.''

Vi is a prime example of this philosophy: it is probably the best text editor for console-based file manipulation. With a powerful regular expression engine, a script API, and a wide variety of concise commands, I am convinced vi is the best way to create or alter files on the disk. It does one thing very well.

Lisp software is different. Instead of the proverbial one thing done well, Lisp software is open-ended. It's viewed as perpetually incomplete: it can always be expanded on; it frequently involves some sort of interactive prompt at which the software itself can be changed by the end user; and meddling with it is generally expected, if not encouraged.

Emacs is Lisp software, and it shows: it's the open-ended text editor, if it can really be called a "text editor" at all. In emacs, I can write some emacs-lisp expressions in a buffer and evaluate them on the fly. Those expressions can do anything from some simple calculations to altering the behaviour of emacs itself. When several of those expressions are used together, they can be packaged as an application: an application that runs atop emacs.

For system administration work, I reach for vi. Between ubiquity (vi is on every system) and sed-compatible regular expressions, vi fits perfectly into the Unix administrator's toolkit and mode of thought.

And as a side note, let me point out that if you don't know sed and awk, you're not a complete Unix administrator. These tools are essential, and you need to learn them. Perl is great, and you need to know Perl too, but sed and awk are indispensible to Unix administration.

When I write code, I always reach for emacs. Emacs is the perfect coder's editor. Let's be honest, emacs is more IDE than editor. Emacs allows me to see my code clearly highlighted, it has sensible (and adjustable) code formatting rules, and it lets me write and run the code all in one place. I get a kick out of the discussions on Perl Monks about code formatting: I always just use emacs and let it format the code as it sees fit. It always does a good job.

And as I've been using more and more Lisp to solve my data problems, I've found emacs + SLIME + SBCL to be the only way to code.

Tuesday, February 17, 2009

New Workstation

I'm blogging from my shiny new FreeBSD 7.1 workstation. This is probably not of general interest: I'm using my blog to document how I built this workstation, as there were some hiccups along the road, and maybe some other poor soul will find it useful to know how I worked around them.

So this workstation is a Dell Optiplex 760. It's running a 3GHz Intel Core 2 Duo with 4 GB RAM and two 215 GB hard drives.

So I installed FreeBSD 7.1 from the DVD image via BitTorrent. I followed the instructions on ish and got it booting into a root filesystem on ZFS. Note that the instructions suggest first installing into a 1GB UFS partition, but I found 500MB more than enough room.

So now my filesystem looks like:

> df -h
Filesystem Size Used Avail Capacity Mounted on
tank/root 213G 165M 213G 0% /
devfs 1.0K 1.0K 0B 100% /dev
tank/tmp 213G 128K 213G 0% /tmp
tank/usr 214G 456M 213G 0% /usr
tank/usr/home 218G 4.3G 213G 2% /usr/home
tank/usr/local 216G 2.6G 213G 1% /usr/local
tank/usr/ports 216G 2.4G 213G 1% /usr/ports
tank/usr/ports/packages 215G 1.2G 213G 1% /usr/ports/packages
tank/var 213G 38M 213G 0% /var
> cat /etc/fstab
# Device Mountpoint FStype Options Dump Pass#
/dev/ad8s1b none swap sw 0 0
/dev/ad10s1b none swap sw 0 0
/dev/ad8s1a / ufs rw 1 1
/dev/acd0 /cdrom cd9660 ro,noauto 0 0

The zpool labelled "tank" is actually mirrored across both hard drives:

> sudo zpool status
pool: tank
state: ONLINE
scrub: none requested

tank ONLINE 0 0 0
mirror ONLINE 0 0 0
ad8s1d ONLINE 0 0 0
ad10s1d ONLINE 0 0 0

errors: No known data errors

I haven't yet mirrored the boot partitions between the drives, but I do have both swap partitions enabled, which I suspect is actually too much swap.

The real difficulty was in getting my dual monitor configuration to work. Without a configuration file, Xorg started the monitors in mirrored mode. After I ran
Xorg -configure
the monitors came up with much better resolution, but were still mirrored. Thus began a rather tedious "let's configure X" session that involved a great deal of Google searches and too many false leads.

The final breakthrough came when I found a post on a message board (which I can't find now) and read through the manpage for the Radeon driver a few times. Here's what I did:

  1. I started xfce4 with the automatically-created Xorg configuration.

  2. I used xrandr to discover what monitors the driver saw:

    > xrandr
    Screen 0: minimum 320 x 200, current 3200 x 1200, maximum 3200 x 1200
    TV_7PIN_DIN disconnected
    DVI-I_1/digital connected 1600x1200+0+0 367mm x 275mm
    1600x1200 60.0*+ 59.9
    1280x1024 75.0 59.9
    1152x864 74.9
    1024x768 75.1 60.0
    800x600 75.0 60.3
    640x480 75.0 60.0
    720x400 70.1
    DVI-I_1/analog disconnected
    DVI-I_2/digital connected 1600x1200+1600+0 367mm x 275mm
    1600x1200 60.0*+ 59.9
    1280x1024 75.0 59.9
    1152x864 74.9
    1024x768 75.1 60.0
    800x600 75.0 60.3
    640x480 75.0 60.0
    720x400 70.1
    DVI-I_2/analog disconnected

  3. I used xrandr to turn the monitors off and on, and to arrange them side-by-side to my liking, paying attention to the warnings it gave me.

  4. I wrote the new configuration to the Xorg configuration file:

    > cat /etc/X11/xorg.conf
    Section "Files"
    RgbPath "/usr/local/share/X11/rgb"
    ModulePath "/usr/local/lib/xorg/modules"
    FontPath "/usr/local/lib/X11/fonts/misc/"
    FontPath "/usr/local/lib/X11/fonts/TTF/"
    FontPath "/usr/local/lib/X11/fonts/OTF"
    FontPath "/usr/local/lib/X11/fonts/Type1/"
    FontPath "/usr/local/lib/X11/fonts/100dpi/"
    FontPath "/usr/local/lib/X11/fonts/75dpi/"

    Section "Module"
    Load "GLcore"
    Load "extmod"
    Load "record"
    Load "glx"
    Load "xtrap"
    Load "dri"
    Load "dbe"
    Load "freetype"
    Load "type1"

    Section "InputDevice"
    Identifier "Keyboard0"
    Driver "kbd"

    Section "InputDevice"
    Identifier "Mouse0"
    Driver "mouse"
    Option "Protocol" "auto"
    Option "Device" "/dev/sysmouse"
    Option "ZAxisMapping" "4 5 6 7"

    Section "Monitor"
    Identifier "leftMonitor"
    VendorName "Dell"
    ModelName "Dell 2007FP"
    Option "DPMS" "on"

    Section "Monitor"
    Identifier "rightMonitor"
    VendorName "Dell"
    ModelName "Dell 2007FP"
    Option "DPMS" "on"
    Option "RightOf" "leftMonitor"

    Section "Device"
    Identifier "Card0"
    Driver "radeonhd"
    VendorName "ATI Technologies Inc"
    BoardName "Unknown Board"
    BusID "PCI:1:0:0"
    Option "Monitor-DVI-I_1/digital" "leftMonitor"
    Option "Monitor-DVI-I_2/digital" "rightMonitor"

    Section "Screen"
    Identifier "Screen0"
    Device "Card0"
    Monitor "leftMonitor"
    Monitor "rightMonitor"
    DefaultDepth 24
    SubSection "Display"
    Viewport 0 0
    Depth 24
    Virtual 3200 1200

Notice I don't have two video cards listed: I simply have both monitors listed as options on the one card. Notice also the names are "Monitor-DVI-I_1/digital" and "Monitor-DVI-I_2/digital" on the card. Apparently the driver probes them as "DVI-I_1/digital" and "DVI-I_2/digital", and Xorg prepends those strings with "Monitor-".

My new setup is working perfectly now.

The one problem I haven't totally resolved is the USB mouse and keyboard. The mouse wasn't working at all (a Logitech MX Laser wireless) until I plugged it into a USB port on the back of the box. Then there was contention with the keyboard, where one or the other would spontaneously stop working. I finally got them to work together, almost consistently, by plugging them both in the back, the keyboard to the right of the mouse. I'm not 100% sure this is fixed, but it's working fairly reliably now.

I did have to twiddle some things in /etc/rc.conf, and got some help from sysinstall on that too:

> sed -n -e '/^[^#]/p' /etc/rc.conf

FreeBSD has come a long way since I started using it in the 4.0-RELEASE days. It's been a couiple years since I've messed with it, and it's not been disappointing to return. I'm looking forward to a long and productive relationship with this great operating system.

I'm not done yet: I have some Perl modules and some ASDF packages to install, as well as migrating PGP and ssh keys from my MacBook. But we're well on the way here.

And I'm having a blast.

Saturday, February 7, 2009

Lying for Mother Earth

For various reasons, I ride the bus and/or my bike to work most of the time. The idea is actually to phase out the bus and ride just my bike sometime this spring, but it's still a little cool and wet for that just now.

The university where I work is part of a county-wide campaign to reduce single-passenger cars on the roads (largely in the name of going green), and so we're all encouraged to log our trips to/from work with Pierce Trips Commute Calendar, apparently a county-run logger to track one's travels in non-single-driver commutes. This calendar makes some calculations of how much gas, money, and emissions you save over 'if you had driven alone.' There are incentives and prizes given out, both by the school and the county based on people's participation in the program. HR encourages the use of the calendar, which is fine by me.

I started using the calendar sometime in November.

You can actually see their calculations on a "Results" page, so every once in a while I take a look. I'm always struck by the patent dishonesty of their calculations. Here are my results as of today:

You've entered 32 trips since November 19, 2008
You eliminated 659.2 drive-alone miles.

You prevented the following from being emitted into the environment*:
535.78 pounds of carbon dioxide (contributes to global warming)
1.97 pounds of hydrocarbons (contributes to smog)
18 pounds of carbon monoxide (poisonous gas)

You saved 27.49 gallons of gasoline and $116.83 dollars*.

*Statistics based on average emissions for passenger cars and average gas price of $4.25.

So let's examine these numbers.

  1. Not every trip has been correctly entered, as I occasionally forget to do so until after I have forgotten details. So I probably actually saved a good deal more. But that's not the calendar's fault.

  2. The mileage is correct, according to Google Maps. Actually, my bike rides are longer than my bus and car rides (the direct route is not safe on a bike), but I log them as the same distance, because it's how many miles I saved, not how many I rode. So I've "saved" 659.2 miles.

  3. They calculate I've saved $116.83 dollars (sic). Let's ignore the redundancy for now, and examine this number. It's "based on... average gas price of $4.25". Gas hasn't cost $4.25/gallon since long before November 2008. It hung around $1.70/gallon for several weeks, and is now somewhere less than $2.50/gallon. So they're basing the calculation on some fantasy gas price.

  4. The "savings" in gas are based on $4.25/gal * 27.49 gal = $116.83. My calculator agrees with them, although as we've already seen, the actual average gas price is about half the number they calculate. But let's not be hasty: bus fare is $1.75/trip. So those 32 trips theoretically cost me 32 * $1.75 = $56.00. So when I subtract my bus fare, I actually "saved" only $60.83. Hmmm... not such a great savings. And if we correct for a more realistic gas price, it looks like I actually only came out ahead about $12.73.

  5. I have a bus pass, which lists at $63 / month. That's cheaper than paying regular fare, but it still means I paid (theoritically) $189 for November--January. So I'm actually coming out $72.17 behind for those three months.

  6. The $12.73 I saved (correcting for bus fare) aren't that impressive, given my commute time is about 25 minutes one-way in my car to work. But the bus trip takes me about 1:15 hours, so I take about 50 minutes longer one way to get to work. On 32 trips, that's 1600 minutes, or 27:40 hours. So I saved $12.73 in exchange for 27:40 hours. That means my time is valued at about $0.50/hr.

  7. I notice the emissions calculation also ignores the emissions the city bus makes in taking me to work, which entails at least one stop and subsequent restart specifically for me each way. The bus burns a great deal more fuel than my car, and emits a great deal more nasty stuff too. I don't have any real numbers for that, but I suspect it's at least 10-25% lower than they glowingly report, once it's amortized across all the bus riders (I've frequently been one of three passengers on a bus).

So they're pretty much lying to me about my "achievement".

The situation's not quite so dim as they raw numbers indicate. I get a serious discount on a monthly bus pass as a perk of working on campus, so I pay much less than $1.75 fare. And my car's a Suburban, so I burn more gas than they think. And honestly, I've used pretty rough numbers here. Not all my "saving" rides are on the bus: some are on my bike, which has no fare. And neither the calendar nor I have amortized vehicle maintenance over those trips. Of course, we'd need to amortize bike maintenance costs too.

But the point is, they don't actually know that. This is not my employer's project, it's a county project: not everyone on it gets discounted bus passes, not everyone drives an enormous beast when they do drive. And the gas price "estimates" are blatantly dishonest.

I intend to continue to ride my bike and/or the bus to work for reasons of my own. Not because of global warming (anthropogenic global warming is a political myth), but for reasons like a general disapproval of wastefulness and the fact that every mile I bike is a little less Ox.

But I thought it interesting how disingenuous this particular [county] government project is.

Thursday, February 5, 2009

Our tax dollars at work!

I rode my bike to work today: it was cold and I'm still in pretty bad shape, so I succumbed to temptation and cut a corner, riding down 'A St.' in Tacoma.

Right at the corner of A St. and 171 Ave, there was a stopped school bus. The lights weren't flashing: it was pulled over on the side of the road. The house on the corner has a privacy fence, and the bus was between the street and the fence.

I was peddling slowly: there is a slight rise there, and "slight rise" was enough this morning, so I was geared down a little, and just sort of puffing up the hill.

I saw the bus driver get out of the bus, and I noticed a bunch of kids on the bus, probably a half dozen, maybe more. It's hard to tell with the tinted windows.  They looked like they were six or seven years old.

As I rounded the corner, I looked back to see the bus driver standing at the rear tire, urinating. He appeared actually to be urinating on rear tire of the bus. In front of about a dozen primary schoolers.

Class, all class.

That's our tax dollars at work.

And people ask us why we home-school our kids.