briancarper.net (λ) (Tag: Lisp)

I'm turning into a Lisp snob

Fri, 13 Nov 2009 22:38:50 -0800

Reddit and StackOverflow and other websites I frequent are filled to the brim with discussion of Google's Go. The code snippet on the front page is:

package main

import "fmt"

func main() {
  fmt.Printf("Hello, 世界\n")
}

First thoughts that ran through my head as I looked over the site:

Ugh, look at all that syntax.
Nice(r) type system (than C++ and Java). I'll stick with multimethods though.
Concurrency semantics, hmmm... Shared mutable memory between threads? I think I'll stick with Clojure for now thanks.
Where are the macros?
It has anonymous functions and closures and sort-of first-class functions? Good. Welcome to the 1960's.
len is a special operator? Sigh. (Programming language quality is usually inversely proportional to the number of special forms.)
Cool that they used Japanese in the example though. (That word is sekai, "world", obviously.)

Compared to a Lisp, this language looks indistinguishable to C, Perl, Python, Java etc. It looks like such a small incremental improvement (if it even is an improvement). Yet another imperative, for-loop-wielding, curly-brace-using, pointer-mangling, state-mutating, OOP language.

In fact via Reddit today I read this awesome post to a mailing list which compares Go with ALGOL68, and it gave me a would-you-look-at-that moment. Once you learn a few languages that are significantly different from ALGOL derivatives, all ALGOLish languages start to look eerily similar. Are we really stuck with ALGOL-derived languages being the only viable mainstream languages for all time? How much polish can we possibly apply to the same turd?

Then I realized, I'm turning into a Lisp snob. : ( Learning a Lisp apparently does spoil you for the rest of time. I am without a basis to judge whether this language will be a successful replacement of anything. All I know is I probably won't use it. Honestly I'm much more excited about new things on the horizon in Clojure. And I still have getting better at Haskell on my TODO list.

Emacs Clojure colors

Fri, 30 Oct 2009 19:02:14 -0700

In yet another step along the path of trying to forcibly morph Emacs into Vim, I started a port of my Vim color scheme Gentooish to an Emacs color-theme.

Then I threw a few lines at the bottom of .emacs to highlight a few more Clojure-specific things when in clojure-mode. Result:

This is inspired partly by parenface.el which dims parens so they don't stand out as much. Lispers have this silly meme where they pretend they can't see parens at all, but being a mere human I want Emacs to help blend them into the background a bit.

I want to dim parens but also make braces and brackets pop out a bit more. Coloring braces, brackets and parens slightly differently helps make a few things easier to read in my opinion, especially in ))]}])) kinds of situations (like at the end of the function above).

This is pretty brute-force and I really wish I could figure out how to highlight #{} (sets) differently than {} (hash-maps). Highlighting (), '(), `(), and #() differently would be awesome too. But that is well beyond my elisp skills at the present. Certain areas of Emacs are all but impenetrable, no matter how much documentation I read and how much banging on the keyboard I do. The morass that is font-lock is one of those areas.

p.s. I find an easy way to keep your dotfiles in git is to make a folder (called dotfiles or something) and symlink your dotfiles from there to your home directory. I also put * in .gitignore so git doesn't slurp up anything sensitive by accident.

Sort a Clojure map by two or more keys

Wed, 21 Oct 2009 02:23:30 -0700

Note to self. Given this:

user> (def x [{:foo 2 :bar 11}
              {:bar 99 :foo 1}
              {:bar 55 :foo 2}
              {:foo 1 :bar 77}])

#'user/x

How do you sort by :foo, and where :foo is equal, sort by :bar?

Obviously not like this:

user> (sort x)
; Evaluation aborted.
;; java.lang.ClassCastException: clojure.lang.PersistentArrayMap cannot be cast to java.lang.Comparable

Also not like this:

user> (sort-by #(map % [:foo :bar]) x)
; Evaluation aborted.
;; java.lang.RuntimeException: java.lang.ClassCastException: clojure.lang.LazySeq cannot be cast to java.lang.Comparable

What's a guy got to do to find something Comparable? Well, vectors are Comparable, so we can do this:

user> (sort-by #(vec (map % [:foo :bar])) x)
({:foo 1, :bar 77} {:bar 99, :foo 1} {:foo 2, :bar 11} {:bar 55, :foo 2})

Now I have two problems

Fri, 25 Sep 2009 23:02:14 -0700

I'm converting one of my websites from Ruby on Rails to Clojure in my spare time. I stupidly put a bunch of RoR-style links inline into certain bits of plaintext content, so in my DB there are a bunch of text fields with <%= link_to ... %> in the middle.

It was easy to fix with a regex though:

(defn clean [txt]
  (re-gsub #"<%=\s*link_to\s+(\"[^\"]+\"|'[^']+')\s*(?:,\s*'([^']+)'\s*)?(?:,\s*image_path\(['\"]([^'\"]+)['\"]\)\s*)?(?:,\s*:controller\s*=>\s*(?::(\S+)|['\"]([^\"']+)['\"])\s*)?(?:,\s*:action\s*=>\s*(?::(\S+)|['\"]([^\"']+)['\"])\s*)?(?:,\s*:id\s*=>\s*(?:(\d+)|:(\S+)|['\"]([^\"']+)['\"])\s*)?\s*%>"
           (fn [[_ s & parts]] (let [href (str-join "/" (filter identity parts))]
                           (str "<a href=\"/" href "\">" (re-gsub #"^[\"']|[\"']$" "" s) "</a>")))
           txt))

And by easy, I mean not easy.

Note to self, try something other than a regex next time.

Note to self, don't bury some framework's funky-syntax DSL in the middle of plaintext content. Next time use HTML or do the conversion from DSL to HTML early rather than late.

Silly how two years ago I thought I'd be using Ruby for that site forever.

Church Numerals in Clojure

Wed, 16 Sep 2009 02:14:30 -0700

Church Numerals are a way of encoding natural numbers using lambda expressions. A number n is represented by a function that composes another function f around its argument x, n times. So:

0 ≡ λf.λx. x
1 ≡ λf.λx. f x
2 ≡ λf.λx. f (f x)
...

These can be implemented very easily in Lisps and other languages with first-class functions. In Clojure, this would be:

0 ≡ (fn [f] (fn [x] x))
1 ≡ (fn [f] (fn [x] (f x)))
2 ≡ (fn [f] (fn [x] (f (f x))))
...

The insane thing is that you can do arithmetic on these things using nothing but more anonymous functions. It's lambdas all the way down.

Once you understand this, your brain will explode, and hopefully little bits of it will land next to the remnants of my own brain to keep it company.

So based on the Wikipedia article, here are some functions that convert numbers to and from Church encoding, and then some functions to do basic arithmetic on Church numerals. I understand this down to exp (via lots of pencil and paper expansions) but I gave up understanding at pred.

(defn church
  "Convert a natural number to a Church numeral."
  [n]
  (loop [n n, acc (fn [f] (fn [x] x))]
    (if (zero? n)
      acc
      (recur (dec n)
             (fn [f]
               (fn [x]
                 (f ((acc f) x))))))))

(defn unchurch
  "Convert a Church numeral to a plain integer."
  [n]
  ((n inc) 0))


(def plus
     (fn [m]
       (fn [n]
         (fn [f]
           (fn [x]
             ((m f) ((n f) x)))))))

(def mult 
     (fn [m]
       (fn [n]
         (fn [f]
          (n (m f))))))

(def exp
     (fn [m]
       (fn [n]
         (n m))))

(defn pred [n]
  (fn [f]
    (fn [x]
      (((n (fn [g]
             (fn [h]
               (h (g f)))))
        (fn [u] x))
       (fn [u] u)))))

(def sub
     (fn [m]
       (fn [n]
         ((n pred) m))))

(def is-zero?
     (fn [n]
       ((n (fn [x] 'FALSE)) 'TRUE)))

And this somehow actually works:

user> (unchurch (fn [f] (fn [x] (f (f (f (f (f (f (f x))))))))))
7
user> (unchurch (church 7))
7
user> (unchurch ((plus (church 2)) (church 3)))
5
user> (unchurch ((mult (church 2)) (church 3)))
6
user> (unchurch ((exp (church 2)) (church 3)))
8
user> (unchurch ((sub (church 7)) (church 3)))
4
user> (is-zero? (church 0))
TRUE
user> (is-zero? (church 7))
FALSE

This is the kind of thing I wish I'd studied more in college. I covered some cool stuff, some small amount of theory of computation and whatnot, but not nearly enough of this. Not enough functional programming, not enough Lispy goodness.

A lot of people say all programmers should learn assembler language so you understand what's happening at a low level. But shouldn't it be just as important to learn lambda calculus? It's another kind of low level, a very important one at that.

Adventures in blogging

Tue, 04 Aug 2009 00:24:23 -0700

So my newest site is about pareidolia which I think is an interesting subject. It's still in its early stages, using a data import from yet another of my old sites that I ran with a friend of mine. This makes three websites I'm running out of the same JVM on my server now. It'll be interesting to see how many more I can cram in there before my VPS explodes. CPU and RAM utilization are still supposedly minimal. I actually plan to redo my old faltering FF1 site entirely in Clojure too, which is probably the largest Clojure project I plan to undertake in the foreseeable future.

I'm still trying to come up with some kind of "how to make a blog in Clojure" tutorial once I nail down a good way to do it. The code for my blog has "grown organically" (i.e. become a mess) beyond the point where I'd inflict it upon the world without substantial cleanup. Probably easier to start over from scratch with something that's more demonstrative.

I'm also not too savvy on many of the "right ways" to do things in Compojure, as per this discussion. That's the good and bad thing about using a Lisp. You can pull off almost anything. There is no underlying structure to the code other than structure you impose. It's easy to write a mess.

Making image thumbnails in Clojure

Sun, 02 Aug 2009 23:32:38 -0700

I'm making a new website (in Clojure of course) and I have a need to resize uploaded images to make thumbnails. At first I tried to use JMagick because I'm familiar with ImageMagick already and it seemed like an OK library. But on the crusty old OS my VPS uses, I had a really hard time getting it to build, and even once it built it started segfaulting like mad.

Should've gone with something simpler first. Java has built-in libraries for this. It only took a few seconds to adapt this code on Stack Overflow into Clojure code.

(use 'clojure.contrib.java-utils)
(defn make-thumbnail [filename new-filename width]
  (let [img (javax.imageio.ImageIO/read (as-file filename))
        imgtype (java.awt.image.BufferedImage/TYPE_INT_ARGB)
        width (min (.getWidth img) width)
        height (* (/ width (.getWidth img)) (.getHeight img))
        simg (java.awt.image.BufferedImage. width height imgtype)
        g (.createGraphics simg)]
    (.drawImage g img 0 0 width height nil)
    (.dispose g)
    (javax.imageio.ImageIO/write simg "png" (as-file new-filename))))

Tokyo Cabinet, Engage

Sun, 05 Jul 2009 20:50:51 -0700

I posted recently about being dissatisfied with how my blog was playing with MySQL. I was going to try for some kind of file-based storage, but in the end I decided to go with Tokyo Cabinet, which is a very lightweight key/value store.

This simplifies a lot of code, because a blog is pretty much one list of objects (posts) with a bunch of sub-objects (tags, categories, comments). An SQL DB is made for independent objects that are related to each other via keys. So storing a post means deconstructing it into its parts and stuffing all the parts into their own tables, and fetching a post means fetching all the parts and putting it back together. A bit of a bother.

With a key/value store, a post is a hashmap, and it has sub-lists of comments and tags etc., and you serialize the whole thing and stuff it into the DB; when fetching you un-serialize it and you're good. Clojure being a Lisp, there's already a nice serialization format (s-expressions), so there's hardly any work to be done.

I'll give it a few days and if everything actually works, I'll push the code to github. The code is still pretty nasty and ad-hoc; I did this rewrite in the matter of a couple of hours on a weekend. But parts of it may be useful to someone.

I also plan to do some kind of simplified "How to make a blog in Clojure" tutorial in the near future. The code for this blog is bloated with a lot of functionality that is pretty specific to my site and that most people don't need. A more to-the-point tutorial would probably be more helpful.

Clojure, SLIME, ODBC, SQL Server

Fri, 26 Jun 2009 11:27:22 -0700

I had a lot of trouble connecting to an MS SQL Server at work via Clojure. Java 6 comes with a JDBC-ODBC bridge which worked fine from a Clojure REPL at a command prompt, or from inferior-lisp in Emacs, but in SLIME it would hang every time I tried to connect and I'd have to kill Java. Couldn't for the life of me figure out why.

I got it to work eventually by using Microsoft's own JDBC driver, which you can download here.

Once you put the downloaded .jar file on your CLASSPATH (in my case, sqljdbc4.jar) you can connect like this:

user> (def db {:classname "com.microsoft.sqlserver.jdbc.SQLServerDriver"
               :subprotocol "sqlserver"
               :subname "//server_hostname;database=SomeDatabase;user=SomeUser;password=SomePassword"})
#'user/db
user> (use 'clojure.contrib.sql)
nil
user> (with-connection db 
        (with-query-results rs ["SELECT * FROM whatever"] (prn rs)))
... results ...

Posted for the sake of Googlebot and for my own future sanity.

Five Things that Mildly Annoy Me in Clojure

Tue, 16 Jun 2009 21:56:21 -0700

This infamous blog post suggests that someone familiar with a language should be able to name five things they hate about it. "Hate" is a strong word, but I decided to think of five things I find mildyly annoying about Clojure, my favorite language of the moment.

Hashing integers

Clojure automatically converts integers between Integer, Long and BigInteger as needed to prevent overflow. This is good. Integers of the various classes test as equal too. This is also good.

user> (= 123 (int 123) (long 123) (bigint 123))
true

So would you expect this?

user> (hash-map (int 123) :foo (long 123) :bar (bigint 123) :baz)
{123 :foo, 123 :bar, 123 :baz}

Yes, each of the integer classes, though equal via =, do not have the same hash value when put into a hash-map. This is because:

user> (.equals (int 123) (long 123))
false

This is a wart inherited from the JVM. See here for discussion and explanation.

What's more, if you print this map and then try to read it back in, the integers will be read as int, long or bigint arbitrarily depending how big they are. This means you may not get the same class of object back that you output originally.

user> (def x {(bigint 123) :foo})
#'user/x
user> (= x x)
true
user> (def y (read-string (pr-str x)))
#'user/y
user> (= x y)
false
user> (class (first (keys y)))
java.lang.Integer
user> (class (first (keys x)))
java.math.BigInteger

This means that if you ever use integers as hash keys, you must be very careful to cast them all to the same integer type manually.

Metadata doesn't work on everything

Clojure lets you stick arbitrary metadata on various objects. This is higly useful; you can decorate objects with information that doesn't affect the value of the object. However metadata doesn't work everywhere.

user> (with-meta "foo" {:bar :baz})
java.lang.ClassCastException: java.lang.String cannot be cast to clojure.lang.IObj (NO_SOURCE_FILE:0)

You can only stick metadata on certain Clojure objects like Symbols, Vars, Refs, Agents, all of the Clojure collections and so on. You can't stick metadata on, say, a String or an Integer, because those are closed Java classes and can't be touched. It would be nice if you could.

use vs. require vs. import vs. load vs. ...

There are a startling number of ways to import a library into your code in Clojure. You have to choose from load, import, require, use, refer, and so on. Some work on Java classes, some work on Clojure libs. Some of them import symbols into your namespace, some of them don't. Some of them take strings as arguments, some take symbols, some take quoted lists of symbols, some take quoted lists of symbols with sub-lists of arguments. And all of these can be and usually are weirdly inlined into a namespace declaration, with a completely different list-quoting style.

So in Ruby you can do this:

require 'util'
require 'config'
require 'whatever'

Whether it's a gem, or a Ruby source file sitting locally, it all works the same as long as the load path is set up right.

But in Clojure, you do this (actual code from an IMAP library I wrote):

(ns qt4-mailtray.mail
  (:import (java.util Properties)
           (javax.mail Session Store Folder Message Flags Flags$Flag FetchProfile FetchProfile$Item)
           (javax.mail.internet InternetAddress))
  (:use clojure.contrib.str-utils))

This can quickly become unwieldy, especially if you start using the :as or :only or :rename arguments. It's made worse by Java's insane API's full of a billion classes that you need to import to do simple things. (And those things with dollar signs are mangled Java inner class names.) Clojure also lacks the ability to import a whole package worth of classes at once using java.io.* syntax, so you must name all of the classes explicitly.

every? vs some.

This is such a trite pet-peeve that it's barely worth mentioning, but it seems to be brought up repeatedly and endlessly on the Clojure mailing list so at least I'm not the only one bugged by it.

Clojure has a function (every? pred coll) which tests whether every item in a collection tests true via some predicate. To test whether every item in a collection tests false, we have not-any?. And we have a not-every? which tests whether any item tests false.

user> (every? even? [2 4 6])
true
user> (not-every? even? [2 4 6])
false
user> (not-any? even? [2 4 6])
false

Now what would you expect a function to be called which tests whether any item in a collection tests true via some predicate? If you said any? you are wrong! It's some.

Note that some isn't a predicate (hence no question mark in the name); it doesn't return true or false, as above, but rather returns the result of running pred on an item in coll.

user> (some identity [nil 1 2 3])
1

any? is pretty easy to write so it doesn't matter that much. Probably many people have an identical function sitting in some utils.clj file on their systems.

(defn any? [pred coll]
  (when (seq coll)
    (if (pred (first coll))
      true
      (recur pred (next coll)))))

Stack trace madness

Give this function:

(defn foo []
  (throw (Exception. "BARFED")))

What does the stack trace look like in SLIME when you call foo? Like this:

java.lang.Exception: BARFED (NO_SOURCE_FILE:0)
  [Thrown class clojure.lang.Compiler$CompilerException]

Restarts:
 0: [ABORT] Return to SLIME's top level.
 1: [CAUSE] Throw cause of this exception

Backtrace:
  2: swank.commands.basic$eval_region__729.invoke(basic.clj:36)
  3: swank.commands.basic$listener_eval__738.invoke(basic.clj:50)
  4: clojure.lang.Var.invoke(Var.java:346)
  5: user$eval__1506.invoke(NO_SOURCE_FILE)
  6: clojure.lang.Compiler.eval(Compiler.java:4580)
  7: clojure.core$eval__4016.invoke(core.clj:1728)
  8: swank.core$eval_in_emacs_package__336.invoke(core.clj:55)
  9: swank.core$eval_for_emacs__413.invoke(core.clj:123)
 10: clojure.lang.Var.invoke(Var.java:354)
 11: clojure.lang.AFn.applyToHelper(AFn.java:179)
 12: clojure.lang.Var.applyTo(Var.java:463)
 13: clojure.core$apply__3269.doInvoke(core.clj:390)
 14: clojure.lang.RestFn.invoke(RestFn.java:428)
 15: swank.core$eval_from_control__339.invoke(core.clj:62)
 16: swank.core$eval_loop__342.invoke(core.clj:67)
 17: swank.core$spawn_repl_thread__474$fn__505$fn__507.invoke(core.clj:173)
 18: clojure.lang.AFn.applyToHelper(AFn.java:171)
 19: clojure.lang.AFn.applyTo(AFn.java:164)
 20: clojure.core$apply__3269.doInvoke(core.clj:390)
 21: clojure.lang.RestFn.invoke(RestFn.java:428)
 22: swank.core$spawn_repl_thread__474$fn__505.doInvoke(core.clj:170)
 23: clojure.lang.RestFn.invoke(RestFn.java:402)
 24: clojure.lang.AFn.run(AFn.java:37)
 25: java.lang.Thread.run(Thread.java:619)

Yeouch. Now imagine that the above error is coming not from a simple function, but from some random line among hundreds of lines of source code.

Stack traces in Clojure will often tell you little to nothing about what is causing the error, or more importantly, where it's coming from in your code. Clojure functions are translated into Java classes when they're run through the JVM. Often can't even see the name of the function that's throwing the error; names are mangled into things like user$eval__1473.invoke, which is really really confusing when you use anonymous functions.

Per Jason Wolfe and Randall Schulz sometimes you can get a better stack trace if you dig a bit deeper:

user> (.printStackTrace (.getCause *e))

java.lang.Exception: BARFED
    at user$foo__1503.invoke(NO_SOURCE_FILE:1)
    at user$eval__1509.invoke(NO_SOURCE_FILE:1)
    at clojure.lang.Compiler.eval(Compiler.java:4580)
    at clojure.core$eval__4016.invoke(core.clj:1728)
    at swank.commands.basic$eval_region__729.invoke(basic.clj:36)
    at swank.commands.basic$listener_eval__738.invoke(basic.clj:50)
    at clojure.lang.Var.invoke(Var.java:346)
    at user$eval__1506.invoke(NO_SOURCE_FILE)
    at clojure.lang.Compiler.eval(Compiler.java:4580)
    at clojure.core$eval__4016.invoke(core.clj:1728)
    at swank.core$eval_in_emacs_package__336.invoke(core.clj:55)
    at swank.core$eval_for_emacs__413.invoke(core.clj:123)
    at clojure.lang.Var.invoke(Var.java:354)
    at clojure.lang.AFn.applyToHelper(AFn.java:179)
    at clojure.lang.Var.applyTo(Var.java:463)
    at clojure.core$apply__3269.doInvoke(core.clj:390)
    at clojure.lang.RestFn.invoke(RestFn.java:428)
    at swank.core$eval_from_control__339.invoke(core.clj:62)
    at swank.core$eval_loop__342.invoke(core.clj:67)
    at swank.core$spawn_repl_thread__474$fn__505$fn__507.invoke(core.clj:173)
    at clojure.lang.AFn.applyToHelper(AFn.java:171)
    at clojure.lang.AFn.applyTo(AFn.java:164)
    at clojure.core$apply__3269.doInvoke(core.clj:390)
    at clojure.lang.RestFn.invoke(RestFn.java:428)
    at swank.core$spawn_repl_thread__474$fn__505.doInvoke(core.clj:170)
    at clojure.lang.RestFn.invoke(RestFn.java:402)
    at clojure.lang.AFn.run(AFn.java:37)
    at java.lang.Thread.run(Thread.java:619)

This one at least mentions foo by name but you're still going to have a headache after a few hours of those stack traces.

Conclusion

So that's five things. You will notice a common theme. Most of these issues are inherited from the JVM. This is to be expected, I suppose. There's no way you can wrap one language in another without a few compromises.

But these things aren't show-stoppers. They are minor annoyances compared to the benefits you get from using the JVM, i.e. the good performance, tons of libraries, cross-platformness, and so on. Clojure is fun enough to work with and wart-less enough that it took me well over two weeks to write this post.

(If you were expecting me to mention loop/recur and the lack of native TCO in the JVM, you were PAINFULLY WRONG. No one who uses Clojure loses sleep over native TCO. It's largely a non-issue that's endlessly repeated by people looking for an excuse to pass up Clojure in favor of $their_pet_language. To each his own, but I have never found myself caring the slightest about loop/recur.)

ABLE

Thu, 04 Jun 2009 10:48:58 -0700

I learned recently of ABLE, which is a minimalistic GUI editor + REPL for Common Lisp. You download it and run it and there you go, Common Lisp with lots of included libraries, no Emacs trial-by-fire necessary.

This is a really great idea in my opinion. I wish I'd had this a year or two ago. Common Lisp has too many barriers for newbies to get started. Anything that removes a few of them can only benefit the Lisp world.

More Clojure Mandelbrot Goodness

Wed, 20 May 2009 23:39:23 -0700

After my brief stint in the world of fractal geometry and Clojure, I decided to make a real Mandelbrot set viewer. The resulting source code is here. Here's a simple output (click for bigger version):

It's a pretty naive implementation, barely 100 lines of code, but even with my brute-force approach, given a liberal sprinkling of type hints it runs fast enough. Programming Swing from Clojure couldn't be easier (though I doubt programming Swing from any language is ever really enjoyable, it's a painful bunch of libraries).

There's a discussion of different coloring algorithms on Wikipedia, but even after reading that, getting this thing to look good was difficult. I don't know enough math for it. I ended up cheating and I colored a couple of them in the GIMP, so I could use them as desktop wallpapers.

There are some more PNGS over here including one that's 16000x16000 (producing it almost melted my CPU last night).

Who needs a DB?

Wed, 20 May 2009 22:53:03 -0700

My blog is still working, in spite of my best efforts to crash it. So that's good. But lately I've been thinking that an SQL database is a lot of overkill just to run a little blog like this.

My blog only has around 450 posts total (over the course of many years), and about an equal number of user comments (thanks to all commenters!). Why do I need a full-blown database for that? All of my posts plus comments plus all meta-data is only 2 MB as a flat text file, 700k gzipped.

By far the most complicated part of my blog engine is the part that stuffs data into the database and gets it back out again in a sane manner (translating Clojure data to SQL values, and back again; splitting up my Clojure data structures into rows for different tables, and then re-combining values joined from multiple tables into one data structure). Eliminating that mess would be nice.

Inevitably I ended up with some logic in the database too: enforcing uniqueness of primary keys, marking some fields as NOT NULL, giving default values and so on. But a lot of other logic was in my Clojure code, e.g. higher-level semantic checking, and some things I wanted to set as default values were impossible to implement in SQL.

Wouldn't it be nice for all the logic to be in Clojure? And the data store on disk to be a simple dump of a Clojure data structure? I can (and did) write a few macros to give me SQL-like field declaration and data validation, for uniqueness of IDs and data types etc. For my limited needs it works OK.

The next question is what format to use for dumping to disk. Happily Clojure is Lisp, so dumping it as a huge s-exp via pr-str works fine, and reading it back in later via read-string is trivial.

Some Java data types can't be printed readably by default, for example java.util.Dates, which print like this:

#<Date Wed May 20 22:39:00 PDT 2009>

The #<> reader macro deliberately throws an error if you try to read that back in, because the reader isn't smart enough to craft Date objects from strings by default. But Clojure is extensible; you can specify a readable-print method for any data type like this:

(defmethod clojure.core/print-method java.util.Date [o w]
  (.write w (str "#=" `(java.util.Date. ~(.getTime o)))))

Now dates print as

#=(java.util.Date. 1242884415044)

and if you try to read that via read-string, it'll create a Date object like you'd expect.

user> (def x (read-string "#=(java.util.Date. 1242884415044)"))
#'user/x
user> (class x)
java.util.Date
user> (str x)
"Wed May 20 22:40:15 PDT 2009"

Storing data in a plain file has another benefit of letting me grep my data from a command line, or even edit the data in a text editor and re-load it into the blog (God help me if that's ever necessary).

Having multiple threads banging on a single file on disk is a horrible idea, but Clojure refs and agents and transactions handle that easily. But I do have to work out how not to lose all my data in case the server crashes in the middle of a file update. (I've lost data (in a recoverable way) due to a server crash in the middle of a MySQL update too, so this is a problem for everyone.) Perhaps I'll keep a running history of my data, each update being a new timestamped file, so old files can't possibly be corrupted. Or use the old write-to-tmp-file-and-rename-to-real-file routine. Or heck, I could keep my data in Git and use Git commands from Clojure. It'd be nice to have a history of edits.

If this idea works out I'll upload code for everything to github, as usual.

Clojure: ASCII Mandelbrot Set

Tue, 12 May 2009 18:46:01 -0700

Did you know there's this neat Lisp message board where from time to time someone posts a short problem similar in spirit to the infamous RubyQuiz?

Not a lot of people have participated so far, hopefully that changes. I participated this time; the problem is to render the Mandelbrot Set in ASCII. Here's my Clojure version (based loosely on this one).

(ns mandelbrot
  (:refer-clojure :exclude [+ * <])
  (:use (clojure.contrib complex-numbers)
        (clojure.contrib.generic [arithmetic :only [+ *]]
                                 [comparison :only [<]]
                                 [math-functions :only [abs]])))

(defn- mandelbrot-seq [x y]
  (let [z (complex x y)]
    (iterate #(+ z (* % %)) z)))

(defn- mandelbrot-char [x y]
  (loop [c 126
         m (mandelbrot-seq x y)]
    (if (and (< (abs (first m)) 2)
             (> c 32))
      (recur (dec c) (rest m))
      (char c))))

(defn- mandelbrot-line [xs y]
  (apply str (map #(mandelbrot-char % y) xs)))

(defn- m-range [min max num-steps]
  (range min
         max
         (/ (+ (abs min)
               (abs max))
            num-steps)))

(defn mandelbrot [rmin rmax imin imax]
  (let [rows 30
        cols 50
        xs (m-range rmin rmax cols)
        ys (m-range imin imax rows)]
    (dorun (map #(println (mandelbrot-line xs %)) ys))))

(comment
  ;Example run:
  (mandelbrot -2.0 1.0 -1.5 1.5)
"
~~~~~~~~~~~~}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}
~~~~~~~~~~~~}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}
~~~~~~~~~~}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}
~~~~~~~~~}}}}}}}}}}}}|||||||||}}}}}}}}}}}}}}}}}}}}
~~~~~~~}}}}}}}}|||||||||||||||||||||}}}}}}}}}}}}}}
~~~~~~}}}}}||||||||||||||{{{{zlxz{{{||||}}}}}}}}}}
~~~~~}}}}|||||||||||||{{{{{zzyxpvlz{{{||||}}}}}}}}
~~~~}}}|||||||||||||{{{{{{zzyxvnpwyzz{{{||||}}}}}}
~~~}}|||||||||||||{{{{{{zzyyws   .vyzzz{{|||||}}}}
~~~}||||||||||||{{{{{zzxwwwvus   muvxyywz{|||||}}}
~~}|||||||||||{{{zzzzyyu= p         oteqpz{|||||}}
~~||||||||||{zzzzzzyyyvtm              oxz{{|||||}
~}|||||{{{zyvwxxxxxxxwrG                vuz{|||||}
~||{{{{{zzzywsMsqRovvs                  pxz{{|||||
~|{{{{{zzzyxsq      pj                  `xz{{|||||
~{{{{yyyxwsrp                           wyz{{|||||
~?:3 3 #                              ovxzz{{|||||
~{{{{yyyxwsrp                           wyz{{|||||
~|{{{{{zzzyxsq      pj                  `xz{{|||||
~||{{{{{zzzywsMsqRovvs                  pxz{{|||||
~}|||||{{{zyvwxxxxxxxwrG                vuz{|||||}
~~||||||||||{zzzzzzyyyvtm              oxz{{|||||}
~~}|||||||||||{{{zzzzyyu= p         oteqpz{|||||}}
~~~}||||||||||||{{{{{zzxwwwvus   muvxyywz{|||||}}}
~~~}}|||||||||||||{{{{{{zzyyws   .vyzzz{{|||||}}}}
~~~~}}}|||||||||||||{{{{{{zzyxvnpwyzz{{{||||}}}}}}
~~~~~}}}}|||||||||||||{{{{{zzyxpvlz{{{||||}}}}}}}}
~~~~~~}}}}}||||||||||||||{{{{zlxz{{{||||}}}}}}}}}}
~~~~~~~}}}}}}}}|||||||||||||||||||||}}}}}}}}}}}}}}
~~~~~~~~~}}}}}}}}}}}}|||||||||}}}}}}}}}}}}}}}}}}}}
~~~~~~~~~~}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}
"
)

And here's some obligatory Jonathan Coulton.

Vim and Emacs modelines

Tue, 05 May 2009 18:22:47 -0700

Both Vim and Emacs have "windows", resizeable panes inside the main app "frame". Windows are a very useful feature (aside from the ill-chosen name; should've been "panes" or something). I find windows superior to tabs in my opinion; you can easily view more than one file at a time, view two files side-by-side for comparison, and such. Both Vim and Emacs use the border under the window to show helpful information like the filename and editing mode such; this is called the "status line" or "modeline". Again, very handy.

How do you set a nice custom modeline in Vim? Here's my config from my .vimrc:

set statusline=%f\ %2*%m\ %1*%h%r%=[%{&encoding}\ %{&fileformat}\ %{strlen(&ft)?&ft:'none'}\ %{getfperm(@%)}]\ 0x%B\ %12.(%c:%l/%L%)

Yikes! What a mess. It's just a big string with a bunch of special codes and literal characters all mixed up. If you do :h 'statusline' you can read about all the options. That string is ugly though.

Or so I thought, until I witnessed the trainwreck that is Emacs. The documentation for Emacs' modeline comprises a small encyclopedia. The configuration itself is spread across a couple dozen configuration variables. This is the default value for just one of them:

(#("%[" 0 2
   (help-echo "Recursive edit, type C-M-c to get out"))
 #("(" 0 1
   (help-echo "mouse-1: Select (drag to resize)\nmouse-2: Make current window occupy the whole frame\nmouse-3: Remove current window from display"))
 (:propertize
  ("" mode-name)
  help-echo "Major mode\nmouse-1: Display major mode menu\nmouse-2: Show help for major mode\nmouse-3: Toggle minor modes" mouse-face mode-line-highlight local-map
  (keymap
   (mode-line keymap
          (down-mouse-3 keymap
                (abbrev-mode menu-item "Abbrev (Abbrev)" abbrev-mode :help "Automatically expand abbreviations" :button
                     (:toggle . abbrev-mode))
                (auto-fill-mode menu-item "Auto fill (Fill)" auto-fill-mode :help "Automatically insert new lines" :button
                        (:toggle . auto-fill-function))
                (auto-revert-mode menu-item "Auto revert (ARev)" auto-revert-mode :help "Revert the buffer when the file on disk changes" :button
                          (:toggle bound-and-true-p auto-revert-mode))
                (auto-revert-tail-mode menu-item "Auto revert tail (Tail)" auto-revert-tail-mode :help "Revert the tail of the buffer when buffer grows" :enable
                           (buffer-file-name)
                           :button
                           (:toggle bound-and-true-p auto-revert-tail-mode))
                (flyspell-mode menu-item "Flyspell (Fly)" flyspell-mode :help "Spell checking on the fly" :button
                       (:toggle bound-and-true-p flyspell-mode))
                (font-lock-mode menu-item "Font Lock" font-lock-mode :help "Syntax coloring" :button
                        (:toggle . font-lock-mode))
                (glasses-mode menu-item "Glasses (o^o)" glasses-mode :help "Insert virtual separators to make long identifiers easy to read" :button
                      (:toggle bound-and-true-p glasses-mode))
                (hide-ifdef-mode menu-item "Hide ifdef (Ifdef)" hide-ifdef-mode :help "Show/Hide code within #ifdef constructs" :button
                         (:toggle bound-and-true-p hide-ifdef-mode))
                (highlight-changes-mode menu-item "Highlight changes (Chg)" highlight-changes-mode :help "Show changes in the buffer in a distinctive color" :button
                            (:toggle bound-and-true-p highlight-changes-mode))
                (outline-minor-mode menu-item "Outline (Outl)" outline-minor-mode :help "" :button
                        (:toggle bound-and-true-p outline-minor-mode))
                (overwrite-mode menu-item "Overwrite (Ovwrt)" overwrite-mode :help "Overwrite mode: typed characters replace existing text" :button
                        (:toggle . overwrite-mode))
                "Minor Modes")
          (mouse-2 . describe-mode)
          (down-mouse-1 menu-item "Menu Bar" ignore :filter
                (lambda
                  (_)
                  (mouse-menu-major-mode-map))))))
 ("" mode-line-process)
 (:propertize
  ("" minor-mode-alist)
  mouse-face mode-line-highlight help-echo "Minor mode\nmouse-1: Display minor mode menu\nmouse-2: Show help for minor mode\nmouse-3: Toggle minor modes" local-map
  (keymap
   (header-line keymap
        (down-mouse-3 keymap
                  (abbrev-mode menu-item "Abbrev (Abbrev)" abbrev-mode :help "Automatically expand abbreviations" :button
                       (:toggle . abbrev-mode))
                  (auto-fill-mode menu-item "Auto fill (Fill)" auto-fill-mode :help "Automatically insert new lines" :button
                          (:toggle . auto-fill-function))
                  (auto-revert-mode menu-item "Auto revert (ARev)" auto-revert-mode :help "Revert the buffer when the file on disk changes" :button
                        (:toggle bound-and-true-p auto-revert-mode))
                  (auto-revert-tail-mode menu-item "Auto revert tail (Tail)" auto-revert-tail-mode :help "Revert the tail of the buffer when buffer grows" :enable
                             (buffer-file-name)
                             :button
                             (:toggle bound-and-true-p auto-revert-tail-mode))
                  (flyspell-mode menu-item "Flyspell (Fly)" flyspell-mode :help "Spell checking on the fly" :button
                         (:toggle bound-and-true-p flyspell-mode))
                  (font-lock-mode menu-item "Font Lock" font-lock-mode :help "Syntax coloring" :button
                          (:toggle . font-lock-mode))
                  (glasses-mode menu-item "Glasses (o^o)" glasses-mode :help "Insert virtual separators to make long identifiers easy to read" :button
                        (:toggle bound-and-true-p glasses-mode))
                  (hide-ifdef-mode menu-item "Hide ifdef (Ifdef)" hide-ifdef-mode :help "Show/Hide code within #ifdef constructs" :button
                           (:toggle bound-and-true-p hide-ifdef-mode))
                  (highlight-changes-mode menu-item "Highlight changes (Chg)" highlight-changes-mode :help "Show changes in the buffer in a distinctive color" :button
                              (:toggle bound-and-true-p highlight-changes-mode))
                  (outline-minor-mode menu-item "Outline (Outl)" outline-minor-mode :help "" :button
                          (:toggle bound-and-true-p outline-minor-mode))
                  (overwrite-mode menu-item "Overwrite (Ovwrt)" overwrite-mode :help "Overwrite mode: typed characters replace existing text" :button
                          (:toggle . overwrite-mode))
                  "Minor Modes"))
   (mode-line keymap
          (down-mouse-3 keymap
                (abbrev-mode menu-item "Abbrev (Abbrev)" abbrev-mode :help "Automatically expand abbreviations" :button
                     (:toggle . abbrev-mode))
                (auto-fill-mode menu-item "Auto fill (Fill)" auto-fill-mode :help "Automatically insert new lines" :button
                        (:toggle . auto-fill-function))
                (auto-revert-mode menu-item "Auto revert (ARev)" auto-revert-mode :help "Revert the buffer when the file on disk changes" :button
                          (:toggle bound-and-true-p auto-revert-mode))
                (auto-revert-tail-mode menu-item "Auto revert tail (Tail)" auto-revert-tail-mode :help "Revert the tail of the buffer when buffer grows" :enable
                           (buffer-file-name)
                           :button
                           (:toggle bound-and-true-p auto-revert-tail-mode))
                (flyspell-mode menu-item "Flyspell (Fly)" flyspell-mode :help "Spell checking on the fly" :button
                       (:toggle bound-and-true-p flyspell-mode))
                (font-lock-mode menu-item "Font Lock" font-lock-mode :help "Syntax coloring" :button
                        (:toggle . font-lock-mode))
                (glasses-mode menu-item "Glasses (o^o)" glasses-mode :help "Insert virtual separators to make long identifiers easy to read" :button
                      (:toggle bound-and-true-p glasses-mode))
                (hide-ifdef-mode menu-item "Hide ifdef (Ifdef)" hide-ifdef-mode :help "Show/Hide code within #ifdef constructs" :button
                         (:toggle bound-and-true-p hide-ifdef-mode))
                (highlight-changes-mode menu-item "Highlight changes (Chg)" highlight-changes-mode :help "Show changes in the buffer in a distinctive color" :button
                            (:toggle bound-and-true-p highlight-changes-mode))
                (outline-minor-mode menu-item "Outline (Outl)" outline-minor-mode :help "" :button
                        (:toggle bound-and-true-p outline-minor-mode))
                (overwrite-mode menu-item "Overwrite (Ovwrt)" overwrite-mode :help "Overwrite mode: typed characters replace existing text" :button
                        (:toggle . overwrite-mode))
                "Minor Modes")
          (mouse-2 . mode-line-minor-mode-help)
          (down-mouse-1 . mouse-minor-mode-menu))))
 #("%n" 0 2
   (local-map
    (keymap
     (mode-line keymap
        (mouse-2 . mode-line-widen)))
    mouse-face mode-line-highlight help-echo "mouse-2: Remove narrowing from the current buffer"))
 #(")" 0 1
   (help-echo "mouse-1: Select (drag to resize)\nmouse-2: Make current window occupy the whole frame\nmouse-3: Remove current window from display"))
 #("%]" 0 2
   (help-echo "Recursive edit, type C-M-c to get out"))
 #("--" 0 2
   (help-echo "mouse-1: Select (drag to resize)\nmouse-2: Make current window occupy the whole frame\nmouse-3: Remove current window from display")))

What the hell?

Instead of a string, it's an s-expression, which is better, right? Well no, it's still just a big construct with arbitrary meanings assigned to its contents. Lists mean one thing, strings mean another thing (and those strings, like Vim's, can contain special escape sequences). Symbols mean something else, symbols that are keywords mean something else, numbers mean something else, and so on.

To decipher this I had to learn this mini-language. And also learn about "text properties" and a bunch of elisp stuff. It also required knowledge about a bunch of minor modes and how they tie into the modeline, all of which is essentially a bowl of spaghetti code. And keymaps, and maps to control mouse click events and such. Eventually I figured out that most of that crap is controlling tooltip text.

If you do M-x customize-apropos in Emacs and search for "mode-line", you'll get a helpful list of all of the configuration values and their values. (The default values contain literal tab characters, which you can't even type into the customize text fields without C-qing, because tab jumps you between fields. Ughhhhhhh.)

I gave up even trying to get Emacs to have all the helpful information my Vim modeline has. Even deleting the default crap to pare this down to something readable took some effort. This what I ended up with:

(mode-line-format (quote ("%e--[" mode-line-buffer-identification "]" (vc-mode vc-mode) "  " mode-line-modes global-mode-string " %-")))
(mode-line-in-non-selected-windows t)
(mode-line-modes (quote ("%[" "(" (:propertize ("" mode-name)) ("" mode-line-process) (:propertize ("" minor-mode-alist)) "%n" ")" "%]")))
(mode-line ((((class color) (min-colors 88)) (:background "#333333" :foreground "#bcbcbc" :box (:line-width -1 :color "#333333")))))
(mode-line-highlight ((((class color) (min-colors 88)) nil)))
(mode-line-inactive ((default (:inherit mode-line)) (((class color) (min-colors 88) (background dark)) (:foreground "#8b8b8b" :weight light))))

What does that mean? Don't ask me, I can no longer read it. If customize hadn't produced a lot of that for me, I probably wouldn't have managed. My favorite part is the four-deep nested list of lists of lists of lists for the colors.

Verdict?

Vim's modeline is less powerful than Emacs. But who cares if you can't even read it to edit it? Most of Emacs' modeline features are annoying. (My motivation for editing this to begin with was to turn off all the mouse buttons and mini-menus and crap.)

Vim's status line is exactly configurable enough. I don't want to build a small GUI app in my modeline. I want it to show certain bits of information about the buffer, that's it. Vim does this.
Vim script is less elegant than elisp. Or is it? Vim's modeline is a custom DSL for formatting modelines. It's hard to think of anything more concise. Concision is a very good thing. Emacs' version is more general, at the expense of horrid verbosity and unreadability.
Vim's modeline is a string, which means you either write it literally, or you construct it by concatenating lots of other strings. This is a faux pas, right? It's like using eval in Ruby or Perl. It's fragile and error-prone. Emacs uses a Lisp, with its macros and quoted lists and code is data and so on.

But who cares? In this case, a simple string is powerful enough. I don't need a whole Turing-complete programming language to configure a modeline. It's massive overkill and you pay a price for it. The minute a human being is supposed to be keeping track that the second element in the 5-deep nested list means "x", something has gone horribly wrong.

Vim wins this round.

Clojure 1.0

Mon, 04 May 2009 21:04:37 -0700

Clojure 1.0 was released today. For those who care about version numbers, this should be good news. Clojure has been good and "ready" for a good while in my opinion but it's kind of nice to see it made somewhat official.

The pace of development and the level of enthusiasm in the Clojure community are something to behold. Here's hoping it holds up. The community is knowledgeable and helpful and overall positive and people are churning out an awful lot of useful code. Kudos to the Clojure devs and contributors, thanks for the wonderfully fun tool and toy to work and play with.

Clojure Reader Macros

Sat, 18 Apr 2009 17:14:50 -0700

Unlike Common Lisp, Clojure doesn't support user-defined reader macros. You can read some of the rationale for why in this chat log, among other places. I think that's probably a good decision; I don't see a lot of need for mangling the reader. Regular macros get you pretty far already and Clojure has built-in reader support for all the good stuff.

But how hard would it be to have custom reader macros in Clojure if you wanted them? Turns out not too hard if you're willing to ruthlessly break encapsulation and rely on implementation details. Here's one way you could define a dispatch reader macro (i.e. one starting with # and some specified second character):

(defn dispatch-reader-macro [ch fun]
  (let [dm (.get (doto (.getDeclaredField clojure.lang.LispReader "dispatchMacros")
                   (.setAccessible true))
                 nil)]
    (aset dm (int ch) fun)))

Pass in a character and an fn and you get a reader macro. For a silly example let's make reader syntax to uppercase a literal string.

(defn uppercase-string [rdr letter-u]
  (let [c (.read rdr)]
    (if (= c (int \"))
      (.toUpperCase (.invoke
                     (clojure.lang.LispReader$StringReader.)
                     rdr
                     c))
      (throw (Exception. (str "Reader barfed on " (char c)))))))

The function is passed a reader and the dispatch character (which you can usually ignore). I cheat and use Clojure's StringReader to do the real work.

Now I can do this:

user> (dispatch-reader-macro \U uppercase-string)
#<user$uppercase_string__1295 user$uppercase_string__1295@9b59a2>

user> #U"Foo bar BAZ"
"FOO BAR BAZ"

user> (println #U"foo\nbar")
FOO
BAR
nil

user> #U(blarg)
java.lang.Exception: Reader barfed on (

(= "FOO" "foo")
false

(= "FOO" #U"foo")
true

Oh sweet Jesus don't use this in real code, because:

The community will rightly hunt you down with torches and pitchforks.
Reader macro characters are reserved and may conflict with later changes to the core language.
These are set globally, not per-namespace.
And so on. Just don't.

But I think it's a nice demonstration. I've read opinions that Clojure isn't a Real Lisp™ because a lot of Clojure is written in Java and isn't extensible in Clojure itself, but that's generally not true. The reader code for Clojure was all written in Java, but above I modify it from Clojure. There is no line separating Java-land and Clojure-land. It's all one big happy family.

Blog and CRUD

Sun, 12 Apr 2009 20:01:35 -0700

I updated my blog source code on github. I also split my CRUD library out into its own clj-crud repo. It is cruddy, so the name is apt.

This code still isn't polished enough for someone to drop it on a server and fire it up, but maybe it'll give someone some ideas. I think the new code is cleaner and it'll be easier for me to add features now.

Beware bugs, I'm positive I introduced some.

EDIT: A word about the CRUD library... persisting data to disk is hard when the data may be mutated by many threads at once and the destination for your data is an SQL database that may or may not even be running. I have more respect for people who've written libraries that actually do this kind of thing and work right. Granted I only spent 3 days on mine but still, it's tricky.

I gave up for a while and tried clj-record, but it was prohibitively slow. It has the old N+1 queries problem when trying to select an object which has N sub-objects. In real life you'd write SQL joins to avoid such things. Ruby on Rails on the other hand gets around this via some nasty find syntax.

I get around it by having all my data in a Clojure ref in RAM already so it doesn't matter. And by using hooks so each object keeps a list of its sub-objects and the list is always up-to-date (updates of sub-objects propagate to their parents). But the crap I have to do to get this to just barely work is pretty painful.

Lisp Syntax Doesn't Suck

Wed, 08 Apr 2009 04:03:12 -0700

I spend a lot of time talking about what I don't like about various languages, but I never talk about what I do like. And I do like a lot, or I wouldn't spend so much time programming and talking about programming.

So here goes. I like the syntax of Lisp. I like the prefix notation and the parentheses.

Common Complaints

A common criticism of Lisp from non-Lispers is that the syntax is ugly and weird. The parentheses are impossible to keep balanced. It ends up looking like "oatmeal with fingernail clippings mixed in".

Also, prefix notation is horrible. 1 + 2 is far superior to (+ 1 2). Infix notation is how everyone learns things and how all the other languages do it. There are countless numbers of people (example) who have proposed to "fix" this, to give Lisp some kind of infix notation. The topic inevitably comes up on Lisp mailing lists and forums.

Partly this is subjective opinion and can't be argued with. I can't say that Lispy parens shouldn't be ugly for people, any more than I can say that someone is wrong to think that peanut butter is gross even though I like the taste of it. But in another sense, does it matter that it's painful? Does it need to be changed? Should the weird syntax stop you from learning Lisp?

Prefix Notation: Not Intuitive?

There is no "intuitive" when it comes to programming. There's only what we're used to and what we aren't.

What does = mean in a programming language? Most people from a C-ish background will immediately say assignment. x = 1 means "give the variable/memory location called X the value 1".

For non-programmers, = is actually an equality test or a statement of truth. 2 + 2 = 4; this is either a true or false statement. There is no "assignment". The notion of assignment statements is an odd bit of programming-specific jargon. In most programming languages we've learned instead that == is an equality test. Of course some have := for assignment and = for equality tests. But = and == seems to be more common. Some languages even have ===. Or x.equals(y). Even less like what we're used to. (Don't get started on such magic as +=.)

Most of us have no problem with these, after a while. But few of us were programmers before we learned basic math. How many of us remember the point in time when we had to re-adjust our thinking that = means something other than what we've always learned it to mean? I actually do remember learning this, over a decade ago. This kind of un-learning is painful and confusing, there's no question.

But it's also necessary, because these kinds of conventions are arbitrary and vary between fields of study (and between programming languages). And there are only so many symbols and words available to use, so we re-use them. None of the meanings for = is "right" or more "intuitive" than the other. = has no inherent meaning. It means whatever we want it to mean. Programming is chock-full of things like this that makes no sense until you memorize the meaning of them.

Consider a recent article that got a lot of discussion, about why all programmers should program in English. How much less intuitive can you get, for a native speaker of another language to program using words in English? Yet they manage. (Have you ever learned to read sheet music? Most of the terms are in Italian. I don't speak a word of Italian, yet I managed.)

The point is that it's very painful to un-learn things that seem intuitive, and to re-adjust your thinking, but it's also very possible. We've all done it before to get to where we are. We can all do it again if we need to.

Prefix notation is unfamiliar and painful for many people. When I first started learning Lisp, the prefix notation was awfully hard to read without effort, even harder to write. I would constantly trip up. This is a real distraction when you're trying to write code and need to concentrate. But it only took me maybe a week of constant use to ingrain prefix notation to the point where it didn't look completely alien any longer.

At this point prefix notation reads to me as easily as infix notation. I breeze right through Lisp code without a pause. In Clojure, you can write calls to Java methods in Java order like (. object method arg arg arg) or you can use a Lispy order like (.method object arg arg arg); I find myself invariably using the Lispy way, as does most of the community, even though the more traditional notation is available.

You can get used to it if you put in a minimal amount of effort. It's not that hard.

Benefits of Prefix Notation

Why bother using prefix notation if infix and prefix are equally good (or bad)? For one thing, prefix notation lets you have variable-length parameter lists for things that are binary operations in other languages. In an infix language you must say 1 + 2 + 3 + 4 + 5. In a prefix language you can get away with (+ 1 2 3 4 5). This is a good thing; it's more concise and it makes sense.

Most languages stop at offering binary operators because that's as good as you get when you have infix operators. There's a ternary operator x?y:z but it's an exception. In Lisp it's rare to find a function artificially limited to two arguments. Functions tend to take as many arguments as you want to throw at them (if it makes sense for that function).

Prefix notation is consistent. It's always (function arg arg arg). The function comes first, everything else is an argument. Other languages are not consistent. Which is it, foo(bar, baz), or bar.foo(baz)? There are even oddities in some languages where to overload a + operator, you write the function definition prefix, operator+(obj1, obj2), but to call that same function you do it infix, obj1 + obj2.

The consistency of Lisp's prefix notation opens up new possibilities for Lispy languages (at least, Lisp-1 languages). If the language knows the first thing in a list is a function, you can put any odd thing you want in there and the compiler will know to call it as a function. A lambda expression (anonymous function)? Sure. A variable whose value is a function? Why not? And if you put a variable whose value is a function in some place other than at the start of a list, the language knows you mean to pass that function as an argument, not call it. Other languages are far more rigid, and must resort to special cases (like Ruby's rather ugly block-passing syntax, or explicit .call or .send).

Consistency is good. It's one less thing you have to think about, it's one less thing the compiler has to deal with. Consistent things can be understood and abstracted away more easily than special cases. The syntax of most languages largely consists of special cases.

Parens: Use Your Editor

The second major supposed problem with Lisp syntax is the parens. How do you keep those things balanced? How do you read that mess?

Programming languages are partly for human beings and partly for computers. Programming in binary machine code would be impossible to read for a human. Programming in English prose would be impossible to parse and turn into a program for a computer. So we meet the computer halfway. The only question is where to draw the line.

The line is usually closer to the computer than to the human, for any sufficiently powerful language. There are very few programing languages where we don't have to manually line things up or match delimiters or carefully keep track of punctuation (or syntactic whitespace, or equivalent).

For example, any language with strings already makes you pay careful attention to quotation marks. And if you embed a quotation mark in a quote-delimited string, you have to worry about escaping. And yet we manage. In fact I think that shell-escaping strings is a much hairier problem than balancing a lot of parens, but we still manage.

This is sadly a problem we must deal with as programmers trying to talk to computers. And we deal with it partly by having tools to help us. Modern text editors do parenthesis matching for you. If you put the cursor on a paren, it highlights the match. In Vim you can bounce on the % key to jump the cursor between matching parens. Many editors go one step further and insert the closing paren whenever you insert an opening one. Emacs of course goes one step further still and gives you ParEdit. Some editors will even color your parens like a rainbow, if that floats your boat. Keeping parens matched isn't so hard when you have a good editor.

And Lisp isn't all about the parens. There are also generally-accepted rules about indentation. No one writes this:

(defn foo [x y] (if (= (+ x 5) y) (f1 (+ 3 x)) (f2 y)))

That is hard to read, sure. Instead we write this:

(defn foo [x y]
  (if (= (+ x 5) y)
    (f1 (+ 3 x))
    (f2 y)))

This is no more difficult to scan visually than any other language, once you're used to seeing it. And all good text editors will indent your code strangely if you forget to close a paren. It will be immediately obvious.

A common sentiment in various Lisp communities is that Lispers don't even see the parens; they only see the indentation. I wouldn't go that far, but I would say that the indentation makes Lisp code easily bearable. As bearable as a bunch of gibberish words and punctuation characters can ever be for a human mind.

When I was first learning Lisp I did have some pain with the parens. For about a week. After learning the features of Vim and Emacs that help with paren-matching, that pain went away. Today I find it easier to work with and manipulate paren-laden code than I do to work with other languages.

Benefits of the Parens

Why bother with all the parens if there's no benefit? One benefit is lack of precedence rules. Lisp syntax has no "order of operations". Quick, what does 1 + 2 * 3 / 4 - 5 mean? Not so hard, but it takes you a second or two of thinking. In Lisp there is no question: (- (+ 1 (/ (* 2 3) 4)) 5). It's always explicit. (It'd look better properly indented.)

This is one less little thing you need to keep in short-term memory. One less source of subtle errors. One less thing to memorize and pay attention to. In languages with precedence rules, you usually end up liberally splattering parens all over your code anyways, to disambiguate it. Lisp just makes you do it consistently.

As I hinted, code with lots of parens is easy for an editor to understand. This make it easier to manipulate, which makes it faster to write and edit. Editors can take advantage, and give you powerful commands to play with your paren-delimited code.

In Vim you can do a ya( to copy an s-exp. Vim will properly match the parens of course, skipping nested ones. Similarly in Emacs you can do C-M-k to kill an s-exp. How do you copy one "expression" in Ruby? An expression may be one line, or five lines, or fifty lines, or half a line if you separate two statements with a semi-colon. How do you select a code block? It might be delimited by do/end, or curly braces, or def/end, or who knows. There are plugins like matchit and huge syntax-parsing scripts to help editors understand Ruby code and do these things, but it's not as clean as Lisp code. Not as easy to implement and not as fool-proof that it'll work in all corner cases.

ParEdit in Emacs gives you other commands, to split s-exps, to join them together, to move the cursor between them easily, to wrap and unwrap expressions in new parens. This is all you need to manipulate any part of Lisp code. It opens up possibilities that are difficult or impossible to do correctly in a language with less regular syntax.

Of course this consistency is also partly why Lisps can have such nice macro systems to make programmatic code-generation so easy. It's far easier to construct Lisp code as a bunch of nested lists, than to concatenate together strings in a proper way for your non-Lisp language of choice to parse.

Conclusion

Yeah Lisp syntax isn't intuitive. But nothing really is. You can get used to it. It's that not hard. It has benefits.

Sometimes it's worth learning things that aren't intuitive. You limit yourself and miss out on some good things if you stick with what you already know, or what feels safe and sound.

Clojure 1, PHP 0

Mon, 16 Mar 2009 18:50:00 -0700

Goodbye Wordpress

As I mentioned many times, I've been working on replacing Wordpress for my blogging needs. Wordpress has been pretty good for the past three years, but it's time to move on, for a bunch of reasons.

Primarily, the way Wordpress automatically mangles my text is annoying. For example, it turns newlines into paragraphs inconsistently (especially when it comes to pre/code blocks). This blog is mostly about programming, which means being able to post code without having my quotes turned into "smart" quotes and my --flags turned into long-dashes is kind of important. HTML is sometimes automatically escaped, and sometimes not. I can't count how many comments I've gotten where someone posted some code, then posted again to inform me that Wordpress ate the code for dinner. There are plugins to fix some of this, which break every time Wordpress releases a new version, and have never really worked that well for me.

Writing a theme for Wordpress means a mix of PHP and HTML and CSS, which is painful to read and even more painful to write. Aside from the considerable ugliness of PHP itself, there's a lot of weird magic involved with themes, based on naming conventions for files, weird fall-through behavior when certain theme files aren't present and so on. The Wordpress API is enormous and not fun to work with if you want to do something other than the standard Wordpressy kind of blog structure. Static pages aren't too much fun to work with in Wordpress either.

Lately I think I was getting hammered with spam partly because Wordpress is such an easy target. Askimet is nice but it wasn't catching enough lately; maybe 10-15 spams per week were slipping through. And there was always the chance that some widely-known exploit in Wordpress was going to leave my site susceptible to some roving bot.

And so on.

Hello Clojure

Why Clojure? Because it's awesome and fun and powerful and I wanted to learn it better.

Compojure is a web framework for Clojure that made a lot of this very easy. Coming here from a Ruby on Rails background, Compojure has a lot going for it in comparison. Compojure is lightweight and more low-level than Rails. For example Compojure doesn't enforce MVC on you, doesn't force a unit testing framework on you, and doesn't care how you access your data. Compojure just lets you route HTTP requests to Clojure functions based on the URL and request method (RESTfully: POST/GET/DELETE/PUT), and it gives you easy access to the request information, session, GET/POST parameters and cookies.

Under the hood it's all servlets and Jetty, both of which are solid, stable, well-tested, well-documented technologies. However, thankfully, all of that Java stuff is under the hood, and well under it. I didn't have to write a single line of Java or interact with single servlet directly. Everything (session, params, headers) is a Clojure hash-map from the perspective of my code.

Compojure also comes with a domain-specific language for writing HTML, which is similar to CL-WHO and myriad other Common Lisp HTML DSL's. All of which are awesome. I can't say enough how much nicer it is to write (or generate) structured s-exps than to write HTML by hand. More on that below.

Compojure doesn't come with any way to interact with a database, so I had to write one. clojure.contrib has an SQL lib which easily lets you interact with a MySQL database. (Clojure can talk to MySQL via MySQL's JDBC connector, of course.) I used clojure.contrib.sql to write a small (192 lines) library which slurps up a bunch of database tables into Clojure refs, and provides a few functions for basic CRUD operations so that any updates to the ref data is also transparently reflected in the database. The database is essentially only for keeping an on-disk cache of the data in case I need to restart the server. The average number of DB queries per page is zero; everything except posting/editing/deleting data just reads out of a Clojure ref.

With possibly multiple users posting data at once, it's nice to have Clojure's built-in concurrency support. Updating the data refs with new data is always safe from multiple threads simply by throwing a (dosync) around all of the write accesses. This was completely painless to write.

I decided I wanted to use Markdown for posting comments and authoring new pages. This was also very simple to do; I outlined how to get Markdown working in Java and Clojure, in a previous post. The real-time previews for comments are largely inspired by / ripped-off from Stack Overflow, implemented mostly using open-source Javascript libraries like Showdown, JQuery, TypeWatch and TextAreaResizer.

A Brief Comparison: Clojure vs. Wordpress

All of my code including the CRUD library, all of the HTML for the templates and layout, admin controls, and all the glue to put it together is 1,253 lines of code. Wordpress is somewhere over 78,000 lines of PHP depending what you count (doesn't include any themes or layout, but does include Wordpress features I didn't need and didn't implement). It's still a pretty nice reduction in code overall, any way you look at it.

As an example, in my old Wordpress site I had a plugin catcloud to generate a "tag cloud". This plugin itself is 226 lines of PHP, not bad. However, here's the Clojure code to generate a similar tag cloud (which you can see here currently):

(defn tag-cloud []
  (let [tags (sort-by #(.toLowerCase (:name (first %))) (all-tags-with-counts))
        counts (map second tags)
        max-count (apply max counts)
        min-count (apply min counts)
        min-size 90.0
        max-size 200.0
        color-fn (fn [val]
                   (let [b (min (- 255 (Math/round (* val 255))) 200)]
                     (str "rgb(" b "," b "," b ")")))
        tag-fn (fn [[tag c]]
                 (let [weight (/ (- (Math/log c) (Math/log min-count))
                                 (- (Math/log max-count) (Math/log min-count)))
                       size (+ min-size (Math/round (* weight
                                                       (- max-size min-size))))
                       color (color-fn (* weight 1.0))]
                   [:a {:href (:url tag)
                        :style (str "font-size: " size "%;" "color:" color)}
                    (:name tag)]))]
    (block nil
           [:h2 "Tags"]
           [:div.tag-cloud
            (apply html (interleave (map tag-fn tags)
                                    (repeat " ")))])))

This is 10 times less code, which is a good reduction in my opinion. Most of the code is the math to generate a weight logarithmically for each tag so they scale nicely. (all-tags-with-counts) fetches a seq of two-item pairs: the tags themselves (which are hash-maps) and a count of posts for each tag. There are two locally-defined functions in the let which generate the text color and the font size and HTML for each tag.

The vectors that look like [:h2 "Tags"] are input for Compojure's HTML-generating DSL; this would be transformed for example into <h2>Tags</h2>. (block ...) is a macro which wraps its content in HTML for the rounded borders of my layout. (Math/log ...) and friends are calls to standard Java math functions.

This whole function is less code than just the horrible boilerplate array declarations at the top of the Wordpress plugin:

$catcloud_field_data = array(
  array('name' => 'Minimum Font Size', 'option' => 'catcloud_min_font_size', 'size' => '4', 'maxlength' => '3',
       'default' => '9', 'note' => 'Used for the least frequent categories', 'validation' => '/^\d{1,3}(\.\d{1,3})?$/'),
  array('name' => 'Maximum Font Size', 'option' => 'catcloud_max_font_size', 'size' => '4', 'maxlength' => '3',
       'default' => '18', 'note' => 'Used for the most frequent categories', 'validation' => '/^\d{1,3}(\.\d{1,3})?$/'),
  array('name' => 'Font Face', 'option' => 'catcloud_font_face', 'size' => '15', 'maxlength' => '254',
       'default' => '', 'note' => 'Set an optional list of font faces', 'validation' => '/.*/'),
  array('name' => 'Font Units', 'option' => 'catcloud_font_units', 'size' => '3', 'maxlength' => '2',
       'default' => 'pt', 'note' => 'Choose one of em, pt, px or %', 'validation' => '/^(%|em|pt|px)$/'),
  array('name' => 'Color Start', 'option' => 'catcloud_color_start', 'size' => '7', 'maxlength' => '6',
       'default' => '0066CC', 'note' => 'For the least frequent categories. Use a hexadecimal RGB triplet. ie. 0066CC',
       'validation' => '/^[\dA-F]{6}$/i'),
  array('name' => 'Color End', 'option' => 'catcloud_color_end', 'size' => '7', 'maxlength' => '6',
       'default' => 'CC6600', 'note' => 'For the most frequent categories. Use a hexadecimal RGB triplet. ie. CC6600',
       'validation' => '/^[\dA-F]{6}$/i'),
  array('name' => 'Before Category', 'option' => 'catcloud_before', 'size' => '3', 'maxlength' => '20',
       'default' => '[', 'note' => 'Set the character(s) to display before category names', 'validation' => '/.*/'),
  array('name' => 'After Category', 'option' => 'catcloud_after', 'size' => '3', 'maxlength' => '20',
       'default' => ']', 'note' => 'Set the character(s) to display after category names', 'validation' => '/.*/'),
  array('name' => 'Show Top N Categories', 'option' => 'catcloud_top_n_cats', 'size' => '5', 'maxlength' => '3',
       'default' => '', 'note' => 'Show only the top N categories (where N is a number like 10 or 25 or whatever. Set to 0 or empty for no limit.',
       'validation' => '/^\d*$/'),
  array('name' => 'Excluded Categories', 'option' => 'catcloud_excluded_cats', 'size' => '15', 'maxlength' => '254',
       'default' => '', 'note' => 'A comma-separated list of category ids.',
       'validation' => '/^[\d, ]*$/'),
)

Ugh. As another example, here's the code that handles a POST request to add a new blog page:

(defn do-new-post []
  (check-login
   (let [post (add-post *params*)]
     (sync-tags post (:all-tags *params*))
     (redirect-to "/"))))

It does exactly what it says: Check to make sure the user is logged in, add the post based on the POST params, sync up the tags for that post and redirect to the front page. Lisp lets you say what you want very concisely, with a bare minimum of boilerplate.

How about speed? My Clojure code is actually generating HTML in the most brute-force and wasteful way possible. The HTML for each page is regenerated from scratch, via a cascade of a couple dozen function and macro calls, every time you load a page. But it's still pretty fast, a couple hundred milliseconds for most page requests. This is slightly faster than the Wordpress version of my site. If I ever have performance issues I can switch to another Clojure HTML library, like clj-html which uses the same vector-style syntax but pre-compiles the HTML.

How hard was it to set up on the server? Wordpress is pretty famous for being dirt-easy to deploy anywhere. My Clojure app by comparison was slightly more difficult, as you might expect, but it wasn't brain surgery. My server runs Debian. First I installed the JVM via apt, then I rsynced a bunch of jar's and clj files to the server, then I installed emacs and screen also via apt. Then I put two lines into an Apache config file to proxy-forward traffic to a local port where jetty would be listening. I started Emacs, did (require 'bcc.blog.server), did (bcc.blog.server/go) to start everything, and that's about it. Took about 15 minutes to set up from scratch. When I find a bug, I SSH in, re-attach to screen, fix it in Emacs, hit C-c C-c to recompile just the functions I need to update, and then detach from screen again.

I'm pretty pleased with this so far. It was fun to write and has all the features I used from Wordpress, plus more, and the building blocks are there to extend things if I imagine up a new feature I like.

Looks like my blog is still running today in spite of my predictions. Still waiting for the JVM to crash though, I know it's coming. I plan to post the source code for some of this once I'm sure it works.

Clojure: 1, Common Lisp: 0

Mon, 19 Jan 2009 02:56:04 -0800

Waaaay back in January 2008 I finished my origami gallery photo-blog, written from scratch in Common Lisp. It took me about a month and half of struggling to get it going.

Shattered Dreams

Ah, to be young again. I was very enthusiastic about learning a Lisp, and Common Lisp was just about the best in town. I'd recently read Practical Common Lisp and maybe I hadn't imbibed the Kool-Aid yet, but I'd sipped it quite a bit.

Then reality came a'knocking. SBCL couldn't run on my server's VPS without hacking and recompiling it (many times) to fix memory-mapping issues and get threading working. Cleanly installing all the necessary libraries via ASDF was a problem and a half. Learning Emacs and SLIME was just about the most painful computer experience I've ever been through.

And those were the big things. There were so many little things. Dealing with pathnames in a language that was written before modern-day pathnames were semi-standardized. Destructive vs. safe list operations. Crappy hash-table support, to the point where I gave up and used lists for everything. CL-SQL, which is pretty good but whose syntax and verbosity and quirkiness leaves much to be desired. (I never got the CLOS stuff in CL-SQL working.)

Trying to wrap my head around macros enough to use CL-WHO effectively (never did get it right). Trying to wrap my brain around CL packages and setting up ASDF to work with my own code (I got this working, but I couldn't tell you how at this point). The list goes on and on.

Once I finished the site, I was proud of slogging through it, but I was also exhausted. Common Lisp is language that looks great on paper but it never clicked with me. I set up some Debian init scripts on my server to make sure SBCL would start if my server restarted, and then I tried my best to forget about it. I was burned out.

Sometime a few weeks ago, my origami gallery stopped working. Nothing but 404's. I'm not sure when or why. I tried running my script to restart the SBCL background process and it died with a Bible-and-a-half worth of errors in SBCL. Sigh. I didn't have the patience or the desire to fix it.

Clojure to the rescue

So, today on a whim I decided to re-write the site in Clojure to get it back up and running, using Compojure, a nice new Clojure web framework.

I'm happy to say I'm already done. My newly-Clojure-powered origami gallery, up and running. Cheerfully untested in Internet Explorer, but works in Firefox and Opera.

Start to finish the whole thing took me 8 hours. That includes time writing that little thumbnail-scrolling strip at the bottom in JQuery, a bit of Javascript to hide and show the comments pane, a new stylesheet, and also a bit of time to resize and touch up a couple of new photos of new models to post. The whole site weighs in at 350 lines of Clojure code, which includes an ORM-ish database layer, all of the HTML (s-exp style) and all the controller code.

It was such a joy to write, compared to my first slog through Common Lisp. But why? I don't feel appreciably smarter than I was a year ago. Why did it take me 8 hours this time but 2 months last time? Probably thanks to Clojure itself.

Deploying SBCL was one of my biggest roadblocks last time. By contrast, installing Clojure is easy, even in its infancy where there are sometimes quirks with SLIME compatibility and such. These problems are always minor and the mailing list is always on top of them.

Installing libs is easy, you throw a jar into a directory and you're done. I deployed everything to my Debian server in 15 minutes, which included installing the JVM for the first time, fetching Clojure and all required libs, compiling them, and setting up an environment. (Pro-tip, there's a new bash launcher script in clojure-contrib now, which makes starting Clojure a bit easier and more standard.)

How do you install MySQL support for Clojure? You don't; you install it for Java. There is official documentation on the MySQL website about getting it to work. It's a single jar file you download and throw into your CLASSPATH. Then get the SQL lib from clojure-contrib; 5 minutes of documentation reading, and I was done. When's the last time you saw official documentation on a vendor's website for a Lisp?

That's how it is with Clojure. Compojure uses Jetty as its HTTP server. Jetty is mature, stable, widely used and very well documented. If we had to wait for someone to write an HTTP server for Clojure from scratch, where would we be? I can't say enough times how great it is to be able to slurp up all the Java resources in the world and play with them in a Lisp.

But it's the little things too, that make Clojure such a joy. How do you concatenate strings in Clojure? (str string1 string2 string3). How do I access the name of a "comment" object? (:name comment). How do I set up Compojure so that when someone accesses the url "/" it calls a function called index-page? (defservlet my-servlet (GET "/" (index-page))).

The Compojure HTML-generating library takes full advantage of Clojure literal syntax so that you can do things like [:a {:href "http://google.com"} (str "Goo" "gle")] to output an HTML link, using a mixture of vector and hash literals and function calls. This alone makes it far more pleasant to use than CL-WHO (and much nicer than writing raw HTML).

And so on. Easy easy easy.

Example one: What time is it?

Here is an issue that exemplifies the kind of hassle I went through in Common Lisp, that I never hope to go through again. How can you get Common Lisp to tell you the current time, and store it in your database? I use this when people post comments, to capture the time the comment was posted. And some other things.

You can read all about getting the current time in Common Lisp here or here. Ignoring that there are two representations of time to choose from (universal vs. internal), the important thing to note is that neither is easily used for anything. CL-SQL meanwhile has types "wall-time" and "universal-time".

So we'll go with universal time. This seemed to be the most popular way to store a time in a database field, back when I researched it. Universal time is a count of milliseconds since 1900. How do you turn this into something a database can understand as a timestamp, or turn it into something readable for human beings? Many languages stores times in a similar way as a huge integer, but most also give you really easy ways to turn millisecond counts into something legible. Not in Common Lisp. Instead, it's as simple as

(multiple-value-bind
    (second minute hour date month year day-of-week dst-p tz)
    (get-decoded-time)
    (format t "It is now ~2,'0d:~2,'0d:~2,'0d of ~a, ~d/~2,'0d/~d (GMT~@d)"
          hour
          minute
          second
          (nth day-of-week *day-names*)
          month
          date
          year
          (- tz)))

In other words, it's not simple. At all. Sure it's just a couple of utility functions (and learning the magic, cryptic FORMAT language) away, but multiply a couple of utility functions by the dozens upon dozens of times I have to do this kind of crap for a simple 300-line website, and it turns out I'm not making a website any longer, instead I'm ad-hoc re-writing Common Lisp to do what any language invented in the past two decades can do out of the box. (e.g., in Ruby it's Time.now).

I never did get this working entirely right for my site. Timestamps worked and were stored in the DB, but they were never in the proper timezone for some reason. I could've fixed it, but it was a low-priority problem below sundry other problems.

So how do you do this in Clojure? When I use the SQL lib from clojure-contrib, TIMESTAMP values in the database end up as Java Timestamp objects in Clojure by the time I see them. You can read about it here in simple, easily-navigable javadoc. I call (.toString timestamp) and it gives me a human-readable version of the time. Or I can just do (str timestamp). Or I can use .toLocaleString (deprecated) or use a DateFormat object if I want anything fancier. The end. Because Java can do it, Clojure can do it.

Example two: Filenames

How do I get a list (or vector) of all the files in a directory? For my photo-blog I use this to get lists of thumbnails for the photos. In Clojure it was simple enough that I wrote it out myself; there may be a shorter way.

(defn glob [dirname] (into [] (.list (new java.io.File dirname))))

For Common Lisp you probably want to asdf-install CL-FAD, which "Returns a 'fresh' list of pathnames corresponding to the truenames of all files within the directory named by the non-wild pathname designator dirname." What the hell does that even mean? In fact I do know what it means, but only after plenty of reading. Completely unnecessary reading, if I was using any other language.

Just be careful with filenames, because CL has two ways of representing directories, and this also varies between implementations. This is enough of a problem that a whole chapter of PCL is devoted to writing a library to fix it.

I think the reason I got this project done so much more quickly this time is mostly because I'm using a better language for the job.

Update: hi Reddit. Thanks for DDOS-murdering my server. :)

Books are our friends

Sun, 30 Nov 2008 22:22:01 -0800

I went on a reading binge over Thanksgiving break. Read on for reviews.

Test-Driven Development By Example

First I read Test-Driven Development By Example. This is a very short book, especially given the hefty price tag. I would probably not have bought this if I'd seen it on a shelf at the store rather than online.

That said, it's a good book if you want to understand the mindset behind the whole test-driven development fad. The book is heavy on mindset and light on mechanics; it doesn't tell you how to set up an environment, how to compile and run tests, or any such thing. It just goes through some examples and explains what the programmer was thinking, and how to tackle the problem from a TDD point of view. The intended audience therefore is someone who has plenty of experience programming and wants to learn a new way to look at problem solving.

The book goes through writing some terrible code (admittedly terrible by the author) to solve a simple problem, then refactoring the code to be less and less terrible by means of tests. The writing style is engaging and casual and he describes mistakes as well as successes, which is a nice way to write a tutorial book. The alternate writing style, belting out the correct answer the first time every time, is not as enlightening, and I appreciate it in this book.

The book places a heavy emphasis on writing tests FIRST, making small incremental changes, and refactoring as you go. It also explains how TDD can be related to various OOP design patterns, which I didn't find all that helpful. And it describes some "test patterns", which I found even less helpful. But it's mostly an aesthetic objection; something about "design patterns" sits less and less well with me the more I get away from Java-like languages. The information is still good, straightforward and to the point, take it or leave it.

I have been unable to drink the TDD Kool-Aid, but I can see where it'd probably result in an improvement in some of my code for certain specific kinds of problems. I'll probably try it out again next time I have to write a little library at work.

This book is expensive and short, but the alternative for learning about TDD are half-baked blog posts, which is why I bought a book. I don't know of a better book because this is the only one I have, and I'm unsure at this point whether I really want to buy another book on this topic. But this book is highly recommended by many TDD ~~fanatics~~ enthusiasts, so I don't know.

The Little Schemer

Next I read The Little Schemer. The style of this book can best be described as "cute". Examples all use food, and it instructs you at various points to go make yourself a sammich (even leaving room in the book for "jelly stains"). This somehow makes the topics, which can be rather intimidating, somehow less scary. The book is very short, and not very dense (not many words on a page), but they somehow cram a lot of information into the book anyways, via the many examples. It's a very focused and methodical book.

This book goes through developing a simplified dialect of Scheme, with a focus on recursion and building up parts of the language from very simple primitives. There are some neat little things along the way, like using s-expressions to represent numbers, and building up a full set of arithmetic functions using nothing but "add 1", "subtract 1", and recursion. It'd be insane to do in real life because of performance, but it's really neat from a "hey look what you can do, isn't this cool!" point of view.

The first half of the book is pretty simplistic (at least for me, with some basic knowledge of Lisp and Scheme), but the second half really starts delving into some crazy things, passing lambdas around to lambdas and writing evaluators and stuff. But you almost don't even notice because of how solidly the foundations are built up to that point, and how well the examples are explained. (You probably will notice once you hit the y-combinator, on account of your brains detonating.)

This not a good book for "learning Lisp" in terms of learning the gritty details of to use a real-life implementation of Lisp. But it's a good book for learning some of the concepts that make Lisp and functional languages powerful. This is a very interesting and unique book, also highly recommended by many in the Lisp world. I'm already planning to try The Seasoned Schemer next.

Real World Haskell

Finally I started plowing through the recently-released Real World Haskell. This book is freely available online, which is great. It also allows readers to make comments on every paragraph in the book, which is a highly collaborative and probably very intelligent way to write a book, and also can provide insight for readers who don't understand the text at any given point. I may buy a hard copy, I haven't decided. (The online version has some FIXME notes and typos that I can only hope are fixed in the real thing.)

I'm only up to chapter 11, but it's good so far. It gives a ton of examples and goes slow enough for people completely new to Haskell to pick the language up. And it includes some of the mechanics of compiling and/or running programs in GHC, which is extremely helpful. I did start getting lost around the time monads were introduced, but that's to be expected. It usually takes me two or three good reads of this kind of book before I grok it all, which is no fault of the book.

Haskell. Last time I used it, I wrote Pong in Haskell in one of my college classes. It was an immensely painful experience. I'm going back to re-learn Haskell now because I find myself edging more and more toward functional languages and away from the C/Java world.

Haskell is far too much to swallow if you have no experience with functional programming, which is probably why I hated it in college. (That said, a smarter person than I may have fewer problems.) After graduating school, I got my first whiff of this world again via Perl's map and grep and friends. Then Ruby pushed it home for me with its widespread use of block parameters (lambdas in disguise). And after learning Lisp (and especially later, Clojure) I can't live without higher-order functions. If I had to write a classic for-loop and manually keep track of a counter variable I'd probably vomit at this point. Haskell takes function manipulation to an even greater extreme, which I still haven't fully wrapped my head around, but I like what I've learned so far.

Laziness is awesome, function currying is awesome, and this is the first time I've read about folding, apart from brief struggles with Ruby's (poorly named) Enumerable#inject. Pattern matching strikes me as vaguely similar to Common Lisp's / Clojure's destructuring-binding, which is one of the nicest features I've seen in any language when it comes to making a programmer's life easier.

Maybe it's all just interesting because it's still new to me. But I like the whole idea of functional programming. I like the idea of functions that always produce the same output from a given input. How and when to handle side-effects and object mutation is a problem that's always nagged at the back of my mind even when writing Ruby code.

That said, Haskell still (at this point in the book) does not strike me as a practical or real-world language in the slightest. You've got to jump through some crazy hoops to get a lot of things to work, especially when it comes to I/O. The book describes how to write a pretty-printer, and how to parse a binary file, both of which require some acrobatics to write concisely in Haskell. In particular the parser library lost me entirely; functions were being chained to functions in all directions and I couldn't follow it.

I think one reason Java is so popular is that you don't have to be a genius to write it. To write Haskell (or even Lisp) well, to take advantage of the language and use it smoothly, you really do need to do some deeper thinking. The abstractions are more powerful and more "abstract". It's not too hard to understand a world made of objects with a bunch of knobs you turn via method calls. Lambdas and recursive functions and monads and typeclasses are a more ephemeral thing.

That said I plan to stick with the book. Practice makes perfect.

Qt4 in Lisp?!

Fri, 31 Oct 2008 16:31:43 -0700

NOTE: Updated 2008-12-12 to reflect Clojure's new syntax.

Imagine Qt4 bindings for Lisp that are:

Officially supported
Thoroughly documented, with tons of examples
Cross-platform
Consistently up-to-date

Keep dreaming, right? Your first thought might be to check CLiki for some Common Lisp bindings. There is indeed a Qt project there. Except at a glance, it's listed as working only for CMUCL. And the last update was 2003. And the download link is 404'd. Oops!

But there's a link to a some QT4 bindings, CFFI style. Much more promising. Except... last update: March 2007. "Project status: dead." Never mind. There's also this and maybe a few others you could Google up, but I haven't tried them and I'm not going to.

Kudos to the people who wrote the above; providing bindings for all of Qt4 is a daunting task and it's no wonder it's not done for Common Lisp. Doesn't change the reality of the fact that Qt4 (like many other libraries) is a no-go in Common Lisp. There's nothing stopping Common Lisp from having great libraries except lack of manpower; but lack of manpower and lack of mature, tested, up-to-date libraries is still a real problem when you want to write an application today.

Enter Clojure

Luckily, it turns out there ARE Qt4 bindings that you can use from Lisp. That Lisp is Clojure, and the bindings are Qt Jambi. Many people are excited about Clojure nowadays, and this is one big reason (of many). It's hard to beat Java when it comes to libraries, and Clojure gets them all for free.

Let's try to write one of the official Qt4 examples / tutorials in Clojure and see how it goes. Keep in mind that I'm still learning Clojure, so this may be far from ideal, but it should give you a bit of a taste of Clojure and Qt Jambi if you've never seen it before.

To spoil the ending a bit: It works! In Vista and Linux (I don't have an OS X box to try, but it'll probably work there too):

Setup

To set up Clojure, follow the directions on the Clojure wiki. Be sure to get the SVN version of Clojure, because it's being developed very rapidly and is often updated many times per week. You can also check out some recent posts on Bill Clementson's blog, and this movie on LispCast which walks through setting up Clojure and Emacs/SLIME.

To make Qt Jambi available to Clojure, you may have some options. If you use Linux, your distro may make this available via your package manager. Otherwise go to the Qt download site and download "Qt for Java" for your OS. This is a freaking huge download, but it includes all the documentation and source code and plugins; all you need out of it are a couple of JAR files. There are two JARs you need, one cross-OS and one specific to your OS.

You have to put these JARs on your CLASSPATH so Clojure can find them. There are lots of ways to do this; one way is on the commandline:

java -cp qt-jambi-4.4.3_01.jar:qtjambi-linux32-gcc-4.4.3_01.jar:clojure-lang-1.0-SNAPSHOT.jar clojure.lang.Repl

There's also add-classpath in Clojure, which lets you edit the classpath while Clojure is running. I was unable to get this to work with Qt Jambi, but I didn't try very hard. In reality I just use a shell script so I can dump a bunch of JARs into a directory, and they will all be added to CLASSPATH automatically when I start Clojure.

That's it. Installing libraries in Clojure couldn't be much easier.

The Code

Here's the complete example code for download. Let's walk through it a bit. First we have to import the Qt Jambi apps into Clojure. Qt Jambi is on our CLASSPATH and available to Clojure, but we still have to explicitly import the bits of it we want:

(ns clojure-qt4-demo
  (:import (com.trolltech.qt.gui QApplication QPushButton QFont QFont$Weight)
           (com.trolltech.qt.core QCoreApplication)))

ns creates a new namespace, and provides convenience syntax to import Java classes at the same time. It can also import other Clojure files or Clojure libraries, among other things.

Note immediately how this is nicer than Java. In Java you would be writing something like

import com.trolltech.qt.gui.QApplication;
import com.trolltech.qt.gui.QPushButton;
import com.trolltech.qt.gui.QFont;
...

And so on, over and over. If you're lucky you have a bloated IDE to type those things for you. In Clojure, macros let us factor out the repetition; we name the big long path once, and pluck out the things we want. The other option in Java is to import com.trolltech.qt.gui.*, which pollutes your namespace unnecessarily; Clojure lets us succinctly take just what we want.

One thing I couldn't find documentation for is the way to import a static inner subclass of another class. To do this you have to use syntax like SomeClass$SomeSubClass. My first thought was QFont/Weight or QFont.Weight, but those don't work.

If you're typing all of this at a REPL, you now have to do:

(in-ns 'clojure-qt4-demo)

to switch to the namespace we just created. Next we set up some convenience functions:

(defn init []
  (QApplication/initialize (make-array String 0)))

In Qt4, the first thing you always do is initialize QApplication. QApplication is essentially a singleton class, and it has to be initialized via QApplication.initialize(), a static method call, before you do anything else. In Clojure this becomes (QApplication/initialize), which says to call the function or static method called "initialize" in the namespace called "QApplication". Static methods in Java become functions in a class-namespace in Clojure.

This function expects an array of Strings, which in a normal app would be commandline parameters. In Clojure we can make a native Java Array of Strings via make-array. I just pass in an empty one because I'm running this from a REPL. Note, a native Java Array of Strings is different from a Clojure persistent array, [ ]. This is a bit nasty, but necessary for Java interop. Most of the time you will be able to get by with [ ]. Next:

(defn exec []
  (QApplication/exec))

This function (again a static method call) is generally the LAST thing you do in a Qt application. It starts displaying widgets and fires up the event loop.

A gotcha here is that certain things must happen in a certain order. A Qt app looks something like this:

QApplication.initialize()
Initialize and set up all of your widgets etc.
QApplication.exec()
Either the program exits, or go to step 1 to start over.

For example if you try to exec() before you initialize(), it does nothing at all. If you call initialize() more than once, it dies with a RuntimeException. When I was playing around with this at the REPL, it was very common that I called initialize multiple times by mistake. It's often safe to swallow the exception though.

All of that bookkeeping really wants to be made into a macro. Here's a simple macro:

(defmacro qt4 [& rest]
  `(do
     (try (init) (catch RuntimeException e# (println e#)))
     ~@rest
     (exec)))

This kind of thing lets you avoid the insanity of Java, which forces you to re-type exception-handling code and other boilerplate code over and over and over. You can abstract that all away in Clojure.

Macros in Clojure are much like Common Lisp macros. ~@ is a splicing-unquote, like ,@ in Common Lisp. Another interesting thing here is e#, which is a shorthand way to create a gensym. It creates a unique symbol; this prevents our macro from stepping on the toes of any other symbol called "e".

Using this macro we can finish our simple example:

(defn hello-world []
  (qt4
   (let [app (QCoreApplication/instance)
         button (new QPushButton "Go Clojure Go")]
     (.. button clicked (connect app "quit()"))
     (doto button
       (.setFont (new QFont "Deja Vu Sans" 18 (.. QFont$Weight Bold value)))
       (.setWindowTitle "Go Clojure Go")
       (.resize 250 100)
       (.show)))))

EDIT: Updated doto's to reflect Clojure's latest syntax.

The hello-world function is pretty straightforward. It makes a button, sets up an event handler that closes our app when the button is clicked, then sets its title and font, resizes it and shows it. Note how seamless this is. Aside from all the new's and .'s, you could barely tell this was Java. It's all tasty Lisp.

Clojure has a bunch of simple convenience macros that make writing Java less painful, two of which I show here. One of the biggest problems with Java is its extremely verbose and punctuation-heavy C-like syntax. Compare:

button.resize(250, 100);
button.setFont(new QFont("Deja Vu Sans", 18, QFont.Weight.Bold.value()));
button.setWindowTitle("Go Clojure Go");
button.show();

Look how much repetition there is in the code. button.blah, button.blah, button.blah. Repetition is bad. Clojure has the doto macro which says "take this thing, and do a bunch of stuff to it" without having to retype it every time:

(doto button
  (.resize 250 100)
  (.setFont (new QFont "Deja Vu Sans" 18 (.. QFont$Weight Bold value)))
  (.setWindowTitle "Go Clojure Go")
  (.show))

EDIT: Updated doto's to reflect Clojure's latest syntax.

Another macro is .. which says "take these things and put dots between everything", chaining method calls. So instead of QFont.Weight.Bold.value(), you can say (.. QFont$Weight Bold value). Handy.

To run this from a REPL, type

(hello-world)

Verdict

This app of course is a silly toy and just scratches the surface. But in my opinion, Clojure already beats all other Lisps by providing access libraries like this one. This is one of its killer features for me. The fact that you can seamlessly and easily use any Java library from Clojure is pretty amazing.

Why would you want to run Qt4 apps from Lisp anyways? Java (and thus Clojure) already have other GUI frameworks, like Swing and SWT and AWT, so why bother with this? Well, a few reasons...

Because I can.
Qt4 has a lot of impressive features nowadays. Run this Qt Jambi example to see some of those features.
Qt's design may appeal more to you than Swing's, which can be clunky at times.
Maybe you want to integrate well into KDE. I first started exploring GUI programming in Clojure when trying to make an icon sit in the system tray, and while Swing can do this, it's a bit painful to get it to look just right. It's trivial to do in Qt4.
If I'm forced to write a Qt4 app, I'd much rather write it in a highly interactive and iterative way using Clojure than struggle with a write/compile/debug/recompile/wait-for-hours cycle. At the very least Clojure would be ideal for prototyping.
Because I can!

Verdict: Clojure rules. But Java interop is just one reason it rules. Clojure has native Perl/Ruby-like reader support for hashes/arrays/sets/regexes/etc., support for easy and safe concurrency, cross-platform-ness galore, and a much more modern feel than Common Lisp. It's being worked on by some very smart people and the community is vibrant, enthusiastic, and welcoming. It has all the strengths of Java and Common Lisp, and very few of their weaknesses.

Practicality: PHP vs. Lisp?

Mon, 22 Sep 2008 01:17:25 -0700

Eric at LispCast wrote an article about why PHP is so ridiculously dominant as a web language, when arguably more powerful languages like Common Lisp linger in obscurity.

I think the answer is pretty easy. In real life, practicality usually trumps everything else. Most programmers aren't paid to revolutionize the world of computer science. Most programmers are code monkeys, or to put it more nicely, they're craftsmen who build things that other people pay them to create. The code is a tool to help people do a job. The code is not an end in itself.

In real life, here's a typical situation. You have to make a website for your employer that collects survey data from various people out in the world, in a way that no current off-the-shelf program quite does correctly. If you could buy a program to do it that'd be ideal, but you can't find a good one, so you decide to write one from scratch. The data collection is time-sensitive and absolutely must start by X date. The interface is a web page, and people are going to pointy-clicky their way through, and type some numbers, that's it; the backend just doesn't matter. For your server, someone dug an old dusty desktop machine out of a closet and threw Linux on there for you and gave you an SSH account. Oh right, and this project isn't your only job. It's one of many things you're trying to juggle in a 40-hour work week.

One option is to write it in Common Lisp. You can start by going on a quest for a web server. Don't even think about mod_lisp, would be my advice, based on past experience. Hunchentoot is good, or you can pay a fortune for one of the commercial Lisps. If you want you could also look for a web framework; there are many to choose from, each more esoteric, poorly documented and nearly impossible to install than the last. Then you get to hunt for a Lisp implementation that actually runs those frameworks. Then you get to try to install it and all of your libraries on your Linux server, and on the Windows desktop machine you have to use as a workstation. Good luck.

Once you manage to get Emacs and SLIME going (I'm assuming you already know Emacs intimately, because if you don't, you already lose) you get to start writing your app. Collecting data and moving it around and putting it into a database and exporting it to various statistics packages is common, so you'd do well to start looking for some libraries to help you out with such things. In the Common Lisp world you're likely not to find what you need, or if you're lucky, you'll find what you need in the form of undocumented abandonware. So you can just fix or write those libraries yourself, because Lisp makes writing libraries from scratch easy! Not as easy as downloading one that's already been written and debugged and matured, but anyways. Then you can also roll your own method of deploying your app to your server and keeping it running 24/7, which isn't quite so easy. If you like, you can try explaining your hand-rolled system to the team of sysadmins in another department who keep your server machine running.

Don't bet on anyone in your office being able to help you with writing code, because no one knows Lisp. Might not want to mention to your boss that if you're run over by a bus tomorrow, it's going to be impossible to hire someone to replace you, because no one will be able to read what you wrote. When your boss asks why it's taking you so long, you can mention that the YAML parser you had to write from scratch to interact with a bunch of legacy stuff is super cool and a lovely piece of Lisp code, even if it did take you a week to write and debug given your other workload.

Be sure to wave to your deadline as it goes whooshing by. If you're a genius, maybe you managed to do all of the above and still had time to roll out a 5-layer-deep Domain Specific Language to solve all of your problems so well it brings tears to your eye. But most of us aren't geniuses, especially on a tight deadline.

Another option is to use PHP. Apache is everywhere. MySQL is one simple apt-get away. PHP works with no effort. You can download a single-click-install WAMP stack nowadays. PHP libraries for everything are everywhere and free and mature because thousands of people already use them. The PHP official documentation is ridiculously thorough, with community participation at the bottom of every page. Google any question you can imagine and you come up with a million answers because the community is huge. Or walk down the hall and ask anyone who's ever done web programming.

The language is stupid, but stupid means easy to learn. You can learn PHP in a day or two if you're familiar with any other language. You can write PHP code in any editor or environment you want. Emacs? Vim? Notepad? nano? Who cares? Whatever floats your boat. Being a stupid language also means that everyone knows it. If you jump ship, your boss can throw together a "PHP coder wanted" ad and replace you in short order.

And what do you lose? You have to use a butt-ugly horrid language, but the price you pay in headaches and swallowed bile is more than offset by the practical gains. PHP is overly verbose and terribly inconsistent and lacks powerful methods of abstraction and proper closures and easy-to-use meta-programming goodness and Lisp-macro syntactic wonders; in that sense it's not a very powerful language. Your web framework in PHP probably isn't continuation-based, it probably doesn't compile your s-expression HTML tree into assembler code before rendering it.

But PHP is probably the most powerful language around for many jobs if you judge by the one and only measure that counts for many people: wall clock time from "Here, do this" to "Yay, I'm done, it's not the prettiest thing in the world but it works".

The above situation was one I experienced at work, and I did choose PHP right from the start, and I did get it done quickly, and it was apparently not too bad because everyone likes the website. No one witnessed the pain of writing all that PHP code, but that pain doesn't matter to anyone but the code monkey.

If I had to do it over again I might pick Ruby, but certainly never Lisp. I hate PHP more than almost anything (maybe with the exception of Java) but I still use it when it's called for. An old rusty wobbly-headed crooked-handled hammer is the best tool for the job if it's right next to you and you only need to pound in a couple of nails.

Perl6 features borrowed from Lisp

Tue, 09 Sep 2008 15:28:57 -0700

Via PerlMonks I found a couple of articles discussing in good detail some of the new features of Perl6.

Perl6 steals even more things from Common Lisp than Perl5 did: it has multimethods / multiple dispatch for example, which is a huge plus. Via this interview with Damian Conway we learn that Perl6 will also have named, optional, and "rest" parameters to subs, just like in CL. That's also a good thing; CL's parameter-passing styles are nice, and it's awesome how you can combine them. Certainly better than Perl5 (but everything is better than Perl5). There's also apparently special Perl6 syntax for applying functions to lists and currying functions, and weird Capture objects to explicitly deal with multiple-value returns from subs. Good stuff.

Perl6 is also apparently taking first-class functional objects to an extreme; blocks, subs, and methods are all objects and there are all kinds of metaprogramming hooks to screw around with them. This is one area where Ruby is just a little bit lacking: functions and methods aren't quite first-class enough in Ruby. Most people seem to pass around symbols / names of methods rather than pass around methods as objects themslves. Anonymous blocks are used liberally but mostly via yield, limiting you to one block per method and largely hiding away the block objects themselves.

I'm honestly a bit excited about Perl6, but largely as a curiosity or new toy to play with. It is kind of interesting how languages keep creeping more and more toward Common Lisp. If Perl is a nicer-looking Common Lisp which I can edit properly in Vim, it'll be almost a dream come true; I hate Emacs and Common Lisp tends to be butt-ugly. (Not talking about the parens, mostly about the verbosity and cruft and inconsistencies. Larry Wall famously said that Common Lisp looks like (paraphrased) "oatmeal with toenail clippings mixed in". Perl is certainly at the other extreme.)

http://rakudo.org/ is a good site for keeping up on Perl6 news. It's pretty active. Here's hoping we see a real release of Perl6 someyear.