briancarper.net (λ) (Tag: Clojure)

Keyword Arguments: Ruby, Clojure, Common Lisp

Fri, 24 Jun 2011 17:22:48 -0700

And suddenly I return to blogging, rising from the ashes like some kind of zombie phoenix. Turns out writing a book is a good absorber of time, like some sort of heavy-duty temporal paper towel. Now that I've gotten the terrible similes out of my system, let's talk about keyword arguments, one of my favorite features in any language that supports them.

Ruby, Clojure, and Common Lisp are all languages I enjoy to some degree, and they all have keyword arguments. Let's explore how keyword args differ in those languages.

Why keyword arguments?

Why are keyword arguments good?

You can omit arguments.
You can supply arguments in an arbitrary order.
Arguments are labeled, so you know what argument means what.

Positional arguments require mentally lining up the 7th argument in your function call with the 7th argument in the function signature, and so on. Keyword arguments become more and more attractive the more arguments you have in your function signature.

Keyword arguments trade a bit of verbosity for added explicitness, clarity and reduced mental burden. (Kind of like Lisps do overall. Fancy that.)

Ruby

Ruby doesn't have special support for keyword arguments. But Ruby likes its hashes, so you can just pass one in as an argument to a function. As some syntax sugar, if you pass a "flat" list of :key => val pairs, Ruby slurps them all together and stuffs them into a hash for you.

def foo(arg)
  p arg
end

foo({:x => 123})                  # => {:x=>123}
foo(:x => 123)                    # => {:x=>123}

With even more added sugar, you can leave off the parens in Ruby function calls. So this is pretty common in Ruby:

foo :x => 123                     # => {:x=>123}

How nice and punctuation-less. But then things get ugly. What about this?

foo {:x => 123}

That won't even compile. Ruby thinks {:x => 123} is a code block, and a bare key/value pair isn't valid syntax as the first thing in a code block. You need the parens. A bit unfortunate, but it gets worse...

def bar(arg1,arg2)
  puts "#{arg1} #{arg2}"
end

bar :x => 123, :y => 456          # Runtime error
bar {:x => 123}, {:y => 456}      # Won't compile

In the first example, all of the key/value pairs are slurped into one hash and end up in arg1. There's nothing left for arg2, so you get a "wrong number of arguments" exception. The second example won't even compile, of course, because again Ruby thinks {:x => 123} must be a code block with invalid syntax.

It gets worse if you change the argument list for bar slightly...

def bar2(arg1 = {}, arg2 = {})
  puts "#{arg1} #{arg2}"
end

This way, you don't even have to supply any arguments. This is nice, if all of your argument are optional.

bar2    # => {} {}

Now suppose you want arg1 to be {:x => 123}, and arg2 to be {:y => 456}. You might naively try this:

bar2 :x => 123, :y => 456         # => {:x=>123, :y=>456} {}

Oops, it all got dumped into arg1, and now instead of a missing argument error, arg2 silently ends up with a default, empty hash. You have to explicitly pass in an empty value for arg1 so that everything is slurped into arg2.

bar2 {}, :x => 123, :y => 456     # => WRONG!  Ruby thinks {} is a code block again.

bar2({}, :x => 123, :y => 456)    # => {} {:x=>123, :y=>456}

So much for syntax sugar. You might think you'd be unlikely to find this kind of thing in the wild, but in Ruby on Rails for example, there are quite a few functions whose argument lists look exactly like this. One signature for link_to is:

link_to(body, url_options = {}, html_options = {})

So...

link_to "foo", :controller => :x                          # OK
link_to "foo", :controller => :x, :class => "css_class"   # WRONG! 
link_to "foo", {:controller => :x}, :class => "css_class" # OK

What about support for default arguments? We might want to say that if you didn't pass in an :x argument, we want it to have some default value. You might think this would work:

def baz(x = {:x => 123})
   p x
end

But you would be sadly mistaken.

baz                               # => {:x=>123}
baz :y => 456                     # => {:y=>456} ... oops

Ruby doesn't merge your keyword arguments into the map in the parameter list. It uses that map if you don't supply any arguments, otherwise your map replaces the default entirely. So to get default arguments, you need something like

def baz2(args = {})
  args = {:x => 123}.merge(args)
  p args
end

baz2                              # => {:x=>123}
baz2 :x => 555                    # => {:x=>555}
baz2 :y => 456                    # => {:x=>123, :y=>456}

Kind of messy, but that's OK.

One last subtle ambiguity in Ruby is determining whether someone passed a nil argument for a keyword explicitly, or omitted a keyword entirely. It might make a difference in some circumstances.

def quux(args={})
  p args.include? :x
  args = {:x => nil}.merge(args)
  p args
end

quux                              # true, {:x=>nil}
quux :x => nil                    # false, {:x=>nil}

Pretty messy, but such is life.

Clojure

Part of the fun of Lisps is lack of ambiguity. Everything is spelled out in all its parenthesized glory. In Clojure, when you call a function like (foo :x 123), Clojure requires you to specify what you want to happen with those arguments.

It used to be that Clojure didn't have much support for keyword args at all. Clojure did have support for allowing variable numbers of arguments to functions though. So in the beginning, people used to slurp all of their arguments together into a list, and then turn it into a map inside the function.

user> (defn foo [& args]
        (let [args (apply hash-map args)]
          (prn args)))
#'user/foo
user> (foo)
{}
user> (foo :x 123)
{:x 123}
user> (foo :x 123 :y 456)
{:y 456, :x 123}

That worked. It still works today. But nowadays there's a better way. Why not slurp your arguments directly into a hash-map?

user> (defn foo [& {:as args}]
        (prn args))
#'user/foo
user> (foo)
nil
user> (foo :x 123)
{:x 123}
user> (foo :x 123 :y 456)
{:y 456, :x 123}

This is an example of destructuring. In this case, all of our arguments are slurped into a list (thanks to &), then this list is matched against our destructuring pattern, in this case {:as args}. :as says to take everything in the list, turn it into a map and give the resulting map the name args.

This isn't a special feature of defn. Destructuring works anywhere you're setting up bindings, for example in let, for, doseq etc. Like so:

user> (let [{:as args} (list :x 123 :y 456)] args)
{:y 456, :x 123}

It just so happens that & creates the list for you, out of the arguments you pass.

Destructuring does lots more than that though. We can immediately pull out the values for keywords we care about, so they'll be bound to names in our function body.

user> (defn foo [& {:keys [x y z]}]
        (prn x y z))
#'user/foo

user> (foo :z 123 :x 456)
456 nil 123

It's certainly a bit more verbose than Ruby in the function signature, but it lacks ambiguity and it saves you some repetition in the function body.

Clojure's approach also has the benefit of specifying directly in the function signature which keywords you're expecting. In a smart editor, like Emacs, you get an indicator of what kinds of keywords you should be passing in. See at the bottom?

This is also available at the Clojure REPL via the doc function.

user> (doc foo)
-------------------------
user/foo
([& {:keys [a b c]}])
  nil
nil

There's no better documentation than live, built-in documentation. There's nothing more distracting when programming than context shifts, and having to dig into a web browser to check a function signature is a huge mental page fault.

What about default values? Sure. You can use :or to specify defaults for some or all of your keywords. This works much more like I'd expect, compared to Ruby.

user> (defn foo [& {:keys [x y z] :or {x 1 y 2 z 3}}]
        (prn x y z))
#'user/foo

user> (foo :y 555)
1 555 3

What about determining whether the user passed nil for a keyword or whether they omitted the keyword entirely? For that, you have to resort to testing the argument map for the existence of the key, which isn't fun, but at least it's possible.

user> (defn foo [& {:keys [x y z] 
                    :or {x 1 y 2 z 3}
                    :as args}]
        (prn x y z)
        (prn args)
        (doseq [k [:x :y :z]]
          (println "Contains" k "=>" (contains? args k))))
#'user/foo

user> (foo :x 1 :y 2)
1 2 3
{:x 1, :y 2}
Contains :x => true
Contains :y => true
Contains :z => false

Why do I keep mentioning this? See Common Lisp below.

In any case, our parameter list is becoming huge and unweildy. The first time I saw sample Clojure code like this, I almost did a spit-take. But after a bit of getting used to, I'm finding that this syntax is pretty comfortable.

The simplest case may not be as concise as Ruby, but aside from lack of ambiguity, the benefit of Clojure's approach is being able to safely do powerful (and borderline insane) things, like nested destructuring. And this works everywhere you're setting up a binding.

(defn ow-my-eyes [a & {[b {:keys [x y]}
                        & {[g h] :r
                           :keys [p q]
                           :or {q 123}}]
                       :x}]
  (prn a b g h p q x y))



user> (ow-my-eyes 444 :x [1 {:x 555 :y 666} :p 3 :r [888 999]])
444 1 888 999 3 123 555 666

If you wrote code like that in real life, you'd likely be defenestrated, but at least you know you can do it. And destructuring in Clojure could likely be extended even further in the future, if someone came up with a use case for something that isn't supported.

Official documentation for all of this is here.

Common Lisp

When it comes to keyword arguments, Common Lisp supports mostly everything that Clojure does, and some things it doesn't. CL was likely a big inspiration for Clojure's destructuring. I highly recommend reading Practical Common Lisp to learn more about CL keyword arguments and list destructuring. (Read the rest of the book while you're at it.)

Keyword arguments in CL look like this:

> (defun foo (&key x (y 123) (z 456 z-supplied-p))
    (pprint (list x y z z-supplied-p)))
FOO

> (foo)
(NIL 123 456 NIL)

> (foo :z nil)             ; note, z-supplied-p tells us whether z was omitted or not
(NIL 123 NIL T)

> (foo :z 555 :x 666 :y 777)
(666 777 555 T)

> (foo :x 123)
(123 123 456 NIL)

> (foo :x 555)
(555 123 456 NIL)

Yeah, there's direct support for distinguishing supplied keys with nil values, and un-supplied keys. That's pretty nice.

Common Lisp also supports insane things like having keyword arguments' default values be a function of other arguments in the parameter list.

> (defun bar (&key (x 123) (y (+ x 1000)))
    (pprint (list x y)))
BAR

> (bar)
(123 1123)

> (bar :x 5)
(5 1005)

> (bar :x 5 :y 6)
(5 6)

Clojure destructuring can't do this by default, so you'd have to resort to a let in the function body.

(defn bar [& {:keys [x y] :or {x 123}}]
  (let [y (or y (+ x 1000))]
    (prn x y)

There's nothing stopping Clojure's destructuring from being patched to support this, of course. But this isn't a feature I've ever found myself wanting particularly badly.

As for how to destructure CL keyword arguments into a vector or hash-map, CL doesn't directly support doing it in function parameter lists like Clojure does. (Though there's nothing stopping someone from throwing together a reader macro to let CL do this, of course.) This isn't really surprising, because CL loves its cons cells, while Clojure embraces maps (and vectors and sets etc.), offering them default reader syntax and lots of other built-in support.

So there you have it. Keyword arguments. Use them. Love them.

Lessons:

Sugar can be bad for your health.
Ambiguous, bad. Explicit, good.

2010 in review

Wed, 05 Jan 2011 10:10:03 -0800

Another year down the drain. A good year, in the end.

2010 Geek Achievements

Wrote some code...
- cow-blog - The engine running this blog.
- oyako - Clojure ORM library.
- gaka - CSS compiler for Clojure
Finished a huge project for work, my first AJAX-y web app (in Rails). That was fun, albeit stressful.
Learned a lot of git.
Learned a lot of Clojure.
Learned a lot of Emacs.
Learned a lot of Javascript.
Learned a lot of PostgreSQL. It's good to be free of MySQL.
Switched to ZSH. This was a good switch.
Tried to learn a lot of Japanese, but kind of fizzled out at the end of the year.
Alllllllmost got a Clojure gold badge on Stack Overflow. I'll get it soon though. Not losing any sleep over it either way.
Read a lot of books. The best: probably Feynman's books of anecdotes.
Blogged a bit. Got an article in Hacker Monthly. Was flamed repeatedly. Learned a lot in the process.

2010 Non-Geek Achievements

Immigrated to Canada. A good move, without a doubt.
Lost 25ish lbs. :)
Learned how to cook better.
My most important achievement from 2010 is actually non-geek: I finally obtained a bit of an offline social life. This is not an easy task for one such as myself.
Continued to learn to appreciate good beer. Longwood Dunkelweizen, mmm.

2010 Failures

Did not blog enough.¹
Did not write enough code.¹
Missed the first Clojure Conj. Maybe next year.
Re-gained 10ish lbs. :( ²

Plans for 2011

Re-lose 25ish pounds. I'd like to reach the weight I had in college.
Finish my rewrite of oyako. I have ambitious plans for it, if I can just find the time.
Finish my rewrite of cow-blog to match oyako.
Keep working on the RPG my wife and I are creating (in Clojure).
Attend the next Clojure Conj, I hope.
Learn more Clojure.
Learn Haskell? Trying and failing to learn Haskell has become somewhat of a tradition, no sense stopping now.
Learn all 2000+ jouyou kanji by the end of the year.
Supar sekrit projekt. But I haven't signed the contract for it yet so I won't talk about it until I do.
Maintain social life at acceptable levels.
Buy a house.

I feel like I have solid plans for completing each of these things. Blogging more often and finishing oyako are high on my list of priorities. I expect 2011 to be my most productive year to date.

See also, non-geek achievement #4, "Obtained social life". ↩
See also non-geek achievement #3, "Learned how to cook better". ↩

gaka 0.2.0

Thu, 29 Jul 2010 13:59:22 -0700

Per many commenters' suggestions and thanks to code from Steve Purcell, you can now use maps for CSS attributes in gaka.

user> (println (gaka/css [:a {:color :red}]))
a {
  color: red;}

This looks more like vanilla CSS thanks to the curlies, which is nice. You just have to keep in mind that your key/value pairs could end up being printed in random order, and order is significant¹ in CSS.

It just so happens that maps are implemented in Clojure right now such that if they only have a few entries (16 key/value pairs), the order will be preserved, because you get a PersistentArrayMap instead of a PersistentHashMap. But it's highly dangerous to rely on such a thing. It could change at any time in the future.

In any case, you can also mix and match maps, lists and "flat" keyvals. They'll all be flattened That can help preserve attribute order in those cases where you need to.

user> (println (gaka/css [:a :color "red" {:padding 0} (list :margin 0)]))
a {
  color: red;
  padding: 0;
  margin: 0;}

I've also enhanced "mixins" a bit further. You can now mixin entire tags as well as attributes. Or a combination of both. Say you want a mixin that means "Make my element have no padding, and make links within the element be red":

user> (println (gaka/css [:div.foo mixin :margin 0]
                         [:div.bar mixin]))
div.foo {
  padding: 0;
  margin: 0;}

  div.foo a {
    color: red;}

div.bar {
  padding: 0;}

  div.bar a {
    color: red;}

You can get gaka from github or Clojars.

Order is only significant in cases where you're doing things like padding: 0; padding-left: 1px. This is arguably bad CSS style, but it's valid, and it's also possible you'll have this kind of thing if you're generating CSS procedurally. But most of the time, order is not significant. e.g. it doesn't matter if you set text color first and background color second, or vice versa. So maybe this isn't so much of a problem in practice. ↩

Clojure syntax highlighting via SyntaxHighlighter

Wed, 21 Jul 2010 15:17:42 -0700

How do you syntax-highlight Clojure code for display on a website? The best way I can find is SyntaxHighlighter.

Daniel Gómez wrote a brush to give SyntaxHighlighter Clojure support. I tweaked it a bit myself and integrated it into cow-blog. I also converted my favorite color scheme to a SyntaxHighlighter theme. So when I write this code:

(defn- ip
  "Given a request, return the IP.  Looks for an x-forwarded-for
  header, falls back to :remote-addr on the request."
  [request]
  (or (get-in request [:headers "x-forwarded-for"])
      (request :remote-addr)))

You should see something like this:

...unless you're reading this via RSS, or in a browser without Javascript enabled, in which case you'll see plain, depressing black and white. But that's one nice thing about SyntaxHighlighter. It degrades nicely.

One bad thing about SyntaxHighlighter is that it doesn't play nicely with Markdown. Or rather, Markdown isn't powerful enough to let you specify the class of any markdown-generated HTML tags. If you want the <pre class="brush: clojure"> that SyntaxHighlighter requires, you have to write out the HTML by hand. But I hacked Showdown a bit to let me specify classes more easily, so I can avoid having to do that.

The code for all of this is on github with the rest of my blog.

Goodbye Tokyo Cabinet, hello PostgreSQL?

Tue, 29 Jun 2010 11:32:58 -0700

The first version of this blog used MySQL; then I switched to Tokyo Cabinet. But now I've switched back to PostgreSQL. Here's why.

Why did I switch to TC to begin with?

There weren't any good ORM-type libraries for Clojure at the time (over a year ago). So there was a bit of an impedance mismatch trying to query and work with my data. In the DB I have separate tables for posts, comments, tags, categories. But 90% of the time I want to fetch a post and end up in Clojure with all related tags, comments etc. (A JOIN won't work here without a lot of work to un-mangle the results; you usually need multiple queries.)

With TC I could store anything in the DB, so I just dumped a post into the DB as serialized hash-maps with all the comments, tags, and categories as sub-keys. So querying was easy. (Or was it? More below.)
MySQL was really slow. This is largely because my queries were terrible, as I tried to solve the problem from #1 via brute force. TC on the other hand is fast.
I tried to solve #2 by using a Clojure ref as a cache. But tying the STM to a database's transaction system is as far as I know difficult or impossible right now (per many threads on the mailing list). I had a lot of potential race conditions, which (as far as I know) never bit me, but probably would've eventually. I had to deal with keeping the cache up-to-date as comments were posted and posts were added and deleted and renamed. Remember:

"There are only two hard problems in Computer Science: cache invalidation and naming things." --Phil Karlton

So why did I stop using TC?

I have no idea how to use a key/value store database properly. TC will take anything you dump into it, which is both a strength and a weakness.

There's a lot of crap you have to do by hand that a proper database does for you. Consider checking for null values, for example; I ended up with a lot of nils in my data because my validations weren't 100% foolproof, or because I imported data via code that didn't run the validations and I never noticed.. Or enforcing uniqueness of values; I had tag objects in the database with the same key but different values (due to capitalization differences), which screwed up a lot of stuff.

On the other hand, there's a lot of information about how to use RDBMS properly, and I have a lot of experience with it already. Constraints are easy to set up. Columns have types, which is nice. (Strange that I gravitate toward statically-type databases while I gravitate toward dynamically-typed programming languages.)
I have to compile and install Tokyo Cabinet by hand on my Linux distro. It's probably not worth distro maintainers to maintain a package that so few people use. MySQL and PostgreSQL have lots of people working on keeping them running OK on most Linux distros.
Some kinds of queries were still awkward in TC. "Give me post X" was great: I'd also get all the tags, categories etc. for free. But then how do you query to get all tags across all posts? Or all comments? Fetch all the posts and iterate over them, collecting their tags, then uniquify the resulting list? Not so pretty, and not so fast. So I was back to caching again, which still gave me nightmares about race conditions and dirty data.

So now why am I using an RDBMS again?

An RDBMS is exactly what I really need, if I could just query the thing concisely and get it to run fast. Thankfully there are some ORM-like libraries for Clojure in the works nowadays, already usable for a hobby project like this blog. There are clj-record, Carte, ClojureQL, and my own Oyako, and possibly others in the works.
For my tiny blog's database, Oyako gives me slightly slower performance than TC, but along the same order of magnitude, which is good enough.
Via Oyako I can (fairly concisely) fetch posts and get the associated tags, comments etc. But I can also easily fetch all tags, or all comments, since they're in their own tables. The "relational" part of RDBMS does come in handy sometimes.

Summary version

I switched to TC to begin with because I was using SQL wrong, and it was too slow and clumsy. Once I figured out how to use SQL correctly, it was a no-brainer to go back.

Introducing Gaka

Mon, 28 Jun 2010 17:59:26 -0700

The CSS for my blog is now being generated via gaka, a CSS-generating library I wrote this afternoon. It's extremely simple, but it got the job done for me. I turned around 600 lines of CSS into around 250 lines of Clojure without much effort. It looks like this:

user> (require '(gaka [core :as gaka]))
nil
user> (def rules [:div#foo
                  :margin "0px"
                  [:span.bar
                   :color "black"
                   :font-weight "bold"
                   [:a:hover
                    :text-decoration "none"]]])
#'user/rules
user> (println (gaka/css rules))
div#foo {
  margin: 0px;}

  div#foo span.bar {
    color: black;
    font-weight: bold;}

    div#foo span.bar a:hover {
      text-decoration: none;}

Gaka is partly inspired by Sass, which I found very pleasant to work with recently. And it's partly inspired by Hiccup, which is a delicious way to generate HTML in Clojure.

There's more info and more examples on github.

New blog engine up and running

Wed, 23 Jun 2010 15:48:24 -0700

Well, my new blog is up and running. Sorry for the temporary lack of cows in my layout. I'm dogfood-testing the blog engine in a fairly vanilla state until I work out some of the bugs. This layout is based upon barecity, a minimalist Wordpress theme that I adapted easily enough to my blog.

As a bonus, I applied a dirty hack to my RSS feed that I think should help avoid screwing up people's RSS readers with duplicate entries.

I'll write again soon with some info about the blog engine and some things I learned writing it.

(As mentioned previously, here's the code.)

Breaking links is easy to do

Tue, 22 Jun 2010 23:23:27 -0700

I apologize in advance to everyone who subscribes to my blog's RSS feed, but this week your RSS reader is probably going to suddenly find 25 "new" posts from me.

My blog currently uses /blog/title as the URL scheme, with similar URLs for categories and tags etc. Soon, I'm probably going to change it to /blog/123/title, as part of the impending release of version 0.2 of my blog engine. (The code-in-progress is in a branch on github, for the daring and foolish among you.)

This way, I can change the title of a post without breaking everything. I have heretofore lacked this ability. It's easy to code, you just tell Compojure to ignore everything after the number in a route. Something like this:

(defroutes foo
  (GET ["/blog/:id:etc" :id #"\d+" :etc #"(/[^/]*)?"] [id]
    (pages/post-page id)))

It's only a few lines of code to change, but the ramifications are widespread. It'll instantly break every link to my blog, for example. At least it's pretty easy to set up a bunch of redirects in Compojure to avoid that. I think this'll work:

(require (blog [db :as db]
               [link :as link])
         (oyako [core :as oyako])
         (ring.util [response :as response]))

(defn redirect-post [name]
  (when-let [post (oyako/fetch-one db/posts :url name)]
    (response/redirect (link/url post))))

(defroutes redirect-routes
  (GET ["/blog/:name" :name #"[^/]+$"] [name]
    (redirect-post name)))

(Oyako here is the experimental ORM-like library I'm using to interface with PostgreSQL nowadays, having ditched Tokyo Cabinet.)

Changing my URL scheme is also going to mess up RSS though, because I (foolishly) used post URLs as the GUIDs in my RSS feed up to this point. This problem I don't know how to avoid. I might reduce the number of posts included in my feed temporarily, to limit the damage.

Emacs creating zombie buffers

Mon, 14 Jun 2010 15:44:18 -0700

Note to self:

Using bleeding-edge swank-clojure and Slime, Emacs was creating buffers that didn't point to any file, every time C-c C-k wanted to display warnings. This was really quite annoying as it resulted in a mess of empty buffers being created. This fixed it:

(setq slime-highlight-compiler-notes nil)

Clojure, from a Ruby perspective

Wed, 09 Jun 2010 13:22:58 -0700

Fogus' recent article "clojure.rb" speculates about why there seem to be so many Ruby users adopting Clojure. As a Ruby user who adopted Clojure, I figured I'd write about my experiences.

What do Ruby and Clojure have in common, that would attract a Rubyist to Clojure? A lot. Obviously, this is somewhat subjective and I don't expect anyone else to agree, but this is what did it for me.

Semantic consistency

In Ruby, everything is an object. It makes it simple to write code without worrying much about what kind of thing you have. foo.some_method(1,2,3) will generally work for any foo.

In Clojure, everything is not an object, specifically because it inherits primitives from Java land (though this doesn't hurt much in the kind of everyday use I put Clojure to). But also because Clojure by design doesn't even attempt to be object-oriented.

Clojure does have abstractions though. For example there's an abstraction that says "this thing can be called like a function". And then you can treat any callable-thing as a function without worrying about what it is.

In Ruby, a lot of things are Enumerable, which means you can do foo.each{} and other similar things for a lot of different types of foo. Clojure has something similar with its seq abstraction. Similarly, most Clojure data types and many Java ones are seq-able, and most built-in core functions can iterate over the guts of various things using seqs. This includes regex matches, strings, directories of files, and so on.

Ruby-style OOP brings a lot of complexity and baggage which Clojure avoids by not being OOP. For example (ignoring Java for the moment), in Clojure land you don't have to worry about a member being public/private/protected, and there are few times when you have to worry about inheritance and class hierarchies. (And there's the whole thread-safety thing.) In Clojure data is stupid and immutable, and functions are just things that take input and give output, usually side-effect free. The separation is clean and this results in programs that are very easy to reason about.

Another example of consistency: Expressions. In both Ruby and Clojure, everything has a value. Things that are "statements" in other languages are instead expressions that return something. This alone makes a lot of programs just a little bit better/easier to write.

Aesthetics

Like Ruby, Clojure code tends to be terse and expressive.

Ruby reads like poetry, because it's mostly words and not so much punctuation. In Ruby, you have none of Perl's sigils, very little of C's semi-colon line-endings and curly-delimited blocks. Especially when you start omitting optional parentheses, it has a very minimalistic vibe which is appealing to many.

    def foo(bar)
      bar.each do |x|
        puts x
      end
    end

Clojure's s-expressions are another story, of course. Some love them, some hate them. Personally, I love them. The tired old trope about Lispers not paying attention to parentheses is true; after a while they blend into the background.

    (defn foo [bar]
      (doseq [x bar]
        (prn x)))

What I see:

    defn foo [bar]
      doseq [x bar]
        prn x

Clojure has the same minimalistic feel to it, in my eyes. It also helps that so many Clojure function names are short and concise. And being able to use punctuation like ? and ! in variable/function names (true? and false? are function names), hyphens instead of underscores... these things help Clojure read smoothly.

But along with aesthetics, in Clojure you get the benefits of s-exps: no order of operations to deal with, and absolute consistency. Everything is (function param1 param2 param3). Look at how many syntax rules you have to memorize for the Ruby code above to make sense. Dot means method call, blocks are do/end delimited and have those weird pipes in there, etc.

Literals and syntax sugar

Ruby has literal syntax for many types. This includes:

:symbols
[arrays]
{:hash => :maps}
/regex/
do block end and {block}

This really is a big deal. I'm spoiled and I can't use a language without these things nowadays. It saves a ton of typing and it makes those things stand out in the code, making it both easier to write and to read. I use those structures all the time in every program I write, so they should have a terse representation.

Clojure has literal support for the same types as Ruby, and they even look mostly the same (with #"regex" being a change I can live with). And then it also has (among others):

#{sets}
#(function-literals %)
'(quoted forms)
`(quasi-quoted ~forms)

Is this a contradiction of my last point? What happened to s-expressions and consistency? Well, in Clojure, the reader shortcuts are just sugar that reduce to s-exps. You can avoid all use of that sugar if you hate it. (hash-map :key "val"), (vector 1 2 3), (quote foo) etc.

But more importantly, let's draw a(n arbitrary) distinction between "good" syntax and "bad" syntax.

Clojure's reader-macro sugar makes your code shorter, but doesn't change the structure of your code. Take a function call (f x y z), and you can always substitute a vector or regex literal or quoted form into it. (f [vector] #{set} #"regex"). The syntax sugar is very local, very self-contained. It doesn't leak into the surrounding code. And of course you can combine them in nearly arbitrary ways: '[quoted vector], {:hash-map-containing-a #{:set 'of #(functions)}}. This is "good" syntax.

Compare this to things like the x ? y : z construct, or heredocs. These things are not as orthogonal. They not only mean something on the "inside", they also influence and interact with the code before and after them, thanks to precedence rules and special parsing rules. Can you stick a heredoc in the middle of a function call? Maybe (I don't even know), but have fun with the indentation and line-breaks if so. When should you use do/end and when should you use {} for blocks? When do you need parens around your ternary if-then-else construct and when don't you? When do you need to use and and when &&? That's the "bad" kind of syntax. Sometimes, maybe even most of the time, it makes your code shorter, but there are a lot of rules to memorize and you never know when you'll be bitten.

Clojure largely avoids the "bad" syntax while taking advantage of the "good". Reader macros make your code shorter and visually easier to scan, but they rarely require you to do backflips to get your code to compile or run properly.

First-class functions

Ruby's blocks and yield and friends let you deal with first-class functions. This is a huge step in the Lisp direction already, and it's one of the things that makes Ruby great. But there are limits. Blocks use funky, special syntax, and in idiomatic Ruby, you will pass around only one block per method. There's the whole lambda and proc mess, and then there are methods-as-objects which are different still. And a lot of Ruby just calls .send on an object and passes in a method name as a symbol.

Being a Lisp, Clojure takes this a bit further. First-class functions are ingrained in nearly everything you do in Clojure. And they are easy to define and easy to call. Define f via defn (to make it top-level), fn (for a local function), or #() (sugar for fn), and then call it like (f).

Clojure also takes advantage of some functional-programming mainstays like partial and complement and comp(osition). We're not in full-blown Haskell territory, but it's a lot more FP than idiomatic Ruby.

And hash-maps, vectors, sets, keywords, and symbols are also callable as functions in Clojure. ({:foo 1} :foo) => 1. Many things can be treated as functions.

Metaprogramming

In Ruby you can mess with the innards of any class you want. There are facilities for defining methods dynamically, opening and inspecting classes at runtime, catch-all handlers for undefined methods, and all kinds of other dark magic. But again there are limits... Ruby needs to make use of eval to get certain things done. And monkey-patching is a shotgun aimed at your foot.

Well, if you like metaprogramming, Lisp macros are top of the line. You can abstract away boilerplate with a vengeance. Macros are the ultimate application of DRY.

Clojure doesn't deal much with classes, so there isn't much of that kind of introspection, but the Lisp principle of code-as-data enables a kind of introspection that you won't find in Ruby. The line between compile-time and run-time is very blurry, which enables all kinds of magic.

Java itself does offer Ruby-style reflection and such, if you need it, but you won't often, while in Clojure land.

Multimethods (and soon, protocols and defrecord) let you avoid monkey-patching and get some of the same kinds of "extend a class" things done in a saner and safer way.

So?

Fogus suspects:

Ruby programmers being the adventurous lot to begin with, are not satisfied with “halfway to Lisp”. Instead, they want it all.

This is true in my case. I like Ruby largely insofar as it borrowed and adapted many great features of Lisp. It only makes sense that I would like Clojure, which takes most of those things one step further. Clojure in particular, as a "modern" Lisp with vaguely Ruby-like syntax in certain places, is an obvious choice.

On top of that, Clojure is fast, thanks to the JVM. Ruby has JRuby too, but vanilla Ruby is not known for its speed. Clojure integrates with a REPL in a way that Ruby really doesn't, making interactive development enjoyable. Clojure is a compiled language, which has benefits for deployment. And again, there's the whole thread-safety thing. Clojure is awesome for writing sane, safe multi-threaded programs. These things are rather appealing.

I do still use Ruby though. Ruby is great for scripting, Clojure not so much, thanks to the JVM startup time, among other things. Ruby can be banged out quickly in any editor, but Clojure isn't much fun to edit in any editor that lacks good paren-matching support and REPL integration.

Rubygems offers dead-simple install of a ton of libraries, whereas Clojure is still working out the details of a standard build tool and install tool. Ruby has a library for anything, and while Clojure can use Java libraries, Java libraries tend to be huge and feature-rich, sometimes too huge for one-off tasks where a small Ruby library is a perfect fit.

Emacs isn't for everyone

Tue, 08 Jun 2010 11:26:15 -0700

Chas Emerick recently posted the results of his State of Clojure survey. It turns out that the (self-selected) group of Clojure-using respondents happen to prefer Emacs as their IDE of choice, eclipsing all other editors by a large margin.

Chas then has this to say:

I continue to maintain that broad acceptance and usage of Clojure will require that there be top-notch development environments for it that mere mortals can use and not be intimidated by...and IMO, while emacs is hugely capable, I think it falls down badly on a number of counts related to usability, community/ecosystem, and interoperability.

As an avid, die-hard Vim and Emacs user for life, I'm going to agree.

Mere mortals?

Emacs isn't difficult to learn. Not in the sense of requiring skill or cleverness. It is however extremely painful to learn. I think there's a difference.

The key word is tedium. Learning Emacs is a long process of rote memorization and repetition of commands until they become muscle memory. If you're smart enough to write programs, you can learn Emacs. You just have to keep dumping time into the task until you become comfortable.

Until you're comfortable, you face the unpleasant task of un-learning all of your habits and forming new ones. And you're trying to do this at the same time you're undertaking another, even harder task: writing programs. And if you're a new Clojurist, and you're learning Emacs and Clojure from scratch at the same time, well, get the headache medication ready.

As a programmer and someone who sits in front of a computer 12+ hours a day, I consider myself pretty flexible and capable of picking up a new user interface. As someone who had been using Vim for years prior to trying Emacs, I considered myself more than capable of learning even a strange and foreign interface. I'd done it once before.

But learning Emacs still hurt. Oh how it hurt. I blogged while I was learning it, and you can see my pain firsthand. I sometimes hear people say "I tried Emacs for a whole month and I still couldn't get it". Well, it took me over a year to be able to sit down at Emacs and use it fluidly for long periods of time without tripping over the editor.

To be fair, I'm talking here about using Emacs as a programming environment. Using Emacs as a Notepad replacement could be learned in short order. C-x C-f, C-x C-s, or use the menus, there you go. Using it comfortably as a full-fledged IDE is significantly harder and requires you to touch (and master) many more features. Syntax highlighting, tab-completion, directory traversal and cwd issues, enabling line numbers, version-control integration, build tool integration, Emacs' funky regex syntax for search/replace, Emacs' bizarre kill rings and undo rings, the list goes on. These things are very flexible in Emacs, which is a great thing, but it's also an impediment to learning how to configure and use them. There's no getting around the time investment.

And it's not just a matter of learning some new keyboard shortcuts. There's a new vocabulary to learn. You don't open files, you visit them. What's a buffer? What's a window? (Not what you think it is.) What's a point? What's a mark? Kill? Yank? "Apropos"? Huh? C-c M-o means what exactly? My keyboard doesn't have a Meta key. Yeah, you can use CUA mode and get your modernized Copy/Cut/Paste shortcuts back, but that's the tip of the iceberg. It's hard even to know where to begin looking for help.

Yeah, Emacs came first, before our more common and more modern conventions were established, and that explains why it's so different. That doesn't change the fact that Emacs today is a strange beast.

Community and ecosystem

Personally I find the Emacs community to be a pretty nice bunch. In the highest tradition of hackerdom and open source software, Emacs users seem to be eager and willing to share their elisp snippets and bend over backwards to help other people learn the editor. I got lots of help when I was struggling and learning Emacs.

The Emacs wiki is an awesome resource. The official documentation is so complete (and so long) that it leaves me speechless sometimes. And there are a million 3rd-party scripts for it. Whatever you want Emacs to do is generally a short google away.

If there's anything wrong with the Emacs community, it'd be people who take Emacs evangelism overboard. The answer to "I don't want to have to use Emacs to use your language" can't be "Be quiet and learn more Emacs", or "If you're too dumb to learn Emacs, go away". In some communities there is certainly some of that. But thankfully I don't see it much in the Clojure community. Let's hope it stays that way.

Interoperability

Once someone spends the time to write a suitable amount of elisp, Emacs can interoperate with anything. I think so many people use SLIME for Clojure development precisely because it interoperates so darned well with Lisps. SLIME is amazing. You probably can't beat Paredit either, and Emacs' flexibility is precisely what makes things like Paredit possible.

The problem is the amount of time you have to spend to get that interoperability set up and to learn how to use it. After two years of using Emacs and Clojure together, every once in a while I still find myself bashing my face on my desk trying to get the latest SLIME or swank to work just right, or trying to get a broken key binding fixed, or tweaking some other aspect of Emacs that's driving me crazy. One day, curly braces stopped being recognized as matched pairs by Paredit. Why? No idea; I fixed it, but it was a half hour of wasted time.

Emacs is good at integrating with Git too. So good that there are four or five different Emacs-Git libraries, each with a different interface and feature set. I gave up eventually and went back to using the command line. (You can embed a shell / command line right in Emacs. There are three or four different libraries to do that too.)

The wealth of options of ways to do things in Emacs is simultaneously a good thing, overwhelming and confusing. If all you want is something that works and gets out of your way, too many options can be worse than one option, even if that one option isn't entirely ideal.

Emacs' Java interop, I know nothing about. Almost certainly, Emacs can come close to a modern Java IDE for fancy features like tab-completion and document lookups and project management. But how long is it going to take you to figure out that tab-completion is called hippie-expand in Emacs? That and a million other surprises await you.

What's my point?

There was a pithy quote floating around on Twitter a while back (I think quoting Rich Hickey):

One possible way to deal with being unfamiliar with something is to become familiar with it.

That's true, and you could say that of Emacs. I strongly believe that when it comes to computers, there's no such thing as "intuitive". There's stuff you've already spent a lot of time getting used to, and there's stuff you haven't.

But certain things require more of a time investment than others. Could I learn Clojure if all the keywords were in Russian or Chinese instead of my native English? Sure, but it'd take me a long time. I'd certainly have to have a good reason to attempt it.

I learned Emacs partly because it was hard. I saw it as a challenge. It was fun, yet painful, but more pain, more glory. Mastering it makes me feel like I've accomplished something. I'd encourage other people to learn Emacs and Vim too. I think the benefits of knowing them outweigh the cost and time investment of learning them.

But I didn't learn Emacs with the goal of being productive. I learned it for the same reason some people build cars in their garages, while most people just buy one and drive it to and from work every day. I learned Emacs because I love programming and I love playing with toys, and Vim or Emacs are as nice a toy as I could ask for. (I love programming enough to form strong opinions and write huge blog posts about text editors.) For me, productivity was a beneficial side-effect.

There are only so many hours in a day. There are a lot of other challenges to conquer, some of which offer more tangible benefits than Emacs mastery would get you. Mastering an arcane text editor isn't necessarily going to be on the top of the list of everyone's goals in life, especially when there are other editors that are easier to use and give you a significant subset of what Emacs would give you. We have to pick our battles.

So I understand when people say they don't want to learn Emacs. I think maybe so many Clojurists use Emacs right now because we're still in the early adopter stage. If you're using Clojure today, you're probably pretty enthusiastic about programming. You're likely invested enough to be willing to burn the required time to learn Emacs.

If Clojure becomes "big", there are going to be a lot of casual users. A casual user of Clojure isn't going to learn Emacs. They're going to silently move on to another language. And I really think that new blood is vital to the strength of a community and necessary for the continued healthy existence of a programming language.

So Clojure does need alternatives. I'll stick with Emacs myself, but there should be practical alternatives. I'd encourage the Clojure community to continue to support and enjoy Emacs, but don't push it too hard.

Printing a nicely formatted plaintext table of data in Clojure

Fri, 16 Apr 2010 11:49:02 -0700

I use this fairly often while data-munging, when I want to quickly view a list of hash-maps of data in a simple table format. Usually this is coming out of database. Maybe someone will find it useful.

(defn table
  "Given a seq of hash-maps, prints a plaintext table of the values of the hash-maps.
  If passed a list of keys, displays only those keys.  Otherwise displays all the
  keys in the first hash-map in the seq."
  ([xs]
     (table xs (keys (first xs))))
  ([xs ks]
     (when (seq xs)
       (let [f (fn [old-widths x]
                 (reduce (fn [new-widths k]
                           (let [length (inc (count (str (k x))))]
                             (if (> length (k new-widths 0))
                               (assoc new-widths k length)
                               new-widths)))
                         old-widths ks))
             widths (reduce f {} (conj xs (zipmap ks ks)))
             total-width (reduce + (vals widths))
             format-string (str "~{"
                                (reduce #(str %1 "~" (%2 widths) "A") "" ks)
                                "~}~%")]
         (cl-format true format-string (map str ks))
         (cl-format true "~{~A~}~%" (repeat total-width \-))
         (doseq [x xs]
           (cl-format true format-string (map x ks)))))))

Then you can do this:

user> (def data [{:name "Brian" :job "Code monkey" :age 29}
                 {:name "Bob" :job "Janitor" :age 97}
                 {:name "Johnny McLongname" :job "None" :age 3}])
#'user/data
user> (table data)
:name             :job        :age 
-----------------------------------
Brian             Code monkey 29   
Bob               Janitor     97   
Johnny McLongname None        3    
nil
user> (table data [:age :name])
:age :name             
-----------------------
29   Brian             
97   Bob               
3    Johnny McLongname 
nil

Installing Clojure with Emacs and SLIME: so easy, yet so hard

Wed, 14 Apr 2010 16:02:55 -0700

How do you install Clojure with SLIME/Emacs support? The answer is either really easy or really hard, depending.

Easy!

If you're new to Clojure and want to get it working fast, you are in luck. Go to swank-clojure's home on github and follow the directions to install it via ELPA. This is very easy and fast and it works.

If you want more documentation, there is (thankfully) a single, central, official-ish location where documentation for installing Clojure is slowly but surely accumulating. You can read about it at http://www.assembla.com/wiki/show/clojure/Getting_Started. This site includes instructions for Emacs, Netbeans, Eclipse, Maven, and everything else under the sun. This is a great resource, and attempts to solve the problem of out-of-date Clojure documentation. (This problem is natural given Clojure's young age and rapid growth, but it's painful for people trying to get started, I bet.)

Or is it...

This is all great. But if you want to do anything non-standard, you run into a bit of a wall. How do you install Clojure+SLIME on Windows? I find that sometimes it works, but sometimes things mysteriously fail, and I have no idea where even to begin to debug it, because there's so much automation going on in the background nowadays. What if you want to build a .emacs.d directory you can simply copy from one system to another and have everything work? (You can do this with most Emacs libs.)

Or how do you install and use a bleeding-edge Clojure or clojure-contrib tree with SLIME? There's good reason to want to, because it has some awesome new features, e.g. deftypes and protocols and rest-parameter destructuring. But most of the "official" methods of installing things give you stable versions of everything.

And then, you generally end up with a bunch of opaque .jar files, auto-downloaded all over your system, which isn't helpful if you want to hack on anything yourself. Most of these jars end up in ~/.m2 nowadays, I think? Used to be ~/.swank-clojure or ~/.clojure or something? And ~/.emacs.d/elpa gets stuff too? Color me continually confused.

The automated build tools we have nowadays (ELPA, Lein, Maven) are awesome when they work, but extremely painful when they fail or when you don't know which knobs to turn when you want to step slightly outside of the mainstream. This is incidental complexity if I ever saw it. While it's far more manual labor, I find it to be so much simpler to download things myself, build everything myself, put it all somewhere on my system myself, and configure Emacs to tell it where to find everything.

Well, this site is as close as I can find to good up-to-date documentation, and what I did to get set up was almost identical to this. But it will likely stop working in a month or two. There's also a recent thread on the Clojure mailing list which has some more tips (and valid complaints).

I will push my Emacs setup to my github repo once I get the kinks worked out. But if you want to get bleeding-edge everything working yourself, all I have to say is God help you, because it's largely unsupported. Be prepared to hack and patch until it works. I'm looking forward to the day when most of the kinks are worked out and things settle down a bit.

Making an RPG in Clojure (part one of many?)

Sat, 20 Feb 2010 16:27:40 -0800

What do you get when you combine old-school Final Fantasy-style RPGs with Clojure? Fun times for all. Well, for me at least.

I'm working on a sort of RPG engine in Clojure so I can make my own RPG. Click the thumbnail for a very preliminary video demo (6.5 MB) showing the engine in action.

All I do in the video is walk around, and eventually start adding random NPCs to the map to test collision detection. Not all that exciting, but I'm proud nonetheless.

To forestall questions, yes I'll eventually post the source code, but no, not yet. It barely works. Just a proof of concept so far.

Right now I can walk around a map while NPCs also randomly walk around the map, not much more. So there isn't much to talk about. But not bad for 4 days and 600 lines of code (one tenth of which is ASCII art... more on that later). Keep in mind that I have no idea what I'm doing.

Collision detection works so people don't walk through walls or each other, and after endless tweaking I got all the animations to be very smooth, even when I add a few dozen NPCs to the map (as I do in the video). The video is a bit jerky but it looks better in person. All of the admittedly poor artwork is also created by myself, thanks to the GIMP and some hastily-read tutorials on pixel art.

It all runs on plain old Swing in Clojure. Here's some of what went right and what went wrong so far.

Background

I've played a lot of RPGs, but I've never programmed a game more complex than Pong. I knew what double-buffering is, and that's as far as my knowledge of game programming went when I started.

My first idea was to use a plain old Swing JFrame. I'd make a bunch of sprites saved as PNG files, read them all in and draw them on the JFrame, as many times per second as I could manage. Then there's some global state to keep track of where everything is. Simple enough.

(PS I have no idea what I'm doing.)

Failures

My first version used Clojure agents (i.e. threads) for everything. The game logic was a thread, the renderer ran in a thread, every NPC was its own thread. The world itself was a single ref that all of these agents banged on. So the NPCs would tell the ref "I want to move down one square", another thread might say "Brian just pushed 'left' on the keyboard, so start scrolling the map". The world-ref would kindly oblige while preventing two NPCs from standing on each other or letting the PC walk through the walls.

Clojure is awesome in letting you do this in a completely safe and coordinated way. Everything worked well. But even for the crappy 2D graphics I'm using, all those threads caused way too much lag. I could get a good framerate if everyone was standing still, but if I was walking while the NPCs were walking, I'd get lots of lag. It was even worse on a slower computer.

My best guess is that the reason for the lag was the constant restarting of canceled transactions due to multiple threads trying to edit the world ref 50+ times per second. My sucky 2-core CPU couldn't keep up. It isn't surprising that this failed, in hindsight.

OpenGL?

Next, I decided to try out OpenGL. There are multiple options for OpenGL in Java. One is JOGL and another is lwjgl.

I program in Linux, and my video card is ancient. I can barely get OpenGL to work in the best of times. Installing JOGL was a slight chore (Gentoo doesn't even include it in its repo). JOGL is not just a JAR you throw onto CLASSPATH, you need some native extensions, obviously. I got it running somehow, but I wouldn't want to explain to someone else how I did it.

I did get jwjgl to work too, eventually, which was nice. There is a really nice Java 2D game framework called Slick which uses lwjgl. Some good games were created using this, for example Stickvania, which recently hit Reddit recently. I got Slick up and running in short order.

Unfortunately Slick doesn't play nicely with a Clojure REPL. I could build and start a game, but once the game is stopped, it never runs properly again without restarting the REPL. Slick caches images to try to be speedy, and it seems like the cache is either corrupted or destroyed when you close down your game, for one thing. This is not conducive to Clojure REPL-style development, and I didn't want to spend a lot of time fixing it.

Another issue is that I'd really like this game to be cross-platform and available to non-hackers, and the thought of trying to tell the average gamer how to install JOGL or lwjgl was daunting, given the bullcrap I had to go through. Not sure if I can just throw JOGL into a JAR and distribute it, maybe I can, but I didn't want to bother reading about it. Swing on the other hand runs everywhere with no effort.

My main problem is that I know even less about OpenGL programming than I do about Swing, and don't have a month to learn. Back to the drawing board.

Success

It turns out I don't need OpenGL anyways. All I need is program more intelligently. (Did I mention I have no idea what I'm doing?) Instead of dozens of threads, my current (working) version has one thread. It updates the game logic, then renders the game, then waits 10 milliseconds or so, then repeats this (forever).

With the single-threaded version, the logic is actually more complex than the multi-threaded version. Agents and refs were really nice and braindead-easy to work with. But such is life.

Once I learned about keeping track of things in realtime by counting milliseconds instead of counting frames or using timeouts to do logic/render updates, things worked better. (Did I mention I have no idea what I'm doing?) The game loop looks like this now:

(defn game-loop [#^Canvas canvas]
  (loop [last-time (get-time)]
    (let [curr-time (get-time)
          delta (- curr-time last-time)]
      (do-logic delta)
      (do-render canvas)
      (when @RUNNING
        (Thread/sleep 10)
        (recur curr-time)))))

The code to actually start the game, to give you an idea:

(defn start-world [world]
  (let [#^JFrame frame (doto (JFrame. "Game")
                         (.addWindowListener (proxy [WindowAdapter] []
                                               (windowClosing [e] (stop)))))
        #^JPanel panel (doto (.getContentPane frame)
                         (.setPreferredSize (Dimension. REAL-WIDTH REAL-HEIGHT))
                         (.setLayout nil))
        #^Canvas canvas (Canvas.)]
    (doto canvas
      (.setBounds 0 0 REAL-WIDTH REAL-HEIGHT)
      (.setIgnoreRepaint true)
      (.addKeyListener (proxy [KeyAdapter] []
                         (keyPressed [e] (handle-keypress e))))
      (.addMouseListener (proxy [MouseAdapter] []
                           (mouseClicked [e] (handle-mouse e))))
      )
    (.add panel canvas)
    (doto frame
      (.pack)
      (.setResizable false)
      (.setVisible true))
    (.createBufferStrategy canvas 2)
    (dosync (ref-set RUNNING true)
            (ref-set PAINTER (agent canvas))
            (ref-set WORLD world))
    (send-off @PAINTER game-loop)))

That's about it for the Swing side of things, aside from scribbling on the Canvas in the render function.

The agent I wrap around the Canvas controls the thread that runs the game loop. The agent helpfully keeps track of any exceptions that happen during the loop, and I can view those exceptions via agent-error, which is handy for debugging.

Note how little code it is to set up a keyboard event handler, thanks to proxy:

    (doto canvas
      ...
      (.addKeyListener (proxy [KeyAdapter] []
                         (keyPressed [e] (handle-keypress e))))

A couple lines of Clojure for what would be a lot of senseless boilerplate in Java. Notice how you the keyboard handler calls a normal Clojure function handle-keypress. Clojure / Java interop really is seamless.

Maps

My maps are made using ASCII art. (Did I mention I have no id-... never mind.) Here's the code for the test map, for example. Map here is a deftype (available in bleeding-edge Clojure), which takes a map of tiles, a "pad" tile, the map, and then a mask showing walls / tiles where the player shouldn't be allowed to walk. My Map type acts like a Clojure hash-map most of the time, but it also lets me name the fields that all maps should share, and has a proper "type", among other things.

(deftype Map [tileset pad tiles walls]
  clojure.lang.IPersistentMap)

(def test-map (cache-map
               (Map {\  (tile "ground")
                     \/ (tile "ground-shadow-botright")
                     \< (tile "ground-shadow-left")
                     \d (tile "dirt")
                     \| (tile "wall_vertical")
                     \- (tile "wall_horizontal")
                     \1 (tile "wall_topleft")
                     \2 (tile "wall_topright")
                     \3 (tile "wall_bottomleft")
                     \4 (tile "wall_bottomright")
                     \u (tile "below-wall")
                     \v (tile "below-wall-shadow")
                     \w (tile "wood-floor")
                     \c (tile "cobble")
                     \b (tile "bush")}

                    (tile "ground")

                    ["                 1----2                            "
                     "                 |vuuu|                            "
                     "1----------------4<bbb3--------------------------2 "
                     "|vuuuuuuuuuuuuuuuu/   uuuuuuuuuuuuuuuuuuuuuuuuuuu|<"
                     "|<1------------2               cccccccc d        |<"
                     "|<|vuuuuuuuuuuu|<       d      c      c          |<"
                     "|<|<           |< cccccccccccccc    d ccccccc d  |<"
                     "|<3------ -----4< c                         ccccc|<"
                     "|<uuuuuuu/uuuuuu/ c  1-------2    1------------2 |<"
                     "|<1---2 bcb       c  |vuuuuuu|<   |vuuuuuuuuuuu|<|<"
                     "|<|vuu|< c  d     c  |< 1-- -4<   |<           |<|<"
                     "|<3- -4<bcb       c  |< |vucuu/   3------ -----4<|<"
                     "|<uu/uu/ c        c  3--4< c      uuuuuuu/uuuuuu/|<"
                     "|<  c  dbcb       c  uuuu/ cccc         bcb      |<"
                     "|<  cccccccccccccccccccccccc  cccccccc   c   d   |<"
                     "|<  d        d                     d ccccc       |<"
                     "|<                    1--------------------------4<"
                     "3---------------------4uuuuuuuuuuuuuuuuuuuuuuuuuuu/"
                     "uuuuuuuuuuuuuuuuuuuuuuu/                           "]

                    ["                 xxxxxx                            "
                     "                 x    x                            "
                     "xxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx "
                     "x                                                x "
                     "x xxxxxxxxxxxxxx                                 x "
                     "x x            x                                 x "
                     "x x            x                                 x "
                     "x xxxxxxx xxxxxx                                 x "
                     "x                    xxxxxxxxx    xxxxxxxxxxxxxx x "
                     "x xxxxx x x          x       x    x            x x "
                     "x x   x              x  xxx xx    x            x x "
                     "x xx xx x x          x  x         xxxxxxx xxxxxx x "
                     "x                    xxxx                        x "
                     "x       x x                             x x      x "
                     "x                                                x "
                     "x                                                x "
                     "x                     xxxxxxxxxxxxxxxxxxxxxxxxxxxx "
                     "xxxxxxxxxxxxxxxxxxxxxxx                            "]
                    )))

This lets me make all of my tiles be simple PNG files with sane names. I could dork around with sprite sheets, but why bother? Emacs or Vim column-editing and overwrite modes make it easy enough to make a map this way. Swing thankfully handles transparency for me if I use PNGs, so it's a no-brainer to make multiple map layers later, which I'll need later. The code for reading in PNGs is brain-dead simple thanks to Java's ImageIO:

(defn tile [name]
  (ImageIO/read (File. (str "img/" name ".png"))))

The cache-map function (below) iterates over the ASCII art (via seqs on the Strings), using each character as a key into the map of real images, and builds a BufferedImage out of it. This is cached, since the renderer needs to re-draw the background every frame. In this function I'm also drawing a "padding" layer to sit under the background, so that I see endless fields of grass when I walk close to the edge of the map, instead of garbled, smeared-out graphical artifacts.

(defn cache-map [amap]
  (let [height (count (:tiles amap))
        width (count (first (:tiles amap)))

        #^BufferedImage img (BufferedImage. (tile-to-real width) (tile-to-real height) BufferedImage/TYPE_INT_ARGB)
        #^Graphics2D g (.createGraphics img)

        #^BufferedImage pad (BufferedImage. (tile-to-real width) (tile-to-real height) BufferedImage/TYPE_INT_ARGB)
        #^Graphics2D padg (.createGraphics pad)]
    (doseq [[y row] (cseq/indexed (:tiles amap))
            [x tile] (cseq/indexed row)]
      (.drawImage padg (:pad amap) (tile-to-real x) (tile-to-real y) REAL-TILE-SIZE REAL-TILE-SIZE nil)
      (if-let [#^Image img ((:tileset amap) tile)]
        (.drawImage g img (tile-to-real x) (tile-to-real y) REAL-TILE-SIZE REAL-TILE-SIZE nil)
        (throw (Exception. (str "Missing tile " tile ".")))))
    (assoc amap
      :map-image img
      :pad-image pad
      :max-map-x width
      :max-map-y height)))

This code is nasty, mostly due to converting between tile-based coordinates and pixel-based coordinates. This code needs to be cleaned up a bit. But that's about the most complex code you'll find in my program so far.

Who needs mutable state?

Clojure is a mostly functional language, in the sense of strongly discouraging unnecessary use of mutable state, and this program is no different. I'm sometimes amazed how far I can get before I need mutable state at all. The vast majority of my functions take a world value (a plain old hash-map) as an argument, and return a new world value after making changes to it. The current state of the world is whatever value is currently in the global WORLD ref.

The render loop grabs a snapshot of the world from the ref on each iteration, and then draws it. Thanks to Clojure refs, the snapshot of the world is guaranteed to be consistent (e.g. no NPC objects in the middle of mutating themselves) and persistent (the world value sticks around as long as the renderer needs it, even if the WORLD ref is changing in another thread). Once it's been drawn, the renderer throws the world snapshot away and it's garbage-collected later.

This all happens around 50-100 times per second in my game, and there's no noticeable lag. So that's a good thing.

Conclusion

That's about it. I'll post the full source code once it doesn't suck as much.

In my opinion, exploring a new area of study is some of the best fun a person can have. It's a flood of information and a surprise every 10 seconds.

Making a game has been like that. There's a huge wealth of knowledge about this kind of programming that I never knew existed. Everything I read on this topic is new to me and fascinating. The saddest(?) part of this whole thing is that I'm going to have more fun programming this game than I usually have playing games I buy in the store.

Out of memory... ouch

Wed, 20 Jan 2010 12:12:05 -0800

I've written before about how I'm running four Clojure-driven websites out of a single JVM on my VPS. No problems for many months, but today I tried to make a blog post and got all kinds of out-of-memory errors. Hopefully I didn't lose any / many user comments on this blog in the past couple days, but it's possible.

I restarted the JVM and gave it a bit more RAM to play with, I imagine this will fix things. But we'll see. It occurs to me now that there may be such a thing as too much caching.

Deploying Clojure websites

Mon, 04 Jan 2010 20:42:17 -0800

On my server I'm running one Java process, which handles four of my websites on four different domains. These are all running on Clojure + Compojure. Some people asked for details of how to do this, so here's a rough outline. For the sake of brevity I'm only going to talk about two domains here, though it scales up to however many you want pretty easily.

This is surely not the only way to do this, and probably not the best way, but it's what I've arrived at after a year of goofing off.

Summary: Emacs + SLIME + Clojure running in GNU Screen; all requests are handled by Apache and mod_proxy sends them to the appropriate Jetty instance / servlet.

Directory structure

First you have to decide on a directory structure. There's a sort of Clojure convention to have a src and deps directory, though Clojure is so young that this convention may or may not stick. Behold my ASCII line-art directory tree:

~/public_html/base/
|
|----src/
|    |
|    |----common/
|    |
|    |----net/
|         |
|         |----briancarper/
|         |    |
|         |    |----<lots of .clj files>
|         |
|         |----ffclassic/
|              |
|              |----<lots of .clj files>
|
|----deps/
|    |
|    |----<lots of .jar files>
|
|----briancarper.net/
|    |
|    |----public/
|    |
|    |----db/
|    |
|    |----apache/
|
|----ffclassic.net/
|    |
|    |----public/
|    |
|    |----apache/
|

src holds the Clojure source code for all of my sites. Java (therefore Clojure) forces your directory structure to match your namespaces, and traditionally people use domain names in reverse order for both. Whether you like or dislike this convention, you must admit that if you're writing code that's actually going to be used to host websites on domains, the convention probably makes sense. There are some utility libraries I wrote that I like to share between sites, so I put those in some common namespace in their own directory.

deps holds a bunch of .jar files. This includes clojure.jar, clojure-contrib.jar, compojure.jar, swank-clojure.jar (for SLIME), and all the Java libraries I use, like Jetty, a bunch of Apache Commons libraries, Tokyo Cabinet, Rhino and so on. This means that all of my sites are sharing libraries, and therefore must be running the same version of Clojure and all other libraries. So when I upgrade a library that one site is using, I must upgrade all the other sites. This is sometimes a hassle, but it forces me not to be lazy.

Then I have one more directory for each of my domains, for static files and such. These directories have different subdirectories depending on how I'm running them. For the sites that use Tokyo Cabinet, I have a db directory to hold the database files. On some sites I want to run a PHP or CGI script, so I have a directory for Apache to use. All of them have a public folder, from which Clojure serves images, JS and CSS files etc.

Phil Hagelberg's recent Leiningen project can possibly help with some of this. I don't use Lein but I may switch someday.

Emacs + SLIME + CLASSPATH setup

Building or installing Clojure and Emacs and SLIME and friends is beyond the scope of this blog post. swank-clojure has come a long way in being auto-installable via ELPA nowadays, if you like.

The most important thing is to set up CLASSPATH correctly. src should be on the CLASSPATH, as well as every .jar file in deps. My .emacs contains this, among other things:

(setq swank-clojure-classpath (list "." "./src" "./deps/*"))
(setq swank-clojure-library-paths (list "/usr/local/lib"))

Java will take a glob and expand it to include every jar file in a directory, thankfully. But this seems to happen right when you start Java, so to add a new .jar to your CLASSPATH you generally need to restart the JVM. This is annoying, but oh well.

If you AOT-compile your Clojure code you might also have a classes folder on your classpath, but I don't bother. You could also compile your whole site into its own .jar file and use that. I don't do this because I want to be able to edit my Clojure source files on the server if I need to.

swank-clojure-library-paths is necessary for me because Tokyo Cabinet uses some C libraries. You might not need it.

Then when I start Emacs, I M-x cd to the base folder and start SLIME there. It'll find Clojure in deps and give me a REPL. If you can get this working, you're well on your way.

Clojure code

I have a file server.clj for each domain, which are in charge of setting up and starting / stopping the servlets. So in src/net/briancarper/server.clj I have something like:

(ns net.briancarper.server
  (:use (compojure.http request servlet session routes)
        (compojure.server jetty)
        (compojure control)
        (net.briancarper ...)))  ;; lots of files containing the guts of the site

(defroutes some-routes ...)
(defroutes other-routes ...)

(defroutes all-routes
  some-routes
  other-routes)

(defserver blog-server
  {:port 8080 :host "localhost"}
  "/*" (servlet all-routes))

src/net/ffclassic/server.clj is similar, except with different routes (obviously) and a different port for the servlet. Note that we bind to localhost here, so that people can't connect directly to port 8080 and bypass Apache.

One of the routes should be set up to point to the proper public folder. Generally some catch-all route near the end of your list of routes, like:

(GET "/*" (static-file (params :*)))

where this static-file knows to serve files from briancarper.net/public, and the other server.clj for the other domains point to different directories.

Compojure has the compojure.http.helpers/serve-file function you can use to serve static files too. I use my own function for various reasons. Note that Compojure generally doesn't set any HTTP headers on your responses. You should probably set some kind of cache control headers unless you want your users to re-download all of your image files and stylesheets every time they hit a new page. You might also have to fiddle with Content-Type sometimes, though Compojure is usually pretty good about guessing them.

Note that Compojure's hands-off approach has its upsides too. For example, I have a bunch of CSS files for my blog. In Clojure, I read and concatenate all of those CSS files into one blob of text and have Clojure serve that as http://briancarper.net/combined.css. combined.css doesn't actually exist. I do the same for JS. This speeds up user requests quite a bit, without having to merge all the files on the filesystem. I could further compactify them if I cared.

If your Clojure app is handling arbitrary request URIs, you should be careful you aren't serving up ../../../../etc/passwd or something. Pretty sure Apache protects you against this by default, but Compojure uses compojure.http.helpers/safe-path? to test for it too if you use the built-in serve-file.

Finally, for convenience, in the base directory I have a file called server.clj which contains this:

(ns server
  (:require [net.briancarper server]
            [net.ffclassic server]))

(defn go []
  (.start net.briancarper.server/blog-server)
  (.start net.ffclassic.server/ffclassic-server))

Then, to start everything, from a REPL:

user> (require 'server)
2010-01-04 18:10:50.046::INFO:  Logging to STDERR via org.mortbay.log.StdErrLog
net.briancarper.server.proxy$javax.servlet.http.HttpServlet$0
net.ffclassic.server.proxy$javax.servlet.http.HttpServlet$0
nil
user> (server/go)
2010-01-04 18:11:02.260::INFO:  jetty-6.1.15
2010-01-04 18:11:02.358::INFO:  Started SocketConnector@localhost:8080
2010-01-04 18:11:02.707::INFO:  jetty-6.1.15
2010-01-04 18:11:02.789::INFO:  Started SocketConnector@localhost:8087
nil
user>

On the server (Apache)

To deploy this to my server I just rsync it all over SSH. On the server, I have Emacs and Java installed, as well as Apache and GNU Screen.

Why Apache? In case I want to run something that isn't Clojure, like webmail or some CGI script. Plus it's easy to run Clojure as a non-priviledged user on a non-priviledged port, and have Apache forward requests to the proper servlet based on the domain in the request. Apache is also nice for doing HTTPS.

My Apache setup is a pretty standard Debian-ish setup. You need mod_proxy installed. My Apache config for one domain looks a bit like this:

<VirtualHost *:80>
        ServerName ffclassic.net
        ServerAlias www.ffclassic.net

        <Directory /home/user/clj/ffclassic.net/apache/>
           AllowOverride All
           Order Allow,Deny
           Allow from all
        </Directory>

        DocumentRoot /home/user/clj/ffclassic.net/apache/

        ProxyPass /foo !
        ProxyPass / http://localhost:8087/
        ProxyPassReverse / http://localhost:8087/
</VirtualHost>

Yeah, that's about it. By default, every request will be passed to Clojure. The ports specified here should obviously match the ports you specify in your Clojure code for your servlets.

ProxyPass /foo ! tells Apache to handle requests to http://ffclassic.net/foo itself rather than pass them to Clojure. I override URIs on a case-by-case basis like this, to tell Apache to run whatever CGI or PHP scripts I need.

One oddity of running a setup like this is that thanks to mod_proxy, the "remote address" for every request is always going to be 127.0.0.1 by the time they get to Clojure. So if you want to see the IP address of your users, you have to do something like:

(defn- ip [request]
  (or ((:headers request) "x-forwarded-for")
      (:remote-addr request)))

Another oddity is that sometimes if Apache is running, and then I start Clojure, I have to restart Apache before it "finds" Clojure and starts forwarding traffic. No big deal though.

This is otherwise transparent from the Clojure end.

Starting it up

You probably want to SSH to your server, start a REPL, start your code running, and then log off the server. There are various ways you can run keep Emacs running in the background. You could run Emacs in server mode and attach to it with emacsclient. I use GNU Screen because it's easy and I have other things running in Screen instances that I like to switch between. So I start Screen, start Emacs, start SLIME, and start my site from there. Then detach from Screen when I'm done. It's easier than it sounds.

Why SLIME and not a normal commandline REPL? Because in Emacs I can patch the code easily on a function-by-function basis, among other things. You can do this from a commandline REPL but not nearly as easily. Emacs is a much nicer place to type than any command line. When I need to make an update to the site or fix a bug, my workflow is typically:

Fix it locally on my test machine, make sure it works
rsync the changed files to the server
SSH to the server
screen -r to bring up Emacs
Open the .clj file(s) I changed
C-c C-c to recompile and reload the individual function(s) I changed, or C-c C-k to recompile and load a whole .clj file, or whatever.
C-a d to detach from Screen

I can do any database inspecting or fiddling I need to do from the REPL. I can also do some live testing of the site or whatever else I want. It's good for debugging.

I've had Emacs running for months this way without having to restart it, it seems to work OK.

That's it

It's not that hard. It's clearly more steps than dumping some PHP files into a directory and having them work. But with Clojure, you get the benefit of a persistently running environment with a nice REPL interface, all in a nice fast multi-threaded JVM. Plus you get to write all your code in Clojure instead of PHP. In terms of fun and long-term maintenance costs, Clojure comes out way ahead in the end, for me anyways.

Clojure and Compojure to the rescue, again

Sun, 03 Jan 2010 15:57:52 -0800

I haven't posted here much recently because I've been hacking on another recently-sort-of-completed website. One of my favorite hobbies is old 8-bit video games. The first thing I ever programmed was a website about Final Fantasy for the old NES, and I've fiddled with it for the past 10 years or so.

A while back I decided to rewrite the whole thing using Clojure + Compojure with data in mysql. This went really well. I know lines of code isn't that great a metric, but it can give a rough estimate: this whole website is done in 3,400 lines of Clojure, which includes all of the HTML "templates" and the DB layer I had to write. And it's ~~turtles~~ Clojure all the way down. The only thing not written in Clojure are a couple bits of Javascript here and there and the stylesheet.

I suspect the target audience of this blog and the target audience of that website don't overlap that much, but I figured someone might be interested in some of the detail of how it's implemented. A few things I learned...

I didn't write a single line of HTML by hand. I can't overemphasize how wonderful this is. Compojure's HTML library lets you write HTML as Clojure data structures, with vectors being tags, keywords being tag names, hash-maps being attribute-value pairs etc.

Not only is this easier to write, it's easy to generate and manipulate. I have a ton of functions where given a list of hash-maps of data, it spits out a nicely formatted chart of that data. Here's an example:
```
(chart "Armor" (armor game)
       ["" #(name-with-image %)
        "Locations" (comp ulist location-links :locations)
        "Buy" (comp g :buy)
        "Sell" (comp g :sell)
        "Absorb" :abs
        "Evade%" (comp percent :ev)
        "Element" (comp ulist #(map :name %) :elements)
        "Casts" #(-> % :magic :name)
        "Usable by" (comp (partial equipable-by game ps) :pcs)])
```
chart is a function taking a chart title, some data, and then a vector of column-header names and functions for the data in each column. Anonymous functions are a great way to format each column differently depending on the kind of data it contains. chart also does some other stuff like standardizing the text of the chart title, alternating the background color of rows of data, re-inserting the header every so often, and formatting the first column specially (expecting it to be a title) and so on.

Sample output of this function is here. A screenshot for the lazy (click for bigger version):
Who needs an ORM? I wrote about this previously, but for my needs, an ORM is overkill. The vast majority of my interaction with a DB is reading data. All you need for that is an easy way to write SQL queries and get the results into a hash-map. I wrote a bunch of macros for this, and there's ClojureQL and clojure.contrib.sql and friends to help. For example I run some polls that users can vote on, and here's the entirety of the code that fetches it out of the DB.
```
(defn polls []
  (with-tables [polls poll_options poll_votes]
    (map (fn [p] (assoc p
                   :total (count (:votes p))))
         (-> polls
             (one-to-many :options poll_options (key= :id :poll_id))
             (one-to-many :votes poll_votes (key= :id :poll_id))))))
```
with-tables is a simple macro I wrote that fetches data from the DB for some table (possibly putting a WHERE clause on the query if specified, unspecified here). It takes care to run the minimal number of queries necessary, generally one query per table of data. Then I use other functions like one-to-many to join the data together. The result is a list of hash-maps, with the value of some keys (here :options, :votes) being sub-lists of hash-maps, for data that "belongs" to the toplevel hash-map.

Doing it this way helps me avoid the infamous "N+1 queries" which plagues naive use of Rails (where each of N objects would query the DB again to get its sub-lists of objects). I think Rails uses an :include option in its find functions to do that same kind of thing.

This is dirt-simple, but it gives me a lot of control. I can easily map or filter over any of these intermediary table lists before or after putting the results together. I can add keys with values that are calculated from other values, so it looks like the records have fields that don't really exist in the DB. I can do a sort-by on some key. I can take a user parameter and return different results based on whether that user is an admin or not. I can put metadata on the hash-maps, e.g. to indicate which table the data was fetched from. And so on.

Thanks to memoize (see below) this is really fast too. The most complicated query I have is for my walkthrough. After priming the cache:
```
net.ffclassic.db.charts> (time (dorun (walkthrough-chapters :nes)))
"Elapsed time: 0.074587 msecs"
```
Speaking of metadata: I like metadata.

For example I have a multimethod called link, which dispatches on the :table metadata key set from my SQL macros above, and outputs an appropriate link for something based on what it is. So (link some-user) links to a user's profile on the site, using the user's username as the anchor text, and styles it differently if it's an admin user or normal user. (link some-forum-post) links to the forum thread which contains the post. (link some-poll) links to a poll using the poll's title as the anchor.

Each of these spits out a vector that looks like [:a {:href "/user/view-profile/1"} "Brian"]. And that's changed into HTML eventually via Compojure's delicious HTML library.

This makes it really easy to link to things in a standard way with standard-looking anchor text. It also makes it easy to standardize on a URL scheme. If I want to change the URLs used to access my polls, I only need to change it in one place.

This is kind of like OOP, but not really, since I can have different metadata tags and use different ones depending on what I'm doing. And I can completely ignore the metadata whenever I don't need to use it.
Speaking of multimethods: multimethods are handy.

I have five "categories" of pages of data on my site, with pages in each category that are usually identical across categories. But sometimes a page in one category should look slightly different from the same page in other categories. Sometimes a page only belongs in one category and should 404 for the other categories. What I needed was some way to say "Display this page identically for each category by default, but page Y on category X is an exception".

Multimethods are great for this. Compojure's "routes" take a request URI and dispatch based on a regex pattern matched against it. I dispatch most requests to a series of multimethods. Given a request URI, I split the URI into category + everything else. Then, I first try a multimethod for category-specific pages, dispatching on a vector of [category rest-of-uri]. If there is no method for this, it falls through (returning :next) and Compojure tries the next route. The next route tries a multimethod for pages that look the same category-wide, dispatching only on rest-of-uri this time. If there's no method for that, it falls through again with :next and a list of "static" routes are tried (oddball pages that don't belong to any category). If those all fail, you get a 404.

This is an easy way to split your site into namespaces based on the uri. You can also add new pages without editing your Compojure routes. Just add a new method to the multimethod.

The code looks something like this:

*code edited 2010-01-04 to fix a crapload of typos, thanks Shawn
```
(defmulti category-specific-page (fn [cat page] [(keyword cat) (keyword page)]))
(defmethod category-specific-page :default [& _] :next)
(defmethod category-specific-page [:cat1 :some-page] [cat page]
  (...))


(defmulti cross-category-page (fn [cat page] (keyword page)))
(defmethod cross-category-page :default [& _] :next)
(defmethod cross-category-page :some-page [cat page]
  (...))


(defroutes some-routes
  (GET #"/(cat1|cat2|cat3)/([^/+])/*$" (apply category-specific-page (:route-params request)))
  (GET #"/(cat1|cat2|cat3)/([^/+])/*$" (apply cross-category-page (:route-params request)))
  (GET "/" (default-index-page))
  (ANY "/*" (error-404))) 
```
This was a nightmare to do properly in Rails. Rails wants to dispatch to controllers, which are classes. To make this work in Rails I had to mangle this concept to fit into the idea of a hierarchy of classes and subclasses. There might have been an elegant way to do it, but I couldn't think of one. I suspect someone will leave me a ~~flame~~ comment telling me how to do it.
memoize is great as a poor-man's web cache. I have a bunch of pages of data that never change. So the first request fetches it from the DB, and subsequent requests are cached in RAM and are therefore nearly instantaneous. The data is cached in the form of huge lists of hash-maps (many of which have sublists of more hash-maps, sometimes many levels deep). I was concerned at first that this might take a lot of RAM, but I went with it, and it turned out not to. top tells me:
```
PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
17   0  257m 127m  12m S    0 23.6  48:18.91 java 
```
I'm actually running four websites in the same instance of the JVM there, so it's not that much RAM usage at all.

memoize doesn't work for everything, obviously. There's no easy way to invalidate or refresh the cache with the built-in memoize, so it's not good for data that is updated frequently. If you do need to invalidate some data, you have to re-compile the function. But it works well for static data. The great thing is that it's very fine-grained. Change a single defn to defn-memo (in clojure.contrib.def) and you're done.
Compojure's "decorators" are really quite nice. For example I use them to implement a kind of templating / layout system. I have a with-layout function which, given a response, wraps the HTML for that response in a standard layout (header, sidebar, footer). I group a bunch of routes together and decorate all of those routes with with-layout, and then it's done. This "layout" code is completely separate from the code the generates the HTML for the pages, and lets me change it easily (add a new layout for some specific pages, or whatever). A bunch of other routes, like those that handle POST requests and such, don't get a layout; I just group those routes separately and don't decorate them.

All of this is done with plain old Clojure functions. There's no separate "template library". There's no ERB or Smarty to wrestle with, though you could use something like StringTemplate if you wanted to.

Another example: certain pages are admin-only. So I group all of those pages together into one group of routes, and do the admin check first-thing, immediately failing if someone isn't logged in or if the current user isn't an admin. This thankfully lets me not have to worry about doing permissions checks all over the place. The code looks like this:
```
(defn admin-only [handler]
  (fn [request]
    (when (:admin (:session request))
      (handler request))))


(defroutes admin-routes
  (GET "/some-admin-only-page" (admin-page))
  (GET "/another-admin-only-page" (other-admin-page)))


(defroutes admin-form-routes
  (POST "/some-admin-only-page" (mangle-database)))


(decorate admin-routes with-layout admin-only with-session)
(decorate admin-form-routes admin-only with-session)


(defroutes all-routes
  admin-routes
  admin-form-routes
  other-routes
  etc.)
```

I'm overall pretty pleased with the site. There are still some rough edges and much more work to be done, but it's a joy to code compared to the Rails version.

Comments work again

Thu, 24 Dec 2009 12:20:06 -0800

I broke the ability to leave comments a couple days ago. Thanks to everyone who let me know. It's fixed now.

I broke it while uploading yet another website I finished a couple days ago. It's yet another Compojure/Clojure site, this time a bit more ambitious than my humble blog. I plan to write about that whole experience once I have a bit of time.

Clojure funding

Mon, 14 Dec 2009 16:49:28 -0800

Rich Hickey works on Clojure full-time for free, and he's asking people who get something out of Clojure to contribute some cash. $100 is less than I spend on random books and crap over the course of a year, so I gladly chipped in.

Clojure doesn't make me any money, I use it for hobby websites that actually cost me money every month just to keep going. But it's so much fun I think $100 is small price to pay, if it keeps Clojure development going.

Lame comment spam management that works

Sun, 06 Dec 2009 02:34:08 -0800

It's been nine months since I ditched Wordpress and moved to a blog system I wrote from scratch (in Clojure). This was a great move in so many ways. One of those ways is comment spam. My site is as popular now (or maybe slightly more popular now) as it was when I was running Wordpress, so I think comparing before and after is valid.

With Wordpress, every morning I'd do the ritual of deleting overnight spambot droppings. Typically I got between 1 and 5 every night. I had a default Wordpress install and all I used for spam filtering was Askimet. Askimet did a surprisingly good job, catching literally if not thousands of spams every week which otherwise would've been ruining my site. But inevitably some would still get through. And what's worse, there were a lot more false positives than I could tolerate.

Since I started counting with my new system, which is around 6 months, to the best of my knowledge I've gotten zero spambot-produced comments that made it through my filters. This is pleasant, to say the least.

The system I'm using is stupid. None of it is stuff I thought of myself, I got ideas from other lots of other blogs or articles I read, but the implementation is mine and it's not sophisticated. It would take a bot author a few seconds to work around it. But no one has bothered. Why bother writing a bot for my one-man blog, when you can write a bot for Wordpress and have it work on tens of thousands of blogs? And I can change my system to defeat the bots with a few lines of code just as easily as they can work around it.

So here's why I think it's working.

1. It's not Wordpress

Just by using something slightly different from Wordpress, I'm think I'm already ahead. For example if you have a blog where a form posts comments to /wp-comments-post.php, a bot doesn't even need to look at your site to spam you. They can blast your server with POST data at that URL in a format they already know Wordpress will accept. My site is all custom code, so everything is different enough that default bot attempts fail immediately.

I think this is the reason that only 1853 spam comments have even been POSTed at me in the last six months. That's an improvement of one or two orders of magnitude already.

2. Honeypot text field

So what about the comments that are actually POSTed? They are presumably the result of bots that parse sites' HTML looking for comment forms and try to POST data that satisfies the form.

So in my comment form I have a field called referer. A "How did you find my site?" kind of thing. In fact I don't care how you found this site, this field is a honeypot. The div containing this field is hidden via CSS.

div#referer-row {
    display: none;
}

So you shouldn't ever see it if you're a human using a browser. But bots parsing the HTML see this field, unless they also bother to parse my CSS and see that it's hidden, which would be expensive and apparently they don't do it.

If you put anything in this referer field, your comment will be rejected as spam. Simple enough.

Many blogs require you to fill in every field or else the comment is rejected, so it seems reasonable to expect most bots to fill in all of your form fields. (My blog actually requires you to fill in nothing but the comment text; author will be set to Anonymous Cow if you don't fill it in.)

In fact this seems to be the case; of 1853 spam comments since March, 1810 put something into this field. Most of the time it's a random string of letters. Not even a URL. Sometimes it's a couple words like "insurance quotes" or something about drugs or casinos.

The downside of this is if you're a human using a browser that doesn't understand CSS, you will see this field. Then if you type something into it and try to comment, it'll end up as spam. So Lynx users and time travelers from 1987 trying to leave me comments might be confused at first.

However as far as I can tell, no intelligible data has ever been entered into this field by a human, so I don't think it's a concern. Six times, the word "None" was entered, but I don't think this is a human because that's nonsense answer to "How did you find this site?". But you never know.

3. Lame static CAPTCHA

That leaves 43 spam comments that made it this far. My other anti-spam measure is a word you have to type. But it's always the same word, and the word is COWS. This CAPTCHA caught the remaining 43. It looks like this:

There's a normal text field with a default value of <= Type this word specified right in the HTML.

<input type="text" value="<= Type this word" name="test" id="test"/>

There are no other instructions besides "Type this word". I'm assuming either that commenters are familiar enough with CAPTCHAs to know what I want, or can figure it out using common sense. Given that my target audience is computer geeks and programmers, this should be a safe assumption. In fact I've had less than a dozen false positives in the past six months via people failing this; see below for details.

To post a comment, the value of this field must contain the word "COWS" somewhere in it, case-insensitively. Otherwise it's spam. Easy enough to implement.

If you have Javacsript enabled, clicking on this field will clear out the default value. If you unfocus the field without typing anything, Javascript will put the default value back in. This is only for convenience. If you don't have Javascript enabled, you have to highlight and backspace over the default text. I don't think this is a huge burden.

Of 1853 spam comments, here's the breakdown for what values end up in this field.

1131: "&lt;= Type this word"
691:  "<= Type this word"
21:   Random letters and numbers
6:    empty
2:    A bunch of URLs
2:    Human beings making typos, e.g. "COW" or "COS".

So most bots are too stupid even to remove the default value from the field, and none of them entered the correct value. 691 times the bot was somehow smart enough to un-escape < into <, which is interesting, but didn't help it defeat the filter. A lot of the random words look like they were made by Markov chains, e.g. 'fridwolfur' and 'lyndonvolk' and 'calbertdom'. If I need to write a childerns' poem I'll know where to look for ideas. One time a bot or spammer managed to type "vows" somehow, but this might or might not be coincidence.

I think this is better than a normal CAPTCHA because:

It's always the word "COWS" in a normal font, so it requires no thought or eye strain to figure out.
It's black and white, so hopefully people with minor vision problems and color blindness can see it.
It fits thematically with my blog layout (it's COWS and it has cow spots).
It's kind of silly, so hopefully people chuckle rather than become ticked off.

It's worse than a real CAPTCHA because it requires no effort to break. So it wouldn't work for Wordpress or VBulletin or something with a million users. But I wonder if every Wordpress had a single static word as a CAPTCHA, but a different word for every blog (generated at install-time maybe), would it work better or worse than the random mangled multi-color CAPTCHAs no one can read? Real randomly-generated CAPTCHAs don't work anyways; bots can already beat them via OCR or other means. A simple word would be less annoying for a human, to be sure.

The other downside is that this is not very accessible to the blind or other people using screen readers, or browsers without image support. This is unfortunate and I'm still trying to figure out how to get around this. Right now the ALT text for the CAPTCHA image is This says 'COWS'; I don't know if this is enough help for people in those situations.

Of course I'll never know how many people see my CAPTCHA and storm away in a rage without even trying to post a comment. But I've never heard a complaint. If this level of CAPTCHA ticks you off personally, please swallow your anger and leave me a comment here saying so, if you feel obliged; I'd love to hear it.

False positives

As best I can tell, there are no false positives from people filling in the honeypot field. But even as simple as it is, some people don't succeed at the CAPTCHA image. Either they typo it or they ignore it entirely.

I just checked and I counted around 6 comments by real humans where the CAPTCHA was ignored and the default <= Type this word ended up in the spam DB. 4 of those people re-posted their comment successfully immediately afterwards by filling in the CAPTCHA. I'm not sure I'm ever going to get much better than that.

Spam that makes it through

I have still gotten spam. Maybe a dozen or so in the past six months. It's all been in the form of a human typing a normal-looking and relevant comment, about open source software or BASH for example, but with a spammy URL buried in it, e.g. a link to a really dodgy-looking blog trying to sell something, or some scummy SEO site. It's either a human or a very sophisticated (or lucky) bot; the comment text in these is indistinguishable from a real comment other than the spam URLs. I have to delete these by hand.

But I was getting these with Wordpress too. No automted anti-spam system is going to defeat a human being, so I don't worry about it.

That's it

The moral of this story is that it doesn't take much to protect yourself from comment spam if you write the code yourself. As long as it's unique, you'll probably be fine.

The other moral is that you don't have to annoy the hell out of your users to filter spam effectively. I'm making the assumption here that my COWS method is not that annoying; tell me if I'm wrong.

I don't know how well this scales. Probably not so well. My blog isn't that highly trafficked. If my site were more popular it might be worse for me. But the improvement over Wordpress is unquestionable.

I've seen all kinds of complicated measures suggested elsewhere, like trying to predict if it's a bot by how many milliseconds it takes between page load and comment posting, or measuring keypress speed, or escaping the HTML of your forms and un-escaping it at loadtime it via Javascript, or setting and retrieving cookies and such. But a lot of this stuff seems fragile and if your browser doesn't suppoort Javascript or cookies (or your users block them), you're screwed. I block these things myself, so I expect visitors to do the same.

If everyone wrote their own blog engines, the world would be a slightly less spammy place. Or else we'd have much smarter bots.

I'm turning into a Lisp snob

Fri, 13 Nov 2009 22:38:50 -0800

Reddit and StackOverflow and other websites I frequent are filled to the brim with discussion of Google's Go. The code snippet on the front page is:

package main

import "fmt"

func main() {
  fmt.Printf("Hello, 世界\n")
}

First thoughts that ran through my head as I looked over the site:

Ugh, look at all that syntax.
Nice(r) type system (than C++ and Java). I'll stick with multimethods though.
Concurrency semantics, hmmm... Shared mutable memory between threads? I think I'll stick with Clojure for now thanks.
Where are the macros?
It has anonymous functions and closures and sort-of first-class functions? Good. Welcome to the 1960's.
len is a special operator? Sigh. (Programming language quality is usually inversely proportional to the number of special forms.)
Cool that they used Japanese in the example though. (That word is sekai, "world", obviously.)

Compared to a Lisp, this language looks indistinguishable to C, Perl, Python, Java etc. It looks like such a small incremental improvement (if it even is an improvement). Yet another imperative, for-loop-wielding, curly-brace-using, pointer-mangling, state-mutating, OOP language.

In fact via Reddit today I read this awesome post to a mailing list which compares Go with ALGOL68, and it gave me a would-you-look-at-that moment. Once you learn a few languages that are significantly different from ALGOL derivatives, all ALGOLish languages start to look eerily similar. Are we really stuck with ALGOL-derived languages being the only viable mainstream languages for all time? How much polish can we possibly apply to the same turd?

Then I realized, I'm turning into a Lisp snob. : ( Learning a Lisp apparently does spoil you for the rest of time. I am without a basis to judge whether this language will be a successful replacement of anything. All I know is I probably won't use it. Honestly I'm much more excited about new things on the horizon in Clojure. And I still have getting better at Haskell on my TODO list.

Clojure: redirecting output to a file

Thu, 12 Nov 2009 21:23:37 -0800

In a scripting language, you often run scripts from a command line and print things to STDOUT. For long output you can redirect it to a file via shell-redirection ($ foo.rb > foo.txt).

But if you're using Clojure for ad-hoc scripting, you're probably sitting in a REPL, not running from a command line. REPLs generally don't have shell-redirection. Printed output just gets dumped into your REPL. This can be annoying. (After a good 10,000 lines of output into a REPL buffer, Emacs starts to lag.)

But you can write a macro to handle this easily enough.

(use 'clojure.contrib.duck-streams)
(defmacro redir [filename & body]
  `(spit ~filename (with-out-str ~@body)))

Then:

user> (redir "foo.txt" (foo) (bar) (baz))

Now whatever (foo) and friends would've printed to STDOUT will instead go to foo.txt. I've found this somewhat useful at work lately.

Edit: better version via Graham Fawcett:

(defmacro redir [filename & body]
  `(binding [*out* (writer ~filename)] ~@body))

Clojure ORM-ish stuff

Tue, 03 Nov 2009 18:21:03 -0800

Suppose I have this:

user> (def foo [{:id 1 :foo 123} {:id 2 :foo 456}])
#'user/foo
user> (def bar [{:foo_id 1 :bar 111} {:foo_id 1 :bar 222}])
#'user/bar

What I want is to "join" foo and bar so that each item in foo ends up with a sub-list of bars based on matching key fields.

In real life, these lists-of-hash-maps are coming out of a database via clojure.contrib.sql, so this is something I actually want to do pretty often. This is also vaguely similar to what you get out of a Rails-like ORM, where you end up with an object that has lists of sub-objects anywhere you have a one-to-many relationship.

Here's how I end up doing this in Clojure:

(defn one-to-many
  ([xs name ys f]
      (for [x xs :let [ys (filter (partial f x) ys)]]
        (assoc x name ys))))

Now I can do this:

user> (pprint (one-to-many foo :bars bar #(= (:id %1) (:foo_id %2))))
({:bars ({:foo_id 1, :bar 111} {:foo_id 1, :bar 222}), :id 1, :foo 123}
 {:bars (), :id 2, :foo 456})

And if I define a helper function:

(defn key=
  ([xkey ykey]
     #(= (xkey %1) (ykey %2))))

Then I can write it more concisely:

user> (pprint (one-to-many foo :bars bar (key= :id :foo_id)))
;; same as above

And if I have another "table" of data like this:

user> (def baz [{:foo_id 1 :baz 555} {:foo_id 2 :baz 999}])
#'user/baz

Then I can join them all like this:

user> (pprint (-> foo
                  (one-to-many :bars bar (key= :id :foo_id))
                  (one-to-many :bazzes baz (key= :id :foo_id))))
({:bazzes ({:foo_id 1, :baz 555}),
  :bars ({:foo_id 1, :bar 111} {:foo_id 1, :bar 222}),
  :id 1,
  :foo 123}
 {:bazzes ({:foo_id 2, :baz 999}), :bars (), :id 2, :foo 456})

This is pretty concise. It may be possible to do it in an even more concise way, (if so, do share). If I was willing to adhere to some Rails-y naming convention for my table names and for the id fields in my tables, I could make this shorter by not having to specify the names of the id fields, but I don't want to go there. It's trivial to write similar functions for a one-to-one relationship, or to use a join-table to "join" two tables with a many-to-many relationship.

I am happily surprised sometimes by how simple it is to roll my own version of things that previously seemed like dark magic. I used Rails for a long time and it seemed like a crapload of code must have gone into making the ORM work. But four lines of code gets me 75% of what I ever needed Rails' ORM for.

This may be more thanks to me opening my eyes a bit than to Clojure being awesome, but either way, I'll take it.

Emacs Clojure colors

Fri, 30 Oct 2009 19:02:14 -0700

In yet another step along the path of trying to forcibly morph Emacs into Vim, I started a port of my Vim color scheme Gentooish to an Emacs color-theme.

Then I threw a few lines at the bottom of .emacs to highlight a few more Clojure-specific things when in clojure-mode. Result:

This is inspired partly by parenface.el which dims parens so they don't stand out as much. Lispers have this silly meme where they pretend they can't see parens at all, but being a mere human I want Emacs to help blend them into the background a bit.

I want to dim parens but also make braces and brackets pop out a bit more. Coloring braces, brackets and parens slightly differently helps make a few things easier to read in my opinion, especially in ))]}])) kinds of situations (like at the end of the function above).

This is pretty brute-force and I really wish I could figure out how to highlight #{} (sets) differently than {} (hash-maps). Highlighting (), '(), `(), and #() differently would be awesome too. But that is well beyond my elisp skills at the present. Certain areas of Emacs are all but impenetrable, no matter how much documentation I read and how much banging on the keyboard I do. The morass that is font-lock is one of those areas.

p.s. I find an easy way to keep your dotfiles in git is to make a folder (called dotfiles or something) and symlink your dotfiles from there to your home directory. I also put * in .gitignore so git doesn't slurp up anything sensitive by accident.

Java timestamp (in)equality

Fri, 23 Oct 2009 00:17:22 -0700

user> (let [milli (.getTime (.getTime (java.util.Calendar/getInstance)))
            ts (java.sql.Timestamp. milli)
            d  (java.util.Date. milli)]
        [(= (.getTime d) (.getTime ts))
         (= ts d)
         (= d ts)])
[true false true]

Really, Java? Really? Come on now.

Documenting it doesn't make it any better either:

Note: This type is a composite of a java.util.Date and a separate nanoseconds value. Only integral seconds are stored in the java.util.Date component. The fractional seconds - the nanos - are separate. The Timestamp.equals(Object) method never returns true when passed an object that isn't an instance of java.sql.Timestamp, because the nanos component of a date is unknown. As a result, the Timestamp.equals(Object) method is not symmetric with respect to the java.util.Date.equals(Object) method. Also, the hashcode method uses the underlying java.util.Date implementation and therefore does not include nanos in its computation.

The internet is strewn with the wreckage of people being gnawed upon by this bug.