Clojure and Compojure to the rescue, again

I haven't posted here much recently because I've been hacking on another recently-sort-of-completed website. One of my favorite hobbies is old 8-bit video games. The first thing I ever programmed was a website about Final Fantasy for the old NES, and I've fiddled with it for the past 10 years or so.

A while back I decided to rewrite the whole thing using Clojure + Compojure with data in mysql. This went really well. I know lines of code isn't that great a metric, but it can give a rough estimate: this whole website is done in 3,400 lines of Clojure, which includes all of the HTML "templates" and the DB layer I had to write. And it's turtles Clojure all the way down. The only thing not written in Clojure are a couple bits of Javascript here and there and the stylesheet.

I suspect the target audience of this blog and the target audience of that website don't overlap that much, but I figured someone might be interested in some of the detail of how it's implemented. A few things I learned...

  1. I didn't write a single line of HTML by hand. I can't overemphasize how wonderful this is. Compojure's HTML library lets you write HTML as Clojure data structures, with vectors being tags, keywords being tag names, hash-maps being attribute-value pairs etc.

    Not only is this easier to write, it's easy to generate and manipulate. I have a ton of functions where given a list of hash-maps of data, it spits out a nicely formatted chart of that data. Here's an example:

    (chart "Armor" (armor game)
           ["" #(name-with-image %)
            "Locations" (comp ulist location-links :locations)
            "Buy" (comp g :buy)
            "Sell" (comp g :sell)
            "Absorb" :abs
            "Evade%" (comp percent :ev)
            "Element" (comp ulist #(map :name %) :elements)
            "Casts" #(-> % :magic :name)
            "Usable by" (comp (partial equipable-by game ps) :pcs)])
    

    chart is a function taking a chart title, some data, and then a vector of column-header names and functions for the data in each column. Anonymous functions are a great way to format each column differently depending on the kind of data it contains. chart also does some other stuff like standardizing the text of the chart title, alternating the background color of rows of data, re-inserting the header every so often, and formatting the first column specially (expecting it to be a title) and so on.

    Sample output of this function is here. A screenshot for the lazy (click for bigger version):

    Armor Page

  2. Who needs an ORM? I wrote about this previously, but for my needs, an ORM is overkill. The vast majority of my interaction with a DB is reading data. All you need for that is an easy way to write SQL queries and get the results into a hash-map. I wrote a bunch of macros for this, and there's ClojureQL and clojure.contrib.sql and friends to help. For example I run some polls that users can vote on, and here's the entirety of the code that fetches it out of the DB.

    (defn polls []
      (with-tables [polls poll_options poll_votes]
        (map (fn [p] (assoc p
                       :total (count (:votes p))))
             (-> polls
                 (one-to-many :options poll_options (key= :id :poll_id))
                 (one-to-many :votes poll_votes (key= :id :poll_id))))))
    

    with-tables is a simple macro I wrote that fetches data from the DB for some table (possibly putting a WHERE clause on the query if specified, unspecified here). It takes care to run the minimal number of queries necessary, generally one query per table of data. Then I use other functions like one-to-many to join the data together. The result is a list of hash-maps, with the value of some keys (here :options, :votes) being sub-lists of hash-maps, for data that "belongs" to the toplevel hash-map.

    Doing it this way helps me avoid the infamous "N+1 queries" which plagues naive use of Rails (where each of N objects would query the DB again to get its sub-lists of objects). I think Rails uses an :include option in its find functions to do that same kind of thing.

    This is dirt-simple, but it gives me a lot of control. I can easily map or filter over any of these intermediary table lists before or after putting the results together. I can add keys with values that are calculated from other values, so it looks like the records have fields that don't really exist in the DB. I can do a sort-by on some key. I can take a user parameter and return different results based on whether that user is an admin or not. I can put metadata on the hash-maps, e.g. to indicate which table the data was fetched from. And so on.

    Thanks to memoize (see below) this is really fast too. The most complicated query I have is for my walkthrough. After priming the cache:

    net.ffclassic.db.charts> (time (dorun (walkthrough-chapters :nes)))
    "Elapsed time: 0.074587 msecs"
    
  3. Speaking of metadata: I like metadata.

    For example I have a multimethod called link, which dispatches on the :table metadata key set from my SQL macros above, and outputs an appropriate link for something based on what it is. So (link some-user) links to a user's profile on the site, using the user's username as the anchor text, and styles it differently if it's an admin user or normal user. (link some-forum-post) links to the forum thread which contains the post. (link some-poll) links to a poll using the poll's title as the anchor.

    Each of these spits out a vector that looks like [:a {:href "/user/view-profile/1"} "Brian"]. And that's changed into HTML eventually via Compojure's delicious HTML library.

    This makes it really easy to link to things in a standard way with standard-looking anchor text. It also makes it easy to standardize on a URL scheme. If I want to change the URLs used to access my polls, I only need to change it in one place.

    This is kind of like OOP, but not really, since I can have different metadata tags and use different ones depending on what I'm doing. And I can completely ignore the metadata whenever I don't need to use it.

  4. Speaking of multimethods: multimethods are handy.

    I have five "categories" of pages of data on my site, with pages in each category that are usually identical across categories. But sometimes a page in one category should look slightly different from the same page in other categories. Sometimes a page only belongs in one category and should 404 for the other categories. What I needed was some way to say "Display this page identically for each category by default, but page Y on category X is an exception".

    Multimethods are great for this. Compojure's "routes" take a request URI and dispatch based on a regex pattern matched against it. I dispatch most requests to a series of multimethods. Given a request URI, I split the URI into category + everything else. Then, I first try a multimethod for category-specific pages, dispatching on a vector of [category rest-of-uri]. If there is no method for this, it falls through (returning :next) and Compojure tries the next route. The next route tries a multimethod for pages that look the same category-wide, dispatching only on rest-of-uri this time. If there's no method for that, it falls through again with :next and a list of "static" routes are tried (oddball pages that don't belong to any category). If those all fail, you get a 404.

    This is an easy way to split your site into namespaces based on the uri. You can also add new pages without editing your Compojure routes. Just add a new method to the multimethod.

    The code looks something like this:

    *code edited 2010-01-04 to fix a crapload of typos, thanks Shawn

    (defmulti category-specific-page (fn [cat page] [(keyword cat) (keyword page)]))
    (defmethod category-specific-page :default [& _] :next)
    (defmethod category-specific-page [:cat1 :some-page] [cat page]
      (...))
    
    
    (defmulti cross-category-page (fn [cat page] (keyword page)))
    (defmethod cross-category-page :default [& _] :next)
    (defmethod cross-category-page :some-page [cat page]
      (...))
    
    
    (defroutes some-routes
      (GET #"/(cat1|cat2|cat3)/([^/+])/*$" (apply category-specific-page (:route-params request)))
      (GET #"/(cat1|cat2|cat3)/([^/+])/*$" (apply cross-category-page (:route-params request)))
      (GET "/" (default-index-page))
      (ANY "/*" (error-404))) 
    

    This was a nightmare to do properly in Rails. Rails wants to dispatch to controllers, which are classes. To make this work in Rails I had to mangle this concept to fit into the idea of a hierarchy of classes and subclasses. There might have been an elegant way to do it, but I couldn't think of one. I suspect someone will leave me a flame comment telling me how to do it.

  5. memoize is great as a poor-man's web cache. I have a bunch of pages of data that never change. So the first request fetches it from the DB, and subsequent requests are cached in RAM and are therefore nearly instantaneous. The data is cached in the form of huge lists of hash-maps (many of which have sublists of more hash-maps, sometimes many levels deep). I was concerned at first that this might take a lot of RAM, but I went with it, and it turned out not to. top tells me:

    PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
    17   0  257m 127m  12m S    0 23.6  48:18.91 java 
    

    I'm actually running four websites in the same instance of the JVM there, so it's not that much RAM usage at all.

    memoize doesn't work for everything, obviously. There's no easy way to invalidate or refresh the cache with the built-in memoize, so it's not good for data that is updated frequently. If you do need to invalidate some data, you have to re-compile the function. But it works well for static data. The great thing is that it's very fine-grained. Change a single defn to defn-memo (in clojure.contrib.def) and you're done.

  6. Compojure's "decorators" are really quite nice. For example I use them to implement a kind of templating / layout system. I have a with-layout function which, given a response, wraps the HTML for that response in a standard layout (header, sidebar, footer). I group a bunch of routes together and decorate all of those routes with with-layout, and then it's done. This "layout" code is completely separate from the code the generates the HTML for the pages, and lets me change it easily (add a new layout for some specific pages, or whatever). A bunch of other routes, like those that handle POST requests and such, don't get a layout; I just group those routes separately and don't decorate them.

    All of this is done with plain old Clojure functions. There's no separate "template library". There's no ERB or Smarty to wrestle with, though you could use something like StringTemplate if you wanted to.

    Another example: certain pages are admin-only. So I group all of those pages together into one group of routes, and do the admin check first-thing, immediately failing if someone isn't logged in or if the current user isn't an admin. This thankfully lets me not have to worry about doing permissions checks all over the place. The code looks like this:

    (defn admin-only [handler]
      (fn [request]
        (when (:admin (:session request))
          (handler request))))
    
    
    (defroutes admin-routes
      (GET "/some-admin-only-page" (admin-page))
      (GET "/another-admin-only-page" (other-admin-page)))
    
    
    (defroutes admin-form-routes
      (POST "/some-admin-only-page" (mangle-database)))
    
    
    (decorate admin-routes with-layout admin-only with-session)
    (decorate admin-form-routes admin-only with-session)
    
    
    (defroutes all-routes
      admin-routes
      admin-form-routes
      other-routes
      etc.)
    

I'm overall pretty pleased with the site. There are still some rough edges and much more work to be done, but it's a joy to code compared to the Rails version.

January 03, 2010 @ 7:57 AM PST
Cateogory: Programming

15 Comments

Sean Devlin
Quoth Sean Devlin on January 03, 2010 @ 11:22 AM PST

Brian,

I'm just excited to somebody write a post about Clojure AND Final Fantasy. Knight + Red Wizard + Black Wizard + Black Wizard FTW!

:)

Mark
Quoth Mark on January 03, 2010 @ 5:09 PM PST

I mainly use Django for web development, because it comes with a pretty powerful admin interface that makes it easy for non-programmers to add things to the database. I just can't imagine taking the time to code all that from scratch.

Compojure doesn't come with anything like that, right?

Mark
Quoth Mark on January 03, 2010 @ 5:30 PM PST

BTW, can you tell us more about your web hosting service, and what you had to do to set up the Java server and getting it working with Clojure/Compojure?

Sandeep
Quoth Sandeep on January 03, 2010 @ 7:27 PM PST

care to share your with-tables macro ?

That would be helpful!

Chad Cook (Flying Mullet)
Quoth Chad Cook (Flying Mullet) on January 03, 2010 @ 10:45 PM PST

I'll raise my hand as one of the few people that would be interested in the blog and the website. I've never heard of Clojure or Compojure but they look like very interesting subjects to explore. Did you look into CakePHP when investigating ways to recode your site?

Shashy
Quoth Shashy on January 04, 2010 @ 12:12 AM PST

Are you running four compojure sites on the same JVM instance? Could you describe how you do this please?

Shawn Hoover
Quoth Shawn Hoover on January 04, 2010 @ 4:02 AM PST

Is there a typo in the routing sample? It seems like there should be two defmulti calls and four defmethod calls, two defmethods for category-specific-page and two for cross-category-page.

Brian
Quoth Brian on January 04, 2010 @ 4:32 AM PST

@Sean Good party, I approve. :)

@Shawn Yep, typo, thanks for noticing.

I'll make another blog post tonight about how I deploy my sites, since it seems people are interested.

Michael
Quoth Michael on January 04, 2010 @ 9:39 PM PST

Are you publishing it to github?

Brian
Quoth Brian on January 08, 2010 @ 6:09 AM PST

The code is so specific to my site and so hacky that I don't think it's going to be of any use to anyone. The blog code I put on the site already is simpler and serves as a better example, for what it's worth.

Ronen
Quoth Ronen on January 16, 2010 @ 5:12 AM PST

I really liked the techniques described here, iv pocked around with code blog code in github & found it quite readable & easy to follow.

The only thing I had to fix was to in blog.tokyocabinet/make-dataref (add-watcher got deprecated in 1.1, the alternative is to use add-watch).

Thank you for sharing you knowledge.

Victor
Quoth Victor on June 24, 2010 @ 11:42 PM PDT

I am not able to view your full blog post past the summary. Clicking on "Read On..." simply refreshes the page without taking me to the whole post. This is happening in both Safari and Chrome. Is the post broken?

Thanks

Ted
Quoth Ted on May 02, 2011 @ 7:09 AM PDT

*LIF2*

I'm late to the party and very new to Clojure, but Compojure and FF? Nice! Can't wait to dig into this.

angel
Quoth angel on June 25, 2011 @ 5:48 AM PDT

Brian you came from ruby no?...How do you feel the metric when program using compojure, comparing it to small frameworks like sinatra?, maybe in sinatra it would be more short (and slow!!)...I've been using clojure but I feel it for web development would be complicate, especially compared to alternatives like sinatra or express.js

Brian
Quoth Brian on June 25, 2011 @ 3:45 PM PDT

I've found Clojure for web dev to be awkward. Ring is a good library, but sessions, cookies, and stuff that's braindead simple in Ruby frameworks always ends up being a hassle in Clojure. It might be my own ignorance at work. But I don't think web dev in Clojure has solidified yet into everything it has the potential to be.

On the other hand, libraries like hiccup are just insanely awesome and have no real equivalent in a non-Lisp language.

I have only toyed briefly with sinatra and never used express.js, but used Rails heavily, so take my opinion with a grain of salt.

Speak your Mind

You can use Markdown in your comment.
Email/URL are optional. Email is only used for Gravatar.

Preview