briancarper.net (λ) (Tag: SLIME)

Deploying Clojure websites

Mon, 04 Jan 2010 20:42:17 -0800

On my server I'm running one Java process, which handles four of my websites on four different domains. These are all running on Clojure + Compojure. Some people asked for details of how to do this, so here's a rough outline. For the sake of brevity I'm only going to talk about two domains here, though it scales up to however many you want pretty easily.

This is surely not the only way to do this, and probably not the best way, but it's what I've arrived at after a year of goofing off.

Summary: Emacs + SLIME + Clojure running in GNU Screen; all requests are handled by Apache and mod_proxy sends them to the appropriate Jetty instance / servlet.

Directory structure

First you have to decide on a directory structure. There's a sort of Clojure convention to have a src and deps directory, though Clojure is so young that this convention may or may not stick. Behold my ASCII line-art directory tree:

~/public_html/base/
|
|----src/
|    |
|    |----common/
|    |
|    |----net/
|         |
|         |----briancarper/
|         |    |
|         |    |----<lots of .clj files>
|         |
|         |----ffclassic/
|              |
|              |----<lots of .clj files>
|
|----deps/
|    |
|    |----<lots of .jar files>
|
|----briancarper.net/
|    |
|    |----public/
|    |
|    |----db/
|    |
|    |----apache/
|
|----ffclassic.net/
|    |
|    |----public/
|    |
|    |----apache/
|

src holds the Clojure source code for all of my sites. Java (therefore Clojure) forces your directory structure to match your namespaces, and traditionally people use domain names in reverse order for both. Whether you like or dislike this convention, you must admit that if you're writing code that's actually going to be used to host websites on domains, the convention probably makes sense. There are some utility libraries I wrote that I like to share between sites, so I put those in some common namespace in their own directory.

deps holds a bunch of .jar files. This includes clojure.jar, clojure-contrib.jar, compojure.jar, swank-clojure.jar (for SLIME), and all the Java libraries I use, like Jetty, a bunch of Apache Commons libraries, Tokyo Cabinet, Rhino and so on. This means that all of my sites are sharing libraries, and therefore must be running the same version of Clojure and all other libraries. So when I upgrade a library that one site is using, I must upgrade all the other sites. This is sometimes a hassle, but it forces me not to be lazy.

Then I have one more directory for each of my domains, for static files and such. These directories have different subdirectories depending on how I'm running them. For the sites that use Tokyo Cabinet, I have a db directory to hold the database files. On some sites I want to run a PHP or CGI script, so I have a directory for Apache to use. All of them have a public folder, from which Clojure serves images, JS and CSS files etc.

Phil Hagelberg's recent Leiningen project can possibly help with some of this. I don't use Lein but I may switch someday.

Emacs + SLIME + CLASSPATH setup

Building or installing Clojure and Emacs and SLIME and friends is beyond the scope of this blog post. swank-clojure has come a long way in being auto-installable via ELPA nowadays, if you like.

The most important thing is to set up CLASSPATH correctly. src should be on the CLASSPATH, as well as every .jar file in deps. My .emacs contains this, among other things:

(setq swank-clojure-classpath (list "." "./src" "./deps/*"))
(setq swank-clojure-library-paths (list "/usr/local/lib"))

Java will take a glob and expand it to include every jar file in a directory, thankfully. But this seems to happen right when you start Java, so to add a new .jar to your CLASSPATH you generally need to restart the JVM. This is annoying, but oh well.

If you AOT-compile your Clojure code you might also have a classes folder on your classpath, but I don't bother. You could also compile your whole site into its own .jar file and use that. I don't do this because I want to be able to edit my Clojure source files on the server if I need to.

swank-clojure-library-paths is necessary for me because Tokyo Cabinet uses some C libraries. You might not need it.

Then when I start Emacs, I M-x cd to the base folder and start SLIME there. It'll find Clojure in deps and give me a REPL. If you can get this working, you're well on your way.

Clojure code

I have a file server.clj for each domain, which are in charge of setting up and starting / stopping the servlets. So in src/net/briancarper/server.clj I have something like:

(ns net.briancarper.server
  (:use (compojure.http request servlet session routes)
        (compojure.server jetty)
        (compojure control)
        (net.briancarper ...)))  ;; lots of files containing the guts of the site

(defroutes some-routes ...)
(defroutes other-routes ...)

(defroutes all-routes
  some-routes
  other-routes)

(defserver blog-server
  {:port 8080 :host "localhost"}
  "/*" (servlet all-routes))

src/net/ffclassic/server.clj is similar, except with different routes (obviously) and a different port for the servlet. Note that we bind to localhost here, so that people can't connect directly to port 8080 and bypass Apache.

One of the routes should be set up to point to the proper public folder. Generally some catch-all route near the end of your list of routes, like:

(GET "/*" (static-file (params :*)))

where this static-file knows to serve files from briancarper.net/public, and the other server.clj for the other domains point to different directories.

Compojure has the compojure.http.helpers/serve-file function you can use to serve static files too. I use my own function for various reasons. Note that Compojure generally doesn't set any HTTP headers on your responses. You should probably set some kind of cache control headers unless you want your users to re-download all of your image files and stylesheets every time they hit a new page. You might also have to fiddle with Content-Type sometimes, though Compojure is usually pretty good about guessing them.

Note that Compojure's hands-off approach has its upsides too. For example, I have a bunch of CSS files for my blog. In Clojure, I read and concatenate all of those CSS files into one blob of text and have Clojure serve that as http://briancarper.net/combined.css. combined.css doesn't actually exist. I do the same for JS. This speeds up user requests quite a bit, without having to merge all the files on the filesystem. I could further compactify them if I cared.

If your Clojure app is handling arbitrary request URIs, you should be careful you aren't serving up ../../../../etc/passwd or something. Pretty sure Apache protects you against this by default, but Compojure uses compojure.http.helpers/safe-path? to test for it too if you use the built-in serve-file.

Finally, for convenience, in the base directory I have a file called server.clj which contains this:

(ns server
  (:require [net.briancarper server]
            [net.ffclassic server]))

(defn go []
  (.start net.briancarper.server/blog-server)
  (.start net.ffclassic.server/ffclassic-server))

Then, to start everything, from a REPL:

user> (require 'server)
2010-01-04 18:10:50.046::INFO:  Logging to STDERR via org.mortbay.log.StdErrLog
net.briancarper.server.proxy$javax.servlet.http.HttpServlet$0
net.ffclassic.server.proxy$javax.servlet.http.HttpServlet$0
nil
user> (server/go)
2010-01-04 18:11:02.260::INFO:  jetty-6.1.15
2010-01-04 18:11:02.358::INFO:  Started SocketConnector@localhost:8080
2010-01-04 18:11:02.707::INFO:  jetty-6.1.15
2010-01-04 18:11:02.789::INFO:  Started SocketConnector@localhost:8087
nil
user>

On the server (Apache)

To deploy this to my server I just rsync it all over SSH. On the server, I have Emacs and Java installed, as well as Apache and GNU Screen.

Why Apache? In case I want to run something that isn't Clojure, like webmail or some CGI script. Plus it's easy to run Clojure as a non-priviledged user on a non-priviledged port, and have Apache forward requests to the proper servlet based on the domain in the request. Apache is also nice for doing HTTPS.

My Apache setup is a pretty standard Debian-ish setup. You need mod_proxy installed. My Apache config for one domain looks a bit like this:

<VirtualHost *:80>
        ServerName ffclassic.net
        ServerAlias www.ffclassic.net

        <Directory /home/user/clj/ffclassic.net/apache/>
           AllowOverride All
           Order Allow,Deny
           Allow from all
        </Directory>

        DocumentRoot /home/user/clj/ffclassic.net/apache/

        ProxyPass /foo !
        ProxyPass / http://localhost:8087/
        ProxyPassReverse / http://localhost:8087/
</VirtualHost>

Yeah, that's about it. By default, every request will be passed to Clojure. The ports specified here should obviously match the ports you specify in your Clojure code for your servlets.

ProxyPass /foo ! tells Apache to handle requests to http://ffclassic.net/foo itself rather than pass them to Clojure. I override URIs on a case-by-case basis like this, to tell Apache to run whatever CGI or PHP scripts I need.

One oddity of running a setup like this is that thanks to mod_proxy, the "remote address" for every request is always going to be 127.0.0.1 by the time they get to Clojure. So if you want to see the IP address of your users, you have to do something like:

(defn- ip [request]
  (or ((:headers request) "x-forwarded-for")
      (:remote-addr request)))

Another oddity is that sometimes if Apache is running, and then I start Clojure, I have to restart Apache before it "finds" Clojure and starts forwarding traffic. No big deal though.

This is otherwise transparent from the Clojure end.

Starting it up

You probably want to SSH to your server, start a REPL, start your code running, and then log off the server. There are various ways you can run keep Emacs running in the background. You could run Emacs in server mode and attach to it with emacsclient. I use GNU Screen because it's easy and I have other things running in Screen instances that I like to switch between. So I start Screen, start Emacs, start SLIME, and start my site from there. Then detach from Screen when I'm done. It's easier than it sounds.

Why SLIME and not a normal commandline REPL? Because in Emacs I can patch the code easily on a function-by-function basis, among other things. You can do this from a commandline REPL but not nearly as easily. Emacs is a much nicer place to type than any command line. When I need to make an update to the site or fix a bug, my workflow is typically:

Fix it locally on my test machine, make sure it works
rsync the changed files to the server
SSH to the server
screen -r to bring up Emacs
Open the .clj file(s) I changed
C-c C-c to recompile and reload the individual function(s) I changed, or C-c C-k to recompile and load a whole .clj file, or whatever.
C-a d to detach from Screen

I can do any database inspecting or fiddling I need to do from the REPL. I can also do some live testing of the site or whatever else I want. It's good for debugging.

I've had Emacs running for months this way without having to restart it, it seems to work OK.

That's it

It's not that hard. It's clearly more steps than dumping some PHP files into a directory and having them work. But with Clojure, you get the benefit of a persistently running environment with a nice REPL interface, all in a nice fast multi-threaded JVM. Plus you get to write all your code in Clojure instead of PHP. In terms of fun and long-term maintenance costs, Clojure comes out way ahead in the end, for me anyways.

Clojure, SLIME, ODBC, SQL Server

Fri, 26 Jun 2009 11:27:22 -0700

I had a lot of trouble connecting to an MS SQL Server at work via Clojure. Java 6 comes with a JDBC-ODBC bridge which worked fine from a Clojure REPL at a command prompt, or from inferior-lisp in Emacs, but in SLIME it would hang every time I tried to connect and I'd have to kill Java. Couldn't for the life of me figure out why.

I got it to work eventually by using Microsoft's own JDBC driver, which you can download here.

Once you put the downloaded .jar file on your CLASSPATH (in my case, sqljdbc4.jar) you can connect like this:

user> (def db {:classname "com.microsoft.sqlserver.jdbc.SQLServerDriver"
               :subprotocol "sqlserver"
               :subname "//server_hostname;database=SomeDatabase;user=SomeUser;password=SomePassword"})
#'user/db
user> (use 'clojure.contrib.sql)
nil
user> (with-connection db 
        (with-query-results rs ["SELECT * FROM whatever"] (prn rs)))
... results ...

Posted for the sake of Googlebot and for my own future sanity.

Practical Common Lisp (review)

Tue, 18 Sep 2007 14:24:37 -0700

My current "programming language of the week" is Common Lisp. I got a dead-tree hard copy of Practical Common Lisp as a going-away present from my last job. This week on a trip to Eugene, Oregon (as part of the process of getting my NEXT job, hopefully) I had lots of time to read through it. I "finished" it today, for some definition of "finish" which includes a "skipped over large chunks of chapters that were way over my head" clause.

I was amazed to learn from Peter Seibel's blog that as of May 2007, there were less than 10,000 copies sold. I think I am really clouded in my view of what kinds of things people know in the world at large. I know there must be some overwhelmingly large group of Windows-only programmers who only know C# and/or Java and nothing else, but perhaps because I live in the world of Linux and open source, I never come in contact with them. When I look around me (figuratively, online) I see Linux guys who know Linux and Perl and Ruby and *nix-style C++ and whatnot. So I often find it very difficult to extrapolate anything about the world at large based on what I see around me.

Either way, PCL is a good book. It has an immense amount of information in it. Almost overwhelmingly large amounts of information, somehow. It may be overwhelming simply because of how unfamiliar many of the concepts and terminologies are to one such as myself who doesn't come from a Lisp background. But there are also many things that I'm 100% sure I wouldn't understand at all if I didn't already know Ruby pretty well (largely because Ruby ganked so much stuff from Lisp). If I didn't have a Ruby background, I'd be even more lost than I was.

Whether Lisp is really practical or not, I'm not sure. For example, it takes a lot of housekeeping even to get a Lisp environment set up. Thankfully available on Peter's website is a pre-packaged "Lisp in a Box" that can at least get you to a REPL prompt with no effort. But then you have to start writing code. Lisp code can be compiled, or interpreted, or a combination of both(?), or both simultaneously(?). Or it can be typed manually into the REPL, and then run, or compiled, or dumped to a file. Or you can dump an image of the Lisp instance (image?) to file entirely. Often there's a mess of manual dependency-managing any time you use packages. And then there are special things you need to do to make sure certain kinds of macros and things load correctly when compiled and run from compiled form, or when saved to file in "readable" form and then re-read as lisp code at a later time. And there are certain things in Lisp that can't be output in "readable" format at all. Do I sound confused? I surely am. I found this writeup showing how one person gets started on a Lisp project, which is somewhat helpful.

Yeah, you can get SLIME for Emacs and then start banging away in the REPL, but losing everything when you close it isn't a lot of fun. So I'm getting by with typing stuff into an Emacs buffer and manually running top-level forms one at a time via a bunch of Emacs shortcuts. It still seems like I'm doing a lot of the work that my environment should be doing for me. But I suppose it's no worse than trying to set up a bunch of Makefiles to manage C++ code. And much of my trouble seems to be a product of my inexperience (but that can probably be said of anything).

Lisp does have a lot of things that are really interesting, like multi-methods and of course macros, which the book covers in a wonderful amount of detail. Even the exception/condition system is different (and arguably much more powerful) than other languages'. The Lisp language itself is simply more powerful than most other languages in many ways, is the only conclusion I can come to. It's more extensible and it gives you more options for abstracting things in ways that are impossible to do (easily) in other languages due to the syntax and available constructs the languages provide. At the same time, all this power comes at the expense of code that's nearly unreadable by my somewhat biased standards; nasty Perl-esque punctuation abounds, there are many inconsistencies in how things are called or passed parameters (often for "historical reasons", the bane of my existence), and given the ability of macros to drastically alter the way code looks and works and to define mini-languages inside Lisp itself, you're very often not quite sure what exactly you're looking at at any given moment.

But I highly recommend the book. For non-Lispers like myself, if nothing else it gives you a taste of language that's very different from probably anything you've seen before, and it can point you in many interesting directions you may never have thought possible. On the other hand, is Lisp a revolutionary Zen-like enlightenment-producing revelation of a language which will expand your programmer mind into vast new dimensions of coding efficiency and bliss? I have yet to take sip of the Kool-Aid required to answer that question, but who knows.