briancarper.net (λ) (Tag: Web Design)

Split page vertically in CSS (minus pixels)

Mon, 23 Apr 2012 21:43:06 -0700

I was designing an online database application recently. The layout I wanted was, I thought, fairly simple:

N pixel header at the top
The rest of the page split vertically into two panes
Each pane should scroll independently

Super easy to do in CSS, right? Of course not! You can't do this:

#header {  height: 50px; }

#panels {  height: 100% - 50px; }

#top, #bottom { overflow: auto; }

This is because (of course) you can't do simple arithmetic in CSS.

I can't think of a reason why it's not supported. My browser knows the height of the window at any given point in time. The browser can surely subtract two numbers. If someone knows of a solid reason why we can't do this in CSS, please clue me in.

I can think of many reasons why I would want to do it though. The above use case is just one of them.

I really dislike resorting to this (which does work, as seen here):

#header {  height: 50px; }

#panels {
    position: absolute;
    top: 50px;
    left: 0px;
    right: 0px;
    bottom: 0px;
}

#top, #bottom { overflow: auto; }

Whenever I start using absolute positioning, I know something went off the rails somewhere.

The worst part isn't that CSS doesn't support this, it's that even if CSS did suddenly support it, I couldn't use it until sometime in 2023 when all the major browsers implemented it and everyone using the old browsers switched or died of old age.

gaka 0.2.0

Thu, 29 Jul 2010 13:59:22 -0700

Per many commenters' suggestions and thanks to code from Steve Purcell, you can now use maps for CSS attributes in gaka.

user> (println (gaka/css [:a {:color :red}]))
a {
  color: red;}

This looks more like vanilla CSS thanks to the curlies, which is nice. You just have to keep in mind that your key/value pairs could end up being printed in random order, and order is significant¹ in CSS.

It just so happens that maps are implemented in Clojure right now such that if they only have a few entries (16 key/value pairs), the order will be preserved, because you get a PersistentArrayMap instead of a PersistentHashMap. But it's highly dangerous to rely on such a thing. It could change at any time in the future.

In any case, you can also mix and match maps, lists and "flat" keyvals. They'll all be flattened That can help preserve attribute order in those cases where you need to.

user> (println (gaka/css [:a :color "red" {:padding 0} (list :margin 0)]))
a {
  color: red;
  padding: 0;
  margin: 0;}

I've also enhanced "mixins" a bit further. You can now mixin entire tags as well as attributes. Or a combination of both. Say you want a mixin that means "Make my element have no padding, and make links within the element be red":

user> (println (gaka/css [:div.foo mixin :margin 0]
                         [:div.bar mixin]))
div.foo {
  padding: 0;
  margin: 0;}

  div.foo a {
    color: red;}

div.bar {
  padding: 0;}

  div.bar a {
    color: red;}

You can get gaka from github or Clojars.

Order is only significant in cases where you're doing things like padding: 0; padding-left: 1px. This is arguably bad CSS style, but it's valid, and it's also possible you'll have this kind of thing if you're generating CSS procedurally. But most of the time, order is not significant. e.g. it doesn't matter if you set text color first and background color second, or vice versa. So maybe this isn't so much of a problem in practice. ↩

Footnotes

Tue, 27 Jul 2010 12:30:57 -0700

Did you ever notice how footnotes make your writing seem more important¹ somehow?

Maybe one reason is that "real" books use footnotes. At a glance, it looks like I have references² backing up everything I say. In reality, I don't, but the connotation carries through somehow³. Now my blog seems scholarly and authoritative.

And if you're like me, you can't resist clicking footnotes to see what they refer to⁴. According to my estimates, by utilizing footnotes, in one fell swoop I have decreased my readers' average reading efficiency by 73%.

In any case, I've added experimental, rudimentary support for footnotes to cow-blog.

I'm loosely copying the syntax from Markdown Extra for this. Markdown is great, except when it isn't. The standard doesn't have support for some useful extensions. I use Showdown for Markdown support, and I'm probably going to work on adding more features of Markdown Extra to Showdown in the near future.

I just dread actually doing it. Showdown (like Markdown itself) is implemented as a series of hackish regex transformations of blobs of text. It's not a proper grammar. Implementing more of Markdown Extra means more regex blobbing. It's brittle and fragile and even getting incomplete support for footnotes was less than enjoyable. But at the same time I find myself wanting to do things that Markdown can't so, so I may have to bite the bullet.

(If there's a Showdown Extra out there already, drop me a URL. It'd be most appreciated. But I couldn't find one.)

In reality nothing I say is important. ↩
Does my inner dialog count as a reference? ↩
Via telepathy. ↩
See? ↩

Clojure syntax highlighting via SyntaxHighlighter

Wed, 21 Jul 2010 15:17:42 -0700

How do you syntax-highlight Clojure code for display on a website? The best way I can find is SyntaxHighlighter.

Daniel Gómez wrote a brush to give SyntaxHighlighter Clojure support. I tweaked it a bit myself and integrated it into cow-blog. I also converted my favorite color scheme to a SyntaxHighlighter theme. So when I write this code:

(defn- ip
  "Given a request, return the IP.  Looks for an x-forwarded-for
  header, falls back to :remote-addr on the request."
  [request]
  (or (get-in request [:headers "x-forwarded-for"])
      (request :remote-addr)))

You should see something like this:

...unless you're reading this via RSS, or in a browser without Javascript enabled, in which case you'll see plain, depressing black and white. But that's one nice thing about SyntaxHighlighter. It degrades nicely.

One bad thing about SyntaxHighlighter is that it doesn't play nicely with Markdown. Or rather, Markdown isn't powerful enough to let you specify the class of any markdown-generated HTML tags. If you want the <pre class="brush: clojure"> that SyntaxHighlighter requires, you have to write out the HTML by hand. But I hacked Showdown a bit to let me specify classes more easily, so I can avoid having to do that.

The code for all of this is on github with the rest of my blog.

In which border-radius is abused

Mon, 05 Jul 2010 15:54:24 -0700

I threw together a new blog layout today. I scaled back the level of cows a bit (just a bit, don't worry!) Criticism / feedback welcome. (IE-related feedback should be dropped off in the circular file by my desk.)

In what is surely a prelude to the future of the internets, I'm abusing border-radius pretty heavily in this layout. border-radius is likely to become the new marquee HTML tag or text-shadow CSS attribute: Maybe an OK idea at first, but then everyone uses it so much it makes your eyes bleed.

So I figured I'd best get my border-radiusing in early while it's still cool. IE8 users, you still get pointy corners. Sucks to be you.

Also, if you have any ideas for features I should implement for cow-blog, please let me know. I've been crawling the internet looking at blogs for ideas of things to implement and features to steal, but I'm running out of ideas. It does everything I want now, but I'm not a reader.

Goodbye Tokyo Cabinet, hello PostgreSQL?

Tue, 29 Jun 2010 11:32:58 -0700

The first version of this blog used MySQL; then I switched to Tokyo Cabinet. But now I've switched back to PostgreSQL. Here's why.

Why did I switch to TC to begin with?

There weren't any good ORM-type libraries for Clojure at the time (over a year ago). So there was a bit of an impedance mismatch trying to query and work with my data. In the DB I have separate tables for posts, comments, tags, categories. But 90% of the time I want to fetch a post and end up in Clojure with all related tags, comments etc. (A JOIN won't work here without a lot of work to un-mangle the results; you usually need multiple queries.)

With TC I could store anything in the DB, so I just dumped a post into the DB as serialized hash-maps with all the comments, tags, and categories as sub-keys. So querying was easy. (Or was it? More below.)
MySQL was really slow. This is largely because my queries were terrible, as I tried to solve the problem from #1 via brute force. TC on the other hand is fast.
I tried to solve #2 by using a Clojure ref as a cache. But tying the STM to a database's transaction system is as far as I know difficult or impossible right now (per many threads on the mailing list). I had a lot of potential race conditions, which (as far as I know) never bit me, but probably would've eventually. I had to deal with keeping the cache up-to-date as comments were posted and posts were added and deleted and renamed. Remember:

"There are only two hard problems in Computer Science: cache invalidation and naming things." --Phil Karlton

So why did I stop using TC?

I have no idea how to use a key/value store database properly. TC will take anything you dump into it, which is both a strength and a weakness.

There's a lot of crap you have to do by hand that a proper database does for you. Consider checking for null values, for example; I ended up with a lot of nils in my data because my validations weren't 100% foolproof, or because I imported data via code that didn't run the validations and I never noticed.. Or enforcing uniqueness of values; I had tag objects in the database with the same key but different values (due to capitalization differences), which screwed up a lot of stuff.

On the other hand, there's a lot of information about how to use RDBMS properly, and I have a lot of experience with it already. Constraints are easy to set up. Columns have types, which is nice. (Strange that I gravitate toward statically-type databases while I gravitate toward dynamically-typed programming languages.)
I have to compile and install Tokyo Cabinet by hand on my Linux distro. It's probably not worth distro maintainers to maintain a package that so few people use. MySQL and PostgreSQL have lots of people working on keeping them running OK on most Linux distros.
Some kinds of queries were still awkward in TC. "Give me post X" was great: I'd also get all the tags, categories etc. for free. But then how do you query to get all tags across all posts? Or all comments? Fetch all the posts and iterate over them, collecting their tags, then uniquify the resulting list? Not so pretty, and not so fast. So I was back to caching again, which still gave me nightmares about race conditions and dirty data.

So now why am I using an RDBMS again?

An RDBMS is exactly what I really need, if I could just query the thing concisely and get it to run fast. Thankfully there are some ORM-like libraries for Clojure in the works nowadays, already usable for a hobby project like this blog. There are clj-record, Carte, ClojureQL, and my own Oyako, and possibly others in the works.
For my tiny blog's database, Oyako gives me slightly slower performance than TC, but along the same order of magnitude, which is good enough.
Via Oyako I can (fairly concisely) fetch posts and get the associated tags, comments etc. But I can also easily fetch all tags, or all comments, since they're in their own tables. The "relational" part of RDBMS does come in handy sometimes.

Summary version

I switched to TC to begin with because I was using SQL wrong, and it was too slow and clumsy. Once I figured out how to use SQL correctly, it was a no-brainer to go back.

Introducing Gaka

Mon, 28 Jun 2010 17:59:26 -0700

The CSS for my blog is now being generated via gaka, a CSS-generating library I wrote this afternoon. It's extremely simple, but it got the job done for me. I turned around 600 lines of CSS into around 250 lines of Clojure without much effort. It looks like this:

user> (require '(gaka [core :as gaka]))
nil
user> (def rules [:div#foo
                  :margin "0px"
                  [:span.bar
                   :color "black"
                   :font-weight "bold"
                   [:a:hover
                    :text-decoration "none"]]])
#'user/rules
user> (println (gaka/css rules))
div#foo {
  margin: 0px;}

  div#foo span.bar {
    color: black;
    font-weight: bold;}

    div#foo span.bar a:hover {
      text-decoration: none;}

Gaka is partly inspired by Sass, which I found very pleasant to work with recently. And it's partly inspired by Hiccup, which is a delicious way to generate HTML in Clojure.

There's more info and more examples on github.

New blog engine up and running

Wed, 23 Jun 2010 15:48:24 -0700

Well, my new blog is up and running. Sorry for the temporary lack of cows in my layout. I'm dogfood-testing the blog engine in a fairly vanilla state until I work out some of the bugs. This layout is based upon barecity, a minimalist Wordpress theme that I adapted easily enough to my blog.

As a bonus, I applied a dirty hack to my RSS feed that I think should help avoid screwing up people's RSS readers with duplicate entries.

I'll write again soon with some info about the blog engine and some things I learned writing it.

(As mentioned previously, here's the code.)

Breaking links is easy to do

Tue, 22 Jun 2010 23:23:27 -0700

I apologize in advance to everyone who subscribes to my blog's RSS feed, but this week your RSS reader is probably going to suddenly find 25 "new" posts from me.

My blog currently uses /blog/title as the URL scheme, with similar URLs for categories and tags etc. Soon, I'm probably going to change it to /blog/123/title, as part of the impending release of version 0.2 of my blog engine. (The code-in-progress is in a branch on github, for the daring and foolish among you.)

This way, I can change the title of a post without breaking everything. I have heretofore lacked this ability. It's easy to code, you just tell Compojure to ignore everything after the number in a route. Something like this:

(defroutes foo
  (GET ["/blog/:id:etc" :id #"\d+" :etc #"(/[^/]*)?"] [id]
    (pages/post-page id)))

It's only a few lines of code to change, but the ramifications are widespread. It'll instantly break every link to my blog, for example. At least it's pretty easy to set up a bunch of redirects in Compojure to avoid that. I think this'll work:

(require (blog [db :as db]
               [link :as link])
         (oyako [core :as oyako])
         (ring.util [response :as response]))

(defn redirect-post [name]
  (when-let [post (oyako/fetch-one db/posts :url name)]
    (response/redirect (link/url post))))

(defroutes redirect-routes
  (GET ["/blog/:name" :name #"[^/]+$"] [name]
    (redirect-post name)))

(Oyako here is the experimental ORM-like library I'm using to interface with PostgreSQL nowadays, having ditched Tokyo Cabinet.)

Changing my URL scheme is also going to mess up RSS though, because I (foolishly) used post URLs as the GUIDs in my RSS feed up to this point. This problem I don't know how to avoid. I might reduce the number of posts included in my feed temporarily, to limit the damage.

Getting list of referers out of Apache logs

Sun, 21 Feb 2010 12:46:34 -0800

I use Google Analytics, but it has a noticeable lag in updating its information. When my site is being hammered, I'd like to see where all the traffic is coming from. It'd also be nice to see how many hits my RSS feed is getting, and how many images and static files are being direct-linked, which Google Analytics currently isn't tracking for me at all.

So this script will look in my Apache logs and print referers for some URL, thanks to ApacheLogRegex:

#!/usr/bin/ruby

require 'apachelogregex'

raise "USAGE: #{$0} log_filename desired_url" unless ARGV[0] and ARGV[1]

format = '%v:%p %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"'
parser = ApacheLogRegex.new(format)
pat = Regexp.new(ARGV[1])
refs = {}

File.readlines(ARGV[0]).each do |line|
  x = parser.parse(line)
  if pat.match(x["%r"])
    r = x["%{Referer}i"]
    refs[r] = (refs[r] || 0) + 1
  end
end
refs.sort_by{|k,v| -v}.each do |ref,count|
  puts "%s: %s" % [count,ref]
end

I used to use awstats for this, but it was too heavyweight and a hassle to set up and keep running. Google Analytics is a no-brainer to use, even though the accuracy isn't as good as parsing Apache logs. At least I get an idea of which of my blatherings people are most interested in.

Out of memory... ouch

Wed, 20 Jan 2010 12:12:05 -0800

I've written before about how I'm running four Clojure-driven websites out of a single JVM on my VPS. No problems for many months, but today I tried to make a blog post and got all kinds of out-of-memory errors. Hopefully I didn't lose any / many user comments on this blog in the past couple days, but it's possible.

I restarted the JVM and gave it a bit more RAM to play with, I imagine this will fix things. But we'll see. It occurs to me now that there may be such a thing as too much caching.

Deploying Clojure websites

Mon, 04 Jan 2010 20:42:17 -0800

On my server I'm running one Java process, which handles four of my websites on four different domains. These are all running on Clojure + Compojure. Some people asked for details of how to do this, so here's a rough outline. For the sake of brevity I'm only going to talk about two domains here, though it scales up to however many you want pretty easily.

This is surely not the only way to do this, and probably not the best way, but it's what I've arrived at after a year of goofing off.

Summary: Emacs + SLIME + Clojure running in GNU Screen; all requests are handled by Apache and mod_proxy sends them to the appropriate Jetty instance / servlet.

Directory structure

First you have to decide on a directory structure. There's a sort of Clojure convention to have a src and deps directory, though Clojure is so young that this convention may or may not stick. Behold my ASCII line-art directory tree:

~/public_html/base/
|
|----src/
|    |
|    |----common/
|    |
|    |----net/
|         |
|         |----briancarper/
|         |    |
|         |    |----<lots of .clj files>
|         |
|         |----ffclassic/
|              |
|              |----<lots of .clj files>
|
|----deps/
|    |
|    |----<lots of .jar files>
|
|----briancarper.net/
|    |
|    |----public/
|    |
|    |----db/
|    |
|    |----apache/
|
|----ffclassic.net/
|    |
|    |----public/
|    |
|    |----apache/
|

src holds the Clojure source code for all of my sites. Java (therefore Clojure) forces your directory structure to match your namespaces, and traditionally people use domain names in reverse order for both. Whether you like or dislike this convention, you must admit that if you're writing code that's actually going to be used to host websites on domains, the convention probably makes sense. There are some utility libraries I wrote that I like to share between sites, so I put those in some common namespace in their own directory.

deps holds a bunch of .jar files. This includes clojure.jar, clojure-contrib.jar, compojure.jar, swank-clojure.jar (for SLIME), and all the Java libraries I use, like Jetty, a bunch of Apache Commons libraries, Tokyo Cabinet, Rhino and so on. This means that all of my sites are sharing libraries, and therefore must be running the same version of Clojure and all other libraries. So when I upgrade a library that one site is using, I must upgrade all the other sites. This is sometimes a hassle, but it forces me not to be lazy.

Then I have one more directory for each of my domains, for static files and such. These directories have different subdirectories depending on how I'm running them. For the sites that use Tokyo Cabinet, I have a db directory to hold the database files. On some sites I want to run a PHP or CGI script, so I have a directory for Apache to use. All of them have a public folder, from which Clojure serves images, JS and CSS files etc.

Phil Hagelberg's recent Leiningen project can possibly help with some of this. I don't use Lein but I may switch someday.

Emacs + SLIME + CLASSPATH setup

Building or installing Clojure and Emacs and SLIME and friends is beyond the scope of this blog post. swank-clojure has come a long way in being auto-installable via ELPA nowadays, if you like.

The most important thing is to set up CLASSPATH correctly. src should be on the CLASSPATH, as well as every .jar file in deps. My .emacs contains this, among other things:

(setq swank-clojure-classpath (list "." "./src" "./deps/*"))
(setq swank-clojure-library-paths (list "/usr/local/lib"))

Java will take a glob and expand it to include every jar file in a directory, thankfully. But this seems to happen right when you start Java, so to add a new .jar to your CLASSPATH you generally need to restart the JVM. This is annoying, but oh well.

If you AOT-compile your Clojure code you might also have a classes folder on your classpath, but I don't bother. You could also compile your whole site into its own .jar file and use that. I don't do this because I want to be able to edit my Clojure source files on the server if I need to.

swank-clojure-library-paths is necessary for me because Tokyo Cabinet uses some C libraries. You might not need it.

Then when I start Emacs, I M-x cd to the base folder and start SLIME there. It'll find Clojure in deps and give me a REPL. If you can get this working, you're well on your way.

Clojure code

I have a file server.clj for each domain, which are in charge of setting up and starting / stopping the servlets. So in src/net/briancarper/server.clj I have something like:

(ns net.briancarper.server
  (:use (compojure.http request servlet session routes)
        (compojure.server jetty)
        (compojure control)
        (net.briancarper ...)))  ;; lots of files containing the guts of the site

(defroutes some-routes ...)
(defroutes other-routes ...)

(defroutes all-routes
  some-routes
  other-routes)

(defserver blog-server
  {:port 8080 :host "localhost"}
  "/*" (servlet all-routes))

src/net/ffclassic/server.clj is similar, except with different routes (obviously) and a different port for the servlet. Note that we bind to localhost here, so that people can't connect directly to port 8080 and bypass Apache.

One of the routes should be set up to point to the proper public folder. Generally some catch-all route near the end of your list of routes, like:

(GET "/*" (static-file (params :*)))

where this static-file knows to serve files from briancarper.net/public, and the other server.clj for the other domains point to different directories.

Compojure has the compojure.http.helpers/serve-file function you can use to serve static files too. I use my own function for various reasons. Note that Compojure generally doesn't set any HTTP headers on your responses. You should probably set some kind of cache control headers unless you want your users to re-download all of your image files and stylesheets every time they hit a new page. You might also have to fiddle with Content-Type sometimes, though Compojure is usually pretty good about guessing them.

Note that Compojure's hands-off approach has its upsides too. For example, I have a bunch of CSS files for my blog. In Clojure, I read and concatenate all of those CSS files into one blob of text and have Clojure serve that as http://briancarper.net/combined.css. combined.css doesn't actually exist. I do the same for JS. This speeds up user requests quite a bit, without having to merge all the files on the filesystem. I could further compactify them if I cared.

If your Clojure app is handling arbitrary request URIs, you should be careful you aren't serving up ../../../../etc/passwd or something. Pretty sure Apache protects you against this by default, but Compojure uses compojure.http.helpers/safe-path? to test for it too if you use the built-in serve-file.

Finally, for convenience, in the base directory I have a file called server.clj which contains this:

(ns server
  (:require [net.briancarper server]
            [net.ffclassic server]))

(defn go []
  (.start net.briancarper.server/blog-server)
  (.start net.ffclassic.server/ffclassic-server))

Then, to start everything, from a REPL:

user> (require 'server)
2010-01-04 18:10:50.046::INFO:  Logging to STDERR via org.mortbay.log.StdErrLog
net.briancarper.server.proxy$javax.servlet.http.HttpServlet$0
net.ffclassic.server.proxy$javax.servlet.http.HttpServlet$0
nil
user> (server/go)
2010-01-04 18:11:02.260::INFO:  jetty-6.1.15
2010-01-04 18:11:02.358::INFO:  Started SocketConnector@localhost:8080
2010-01-04 18:11:02.707::INFO:  jetty-6.1.15
2010-01-04 18:11:02.789::INFO:  Started SocketConnector@localhost:8087
nil
user>

On the server (Apache)

To deploy this to my server I just rsync it all over SSH. On the server, I have Emacs and Java installed, as well as Apache and GNU Screen.

Why Apache? In case I want to run something that isn't Clojure, like webmail or some CGI script. Plus it's easy to run Clojure as a non-priviledged user on a non-priviledged port, and have Apache forward requests to the proper servlet based on the domain in the request. Apache is also nice for doing HTTPS.

My Apache setup is a pretty standard Debian-ish setup. You need mod_proxy installed. My Apache config for one domain looks a bit like this:

<VirtualHost *:80>
        ServerName ffclassic.net
        ServerAlias www.ffclassic.net

        <Directory /home/user/clj/ffclassic.net/apache/>
           AllowOverride All
           Order Allow,Deny
           Allow from all
        </Directory>

        DocumentRoot /home/user/clj/ffclassic.net/apache/

        ProxyPass /foo !
        ProxyPass / http://localhost:8087/
        ProxyPassReverse / http://localhost:8087/
</VirtualHost>

Yeah, that's about it. By default, every request will be passed to Clojure. The ports specified here should obviously match the ports you specify in your Clojure code for your servlets.

ProxyPass /foo ! tells Apache to handle requests to http://ffclassic.net/foo itself rather than pass them to Clojure. I override URIs on a case-by-case basis like this, to tell Apache to run whatever CGI or PHP scripts I need.

One oddity of running a setup like this is that thanks to mod_proxy, the "remote address" for every request is always going to be 127.0.0.1 by the time they get to Clojure. So if you want to see the IP address of your users, you have to do something like:

(defn- ip [request]
  (or ((:headers request) "x-forwarded-for")
      (:remote-addr request)))

Another oddity is that sometimes if Apache is running, and then I start Clojure, I have to restart Apache before it "finds" Clojure and starts forwarding traffic. No big deal though.

This is otherwise transparent from the Clojure end.

Starting it up

You probably want to SSH to your server, start a REPL, start your code running, and then log off the server. There are various ways you can run keep Emacs running in the background. You could run Emacs in server mode and attach to it with emacsclient. I use GNU Screen because it's easy and I have other things running in Screen instances that I like to switch between. So I start Screen, start Emacs, start SLIME, and start my site from there. Then detach from Screen when I'm done. It's easier than it sounds.

Why SLIME and not a normal commandline REPL? Because in Emacs I can patch the code easily on a function-by-function basis, among other things. You can do this from a commandline REPL but not nearly as easily. Emacs is a much nicer place to type than any command line. When I need to make an update to the site or fix a bug, my workflow is typically:

Fix it locally on my test machine, make sure it works
rsync the changed files to the server
SSH to the server
screen -r to bring up Emacs
Open the .clj file(s) I changed
C-c C-c to recompile and reload the individual function(s) I changed, or C-c C-k to recompile and load a whole .clj file, or whatever.
C-a d to detach from Screen

I can do any database inspecting or fiddling I need to do from the REPL. I can also do some live testing of the site or whatever else I want. It's good for debugging.

I've had Emacs running for months this way without having to restart it, it seems to work OK.

That's it

It's not that hard. It's clearly more steps than dumping some PHP files into a directory and having them work. But with Clojure, you get the benefit of a persistently running environment with a nice REPL interface, all in a nice fast multi-threaded JVM. Plus you get to write all your code in Clojure instead of PHP. In terms of fun and long-term maintenance costs, Clojure comes out way ahead in the end, for me anyways.

Clojure and Compojure to the rescue, again

Sun, 03 Jan 2010 15:57:52 -0800

I haven't posted here much recently because I've been hacking on another recently-sort-of-completed website. One of my favorite hobbies is old 8-bit video games. The first thing I ever programmed was a website about Final Fantasy for the old NES, and I've fiddled with it for the past 10 years or so.

A while back I decided to rewrite the whole thing using Clojure + Compojure with data in mysql. This went really well. I know lines of code isn't that great a metric, but it can give a rough estimate: this whole website is done in 3,400 lines of Clojure, which includes all of the HTML "templates" and the DB layer I had to write. And it's ~~turtles~~ Clojure all the way down. The only thing not written in Clojure are a couple bits of Javascript here and there and the stylesheet.

I suspect the target audience of this blog and the target audience of that website don't overlap that much, but I figured someone might be interested in some of the detail of how it's implemented. A few things I learned...

I didn't write a single line of HTML by hand. I can't overemphasize how wonderful this is. Compojure's HTML library lets you write HTML as Clojure data structures, with vectors being tags, keywords being tag names, hash-maps being attribute-value pairs etc.

Not only is this easier to write, it's easy to generate and manipulate. I have a ton of functions where given a list of hash-maps of data, it spits out a nicely formatted chart of that data. Here's an example:
```
(chart "Armor" (armor game)
       ["" #(name-with-image %)
        "Locations" (comp ulist location-links :locations)
        "Buy" (comp g :buy)
        "Sell" (comp g :sell)
        "Absorb" :abs
        "Evade%" (comp percent :ev)
        "Element" (comp ulist #(map :name %) :elements)
        "Casts" #(-> % :magic :name)
        "Usable by" (comp (partial equipable-by game ps) :pcs)])
```
chart is a function taking a chart title, some data, and then a vector of column-header names and functions for the data in each column. Anonymous functions are a great way to format each column differently depending on the kind of data it contains. chart also does some other stuff like standardizing the text of the chart title, alternating the background color of rows of data, re-inserting the header every so often, and formatting the first column specially (expecting it to be a title) and so on.

Sample output of this function is here. A screenshot for the lazy (click for bigger version):
Who needs an ORM? I wrote about this previously, but for my needs, an ORM is overkill. The vast majority of my interaction with a DB is reading data. All you need for that is an easy way to write SQL queries and get the results into a hash-map. I wrote a bunch of macros for this, and there's ClojureQL and clojure.contrib.sql and friends to help. For example I run some polls that users can vote on, and here's the entirety of the code that fetches it out of the DB.
```
(defn polls []
  (with-tables [polls poll_options poll_votes]
    (map (fn [p] (assoc p
                   :total (count (:votes p))))
         (-> polls
             (one-to-many :options poll_options (key= :id :poll_id))
             (one-to-many :votes poll_votes (key= :id :poll_id))))))
```
with-tables is a simple macro I wrote that fetches data from the DB for some table (possibly putting a WHERE clause on the query if specified, unspecified here). It takes care to run the minimal number of queries necessary, generally one query per table of data. Then I use other functions like one-to-many to join the data together. The result is a list of hash-maps, with the value of some keys (here :options, :votes) being sub-lists of hash-maps, for data that "belongs" to the toplevel hash-map.

Doing it this way helps me avoid the infamous "N+1 queries" which plagues naive use of Rails (where each of N objects would query the DB again to get its sub-lists of objects). I think Rails uses an :include option in its find functions to do that same kind of thing.

This is dirt-simple, but it gives me a lot of control. I can easily map or filter over any of these intermediary table lists before or after putting the results together. I can add keys with values that are calculated from other values, so it looks like the records have fields that don't really exist in the DB. I can do a sort-by on some key. I can take a user parameter and return different results based on whether that user is an admin or not. I can put metadata on the hash-maps, e.g. to indicate which table the data was fetched from. And so on.

Thanks to memoize (see below) this is really fast too. The most complicated query I have is for my walkthrough. After priming the cache:
```
net.ffclassic.db.charts> (time (dorun (walkthrough-chapters :nes)))
"Elapsed time: 0.074587 msecs"
```
Speaking of metadata: I like metadata.

For example I have a multimethod called link, which dispatches on the :table metadata key set from my SQL macros above, and outputs an appropriate link for something based on what it is. So (link some-user) links to a user's profile on the site, using the user's username as the anchor text, and styles it differently if it's an admin user or normal user. (link some-forum-post) links to the forum thread which contains the post. (link some-poll) links to a poll using the poll's title as the anchor.

Each of these spits out a vector that looks like [:a {:href "/user/view-profile/1"} "Brian"]. And that's changed into HTML eventually via Compojure's delicious HTML library.

This makes it really easy to link to things in a standard way with standard-looking anchor text. It also makes it easy to standardize on a URL scheme. If I want to change the URLs used to access my polls, I only need to change it in one place.

This is kind of like OOP, but not really, since I can have different metadata tags and use different ones depending on what I'm doing. And I can completely ignore the metadata whenever I don't need to use it.
Speaking of multimethods: multimethods are handy.

I have five "categories" of pages of data on my site, with pages in each category that are usually identical across categories. But sometimes a page in one category should look slightly different from the same page in other categories. Sometimes a page only belongs in one category and should 404 for the other categories. What I needed was some way to say "Display this page identically for each category by default, but page Y on category X is an exception".

Multimethods are great for this. Compojure's "routes" take a request URI and dispatch based on a regex pattern matched against it. I dispatch most requests to a series of multimethods. Given a request URI, I split the URI into category + everything else. Then, I first try a multimethod for category-specific pages, dispatching on a vector of [category rest-of-uri]. If there is no method for this, it falls through (returning :next) and Compojure tries the next route. The next route tries a multimethod for pages that look the same category-wide, dispatching only on rest-of-uri this time. If there's no method for that, it falls through again with :next and a list of "static" routes are tried (oddball pages that don't belong to any category). If those all fail, you get a 404.

This is an easy way to split your site into namespaces based on the uri. You can also add new pages without editing your Compojure routes. Just add a new method to the multimethod.

The code looks something like this:

*code edited 2010-01-04 to fix a crapload of typos, thanks Shawn
```
(defmulti category-specific-page (fn [cat page] [(keyword cat) (keyword page)]))
(defmethod category-specific-page :default [& _] :next)
(defmethod category-specific-page [:cat1 :some-page] [cat page]
  (...))


(defmulti cross-category-page (fn [cat page] (keyword page)))
(defmethod cross-category-page :default [& _] :next)
(defmethod cross-category-page :some-page [cat page]
  (...))


(defroutes some-routes
  (GET #"/(cat1|cat2|cat3)/([^/+])/*$" (apply category-specific-page (:route-params request)))
  (GET #"/(cat1|cat2|cat3)/([^/+])/*$" (apply cross-category-page (:route-params request)))
  (GET "/" (default-index-page))
  (ANY "/*" (error-404))) 
```
This was a nightmare to do properly in Rails. Rails wants to dispatch to controllers, which are classes. To make this work in Rails I had to mangle this concept to fit into the idea of a hierarchy of classes and subclasses. There might have been an elegant way to do it, but I couldn't think of one. I suspect someone will leave me a ~~flame~~ comment telling me how to do it.
memoize is great as a poor-man's web cache. I have a bunch of pages of data that never change. So the first request fetches it from the DB, and subsequent requests are cached in RAM and are therefore nearly instantaneous. The data is cached in the form of huge lists of hash-maps (many of which have sublists of more hash-maps, sometimes many levels deep). I was concerned at first that this might take a lot of RAM, but I went with it, and it turned out not to. top tells me:
```
PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
17   0  257m 127m  12m S    0 23.6  48:18.91 java 
```
I'm actually running four websites in the same instance of the JVM there, so it's not that much RAM usage at all.

memoize doesn't work for everything, obviously. There's no easy way to invalidate or refresh the cache with the built-in memoize, so it's not good for data that is updated frequently. If you do need to invalidate some data, you have to re-compile the function. But it works well for static data. The great thing is that it's very fine-grained. Change a single defn to defn-memo (in clojure.contrib.def) and you're done.
Compojure's "decorators" are really quite nice. For example I use them to implement a kind of templating / layout system. I have a with-layout function which, given a response, wraps the HTML for that response in a standard layout (header, sidebar, footer). I group a bunch of routes together and decorate all of those routes with with-layout, and then it's done. This "layout" code is completely separate from the code the generates the HTML for the pages, and lets me change it easily (add a new layout for some specific pages, or whatever). A bunch of other routes, like those that handle POST requests and such, don't get a layout; I just group those routes separately and don't decorate them.

All of this is done with plain old Clojure functions. There's no separate "template library". There's no ERB or Smarty to wrestle with, though you could use something like StringTemplate if you wanted to.

Another example: certain pages are admin-only. So I group all of those pages together into one group of routes, and do the admin check first-thing, immediately failing if someone isn't logged in or if the current user isn't an admin. This thankfully lets me not have to worry about doing permissions checks all over the place. The code looks like this:
```
(defn admin-only [handler]
  (fn [request]
    (when (:admin (:session request))
      (handler request))))


(defroutes admin-routes
  (GET "/some-admin-only-page" (admin-page))
  (GET "/another-admin-only-page" (other-admin-page)))


(defroutes admin-form-routes
  (POST "/some-admin-only-page" (mangle-database)))


(decorate admin-routes with-layout admin-only with-session)
(decorate admin-form-routes admin-only with-session)


(defroutes all-routes
  admin-routes
  admin-form-routes
  other-routes
  etc.)
```

I'm overall pretty pleased with the site. There are still some rough edges and much more work to be done, but it's a joy to code compared to the Rails version.

Comments work again

Thu, 24 Dec 2009 12:20:06 -0800

I broke the ability to leave comments a couple days ago. Thanks to everyone who let me know. It's fixed now.

I broke it while uploading yet another website I finished a couple days ago. It's yet another Compojure/Clojure site, this time a bit more ambitious than my humble blog. I plan to write about that whole experience once I have a bit of time.

Lame comment spam management that works

Sun, 06 Dec 2009 02:34:08 -0800

It's been nine months since I ditched Wordpress and moved to a blog system I wrote from scratch (in Clojure). This was a great move in so many ways. One of those ways is comment spam. My site is as popular now (or maybe slightly more popular now) as it was when I was running Wordpress, so I think comparing before and after is valid.

With Wordpress, every morning I'd do the ritual of deleting overnight spambot droppings. Typically I got between 1 and 5 every night. I had a default Wordpress install and all I used for spam filtering was Askimet. Askimet did a surprisingly good job, catching literally if not thousands of spams every week which otherwise would've been ruining my site. But inevitably some would still get through. And what's worse, there were a lot more false positives than I could tolerate.

Since I started counting with my new system, which is around 6 months, to the best of my knowledge I've gotten zero spambot-produced comments that made it through my filters. This is pleasant, to say the least.

The system I'm using is stupid. None of it is stuff I thought of myself, I got ideas from other lots of other blogs or articles I read, but the implementation is mine and it's not sophisticated. It would take a bot author a few seconds to work around it. But no one has bothered. Why bother writing a bot for my one-man blog, when you can write a bot for Wordpress and have it work on tens of thousands of blogs? And I can change my system to defeat the bots with a few lines of code just as easily as they can work around it.

So here's why I think it's working.

1. It's not Wordpress

Just by using something slightly different from Wordpress, I'm think I'm already ahead. For example if you have a blog where a form posts comments to /wp-comments-post.php, a bot doesn't even need to look at your site to spam you. They can blast your server with POST data at that URL in a format they already know Wordpress will accept. My site is all custom code, so everything is different enough that default bot attempts fail immediately.

I think this is the reason that only 1853 spam comments have even been POSTed at me in the last six months. That's an improvement of one or two orders of magnitude already.

2. Honeypot text field

So what about the comments that are actually POSTed? They are presumably the result of bots that parse sites' HTML looking for comment forms and try to POST data that satisfies the form.

So in my comment form I have a field called referer. A "How did you find my site?" kind of thing. In fact I don't care how you found this site, this field is a honeypot. The div containing this field is hidden via CSS.

div#referer-row {
    display: none;
}

So you shouldn't ever see it if you're a human using a browser. But bots parsing the HTML see this field, unless they also bother to parse my CSS and see that it's hidden, which would be expensive and apparently they don't do it.

If you put anything in this referer field, your comment will be rejected as spam. Simple enough.

Many blogs require you to fill in every field or else the comment is rejected, so it seems reasonable to expect most bots to fill in all of your form fields. (My blog actually requires you to fill in nothing but the comment text; author will be set to Anonymous Cow if you don't fill it in.)

In fact this seems to be the case; of 1853 spam comments since March, 1810 put something into this field. Most of the time it's a random string of letters. Not even a URL. Sometimes it's a couple words like "insurance quotes" or something about drugs or casinos.

The downside of this is if you're a human using a browser that doesn't understand CSS, you will see this field. Then if you type something into it and try to comment, it'll end up as spam. So Lynx users and time travelers from 1987 trying to leave me comments might be confused at first.

However as far as I can tell, no intelligible data has ever been entered into this field by a human, so I don't think it's a concern. Six times, the word "None" was entered, but I don't think this is a human because that's nonsense answer to "How did you find this site?". But you never know.

3. Lame static CAPTCHA

That leaves 43 spam comments that made it this far. My other anti-spam measure is a word you have to type. But it's always the same word, and the word is COWS. This CAPTCHA caught the remaining 43. It looks like this:

There's a normal text field with a default value of <= Type this word specified right in the HTML.

<input type="text" value="<= Type this word" name="test" id="test"/>

There are no other instructions besides "Type this word". I'm assuming either that commenters are familiar enough with CAPTCHAs to know what I want, or can figure it out using common sense. Given that my target audience is computer geeks and programmers, this should be a safe assumption. In fact I've had less than a dozen false positives in the past six months via people failing this; see below for details.

To post a comment, the value of this field must contain the word "COWS" somewhere in it, case-insensitively. Otherwise it's spam. Easy enough to implement.

If you have Javacsript enabled, clicking on this field will clear out the default value. If you unfocus the field without typing anything, Javascript will put the default value back in. This is only for convenience. If you don't have Javascript enabled, you have to highlight and backspace over the default text. I don't think this is a huge burden.

Of 1853 spam comments, here's the breakdown for what values end up in this field.

1131: "&lt;= Type this word"
691:  "<= Type this word"
21:   Random letters and numbers
6:    empty
2:    A bunch of URLs
2:    Human beings making typos, e.g. "COW" or "COS".

So most bots are too stupid even to remove the default value from the field, and none of them entered the correct value. 691 times the bot was somehow smart enough to un-escape < into <, which is interesting, but didn't help it defeat the filter. A lot of the random words look like they were made by Markov chains, e.g. 'fridwolfur' and 'lyndonvolk' and 'calbertdom'. If I need to write a childerns' poem I'll know where to look for ideas. One time a bot or spammer managed to type "vows" somehow, but this might or might not be coincidence.

I think this is better than a normal CAPTCHA because:

It's always the word "COWS" in a normal font, so it requires no thought or eye strain to figure out.
It's black and white, so hopefully people with minor vision problems and color blindness can see it.
It fits thematically with my blog layout (it's COWS and it has cow spots).
It's kind of silly, so hopefully people chuckle rather than become ticked off.

It's worse than a real CAPTCHA because it requires no effort to break. So it wouldn't work for Wordpress or VBulletin or something with a million users. But I wonder if every Wordpress had a single static word as a CAPTCHA, but a different word for every blog (generated at install-time maybe), would it work better or worse than the random mangled multi-color CAPTCHAs no one can read? Real randomly-generated CAPTCHAs don't work anyways; bots can already beat them via OCR or other means. A simple word would be less annoying for a human, to be sure.

The other downside is that this is not very accessible to the blind or other people using screen readers, or browsers without image support. This is unfortunate and I'm still trying to figure out how to get around this. Right now the ALT text for the CAPTCHA image is This says 'COWS'; I don't know if this is enough help for people in those situations.

Of course I'll never know how many people see my CAPTCHA and storm away in a rage without even trying to post a comment. But I've never heard a complaint. If this level of CAPTCHA ticks you off personally, please swallow your anger and leave me a comment here saying so, if you feel obliged; I'd love to hear it.

False positives

As best I can tell, there are no false positives from people filling in the honeypot field. But even as simple as it is, some people don't succeed at the CAPTCHA image. Either they typo it or they ignore it entirely.

I just checked and I counted around 6 comments by real humans where the CAPTCHA was ignored and the default <= Type this word ended up in the spam DB. 4 of those people re-posted their comment successfully immediately afterwards by filling in the CAPTCHA. I'm not sure I'm ever going to get much better than that.

Spam that makes it through

I have still gotten spam. Maybe a dozen or so in the past six months. It's all been in the form of a human typing a normal-looking and relevant comment, about open source software or BASH for example, but with a spammy URL buried in it, e.g. a link to a really dodgy-looking blog trying to sell something, or some scummy SEO site. It's either a human or a very sophisticated (or lucky) bot; the comment text in these is indistinguishable from a real comment other than the spam URLs. I have to delete these by hand.

But I was getting these with Wordpress too. No automted anti-spam system is going to defeat a human being, so I don't worry about it.

That's it

The moral of this story is that it doesn't take much to protect yourself from comment spam if you write the code yourself. As long as it's unique, you'll probably be fine.

The other moral is that you don't have to annoy the hell out of your users to filter spam effectively. I'm making the assumption here that my COWS method is not that annoying; tell me if I'm wrong.

I don't know how well this scales. Probably not so well. My blog isn't that highly trafficked. If my site were more popular it might be worse for me. But the improvement over Wordpress is unquestionable.

I've seen all kinds of complicated measures suggested elsewhere, like trying to predict if it's a bot by how many milliseconds it takes between page load and comment posting, or measuring keypress speed, or escaping the HTML of your forms and un-escaping it at loadtime it via Javascript, or setting and retrieving cookies and such. But a lot of this stuff seems fragile and if your browser doesn't suppoort Javascript or cookies (or your users block them), you're screwed. I block these things myself, so I expect visitors to do the same.

If everyone wrote their own blog engines, the world would be a slightly less spammy place. Or else we'd have much smarter bots.

Godaddy sucks

Sun, 22 Nov 2009 12:56:57 -0800

I'm in the process of moving all my domains the heck off of Godaddy. I'm trying Namecheap which seems slightly less evil, if the sheer amount of ad banners and upselling bullcrap is any indication. But probably only slightly less evil.

Honestly Godaddy has so many ads I can't even find the button to renew my domains. The process of buying anything takes you through 6 or 7 pages of the most garish, fanatical sleaze-peddling that you are likely to encounter on a website.

Domain registrars are the used car salesmen of the internet.

Blog source code updated

Thu, 22 Oct 2009 23:59:25 -0700

I updated the source code of my blog on github. I'm too tired to write much about it at the moment.

Suffice it to say I rewrote it all from scratch for the purpose of sharing, because lots of people were asking for it. It uses Tokyo Cabinet instead of mysql now, which is nice. I gutted the codebase so it's about 700 lines now (down from 1500, not bad). I plan to write up some posts later exploring various parts of it, for those who are interested.

Hope someone gets something out of it. Use at your own risk.

Moved to Linode

Sat, 15 Aug 2009 23:03:33 -0700

My web host for a good long while was Futurehosting. My OS was Debian 4.0 (Etch). Strike one: as of now there's still no option to upgrade to a newer version of Debian. Debian lags so much to begin with, it's really painful ify ou want to use anything released in the past two years.

I had an unmanaged VPS. I ran a bunch of funky non-standard stuff on there and it ran mostly OK. I had to upgrade to get more RAM just so SBCL would run on it, which sucked but I don't know that another host would've been any better.

The good thing about Futurehosting was that they responded very fast to tickets. The bad thing was the fact that I had ample opportunity to know this. The server would go down randomly once every month or two. I'd open a ticket saying "Hi my server is down", then things would be working again in a half hour, but why did this happen so often? I don't know. An awful lot of "failed switches". I wonder how often this happened without my knowing about it, given how often it happened in the middle of my using the server for something.

With all the hardware they were burning through I would've expected upgrades or price reductions over time, given that I was a steady customer for so long and that disk space and memory keeps becoming cheaper and cheaper in the world. But the prices always stayed the same, which was another strike.

Being hosted there was annoying but never annoying enough to switch. And migrating all of my sites and data to another server seemed like a huge pain. Momentum: the worst enemy of progress.

I moved to a new host on a whim recently: Linode. It was far less painful than I expected. Thanks to Linux and plaintext config files, it was mostly a SCP-it-all-over and tweak process. It took me one evening and a bit of time the next morning. Linode offers a lot of OSes which is also nice.

I pay less for Linode than I did at FH (and I get fewer resources at Linode, but I don't need much). Thus far I'm astonished how much faster things are running on the server. Even goofing off at a terminal, the shell is more responsive. My email loads instantly in kmail instead of lagging for a second. I never knew what I was missing. Linode's DNS control panel is also pretty braindead simple to use.

Futurehosting gets a C+ from me. It worked and my website existed, but it didn't knock my socks off. Hopefully Linode is better.

Adventures in blogging

Tue, 04 Aug 2009 00:24:23 -0700

So my newest site is about pareidolia which I think is an interesting subject. It's still in its early stages, using a data import from yet another of my old sites that I ran with a friend of mine. This makes three websites I'm running out of the same JVM on my server now. It'll be interesting to see how many more I can cram in there before my VPS explodes. CPU and RAM utilization are still supposedly minimal. I actually plan to redo my old faltering FF1 site entirely in Clojure too, which is probably the largest Clojure project I plan to undertake in the foreseeable future.

I'm still trying to come up with some kind of "how to make a blog in Clojure" tutorial once I nail down a good way to do it. The code for my blog has "grown organically" (i.e. become a mess) beyond the point where I'd inflict it upon the world without substantial cleanup. Probably easier to start over from scratch with something that's more demonstrative.

I'm also not too savvy on many of the "right ways" to do things in Compojure, as per this discussion. That's the good and bad thing about using a Lisp. You can pull off almost anything. There is no underlying structure to the code other than structure you impose. It's easy to write a mess.

Who needs a DB?

Wed, 20 May 2009 22:53:03 -0700

My blog is still working, in spite of my best efforts to crash it. So that's good. But lately I've been thinking that an SQL database is a lot of overkill just to run a little blog like this.

My blog only has around 450 posts total (over the course of many years), and about an equal number of user comments (thanks to all commenters!). Why do I need a full-blown database for that? All of my posts plus comments plus all meta-data is only 2 MB as a flat text file, 700k gzipped.

By far the most complicated part of my blog engine is the part that stuffs data into the database and gets it back out again in a sane manner (translating Clojure data to SQL values, and back again; splitting up my Clojure data structures into rows for different tables, and then re-combining values joined from multiple tables into one data structure). Eliminating that mess would be nice.

Inevitably I ended up with some logic in the database too: enforcing uniqueness of primary keys, marking some fields as NOT NULL, giving default values and so on. But a lot of other logic was in my Clojure code, e.g. higher-level semantic checking, and some things I wanted to set as default values were impossible to implement in SQL.

Wouldn't it be nice for all the logic to be in Clojure? And the data store on disk to be a simple dump of a Clojure data structure? I can (and did) write a few macros to give me SQL-like field declaration and data validation, for uniqueness of IDs and data types etc. For my limited needs it works OK.

The next question is what format to use for dumping to disk. Happily Clojure is Lisp, so dumping it as a huge s-exp via pr-str works fine, and reading it back in later via read-string is trivial.

Some Java data types can't be printed readably by default, for example java.util.Dates, which print like this:

#<Date Wed May 20 22:39:00 PDT 2009>

The #<> reader macro deliberately throws an error if you try to read that back in, because the reader isn't smart enough to craft Date objects from strings by default. But Clojure is extensible; you can specify a readable-print method for any data type like this:

(defmethod clojure.core/print-method java.util.Date [o w]
  (.write w (str "#=" `(java.util.Date. ~(.getTime o)))))

Now dates print as

#=(java.util.Date. 1242884415044)

and if you try to read that via read-string, it'll create a Date object like you'd expect.

user> (def x (read-string "#=(java.util.Date. 1242884415044)"))
#'user/x
user> (class x)
java.util.Date
user> (str x)
"Wed May 20 22:40:15 PDT 2009"

Storing data in a plain file has another benefit of letting me grep my data from a command line, or even edit the data in a text editor and re-load it into the blog (God help me if that's ever necessary).

Having multiple threads banging on a single file on disk is a horrible idea, but Clojure refs and agents and transactions handle that easily. But I do have to work out how not to lose all my data in case the server crashes in the middle of a file update. (I've lost data (in a recoverable way) due to a server crash in the middle of a MySQL update too, so this is a problem for everyone.) Perhaps I'll keep a running history of my data, each update being a new timestamped file, so old files can't possibly be corrupted. Or use the old write-to-tmp-file-and-rename-to-real-file routine. Or heck, I could keep my data in Git and use Git commands from Clojure. It'd be nice to have a history of edits.

If this idea works out I'll upload code for everything to github, as usual.

Blog and CRUD

Sun, 12 Apr 2009 20:01:35 -0700

I updated my blog source code on github. I also split my CRUD library out into its own clj-crud repo. It is cruddy, so the name is apt.

This code still isn't polished enough for someone to drop it on a server and fire it up, but maybe it'll give someone some ideas. I think the new code is cleaner and it'll be easier for me to add features now.

Beware bugs, I'm positive I introduced some.

EDIT: A word about the CRUD library... persisting data to disk is hard when the data may be mutated by many threads at once and the destination for your data is an SQL database that may or may not even be running. I have more respect for people who've written libraries that actually do this kind of thing and work right. Granted I only spent 3 days on mine but still, it's tricky.

I gave up for a while and tried clj-record, but it was prohibitively slow. It has the old N+1 queries problem when trying to select an object which has N sub-objects. In real life you'd write SQL joins to avoid such things. Ruby on Rails on the other hand gets around this via some nasty find syntax.

I get around it by having all my data in a Clojure ref in RAM already so it doesn't matter. And by using hooks so each object keeps a list of its sub-objects and the list is always up-to-date (updates of sub-objects propagate to their parents). But the crap I have to do to get this to just barely work is pretty painful.

Blog source code released

Fri, 27 Mar 2009 02:14:53 -0700

By popular demand, I've released the source code for my blog. Hope someone finds it useful.

http://github.com/briancarper/cow-blog/tree/master

Feedback and bug reports welcome, email me or post them somewhere on my blog and I'll find them.

Anti-spam field still holding

Mon, 23 Mar 2009 19:38:27 -0700

So far my silly anti-spam measures are working. Since last week I've had 1861 spam comment attempts, of which 0 were successful. 1857 of them didn't even alter the text my the captcha text field at all. Four of them inexplicably HTML-escaped the < into a <.

One feature I didn't implement from Wordpress is subscribing to comments via email. Sending an email from Java is possible but a little bit painful to implement. The Javamail API is a monster.

I do think it's useful to be able to know when someone responds to comment you left, but is spamming your inbox really the best way? I have to think there's a better way.

I did implement an RSS feed for each individual post's comments. And separate RSS feeds for all the tags on my blog, and all the categories. When RSS feeds are generated dynamically, why not? This is all of the code for the tag feeds:

(defn tag-rss [tagname]
  (if-let [tag (get-tag tagname)]
    (rss
        (str "briancarper.net Tag: " (:name tag))
        (str "http://briancarper.net/" (:url tag))
        "briancarper.net"
        (map rss-item (take 25 (all-posts-with-tag tag))))
    (error-404 )))

Plus the routing code:

(GET "/feed/tag/:name" (tag-rss (route :name)))

But I haven't uploaded the comment-feed feature because I don't know if it's overkill. Personally I am liberal with my RSS feeds, I just pop them into my Akregator and off I go. But I don't know if other people take their feeds more seriously, or what. RSS feeds can be a bit heavyweight. Maybe I should make a feed for all of my comments across all posts.

Blog is still going strong

Wed, 18 Mar 2009 22:01:11 -0700

After I implemented that silly CAPTCHA yesterday, the spam was stopped. There's also a honeypot form field (it's hidden via CSS so humans don't know it's there, and if any bot POSTs text for that field, the data is rejected automatically). It's silly and easily defeated, yet it stopped all 262 spam attempts since yesterday. It looks like all the spam is for one site, but it's coming from a huge range of IPs. So it's probably a botnet. Thanks, MS Windows!

I rewrote my whole CRUD layer so that I could use it for more than one database at once, and then rewrote my gallery code to take advantage, and now two hours later I have my origami gallery back up and running. Both sites are running from the same JVM. I wonder how many sites I can have going at once before the server melts into a puddle of Java-inflicted goo.

  PID PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
11338 16   0  512m 128m  12m S    0  0.3   0:28.33 java

Good thing I have plenty of RAM on the server. From looking at before and after shots of the memory usage, 66 MB is the JVM itself, and 40MB more is Jetty and Compojure and my code and all the dependencies. Then the last ~20 MB or so is my database slurped into RAM. So I can probably fit another few tens of thousands of posts and comments in here before I have to worry much. The real test will be letting this thing run for a couple weeks and see how hard it leaks.

Fun with HTTP headers

Tue, 17 Mar 2009 22:14:10 -0700

One fun thing about playing with Compojure is that it doesn't do much with HTTP headers for you, which is a good learning opportunity. RFC 2616 is rather helpful here.

For example I learned that if you don't set a Cache-Control or Expires header, your browser will happily re-fetch files over and over, which is a bit of performance hit. Static files that don't change often like images etc. can be set with a higher Expires value so they're cached.

Another thing to keep in mind (note to self) is that using mod_proxy to forward traffic to a local Jetty server means that the "remote IP" you get from (.getRemoteAddr request) will always be 127.0.0.1. If you want the user's real remote IP, you have to look in the X-Forwarded-For header (easily accessed as (:x-forwarded-for headers) in Compojure. Given that Identicons are generated from a hash of an IP address, this has resulted in some screwed up (wrongly identical) avatars for a bunch of people in posts for the past couple days. Oops. Not much I can do to fix that now.

In other non-news, I just the spam logging for the blog so I can see the kinds of things bots are doing to get around my feeble anti-spam measures. Sadly the spam seems to have stopped entirely, right after I set this up. How annoying.