263 Posts in Category 'Programming' RSS

Deploying Clojure websites

On my server I'm running one Java process, which handles four of my websites on four different domains. These are all running on Clojure + Compojure. Some people asked for details of how to do this, so here's a rough outline. For the sake of brevity I'm only going to talk about two domains here, though it scales up to however many you want pretty easily.

This is surely not the only way to do this, and probably not the best way, but it's what I've arrived at after a year of goofing off.

Summary: Emacs + SLIME + Clojure running in GNU Screen; all requests are handled by Apache and mod_proxy sends them to the appropriate Jetty instance / servlet.

January 04, 2010 @ 12:42 PM PST
Cateogory: Programming

Clojure and Compojure to the rescue, again

I haven't posted here much recently because I've been hacking on another recently-sort-of-completed website. One of my favorite hobbies is old 8-bit video games. The first thing I ever programmed was a website about Final Fantasy for the old NES, and I've fiddled with it for the past 10 years or so.

A while back I decided to rewrite the whole thing using Clojure + Compojure with data in mysql. This went really well. I know lines of code isn't that great a metric, but it can give a rough estimate: this whole website is done in 3,400 lines of Clojure, which includes all of the HTML "templates" and the DB layer I had to write. And it's turtles Clojure all the way down. The only thing not written in Clojure are a couple bits of Javascript here and there and the stylesheet.

I suspect the target audience of this blog and the target audience of that website don't overlap that much, but I figured someone might be interested in some of the detail of how it's implemented. A few things I learned...

January 03, 2010 @ 7:57 AM PST
Cateogory: Programming

Comments work again

I broke the ability to leave comments a couple days ago. Thanks to everyone who let me know. It's fixed now.

I broke it while uploading yet another website I finished a couple days ago. It's yet another Compojure/Clojure site, this time a bit more ambitious than my humble blog. I plan to write about that whole experience once I have a bit of time.

December 24, 2009 @ 4:20 AM PST
Cateogory: Programming

Let's parse

Is there anything more fun than parsing strings? I submit to you that there is not. I'm currently reading my way through Parsing Techniques - A Practical Guide, which has a first edition free online. (I'm hoping Santa brings me a copy of the 2nd edition this year.)

This is a good book, with enough math to be rigorous but not so much that it's completely unreadable. It starts from the absolute basics ("What's a grammar?") and goes through the Chomsky hierarchy and then dives into parsing techniques in great detail, in a language-agnostic way.

Languages and grammars are fascinating. In high school I studied Spanish, French, Latin and German, largely in my spare time. When I was 16, if people asked what I wanted to do for a living, I said "translator".

The plan to become a translator failed partly because the quality of my early education was horrendous and partly because mastering a language is extremely difficult and at 16 I wasn't motived enough. And then computers showed up in my life, which gave me a never-ending supply of languages to play with, while being fun (and profitable) in so many other ways. But I still took two years of Japanese classes in college for no reason other than enjoyment, and I'm still trying (and failing) to learn Japanese in my spare time 8 years later.

Perl was my first favorite language probably for no reason other than regular expressions. I can understand how people call PCRE syntax line-noise, but to me it's beautiful line noise. I live and breathe regular expressions nowadays. My favorite CS class in college was one where we went through and laboriously built finite-state automata and pushdown automata and Turing machines. Seeing the equivalence of these simple machines with the different classes of grammars was a huge epiphany. Such a simple concept with such huge consequences.

Dijkstra said:

Besides a mathematical inclination, an exceptionally good mastery of one's native tongue is the most vital asset of a competent programmer.

I strongly agree with that sentiment. People tell me at times that I'm good at written communication. I have my doubts, and anyways I find it funny because I'm so terrible at verbal communication. I think if I have any success at writing, it's because I view writing as a mechanical process.

I told a prof in college once that I felt like my papers wrote themselves once I had an idea in mind. There are rules of grammar and style, and you learn them and follow them, or break them deliberately if you have a good reason to. You write some prose, then you debug it until it "works" mentally. I don't care about typos and I split infinitives and comma-splice on purpose, but ambiguous or awkward phrases usually stand out to me like compiler bugs in my brain.

What's more important than language? Few things. Language is important enough to be nearly hard-wired into our brains. Children learn it instinctively. Human beings can still easily and effortlessly out-perform the best supercomputer at the task of parsing and interpreting speech. We think in words. The programming languages computers understand are dirt-simple by comparison, but writing code still feels like writing "thoughts for the computer" sometimes.

There are very few times you'll hear me say "What a wonderful world we live in". But one of those times is when I have the opportunity to explore an area of study like language. It's such an enjoyable experience to struggle and try to master such a thing. It's an amazing universe where we have these weird little rules and they work and we can understand them and manipulate them and produce things with them.

December 15, 2009 @ 12:46 PM PST
Cateogory: Programming

Clojure funding

Rich Hickey works on Clojure full-time for free, and he's asking people who get something out of Clojure to contribute some cash. $100 is less than I spend on random books and crap over the course of a year, so I gladly chipped in.

Clojure doesn't make me any money, I use it for hobby websites that actually cost me money every month just to keep going. But it's so much fun I think $100 is small price to pay, if it keeps Clojure development going.

December 14, 2009 @ 8:49 AM PST
Cateogory: Programming
Tags: Clojure

Lame comment spam management that works

It's been nine months since I ditched Wordpress and moved to a blog system I wrote from scratch (in Clojure). This was a great move in so many ways. One of those ways is comment spam. My site is as popular now (or maybe slightly more popular now) as it was when I was running Wordpress, so I think comparing before and after is valid.

With Wordpress, every morning I'd do the ritual of deleting overnight spambot droppings. Typically I got between 1 and 5 every night. I had a default Wordpress install and all I used for spam filtering was Askimet. Askimet did a surprisingly good job, catching literally if not thousands of spams every week which otherwise would've been ruining my site. But inevitably some would still get through. And what's worse, there were a lot more false positives than I could tolerate.

Since I started counting with my new system, which is around 6 months, to the best of my knowledge I've gotten zero spambot-produced comments that made it through my filters. This is pleasant, to say the least.

The system I'm using is stupid. None of it is stuff I thought of myself, I got ideas from other lots of other blogs or articles I read, but the implementation is mine and it's not sophisticated. It would take a bot author a few seconds to work around it. But no one has bothered. Why bother writing a bot for my one-man blog, when you can write a bot for Wordpress and have it work on tens of thousands of blogs? And I can change my system to defeat the bots with a few lines of code just as easily as they can work around it.

So here's why I think it's working.

December 05, 2009 @ 6:34 PM PST
Cateogory: Programming

Adminer, where have you been all my life?

How do you view and edit data in a mysql DB? Lots of ways.

There's always commandline mysql. This is how I do it around 50% of the time. But it could be better. ASCII-art table dumps are not the easiest things to read, and readline-based history and editing only gets you so far.

phpMyAdmin is what I grew up with. It's pretty good but it's way too heavyweight. It also has all kinds of funky Javascript, to the point where I actually use Greasemonkey to remove some of it. Auto-selecting query text when you try to edit it in particular drives me crazy. Auto-selecting ANYTHING on Linux, where selecting text usually equals clobbering the X clipboard, is a really really bad idea. Last I tried it's a one line of Javascript in Greasemonkey to fix this by the way:

document.getElementById('sqlquery').removeAttribute('onfocus');

But phpMyAdmin may have changed since last I used it, it's been a while.

There's the native mysql GUI, called mysql-gui-tools in Gentoo. It's a standalone app that's pretty good, but the Linux version is gimped up compared to the Windows version for some strange reason. In any case it seems to be discontinued or something. There's some new MySQL Workbench thing coming, which I'll probably try once it hits Gentoo, but it looks like overkill for my simple needs.

SQuirreL SQL is another cross-platform GUI app. It connects to lots of different kinds of DBs (pretty much any DB that Java can talk to), so at work where I have get my mysql server to talk to someone else's MS SQLServer (ugh) I use this. But it's very heavyweight and not the most enjoyable interface.

Last week I chanced upon something pretty good. Adminer is a single PHP file you throw on a server and there you go. It gives you something vaguely similar to phpMyAdmin but far more lightweight. There's very little Javascript messing with my query-writing and the styling is minimal and easy to read. I don't know how secure it is, so I don't plan to put it on any public servers, but on my test server doing web development it's good. I love bouncing back and forth between a the database in one browser tab and my website in another tab. This is what I use now.

November 28, 2009 @ 9:47 AM PST
Cateogory: Programming

I'm turning into a Lisp snob

Reddit and StackOverflow and other websites I frequent are filled to the brim with discussion of Google's Go. The code snippet on the front page is:

package main

import "fmt"

func main() {
  fmt.Printf("Hello, 世界\n")
}

First thoughts that ran through my head as I looked over the site:

  • Ugh, look at all that syntax.
  • Nice(r) type system (than C++ and Java). I'll stick with multimethods though.
  • Concurrency semantics, hmmm... Shared mutable memory between threads? I think I'll stick with Clojure for now thanks.
  • Where are the macros?
  • It has anonymous functions and closures and sort-of first-class functions? Good. Welcome to the 1960's.
  • len is a special operator? Sigh. (Programming language quality is usually inversely proportional to the number of special forms.)
  • Cool that they used Japanese in the example though. (That word is sekai, "world", obviously.)

Compared to a Lisp, this language looks indistinguishable to C, Perl, Python, Java etc. It looks like such a small incremental improvement (if it even is an improvement). Yet another imperative, for-loop-wielding, curly-brace-using, pointer-mangling, state-mutating, OOP language.

In fact via Reddit today I read this awesome post to a mailing list which compares Go with ALGOL68, and it gave me a would-you-look-at-that moment. Once you learn a few languages that are significantly different from ALGOL derivatives, all ALGOLish languages start to look eerily similar. Are we really stuck with ALGOL-derived languages being the only viable mainstream languages for all time? How much polish can we possibly apply to the same turd?

Then I realized, I'm turning into a Lisp snob. : ( Learning a Lisp apparently does spoil you for the rest of time. I am without a basis to judge whether this language will be a successful replacement of anything. All I know is I probably won't use it. Honestly I'm much more excited about new things on the horizon in Clojure. And I still have getting better at Haskell on my TODO list.

November 13, 2009 @ 2:38 PM PST
Cateogory: Programming
Tags: Lisp, Go, Clojure

Clojure: redirecting output to a file

In a scripting language, you often run scripts from a command line and print things to STDOUT. For long output you can redirect it to a file via shell-redirection ($ foo.rb > foo.txt).

But if you're using Clojure for ad-hoc scripting, you're probably sitting in a REPL, not running from a command line. REPLs generally don't have shell-redirection. Printed output just gets dumped into your REPL. This can be annoying. (After a good 10,000 lines of output into a REPL buffer, Emacs starts to lag.)

But you can write a macro to handle this easily enough.

(use 'clojure.contrib.duck-streams)
(defmacro redir [filename & body]
  `(spit ~filename (with-out-str ~@body)))

Then:

user> (redir "foo.txt" (foo) (bar) (baz))

Now whatever (foo) and friends would've printed to STDOUT will instead go to foo.txt. I've found this somewhat useful at work lately.

Edit: better version via Graham Fawcett:

(defmacro redir [filename & body]
  `(binding [*out* (writer ~filename)] ~@body))
November 12, 2009 @ 1:23 PM PST
Cateogory: Programming
Tags: Clojure

Clojure ORM-ish stuff

Suppose I have this:

user> (def foo [{:id 1 :foo 123} {:id 2 :foo 456}])
#'user/foo
user> (def bar [{:foo_id 1 :bar 111} {:foo_id 1 :bar 222}])
#'user/bar

What I want is to "join" foo and bar so that each item in foo ends up with a sub-list of bars based on matching key fields.

In real life, these lists-of-hash-maps are coming out of a database via clojure.contrib.sql, so this is something I actually want to do pretty often. This is also vaguely similar to what you get out of a Rails-like ORM, where you end up with an object that has lists of sub-objects anywhere you have a one-to-many relationship.

Here's how I end up doing this in Clojure:

(defn one-to-many
  ([xs name ys f]
      (for [x xs :let [ys (filter (partial f x) ys)]]
        (assoc x name ys))))

Now I can do this:

user> (pprint (one-to-many foo :bars bar #(= (:id %1) (:foo_id %2))))
({:bars ({:foo_id 1, :bar 111} {:foo_id 1, :bar 222}), :id 1, :foo 123}
 {:bars (), :id 2, :foo 456})

And if I define a helper function:

(defn key=
  ([xkey ykey]
     #(= (xkey %1) (ykey %2))))

Then I can write it more concisely:

user> (pprint (one-to-many foo :bars bar (key= :id :foo_id)))
;; same as above

And if I have another "table" of data like this:

user> (def baz [{:foo_id 1 :baz 555} {:foo_id 2 :baz 999}])
#'user/baz

Then I can join them all like this:

user> (pprint (-> foo
                  (one-to-many :bars bar (key= :id :foo_id))
                  (one-to-many :bazzes baz (key= :id :foo_id))))
({:bazzes ({:foo_id 1, :baz 555}),
  :bars ({:foo_id 1, :bar 111} {:foo_id 1, :bar 222}),
  :id 1,
  :foo 123}
 {:bazzes ({:foo_id 2, :baz 999}), :bars (), :id 2, :foo 456})

This is pretty concise. It may be possible to do it in an even more concise way, (if so, do share). If I was willing to adhere to some Rails-y naming convention for my table names and for the id fields in my tables, I could make this shorter by not having to specify the names of the id fields, but I don't want to go there. It's trivial to write similar functions for a one-to-one relationship, or to use a join-table to "join" two tables with a many-to-many relationship.

I am happily surprised sometimes by how simple it is to roll my own version of things that previously seemed like dark magic. I used Rails for a long time and it seemed like a crapload of code must have gone into making the ORM work. But four lines of code gets me 75% of what I ever needed Rails' ORM for.

This may be more thanks to me opening my eyes a bit than to Clojure being awesome, but either way, I'll take it.

November 03, 2009 @ 10:21 AM PST
Cateogory: Programming