briancarper.net (λ) (Category: Programming)

Shuffle lines in Vim

Thu, 07 Jul 2011 14:07:41 -0700

In a pinch, I needed to randomize the order of a few thousand lines of plain text. In Linux you can just pipe the file through sort, even right inside Vim:

:%!sort -R

But I was stuck on Windows. And I don't know how to randomize a file in native Vim script. But doing it in Ruby is pretty easy, and luckily, Vim has awesome Ruby support. Tne minutes' work and a few peeks at :h ruby and we have a successful, working kludge:

function! ShuffleLines()
ruby << EOF
    buf = VIM::Buffer.current
    firstnum =  VIM::evaluate('a:firstline')
    lastnum = VIM::evaluate('a:lastline')
    lines = []
    firstnum.upto(lastnum) do |lnum|
      lines << buf[lnum]
    end
    lines.shuffle!
    firstnum.upto(lastnum) do |lnum|
      buf[lnum] = lines[lnum-firstnum]
    end
EOF
endfunction

2011-07-07 23:32 - Edited to remove a superfluous line.

2011-07-09 21:33 - Wrong parameter for sort, oops.

Keyword Arguments: Ruby, Clojure, Common Lisp

Fri, 24 Jun 2011 17:22:48 -0700

And suddenly I return to blogging, rising from the ashes like some kind of zombie phoenix. Turns out writing a book is a good absorber of time, like some sort of heavy-duty temporal paper towel. Now that I've gotten the terrible similes out of my system, let's talk about keyword arguments, one of my favorite features in any language that supports them.

Ruby, Clojure, and Common Lisp are all languages I enjoy to some degree, and they all have keyword arguments. Let's explore how keyword args differ in those languages.

Why keyword arguments?

Why are keyword arguments good?

You can omit arguments.
You can supply arguments in an arbitrary order.
Arguments are labeled, so you know what argument means what.

Positional arguments require mentally lining up the 7th argument in your function call with the 7th argument in the function signature, and so on. Keyword arguments become more and more attractive the more arguments you have in your function signature.

Keyword arguments trade a bit of verbosity for added explicitness, clarity and reduced mental burden. (Kind of like Lisps do overall. Fancy that.)

Ruby

Ruby doesn't have special support for keyword arguments. But Ruby likes its hashes, so you can just pass one in as an argument to a function. As some syntax sugar, if you pass a "flat" list of :key => val pairs, Ruby slurps them all together and stuffs them into a hash for you.

def foo(arg)
  p arg
end

foo({:x => 123})                  # => {:x=>123}
foo(:x => 123)                    # => {:x=>123}

With even more added sugar, you can leave off the parens in Ruby function calls. So this is pretty common in Ruby:

foo :x => 123                     # => {:x=>123}

How nice and punctuation-less. But then things get ugly. What about this?

foo {:x => 123}

That won't even compile. Ruby thinks {:x => 123} is a code block, and a bare key/value pair isn't valid syntax as the first thing in a code block. You need the parens. A bit unfortunate, but it gets worse...

def bar(arg1,arg2)
  puts "#{arg1} #{arg2}"
end

bar :x => 123, :y => 456          # Runtime error
bar {:x => 123}, {:y => 456}      # Won't compile

In the first example, all of the key/value pairs are slurped into one hash and end up in arg1. There's nothing left for arg2, so you get a "wrong number of arguments" exception. The second example won't even compile, of course, because again Ruby thinks {:x => 123} must be a code block with invalid syntax.

It gets worse if you change the argument list for bar slightly...

def bar2(arg1 = {}, arg2 = {})
  puts "#{arg1} #{arg2}"
end

This way, you don't even have to supply any arguments. This is nice, if all of your argument are optional.

bar2    # => {} {}

Now suppose you want arg1 to be {:x => 123}, and arg2 to be {:y => 456}. You might naively try this:

bar2 :x => 123, :y => 456         # => {:x=>123, :y=>456} {}

Oops, it all got dumped into arg1, and now instead of a missing argument error, arg2 silently ends up with a default, empty hash. You have to explicitly pass in an empty value for arg1 so that everything is slurped into arg2.

bar2 {}, :x => 123, :y => 456     # => WRONG!  Ruby thinks {} is a code block again.

bar2({}, :x => 123, :y => 456)    # => {} {:x=>123, :y=>456}

So much for syntax sugar. You might think you'd be unlikely to find this kind of thing in the wild, but in Ruby on Rails for example, there are quite a few functions whose argument lists look exactly like this. One signature for link_to is:

link_to(body, url_options = {}, html_options = {})

So...

link_to "foo", :controller => :x                          # OK
link_to "foo", :controller => :x, :class => "css_class"   # WRONG! 
link_to "foo", {:controller => :x}, :class => "css_class" # OK

What about support for default arguments? We might want to say that if you didn't pass in an :x argument, we want it to have some default value. You might think this would work:

def baz(x = {:x => 123})
   p x
end

But you would be sadly mistaken.

baz                               # => {:x=>123}
baz :y => 456                     # => {:y=>456} ... oops

Ruby doesn't merge your keyword arguments into the map in the parameter list. It uses that map if you don't supply any arguments, otherwise your map replaces the default entirely. So to get default arguments, you need something like

def baz2(args = {})
  args = {:x => 123}.merge(args)
  p args
end

baz2                              # => {:x=>123}
baz2 :x => 555                    # => {:x=>555}
baz2 :y => 456                    # => {:x=>123, :y=>456}

Kind of messy, but that's OK.

One last subtle ambiguity in Ruby is determining whether someone passed a nil argument for a keyword explicitly, or omitted a keyword entirely. It might make a difference in some circumstances.

def quux(args={})
  p args.include? :x
  args = {:x => nil}.merge(args)
  p args
end

quux                              # true, {:x=>nil}
quux :x => nil                    # false, {:x=>nil}

Pretty messy, but such is life.

Clojure

Part of the fun of Lisps is lack of ambiguity. Everything is spelled out in all its parenthesized glory. In Clojure, when you call a function like (foo :x 123), Clojure requires you to specify what you want to happen with those arguments.

It used to be that Clojure didn't have much support for keyword args at all. Clojure did have support for allowing variable numbers of arguments to functions though. So in the beginning, people used to slurp all of their arguments together into a list, and then turn it into a map inside the function.

user> (defn foo [& args]
        (let [args (apply hash-map args)]
          (prn args)))
#'user/foo
user> (foo)
{}
user> (foo :x 123)
{:x 123}
user> (foo :x 123 :y 456)
{:y 456, :x 123}

That worked. It still works today. But nowadays there's a better way. Why not slurp your arguments directly into a hash-map?

user> (defn foo [& {:as args}]
        (prn args))
#'user/foo
user> (foo)
nil
user> (foo :x 123)
{:x 123}
user> (foo :x 123 :y 456)
{:y 456, :x 123}

This is an example of destructuring. In this case, all of our arguments are slurped into a list (thanks to &), then this list is matched against our destructuring pattern, in this case {:as args}. :as says to take everything in the list, turn it into a map and give the resulting map the name args.

This isn't a special feature of defn. Destructuring works anywhere you're setting up bindings, for example in let, for, doseq etc. Like so:

user> (let [{:as args} (list :x 123 :y 456)] args)
{:y 456, :x 123}

It just so happens that & creates the list for you, out of the arguments you pass.

Destructuring does lots more than that though. We can immediately pull out the values for keywords we care about, so they'll be bound to names in our function body.

user> (defn foo [& {:keys [x y z]}]
        (prn x y z))
#'user/foo

user> (foo :z 123 :x 456)
456 nil 123

It's certainly a bit more verbose than Ruby in the function signature, but it lacks ambiguity and it saves you some repetition in the function body.

Clojure's approach also has the benefit of specifying directly in the function signature which keywords you're expecting. In a smart editor, like Emacs, you get an indicator of what kinds of keywords you should be passing in. See at the bottom?

This is also available at the Clojure REPL via the doc function.

user> (doc foo)
-------------------------
user/foo
([& {:keys [a b c]}])
  nil
nil

There's no better documentation than live, built-in documentation. There's nothing more distracting when programming than context shifts, and having to dig into a web browser to check a function signature is a huge mental page fault.

What about default values? Sure. You can use :or to specify defaults for some or all of your keywords. This works much more like I'd expect, compared to Ruby.

user> (defn foo [& {:keys [x y z] :or {x 1 y 2 z 3}}]
        (prn x y z))
#'user/foo

user> (foo :y 555)
1 555 3

What about determining whether the user passed nil for a keyword or whether they omitted the keyword entirely? For that, you have to resort to testing the argument map for the existence of the key, which isn't fun, but at least it's possible.

user> (defn foo [& {:keys [x y z] 
                    :or {x 1 y 2 z 3}
                    :as args}]
        (prn x y z)
        (prn args)
        (doseq [k [:x :y :z]]
          (println "Contains" k "=>" (contains? args k))))
#'user/foo

user> (foo :x 1 :y 2)
1 2 3
{:x 1, :y 2}
Contains :x => true
Contains :y => true
Contains :z => false

Why do I keep mentioning this? See Common Lisp below.

In any case, our parameter list is becoming huge and unweildy. The first time I saw sample Clojure code like this, I almost did a spit-take. But after a bit of getting used to, I'm finding that this syntax is pretty comfortable.

The simplest case may not be as concise as Ruby, but aside from lack of ambiguity, the benefit of Clojure's approach is being able to safely do powerful (and borderline insane) things, like nested destructuring. And this works everywhere you're setting up a binding.

(defn ow-my-eyes [a & {[b {:keys [x y]}
                        & {[g h] :r
                           :keys [p q]
                           :or {q 123}}]
                       :x}]
  (prn a b g h p q x y))



user> (ow-my-eyes 444 :x [1 {:x 555 :y 666} :p 3 :r [888 999]])
444 1 888 999 3 123 555 666

If you wrote code like that in real life, you'd likely be defenestrated, but at least you know you can do it. And destructuring in Clojure could likely be extended even further in the future, if someone came up with a use case for something that isn't supported.

Official documentation for all of this is here.

Common Lisp

When it comes to keyword arguments, Common Lisp supports mostly everything that Clojure does, and some things it doesn't. CL was likely a big inspiration for Clojure's destructuring. I highly recommend reading Practical Common Lisp to learn more about CL keyword arguments and list destructuring. (Read the rest of the book while you're at it.)

Keyword arguments in CL look like this:

> (defun foo (&key x (y 123) (z 456 z-supplied-p))
    (pprint (list x y z z-supplied-p)))
FOO

> (foo)
(NIL 123 456 NIL)

> (foo :z nil)             ; note, z-supplied-p tells us whether z was omitted or not
(NIL 123 NIL T)

> (foo :z 555 :x 666 :y 777)
(666 777 555 T)

> (foo :x 123)
(123 123 456 NIL)

> (foo :x 555)
(555 123 456 NIL)

Yeah, there's direct support for distinguishing supplied keys with nil values, and un-supplied keys. That's pretty nice.

Common Lisp also supports insane things like having keyword arguments' default values be a function of other arguments in the parameter list.

> (defun bar (&key (x 123) (y (+ x 1000)))
    (pprint (list x y)))
BAR

> (bar)
(123 1123)

> (bar :x 5)
(5 1005)

> (bar :x 5 :y 6)
(5 6)

Clojure destructuring can't do this by default, so you'd have to resort to a let in the function body.

(defn bar [& {:keys [x y] :or {x 123}}]
  (let [y (or y (+ x 1000))]
    (prn x y)

There's nothing stopping Clojure's destructuring from being patched to support this, of course. But this isn't a feature I've ever found myself wanting particularly badly.

As for how to destructure CL keyword arguments into a vector or hash-map, CL doesn't directly support doing it in function parameter lists like Clojure does. (Though there's nothing stopping someone from throwing together a reader macro to let CL do this, of course.) This isn't really surprising, because CL loves its cons cells, while Clojure embraces maps (and vectors and sets etc.), offering them default reader syntax and lots of other built-in support.

So there you have it. Keyword arguments. Use them. Love them.

Lessons:

Sugar can be bad for your health.
Ambiguous, bad. Explicit, good.

org-mode is awesome

Thu, 20 Jan 2011 16:14:44 -0800

I've seen org-mode for Emacs mentioned very frequently around the interwebs, so it went into my mental queue of topics to learn. It finally bubbled to the top this week, so I took a look.

Organizer? Nah.

As an organizer/calendar, well, I doubt I'll need it. Enforced use of MS Outlook is mandated by work. My Post-it-notes-all-over-my-desk method of organization will also continue to serve me well.

There are some nice agenda-related shortcuts that are probably worth using though, like typing C-c . to enter a datestamp, like <2011-01-20 Thu>. Then you can increment or decrement it one year/month/day at a time via S-up and S-down. I like this.

Plaintext editor? Yes!

As a plaintext outline and table editor... wow. org-mode rocks. Do you know how many hours of my life could have been saved by having a good ASCII table/bullet-list editor? org-mode lines everything up and keeps it all nice and neat for you.

You can also make plaintext check boxes and check/uncheck them. And you can insert hyperlinks and footnotes, and click them to open web pages or jump back and forth between footnote and reference.

There are ways to collapse and expand outlines, search for items and only display those items, and so on. The documentation for org-mode is very clear and took me less than an hour to read through. All-in-all a pleasant experience.

* Agenda
** Things to learn
1. [X] Clojure
2. [X] org-mode (see [fn:diagram1])
3. [ ] Haskell
4. [ ] Japanese
   1. [X] Hiragana
   2. [X] Katakana
   3. [ ] Kanji
5. [ ] The true meaning of friendship

* Footnotes
[fn:diagram1]

| Task                              | Annoyance (1-10) |
|-----------------------------------+------------------|
| Making ASCII tables by hand       |              9.5 |
| Making ASCII bullet lists by hand |              7.2 |
| Using org-mode                    |              0.4 |

It looks nice plastered into my blog, but you don't get a real idea of how many cool things you can do with it until you open it in Emacs and start shuffling items around, bumping them up/down a level in headlines, creating properly-numbered bullet items with one key, and seeing the columns in the table auto-resize as you type.

I also highly recommend putting (setq org-startup-indented t) into .emacs to make everything look pretty on-screen. It still saves as the simple plaintext above, but it looks like this in Emacs:

I can definitely see using org-mode for TODO files in some of my projects. (You can mark entries as TODO (just by typing TODO in front), and then toggle between TODO/DONE via C-c C-t.) I can also see using it as a general-purpose note-taker.

org-mode also has a mobile version for iPhone and Android, synced via WebDAV or Dropbox, so you can org-mode on your phone while you're driving to the grocery store¹. Again I don't really need this, but there it is.

The joy of plaintext

Plaintext is awesome.

It's the universal file format. It's readable and writeable by scripting langauges, terminals, text editors, IDEs, word processors, web browsers, even lowly humans.

Plaintext's one shortcoming is its lack of structure. It's just a bunch of letters. It doesn't have a color, it doesn't have a style, it doesn't line up into columns without a lot of effort. There's nothing stopping you from opening a parenthesized list and forgetting the closing paren.

Computers don't care about these problems, but humans are bad at producting plaintext by hand, and bad at editing it once it's produced. Our clumsy, stumpy fingers and inconsistent, chaotic brains can't handle the freedom.

Emacs (and Vim) are awesome because they let you do magical things to plaintext. They enforce structure. They provide shortcuts so you can get your plainext right the first time.

[ ] is just two braces and a space, but org-mode lets me hit C-c C-c and turn the space into an X. This may seem banal, hardly worth caring about, but add to this shortcut thousands upon thousands of others. Things like org-mode, or paredit, or all of Vim's built-in magic... it all adds up to something wonderful.

And best of all, you always still have the option of manually keyboarding over and typing that X between the braces yourself. It's still just plaintext underneath. So you end up with the best of both worlds.

I do not recommend using org-mode while driving, for public safety reasons. ↩

Vim undo tree visualization

Mon, 18 Oct 2010 15:21:16 -0700

I wrote previously about an awsome plugin to give Emacs Vim-style undo trees.

Vim's undo trees are the best thing since sliced bread, but the interface for browsing through the tree is not pleasant. The Emacs undo-tree library has a way to visualize the tree and move through it with your keyboard, which solves this problem.

But now, thanks to Steve Losh, Vim has an undo-tree visualizer too. Delicious. Though it's still beta and promises to eat your babies, it seems to work pretty well. I think the diff view of the changes for the undo is a really good idea.

Thus continues the eternal Vim/Emacs arms race.

Vim :ruby and :rubydo scope

Tue, 31 Aug 2010 10:40:51 -0700

Note to self. In old Vim (tested in 7.2.320), I could do this:

:ruby x='foo'
:rubydo $_=x

Now every line in the file says foo. But in Vim 7.3 I get an error:

NameError: undefined local variable or method `x' for main:Object

The scoping rules for Ruby in Vim must have changed somewhere along the line. I was abusing this feature to do some handy things, so this is sad.

A workaround is to use global variables in Ruby instead. So this still works:

:ruby $x='foo'
:rubydo $_=$x

Phew.

Emacs undo trees

Tue, 17 Aug 2010 14:06:33 -0700

I've said it before: undo in Emacs is horrible. On the other hand, undo in Vim is awesome.

But this is true no longer. Now there are undo trees for Emacs! Yes, this news is so important I had to italicize and bold it. It's like Emacs has been punching me in the face for years, and today I got it to stop. I never thought I'd see the day.

And it works great too. You can even view the tree visually and navigate it with the cursor keys, which is a step up on what Vim offers out of the box.

In other news, Vim 7.3 is out and it now has persistent undo across reloads. It's like an arms race, and gleeful hackers reap the benefits.

gaka 0.2.0

Thu, 29 Jul 2010 13:59:22 -0700

Per many commenters' suggestions and thanks to code from Steve Purcell, you can now use maps for CSS attributes in gaka.

user> (println (gaka/css [:a {:color :red}]))
a {
  color: red;}

This looks more like vanilla CSS thanks to the curlies, which is nice. You just have to keep in mind that your key/value pairs could end up being printed in random order, and order is significant¹ in CSS.

It just so happens that maps are implemented in Clojure right now such that if they only have a few entries (16 key/value pairs), the order will be preserved, because you get a PersistentArrayMap instead of a PersistentHashMap. But it's highly dangerous to rely on such a thing. It could change at any time in the future.

In any case, you can also mix and match maps, lists and "flat" keyvals. They'll all be flattened That can help preserve attribute order in those cases where you need to.

user> (println (gaka/css [:a :color "red" {:padding 0} (list :margin 0)]))
a {
  color: red;
  padding: 0;
  margin: 0;}

I've also enhanced "mixins" a bit further. You can now mixin entire tags as well as attributes. Or a combination of both. Say you want a mixin that means "Make my element have no padding, and make links within the element be red":

user> (println (gaka/css [:div.foo mixin :margin 0]
                         [:div.bar mixin]))
div.foo {
  padding: 0;
  margin: 0;}

  div.foo a {
    color: red;}

div.bar {
  padding: 0;}

  div.bar a {
    color: red;}

You can get gaka from github or Clojars.

Order is only significant in cases where you're doing things like padding: 0; padding-left: 1px. This is arguably bad CSS style, but it's valid, and it's also possible you'll have this kind of thing if you're generating CSS procedurally. But most of the time, order is not significant. e.g. it doesn't matter if you set text color first and background color second, or vice versa. So maybe this isn't so much of a problem in practice. ↩

Footnotes

Tue, 27 Jul 2010 12:30:57 -0700

Did you ever notice how footnotes make your writing seem more important¹ somehow?

Maybe one reason is that "real" books use footnotes. At a glance, it looks like I have references² backing up everything I say. In reality, I don't, but the connotation carries through somehow³. Now my blog seems scholarly and authoritative.

And if you're like me, you can't resist clicking footnotes to see what they refer to⁴. According to my estimates, by utilizing footnotes, in one fell swoop I have decreased my readers' average reading efficiency by 73%.

In any case, I've added experimental, rudimentary support for footnotes to cow-blog.

I'm loosely copying the syntax from Markdown Extra for this. Markdown is great, except when it isn't. The standard doesn't have support for some useful extensions. I use Showdown for Markdown support, and I'm probably going to work on adding more features of Markdown Extra to Showdown in the near future.

I just dread actually doing it. Showdown (like Markdown itself) is implemented as a series of hackish regex transformations of blobs of text. It's not a proper grammar. Implementing more of Markdown Extra means more regex blobbing. It's brittle and fragile and even getting incomplete support for footnotes was less than enjoyable. But at the same time I find myself wanting to do things that Markdown can't so, so I may have to bite the bullet.

(If there's a Showdown Extra out there already, drop me a URL. It'd be most appreciated. But I couldn't find one.)

In reality nothing I say is important. ↩
Does my inner dialog count as a reference? ↩
Via telepathy. ↩
See? ↩

Clojure syntax highlighting via SyntaxHighlighter

Wed, 21 Jul 2010 15:17:42 -0700

How do you syntax-highlight Clojure code for display on a website? The best way I can find is SyntaxHighlighter.

Daniel Gómez wrote a brush to give SyntaxHighlighter Clojure support. I tweaked it a bit myself and integrated it into cow-blog. I also converted my favorite color scheme to a SyntaxHighlighter theme. So when I write this code:

(defn- ip
  "Given a request, return the IP.  Looks for an x-forwarded-for
  header, falls back to :remote-addr on the request."
  [request]
  (or (get-in request [:headers "x-forwarded-for"])
      (request :remote-addr)))

You should see something like this:

...unless you're reading this via RSS, or in a browser without Javascript enabled, in which case you'll see plain, depressing black and white. But that's one nice thing about SyntaxHighlighter. It degrades nicely.

One bad thing about SyntaxHighlighter is that it doesn't play nicely with Markdown. Or rather, Markdown isn't powerful enough to let you specify the class of any markdown-generated HTML tags. If you want the <pre class="brush: clojure"> that SyntaxHighlighter requires, you have to write out the HTML by hand. But I hacked Showdown a bit to let me specify classes more easily, so I can avoid having to do that.

The code for all of this is on github with the rest of my blog.

ANTLR via Clojure

Fri, 16 Jul 2010 17:41:04 -0700

ANTLR is a parser-generator for Java. Can you use it from Clojure? Sure. Would you want to? Maybe.

Here's how to do it, start to finish.

For the impatient among you, all of the code below is on github.

git clone git://github.com/briancarper/clojure-antlr-example.git

Setup

I'm going to use leiningen for this project. Let's make a new project called antlr-example.

$ lein new antlr-example

Now edit project.clj to tell lein to fetch ANTLR (and Swank, since I use Emacs). ANTLR is available in Maven Central, so leiningen can grab it for us.

(defproject antlr-example "1.0.0-SNAPSHOT"
  :description "ANTLR Clojure example"
  :dependencies [[org.clojure/clojure "1.2.0-beta1"]
                 [org.clojure/clojure-contrib "1.2.0-beta1"]
                 [org.antlr/antlr "3.2"]]
  :dev-dependencies [[swank-clojure "1.3.0-SNAPSHOT"]])

Then a simple lein deps will download all of the jars and all of ANTLR's dependencies for you. Handy. You end up with this:

antlr-example
├── classes
├── lib
│   ├── antlr-2.7.7.jar
│   ├── antlr-3.2.jar
│   ├── antlr-runtime-3.2.jar
│   ├── clojure-1.2.0-beta1.jar
│   ├── clojure-contrib-1.2.0-beta1.jar
│   ├── stringtemplate-3.2.jar
│   └── swank-clojure-1.3.0-20100502.112537-1.jar
├── project.clj
├── README
├── src
│   └── antlr_example
│       └── core.clj
└── test
    └── antlr_example
        └── core_test.clj

Interactive development? Kind of...

One weakness of ANTLR via Clojure is that development is no longer REPL-based. The ANTLR workflow is essentially a Java workflow:

Write a grammar
Compile the grammar into Java source code
Compile the Java source into Java .class files

Until and unless someone writes a Clojure backend for ANTLR, so that ANTLR can directly produce Clojure source code (and it seems like it should be possible to do so), you're back to a write/compile/debug cycle. Joy!

This also means restarting your REPL every time you alter and recompile your grammar.

But the good thing is that there's ANTLR Works, a free GUI for writing ANTLR grammars. ANTLR Works has an interactive interpreter, which is nice, and it has a compiler/debugger, which is better. These let you test your grammar as you write it, by trying inputs and seeing the resulting AST in graphical form, which is as good as you could hope for. This is actually even a bit nicer than a plaintext REPL.

Plus, it has a nice editor for ANTLR code. Emacs was freaking out over the indentation on a few grammars I tried.

So a decent workflow might be to write and debug your grammar in ANTLR Works, then fire up Clojure afterwards to consume the parser.

A simple grammar

We're going to use a classic textbook example, simple arithmetic expressions, as a proof of concept.

For Java (hence Clojure) to find this code, it has to be named properly and it has to be put into the proper directory on CLASSPATH. I'm going to create the grammar file src/antlr_example/Expr.g. (My grammar file, Java source, and Clojure source will all be jumbled together in src. You can easily do it differently if this offends your sensibilities.)

The first line in the grammar names the grammar (and the classes we'll consume from Clojure):

grammar Expr;

Clojure being a s-exp based language, it might be nice to have ANTLR generate an AST. ANTLR has good support for this:

options {
    output=AST;
    ASTLabelType=CommonTree;
}

Helper tokens for the grammar:

tokens {
    PLUS = '+';
    MINUS = '-';
    MULT = '*';
    DIV = '/';
}

These next lines are important. We want to generate classes antlr_example.ExprLexer and antlr_example.ExprParser, so we have to set our code up in the proper package, antlr_example.

This code includes both a parser and lexer, so you have to set the package for both in ANTLR.

@header        {package antlr_example;}
@lexer::header {package antlr_example;}

And then the grammar itself, which is simple:

expr: term ((PLUS | MINUS)^ term)* ;
term: factor ((MULT | DIV)^ factor)* ;
factor: INT ;

INT :   '0'..'9'+ ;
WS  :   ( ' ' | '\t' | '\r' | '\n')+ {$channel=HIDDEN;} ;

One thing to note is (PLUS | MINUS)^; the ^ here tells ANTLR to treat (PLUS | MINUS) as the parent of the node created for this rule. This means the term on either side will be children of this node. We do the same for MULT and DIV. This will give us a nice tree. Otherwise you end up with a flat list of tokens.

{$channel=HIDDEN;} for the whitespace rule tells ANTLR to ignore whitespace, also useful.

If you use ANTLR Works with this grammar, on the input 1 + 2 * 3 + 4, you can see that it works OK.

That was an awful lot of boilerplate and setup and ugly syntax though. See how much Clojure spoils you? But for a simple grammar, it's not that much code.

For a longer or more complex grammar, you might end up having to embed inline Java code into your ANTLR grammar. But again, that's not the end of the world.

Compile everything

Once you write your grammar (presumably in ANTLR Works), you can generate the Java code and compile it thusly (run from your project's base directory):

$ java -cp 'lib/*' org.antlr.Tool src/antlr_example/Expr.g
$ javac -cp 'lib/*' -d classes src/antlr_example/Expr{Lexer,Parser}.java

The first command should generate src/antlr_example/ExprLexer.java and src/antlr_example/ExprParser.java.

The second command should spew tons of class files into classes/. Leiningen and other tools generally include classes/ on your CLASSPATH, so that's a good place for them. Again putting them here is a fairly arbitrary decision on my part. You can put the source and class files anywhere, as long as the class files end up on your CLASSPATH when you start Clojure.

If you're really so lazy that you can't run these two commands, you could use ANTLR Works' built-in commands for compiling your grammar. The only way I could find to invoke it was by starting a Debug session, which compiled my code as a side-effect. And it spews files into places I don't want them, so I like the command line version better.

You could probably also set up a leiningen task for this if you're re-running these commands a lot, or configure your build tool of choice to do the same. I didn't bother.

Now (re)start your REPL and let's write some Clojure.

Clojure code

It's pretty easy to consume an ANTLR parser from Clojure. I'm putting the code below into src/antlr_example/core.clj

First import the classes. You probably need some ANTLR helper classes to set up the parser; you also need the classes you just generated.

(ns antlr-example.core
  (:import (org.antlr.runtime ANTLRStringStream
                              CommonTokenStream)
           (antlr_example ExprLexer ExprParser)))

Here's a function to parse a string using our new parser class:

(defn parse-expr [s]
  (let [lexer (ExprLexer. (ANTLRStringStream. s))
        tokens (CommonTokenStream. lexer)
        parser (ExprParser. tokens)]
    (.expr parser)))

.expr is the name of the top-level rule we want to invoke from our grammar. The rest is boilerplate; just plug in the proper class names.

So let's see what we've got.

user> (in-ns 'antlr-example.core)
#<Namespace antlr-example.core>
antlr-example.core> (parse-expr "1 + 2 * 3 + 4")
#<expr_return antlr_example.ExprParser$expr_return@3755e508>

Er, OK. A good way to inspect Java objects is via bean, so let's view the guts of this object:

antlr-example.core> (bean *1)
{:tree #<CommonTree +>,
 :template nil,
 :stop #<CommonToken [@12,12:12='4',<8>,1:12]>,
 :start #<CommonToken [@0,0:0='1',<8>,1:0]>,
 :class antlr_example.ExprParser$expr_return}

I see. We set up our grammar above to generate an AST, so :tree on the bean will give us that. This translates to .getTree on the Java object. So we can edit our function to call this for us, since we always want to have the tree.

(defn parse-expr [s]
  (let [lexer (ExprLexer. (ANTLRStringStream. s))
        tokens (CommonTokenStream. lexer)
        parser (ExprParser. tokens)]
    (.getTree (.expr parser))))

Now:

antlr-example.core> (parse-expr "1 + 2 * 3 + 4")
#<CommonTree +>

OK, it appears we have one node of the tree, the root node. Let's peak inside:

antlr-example.core> (bean *1)
{:children #<ArrayList [+, 4]>,
 :childIndex -1,
 :parent nil,
 :text "+",
 :nil false,
 :token #<CommonToken [@10,10:10='+',<4>,1:10]>,
 :class org.antlr.runtime.tree.CommonTree,
 :ancestors nil,
 :tokenStartIndex 0,
 :type 4,
 :childCount 2,
 :charPositionInLine 10,
 :tokenStopIndex 12,
 :line 1}

Looks like our children are in :children on the bean, which translates to .getChildren on the Java object. And we have .getChildCount to count the children, and .getText to get the text of the current node. (We could've learned all of this by reading the javadocs for ANTLR too, but what fun is that?)

Since we're dealing with a tree, we can use tree-seq in Clojure to get a flat list of all the tokens in our text.

tree-seq takes three arguments. First a function that returns true if the node has children, false otherwise. Second a function that returns the children for a node that has children. Third the root node of our tree.

That'll give us a seq of tree nodes. So finally we call .getText on all the resulting nodes in the list, to turn node objects into strings.

Easy enough:

(defn node-seq [x]
  (map #(.getText %)
   (tree-seq #(not (zero? (.getChildCount %)))
             #(.getChildren %)
             x)))

antlr-example.core> (node-seq (parse-expr "1 + 2 * 3 + 4"))
("+" "+" "1" "*" "2" "3" "4")

But that's no good. We lost our tree structure.

We'd rather have something like nested vectors or lists. It's easy enough to roll something by hand. This should do it:

(defn AST [node]
  (if (zero? (.getChildCount node))
    (.getText node)
    (let [children (map AST (.getChildren node))
          txt (.getText node)]
      (if txt
        (cons txt children)
        children))))

antlr-example.core> (AST (parse-expr "1 + 2 * 3 + 4"))
("+" ("+" "1" ("*" "2" "3")) "4")

That's better, but it's a list of strings. These strings all happen to be literal representations of Clojure objects, so a call to read-string in the proper places should give us something we can work with:

(defn AST [node]
  (if (zero? (.getChildCount node))
    (read-string (.getText node))
    (let [children (map AST (.getChildren node))
          txt (read-string (.getText node))]
      (if txt
        (apply list txt children)
        children))))

antlr-example.core> (AST (parse-expr "1 + 2 * 3 + 4"))
(+ (+ 1 (* 2 3)) 4)

Hey look, now we have something we can evaluate.

antlr-example.core> (eval *1)
11

Yeah I kind of planned that ahead. If you had a more complex grammar, you might not get away with something quite this easy.

Conclusion

The advantage of ANTLR is that it's mature, widely used, actively developed, and well-documented. (There's a whole book about ANTLR, The Definitive ANTLR Reference by Terence Parr.) There's also a lot of tooling available for it, not just ANTLR Works, but plugins for other Java IDEs.

The disadvantage is that it's a Java library, and as always, there will be some friction when consuming it from Clojure. But it's not that bad.

For pure-Clojure parser generator alternatives, there are fnparse and clojure-pg, neither of which I've tried much.

Vim and plaintext data files

Mon, 12 Jul 2010 10:42:48 -0700

I use Vim to work with plaintext datasets. Here are some habits and code snippets I've picked up which make data files a bit easier to deal with.

Tab-delimited files

If you have a file full of tab-separated values, the columns may not line up very well due to the variable display-length of tabs. You'll often end up with something that looks like this:

foo bar
longvaluehere    quux

But things will line up if you set tabstop to a value greater than the longest column.

:setlocal tabstop=16

Then it'll look like this:

foo             bar
longvaluehere   quux

Now you can use visual-block mode to add and remove columns easily.

If you're working with TSV files, you'll want to set the listchars option in such a way that tabs are displayed specially. I use this, stolen/adapted from here:

:set listchars=eol:\ ,tab:»-,trail:·,precedes:…,extends:…,nbsp:‗

Remember also that you can insert a literal tab into your file no matter what you have expandtab set to, by hitting CTRL-v first.

Those things are just about all I need to work with TSV files comfortably.

Dealing with long lines

Sometimes I'll run some data through a statistics program and it'll tell me "Invalid character at line 576, column 9438". How do you jump to that location in your data? Easily enough in normal mode:

576G9438|

G jumps you to a line, | jumps you to a column. A good mnemonic to remember these: "g = goto line" and "| looks like a column".

Suppose (like me) you turn scrollbars off in your GVim window, or you work from a terminal. How do you scroll horizontally? With zL and zH. But those are hard to type and harder to remember, so I map them to CTRL+Shift+Right and Left.

nnoremap <C-S-Right> zL
nnoremap <C-S-Left> zH

If you have a lot of long lines, it behooves you to set up a mapping to turn soft line-wrapping on and off quickly:

nnoremap <Leader>w :setlocal nowrap!<CR>

Now you can jump back and forth between wrapped and unwrapped views of your data via \w.

Diffs on long lines

When using vimdiff, ]c and [c will jump you to lines that contain differences. But if your lines are thousands of characters long, it doesn't always help to know that two lines are different: you want to know where they differ. ]c puts you at the first column in the line, which isn't helpful.

This function will jump you to the column where the difference between two lines starts:

function! IsDiff(col)
    let hlID = diff_hlID(".", a:col)
    return hlID == 24
endfunction

function! FindDiffOnLine()
    let c = 1
    while c < col("$")
        if IsDiff(c)
            call cursor(".", c)
            return
        endif
        let c += 1
    endwhile
endfunction

This seems fragile, so if you know a better way please leave a comment and let me know.

Diff a buffer against itself

Sometimes I have files with lots of lines of data, and some lines might be duplicate. It's really easy to get rid of the duplicate lines:

:sort u

But what if I want to see what lines were just removed? I don't know of a built-in way to do it, so I use this simple function:

function! MarkDuplicateLines()
    let x = {}
    let count_dupes = 0
    for lnum in range(1, line('$'))
        let line = getline(lnum)
        if has_key(x, line)
            exe lnum . 'norm I *****'
            let count_dupes += 1
        else
            let x[line] = 1
        endif
    endfor
    echomsg count_dupes . " dupe(s) found"
endfunction

That'll put a bunch of asterisks at the beginning of every line that's a duplicate of a previous line.

Fix broken punctuation

Ever had to work with textual data that someone else sent you in MS Word or Excel? I have. It hurts. If you copy/paste from an MS document into a plaintext file, you'll probably end up with a bunch of funky unreadable or undisplayable characters.

This function fixes the most common Word-spawned "smart" punctuation characters that you're likely to run across:

function! FixInvisiblePunctuation()
    silent! %s/\%u2018/'/g
    silent! %s/\%u2019/'/g
    silent! %s/\%u2026/.../g
    silent! %s/\%uf0e0/->/g
    silent! %s/\%u0092/'/g
    silent! %s/\%u2013/--/g
    silent! %s/\%u2014/--/g
    silent! %s/\%u201C/"/g
    silent! %s/\%u201D/"/g
    silent! %s/\%u0052\%u20ac\%u2122/'/g
    silent! %s/\%ua0/ /g
    retab
endfunction

This function was built up during years of choking on special characters that crept up in my data, so I'm sure I'll be adding more to it in the future.

\%u#### here is a regex escape to represent a character as a hexidecimal Unicode codepoint.

Many of these characters show up as blank (whitespace) if you lack the font to display them. If you ever run across a character you can't see and you want to inspect it, put the cursor on it and do

:ascii

That's a bad name for the command, since it works on Unicode characters too. That'll give you the numeric code you can use in a regex to replace them all with something sane.

In which border-radius is abused

Mon, 05 Jul 2010 15:54:24 -0700

I threw together a new blog layout today. I scaled back the level of cows a bit (just a bit, don't worry!) Criticism / feedback welcome. (IE-related feedback should be dropped off in the circular file by my desk.)

In what is surely a prelude to the future of the internets, I'm abusing border-radius pretty heavily in this layout. border-radius is likely to become the new marquee HTML tag or text-shadow CSS attribute: Maybe an OK idea at first, but then everyone uses it so much it makes your eyes bleed.

So I figured I'd best get my border-radiusing in early while it's still cool. IE8 users, you still get pointy corners. Sucks to be you.

Also, if you have any ideas for features I should implement for cow-blog, please let me know. I've been crawling the internet looking at blogs for ideas of things to implement and features to steal, but I'm running out of ideas. It does everything I want now, but I'm not a reader.

Goodbye Tokyo Cabinet, hello PostgreSQL?

Tue, 29 Jun 2010 11:32:58 -0700

The first version of this blog used MySQL; then I switched to Tokyo Cabinet. But now I've switched back to PostgreSQL. Here's why.

Why did I switch to TC to begin with?

There weren't any good ORM-type libraries for Clojure at the time (over a year ago). So there was a bit of an impedance mismatch trying to query and work with my data. In the DB I have separate tables for posts, comments, tags, categories. But 90% of the time I want to fetch a post and end up in Clojure with all related tags, comments etc. (A JOIN won't work here without a lot of work to un-mangle the results; you usually need multiple queries.)

With TC I could store anything in the DB, so I just dumped a post into the DB as serialized hash-maps with all the comments, tags, and categories as sub-keys. So querying was easy. (Or was it? More below.)
MySQL was really slow. This is largely because my queries were terrible, as I tried to solve the problem from #1 via brute force. TC on the other hand is fast.
I tried to solve #2 by using a Clojure ref as a cache. But tying the STM to a database's transaction system is as far as I know difficult or impossible right now (per many threads on the mailing list). I had a lot of potential race conditions, which (as far as I know) never bit me, but probably would've eventually. I had to deal with keeping the cache up-to-date as comments were posted and posts were added and deleted and renamed. Remember:

"There are only two hard problems in Computer Science: cache invalidation and naming things." --Phil Karlton

So why did I stop using TC?

I have no idea how to use a key/value store database properly. TC will take anything you dump into it, which is both a strength and a weakness.

There's a lot of crap you have to do by hand that a proper database does for you. Consider checking for null values, for example; I ended up with a lot of nils in my data because my validations weren't 100% foolproof, or because I imported data via code that didn't run the validations and I never noticed.. Or enforcing uniqueness of values; I had tag objects in the database with the same key but different values (due to capitalization differences), which screwed up a lot of stuff.

On the other hand, there's a lot of information about how to use RDBMS properly, and I have a lot of experience with it already. Constraints are easy to set up. Columns have types, which is nice. (Strange that I gravitate toward statically-type databases while I gravitate toward dynamically-typed programming languages.)
I have to compile and install Tokyo Cabinet by hand on my Linux distro. It's probably not worth distro maintainers to maintain a package that so few people use. MySQL and PostgreSQL have lots of people working on keeping them running OK on most Linux distros.
Some kinds of queries were still awkward in TC. "Give me post X" was great: I'd also get all the tags, categories etc. for free. But then how do you query to get all tags across all posts? Or all comments? Fetch all the posts and iterate over them, collecting their tags, then uniquify the resulting list? Not so pretty, and not so fast. So I was back to caching again, which still gave me nightmares about race conditions and dirty data.

So now why am I using an RDBMS again?

An RDBMS is exactly what I really need, if I could just query the thing concisely and get it to run fast. Thankfully there are some ORM-like libraries for Clojure in the works nowadays, already usable for a hobby project like this blog. There are clj-record, Carte, ClojureQL, and my own Oyako, and possibly others in the works.
For my tiny blog's database, Oyako gives me slightly slower performance than TC, but along the same order of magnitude, which is good enough.
Via Oyako I can (fairly concisely) fetch posts and get the associated tags, comments etc. But I can also easily fetch all tags, or all comments, since they're in their own tables. The "relational" part of RDBMS does come in handy sometimes.

Summary version

I switched to TC to begin with because I was using SQL wrong, and it was too slow and clumsy. Once I figured out how to use SQL correctly, it was a no-brainer to go back.

Introducing Gaka

Mon, 28 Jun 2010 17:59:26 -0700

The CSS for my blog is now being generated via gaka, a CSS-generating library I wrote this afternoon. It's extremely simple, but it got the job done for me. I turned around 600 lines of CSS into around 250 lines of Clojure without much effort. It looks like this:

user> (require '(gaka [core :as gaka]))
nil
user> (def rules [:div#foo
                  :margin "0px"
                  [:span.bar
                   :color "black"
                   :font-weight "bold"
                   [:a:hover
                    :text-decoration "none"]]])
#'user/rules
user> (println (gaka/css rules))
div#foo {
  margin: 0px;}

  div#foo span.bar {
    color: black;
    font-weight: bold;}

    div#foo span.bar a:hover {
      text-decoration: none;}

Gaka is partly inspired by Sass, which I found very pleasant to work with recently. And it's partly inspired by Hiccup, which is a delicious way to generate HTML in Clojure.

There's more info and more examples on github.

New blog engine up and running

Wed, 23 Jun 2010 15:48:24 -0700

Well, my new blog is up and running. Sorry for the temporary lack of cows in my layout. I'm dogfood-testing the blog engine in a fairly vanilla state until I work out some of the bugs. This layout is based upon barecity, a minimalist Wordpress theme that I adapted easily enough to my blog.

As a bonus, I applied a dirty hack to my RSS feed that I think should help avoid screwing up people's RSS readers with duplicate entries.

I'll write again soon with some info about the blog engine and some things I learned writing it.

(As mentioned previously, here's the code.)

Breaking links is easy to do

Tue, 22 Jun 2010 23:23:27 -0700

I apologize in advance to everyone who subscribes to my blog's RSS feed, but this week your RSS reader is probably going to suddenly find 25 "new" posts from me.

My blog currently uses /blog/title as the URL scheme, with similar URLs for categories and tags etc. Soon, I'm probably going to change it to /blog/123/title, as part of the impending release of version 0.2 of my blog engine. (The code-in-progress is in a branch on github, for the daring and foolish among you.)

This way, I can change the title of a post without breaking everything. I have heretofore lacked this ability. It's easy to code, you just tell Compojure to ignore everything after the number in a route. Something like this:

(defroutes foo
  (GET ["/blog/:id:etc" :id #"\d+" :etc #"(/[^/]*)?"] [id]
    (pages/post-page id)))

It's only a few lines of code to change, but the ramifications are widespread. It'll instantly break every link to my blog, for example. At least it's pretty easy to set up a bunch of redirects in Compojure to avoid that. I think this'll work:

(require (blog [db :as db]
               [link :as link])
         (oyako [core :as oyako])
         (ring.util [response :as response]))

(defn redirect-post [name]
  (when-let [post (oyako/fetch-one db/posts :url name)]
    (response/redirect (link/url post))))

(defroutes redirect-routes
  (GET ["/blog/:name" :name #"[^/]+$"] [name]
    (redirect-post name)))

(Oyako here is the experimental ORM-like library I'm using to interface with PostgreSQL nowadays, having ditched Tokyo Cabinet.)

Changing my URL scheme is also going to mess up RSS though, because I (foolishly) used post URLs as the GUIDs in my RSS feed up to this point. This problem I don't know how to avoid. I might reduce the number of posts included in my feed temporarily, to limit the damage.

Vim regex - remove kind-of-matching lines

Sun, 20 Jun 2010 22:49:19 -0700

I have a file where every line starts with a number (followed by whitespace and a bunch of other stuff). Every number appears on either one or two lines, and if two, the second line always has a b after the number.

I need to delete every line for which there's a corresponding b line. But if there's no corresponding b line I want to leave the original line there.

Before:

123 foo bar
456 blarg
789 quux
123b foo baz
789b quux blurble

After:

123b foo baz
456 blarg
789b quux blurble

Except in my real file, I have a thousand lines and it'd take a year to do by hand. Vim to the rescue:

:sort
:%s/^\v((\d+).*\n)(\2b.*)/\3/

And that is why Vim is awesome. Can you think of a shorter way to do this, in Vim or Emacs?

Emacs creating zombie buffers

Mon, 14 Jun 2010 15:44:18 -0700

Note to self:

Using bleeding-edge swank-clojure and Slime, Emacs was creating buffers that didn't point to any file, every time C-c C-k wanted to display warnings. This was really quite annoying as it resulted in a mess of empty buffers being created. This fixed it:

(setq slime-highlight-compiler-notes nil)

Clojure, from a Ruby perspective

Wed, 09 Jun 2010 13:22:58 -0700

Fogus' recent article "clojure.rb" speculates about why there seem to be so many Ruby users adopting Clojure. As a Ruby user who adopted Clojure, I figured I'd write about my experiences.

What do Ruby and Clojure have in common, that would attract a Rubyist to Clojure? A lot. Obviously, this is somewhat subjective and I don't expect anyone else to agree, but this is what did it for me.

Semantic consistency

In Ruby, everything is an object. It makes it simple to write code without worrying much about what kind of thing you have. foo.some_method(1,2,3) will generally work for any foo.

In Clojure, everything is not an object, specifically because it inherits primitives from Java land (though this doesn't hurt much in the kind of everyday use I put Clojure to). But also because Clojure by design doesn't even attempt to be object-oriented.

Clojure does have abstractions though. For example there's an abstraction that says "this thing can be called like a function". And then you can treat any callable-thing as a function without worrying about what it is.

In Ruby, a lot of things are Enumerable, which means you can do foo.each{} and other similar things for a lot of different types of foo. Clojure has something similar with its seq abstraction. Similarly, most Clojure data types and many Java ones are seq-able, and most built-in core functions can iterate over the guts of various things using seqs. This includes regex matches, strings, directories of files, and so on.

Ruby-style OOP brings a lot of complexity and baggage which Clojure avoids by not being OOP. For example (ignoring Java for the moment), in Clojure land you don't have to worry about a member being public/private/protected, and there are few times when you have to worry about inheritance and class hierarchies. (And there's the whole thread-safety thing.) In Clojure data is stupid and immutable, and functions are just things that take input and give output, usually side-effect free. The separation is clean and this results in programs that are very easy to reason about.

Another example of consistency: Expressions. In both Ruby and Clojure, everything has a value. Things that are "statements" in other languages are instead expressions that return something. This alone makes a lot of programs just a little bit better/easier to write.

Aesthetics

Like Ruby, Clojure code tends to be terse and expressive.

Ruby reads like poetry, because it's mostly words and not so much punctuation. In Ruby, you have none of Perl's sigils, very little of C's semi-colon line-endings and curly-delimited blocks. Especially when you start omitting optional parentheses, it has a very minimalistic vibe which is appealing to many.

    def foo(bar)
      bar.each do |x|
        puts x
      end
    end

Clojure's s-expressions are another story, of course. Some love them, some hate them. Personally, I love them. The tired old trope about Lispers not paying attention to parentheses is true; after a while they blend into the background.

    (defn foo [bar]
      (doseq [x bar]
        (prn x)))

What I see:

    defn foo [bar]
      doseq [x bar]
        prn x

Clojure has the same minimalistic feel to it, in my eyes. It also helps that so many Clojure function names are short and concise. And being able to use punctuation like ? and ! in variable/function names (true? and false? are function names), hyphens instead of underscores... these things help Clojure read smoothly.

But along with aesthetics, in Clojure you get the benefits of s-exps: no order of operations to deal with, and absolute consistency. Everything is (function param1 param2 param3). Look at how many syntax rules you have to memorize for the Ruby code above to make sense. Dot means method call, blocks are do/end delimited and have those weird pipes in there, etc.

Literals and syntax sugar

Ruby has literal syntax for many types. This includes:

:symbols
[arrays]
{:hash => :maps}
/regex/
do block end and {block}

This really is a big deal. I'm spoiled and I can't use a language without these things nowadays. It saves a ton of typing and it makes those things stand out in the code, making it both easier to write and to read. I use those structures all the time in every program I write, so they should have a terse representation.

Clojure has literal support for the same types as Ruby, and they even look mostly the same (with #"regex" being a change I can live with). And then it also has (among others):

#{sets}
#(function-literals %)
'(quoted forms)
`(quasi-quoted ~forms)

Is this a contradiction of my last point? What happened to s-expressions and consistency? Well, in Clojure, the reader shortcuts are just sugar that reduce to s-exps. You can avoid all use of that sugar if you hate it. (hash-map :key "val"), (vector 1 2 3), (quote foo) etc.

But more importantly, let's draw a(n arbitrary) distinction between "good" syntax and "bad" syntax.

Clojure's reader-macro sugar makes your code shorter, but doesn't change the structure of your code. Take a function call (f x y z), and you can always substitute a vector or regex literal or quoted form into it. (f [vector] #{set} #"regex"). The syntax sugar is very local, very self-contained. It doesn't leak into the surrounding code. And of course you can combine them in nearly arbitrary ways: '[quoted vector], {:hash-map-containing-a #{:set 'of #(functions)}}. This is "good" syntax.

Compare this to things like the x ? y : z construct, or heredocs. These things are not as orthogonal. They not only mean something on the "inside", they also influence and interact with the code before and after them, thanks to precedence rules and special parsing rules. Can you stick a heredoc in the middle of a function call? Maybe (I don't even know), but have fun with the indentation and line-breaks if so. When should you use do/end and when should you use {} for blocks? When do you need parens around your ternary if-then-else construct and when don't you? When do you need to use and and when &&? That's the "bad" kind of syntax. Sometimes, maybe even most of the time, it makes your code shorter, but there are a lot of rules to memorize and you never know when you'll be bitten.

Clojure largely avoids the "bad" syntax while taking advantage of the "good". Reader macros make your code shorter and visually easier to scan, but they rarely require you to do backflips to get your code to compile or run properly.

First-class functions

Ruby's blocks and yield and friends let you deal with first-class functions. This is a huge step in the Lisp direction already, and it's one of the things that makes Ruby great. But there are limits. Blocks use funky, special syntax, and in idiomatic Ruby, you will pass around only one block per method. There's the whole lambda and proc mess, and then there are methods-as-objects which are different still. And a lot of Ruby just calls .send on an object and passes in a method name as a symbol.

Being a Lisp, Clojure takes this a bit further. First-class functions are ingrained in nearly everything you do in Clojure. And they are easy to define and easy to call. Define f via defn (to make it top-level), fn (for a local function), or #() (sugar for fn), and then call it like (f).

Clojure also takes advantage of some functional-programming mainstays like partial and complement and comp(osition). We're not in full-blown Haskell territory, but it's a lot more FP than idiomatic Ruby.

And hash-maps, vectors, sets, keywords, and symbols are also callable as functions in Clojure. ({:foo 1} :foo) => 1. Many things can be treated as functions.

Metaprogramming

In Ruby you can mess with the innards of any class you want. There are facilities for defining methods dynamically, opening and inspecting classes at runtime, catch-all handlers for undefined methods, and all kinds of other dark magic. But again there are limits... Ruby needs to make use of eval to get certain things done. And monkey-patching is a shotgun aimed at your foot.

Well, if you like metaprogramming, Lisp macros are top of the line. You can abstract away boilerplate with a vengeance. Macros are the ultimate application of DRY.

Clojure doesn't deal much with classes, so there isn't much of that kind of introspection, but the Lisp principle of code-as-data enables a kind of introspection that you won't find in Ruby. The line between compile-time and run-time is very blurry, which enables all kinds of magic.

Java itself does offer Ruby-style reflection and such, if you need it, but you won't often, while in Clojure land.

Multimethods (and soon, protocols and defrecord) let you avoid monkey-patching and get some of the same kinds of "extend a class" things done in a saner and safer way.

So?

Fogus suspects:

Ruby programmers being the adventurous lot to begin with, are not satisfied with “halfway to Lisp”. Instead, they want it all.

This is true in my case. I like Ruby largely insofar as it borrowed and adapted many great features of Lisp. It only makes sense that I would like Clojure, which takes most of those things one step further. Clojure in particular, as a "modern" Lisp with vaguely Ruby-like syntax in certain places, is an obvious choice.

On top of that, Clojure is fast, thanks to the JVM. Ruby has JRuby too, but vanilla Ruby is not known for its speed. Clojure integrates with a REPL in a way that Ruby really doesn't, making interactive development enjoyable. Clojure is a compiled language, which has benefits for deployment. And again, there's the whole thread-safety thing. Clojure is awesome for writing sane, safe multi-threaded programs. These things are rather appealing.

I do still use Ruby though. Ruby is great for scripting, Clojure not so much, thanks to the JVM startup time, among other things. Ruby can be banged out quickly in any editor, but Clojure isn't much fun to edit in any editor that lacks good paren-matching support and REPL integration.

Rubygems offers dead-simple install of a ton of libraries, whereas Clojure is still working out the details of a standard build tool and install tool. Ruby has a library for anything, and while Clojure can use Java libraries, Java libraries tend to be huge and feature-rich, sometimes too huge for one-off tasks where a small Ruby library is a perfect fit.

Emacs isn't for everyone

Tue, 08 Jun 2010 11:26:15 -0700

Chas Emerick recently posted the results of his State of Clojure survey. It turns out that the (self-selected) group of Clojure-using respondents happen to prefer Emacs as their IDE of choice, eclipsing all other editors by a large margin.

Chas then has this to say:

I continue to maintain that broad acceptance and usage of Clojure will require that there be top-notch development environments for it that mere mortals can use and not be intimidated by...and IMO, while emacs is hugely capable, I think it falls down badly on a number of counts related to usability, community/ecosystem, and interoperability.

As an avid, die-hard Vim and Emacs user for life, I'm going to agree.

Mere mortals?

Emacs isn't difficult to learn. Not in the sense of requiring skill or cleverness. It is however extremely painful to learn. I think there's a difference.

The key word is tedium. Learning Emacs is a long process of rote memorization and repetition of commands until they become muscle memory. If you're smart enough to write programs, you can learn Emacs. You just have to keep dumping time into the task until you become comfortable.

Until you're comfortable, you face the unpleasant task of un-learning all of your habits and forming new ones. And you're trying to do this at the same time you're undertaking another, even harder task: writing programs. And if you're a new Clojurist, and you're learning Emacs and Clojure from scratch at the same time, well, get the headache medication ready.

As a programmer and someone who sits in front of a computer 12+ hours a day, I consider myself pretty flexible and capable of picking up a new user interface. As someone who had been using Vim for years prior to trying Emacs, I considered myself more than capable of learning even a strange and foreign interface. I'd done it once before.

But learning Emacs still hurt. Oh how it hurt. I blogged while I was learning it, and you can see my pain firsthand. I sometimes hear people say "I tried Emacs for a whole month and I still couldn't get it". Well, it took me over a year to be able to sit down at Emacs and use it fluidly for long periods of time without tripping over the editor.

To be fair, I'm talking here about using Emacs as a programming environment. Using Emacs as a Notepad replacement could be learned in short order. C-x C-f, C-x C-s, or use the menus, there you go. Using it comfortably as a full-fledged IDE is significantly harder and requires you to touch (and master) many more features. Syntax highlighting, tab-completion, directory traversal and cwd issues, enabling line numbers, version-control integration, build tool integration, Emacs' funky regex syntax for search/replace, Emacs' bizarre kill rings and undo rings, the list goes on. These things are very flexible in Emacs, which is a great thing, but it's also an impediment to learning how to configure and use them. There's no getting around the time investment.

And it's not just a matter of learning some new keyboard shortcuts. There's a new vocabulary to learn. You don't open files, you visit them. What's a buffer? What's a window? (Not what you think it is.) What's a point? What's a mark? Kill? Yank? "Apropos"? Huh? C-c M-o means what exactly? My keyboard doesn't have a Meta key. Yeah, you can use CUA mode and get your modernized Copy/Cut/Paste shortcuts back, but that's the tip of the iceberg. It's hard even to know where to begin looking for help.

Yeah, Emacs came first, before our more common and more modern conventions were established, and that explains why it's so different. That doesn't change the fact that Emacs today is a strange beast.

Community and ecosystem

Personally I find the Emacs community to be a pretty nice bunch. In the highest tradition of hackerdom and open source software, Emacs users seem to be eager and willing to share their elisp snippets and bend over backwards to help other people learn the editor. I got lots of help when I was struggling and learning Emacs.

The Emacs wiki is an awesome resource. The official documentation is so complete (and so long) that it leaves me speechless sometimes. And there are a million 3rd-party scripts for it. Whatever you want Emacs to do is generally a short google away.

If there's anything wrong with the Emacs community, it'd be people who take Emacs evangelism overboard. The answer to "I don't want to have to use Emacs to use your language" can't be "Be quiet and learn more Emacs", or "If you're too dumb to learn Emacs, go away". In some communities there is certainly some of that. But thankfully I don't see it much in the Clojure community. Let's hope it stays that way.

Interoperability

Once someone spends the time to write a suitable amount of elisp, Emacs can interoperate with anything. I think so many people use SLIME for Clojure development precisely because it interoperates so darned well with Lisps. SLIME is amazing. You probably can't beat Paredit either, and Emacs' flexibility is precisely what makes things like Paredit possible.

The problem is the amount of time you have to spend to get that interoperability set up and to learn how to use it. After two years of using Emacs and Clojure together, every once in a while I still find myself bashing my face on my desk trying to get the latest SLIME or swank to work just right, or trying to get a broken key binding fixed, or tweaking some other aspect of Emacs that's driving me crazy. One day, curly braces stopped being recognized as matched pairs by Paredit. Why? No idea; I fixed it, but it was a half hour of wasted time.

Emacs is good at integrating with Git too. So good that there are four or five different Emacs-Git libraries, each with a different interface and feature set. I gave up eventually and went back to using the command line. (You can embed a shell / command line right in Emacs. There are three or four different libraries to do that too.)

The wealth of options of ways to do things in Emacs is simultaneously a good thing, overwhelming and confusing. If all you want is something that works and gets out of your way, too many options can be worse than one option, even if that one option isn't entirely ideal.

Emacs' Java interop, I know nothing about. Almost certainly, Emacs can come close to a modern Java IDE for fancy features like tab-completion and document lookups and project management. But how long is it going to take you to figure out that tab-completion is called hippie-expand in Emacs? That and a million other surprises await you.

What's my point?

There was a pithy quote floating around on Twitter a while back (I think quoting Rich Hickey):

One possible way to deal with being unfamiliar with something is to become familiar with it.

That's true, and you could say that of Emacs. I strongly believe that when it comes to computers, there's no such thing as "intuitive". There's stuff you've already spent a lot of time getting used to, and there's stuff you haven't.

But certain things require more of a time investment than others. Could I learn Clojure if all the keywords were in Russian or Chinese instead of my native English? Sure, but it'd take me a long time. I'd certainly have to have a good reason to attempt it.

I learned Emacs partly because it was hard. I saw it as a challenge. It was fun, yet painful, but more pain, more glory. Mastering it makes me feel like I've accomplished something. I'd encourage other people to learn Emacs and Vim too. I think the benefits of knowing them outweigh the cost and time investment of learning them.

But I didn't learn Emacs with the goal of being productive. I learned it for the same reason some people build cars in their garages, while most people just buy one and drive it to and from work every day. I learned Emacs because I love programming and I love playing with toys, and Vim or Emacs are as nice a toy as I could ask for. (I love programming enough to form strong opinions and write huge blog posts about text editors.) For me, productivity was a beneficial side-effect.

There are only so many hours in a day. There are a lot of other challenges to conquer, some of which offer more tangible benefits than Emacs mastery would get you. Mastering an arcane text editor isn't necessarily going to be on the top of the list of everyone's goals in life, especially when there are other editors that are easier to use and give you a significant subset of what Emacs would give you. We have to pick our battles.

So I understand when people say they don't want to learn Emacs. I think maybe so many Clojurists use Emacs right now because we're still in the early adopter stage. If you're using Clojure today, you're probably pretty enthusiastic about programming. You're likely invested enough to be willing to burn the required time to learn Emacs.

If Clojure becomes "big", there are going to be a lot of casual users. A casual user of Clojure isn't going to learn Emacs. They're going to silently move on to another language. And I really think that new blood is vital to the strength of a community and necessary for the continued healthy existence of a programming language.

So Clojure does need alternatives. I'll stick with Emacs myself, but there should be practical alternatives. I'd encourage the Clojure community to continue to support and enjoy Emacs, but don't push it too hard.

Emacs: Yank lines as lines

Fri, 21 May 2010 12:07:13 -0700

One thing nice about Vim is manipulating whole lines at a time. dd deletes a line (including trailing newline), regardless of where the cursor is on the line. Then, p puts that line (with its newline) as a new line after the current line, and P puts it above the current line, again regardless of where your cursor is at the moment. (It also jumps the cursor to the beginning of the text you just inserted, which is nice.)

Emacs has kill-whole-line (C-S-Backspace) which is like Vim's dd. But I didn't find an equivalent of p and P. So here's my version:

(defun yank-with-newline ()
  "Yank, appending a newline if the yanked text doesn't end with one."
  (yank)
  (when (not (string-match "\n$" (current-kill 0)))
    (newline-and-indent)))

(defun yank-as-line-above ()
  "Yank text as a new line above the current line.

Also moves point to the beginning of the text you just yanked."
  (interactive)
  (let ((lnum (line-number-at-pos (point))))
    (beginning-of-line)
    (yank-with-newline)
    (goto-line lnum)))

(defun yank-as-line-below ()
  "Yank text as a new line below the current line.

Also moves point to the beginning of the text you just yanked."
  (interactive)
  (let* ((lnum (line-number-at-pos (point)))
         (lnum (if (eobp) lnum (1+ lnum))))
    (if (and (eobp) (not (bolp)))
        (newline-and-indent)
      (forward-line 1))
    (yank-with-newline)
    (goto-line lnum)))

(global-set-key "\M-P" 'yank-as-line-above)
(global-set-key "\M-p" 'yank-as-line-below)

Just one more step along the path to Vimmify my Emacs setup. Emacs has some weird edge cases because you can move the cursor one "line" past the last real line in the file. But I think I worked out something comfortable for myself.

PS: I've written about this before, but if you use C-S-Backspace a lot in Emacs on Linux, I highly recommend putting this into your X11 config:

Option "DontZap" "True"

It's really easy to mix up C-S-Backspace and C-M-Backspace (the latter of which kills your X server). It's not fun to mix those up. Not fun at all.

PPS: This thread on Stack Overflow has some Emacs equivalents of Vim's o and O which are pretty nice too.

Vim vs. Emacs: Indenting text before copying

Thu, 13 May 2010 16:21:27 -0700

I use Markdown on my blog for posts and comments, and I post at other sites that use Markdown (e.g. Stack Overflow). In Markdown, text indented four spaces is displayed as code, in pre tags.

I find myself often writing code in Vim or Emacs and needing to copy/paste it into a browser in a Markdown-suitable way.it back. This is easy to do in Vim and Emacs, only a few keystrokes. But "a few" is still greater than "one", so the heck with that. Let's script it.

Vim version

This keymapping in Vim will do it all for me:

vmap <Leader>y :s/^/    /<CR>gv"+ygv:s/^    //<CR>

One clumsy thing about Vim is needing to restore the previous visual selection after each regex-replacement. I could use the marks '< and '> as ranges to :s instead, but that's more typing than simply doing gv in the mapping. Copying to the system clipboard is easy because Vim has a register "+ for that purpose.

This took me maybe 45 seconds to write, probably due to being pretty familiar with Vim already. But in Vim, mappings are easy. You just type the characters that you'd type if you were doing it manually.

Emacs version

Trying to do the same in Emacs was painful. My Emacs-fu is sorely inadequate, compared to my Vim-jitsu. This seems to work, but ugh:

;; adapted from http://www.emacswiki.org/emacs/.emacs-ChristianRovner.el
(defun expand-region-linewise ()
  (interactive)
  (let ((start (region-beginning))
        (end (region-end)))
   (goto-char start)
   (beginning-of-line)
   (set-mark (point))
   (goto-char end)
   (unless (bolp) (end-of-line))))

(defun markdown-copy ()
  (interactive)
  (save-window-excursion
   (save-excursion
     (save-restriction
       (expand-region-linewise)
       (narrow-to-region (region-beginning) (region-end))
       (goto-char (point-min))
       (replace-regexp "^" "    ")
       (clipboard-kill-ring-save (point-min) (point-max))
       (goto-char (point-min))
       (replace-regexp "^    " "")))))

Writing this involved a long journey through the Emacs documentation.

One difficulty was getting Emacs to play friendly with my fat-fingered region-marking. I don't always highlight from the beginning of the first line to the end of the last. That's why Vim's visual-line mode is awesome; the cursor can be anywhere on the line, it still selects the whole line. The handy function above (found on the Emacs wiki) takes care of that. I don't know how long it would've taken me to come up with that on my own.

Then it was a matter of rooting through millions of Emacs functions until I found the ones that move the point around and copy text to the clipboard. Along the way I discovered the wonders of "narrowing", which limits Emacs to work on some region of text, and all those macros to undo the messes I make while moving around.

Maybe I could've done this with an Emacs keyboard macro, and then called apply-macro-to-region-lines. And maybe I could use append-next-kill to build up the indented text one line at a time. But my efforts to do this or anything like it failed horribly.

In any case I thought it was an interesting comparison. Improvements to either version are welcome.

EDIT: This works too (thanks Holger Durer):

(defun markdown-copy ()
  (interactive)
  (save-excursion
    (expand-region-linewise)
    (indent-rigidly (region-beginning) (region-end) 4)
    (clipboard-kill-ring-save (region-beginning) (region-end))
    (indent-rigidly (region-beginning) (region-end) -4)))

Printing a nicely formatted plaintext table of data in Clojure

Fri, 16 Apr 2010 11:49:02 -0700

I use this fairly often while data-munging, when I want to quickly view a list of hash-maps of data in a simple table format. Usually this is coming out of database. Maybe someone will find it useful.

(defn table
  "Given a seq of hash-maps, prints a plaintext table of the values of the hash-maps.
  If passed a list of keys, displays only those keys.  Otherwise displays all the
  keys in the first hash-map in the seq."
  ([xs]
     (table xs (keys (first xs))))
  ([xs ks]
     (when (seq xs)
       (let [f (fn [old-widths x]
                 (reduce (fn [new-widths k]
                           (let [length (inc (count (str (k x))))]
                             (if (> length (k new-widths 0))
                               (assoc new-widths k length)
                               new-widths)))
                         old-widths ks))
             widths (reduce f {} (conj xs (zipmap ks ks)))
             total-width (reduce + (vals widths))
             format-string (str "~{"
                                (reduce #(str %1 "~" (%2 widths) "A") "" ks)
                                "~}~%")]
         (cl-format true format-string (map str ks))
         (cl-format true "~{~A~}~%" (repeat total-width \-))
         (doseq [x xs]
           (cl-format true format-string (map x ks)))))))

Then you can do this:

user> (def data [{:name "Brian" :job "Code monkey" :age 29}
                 {:name "Bob" :job "Janitor" :age 97}
                 {:name "Johnny McLongname" :job "None" :age 3}])
#'user/data
user> (table data)
:name             :job        :age 
-----------------------------------
Brian             Code monkey 29   
Bob               Janitor     97   
Johnny McLongname None        3    
nil
user> (table data [:age :name])
:age :name             
-----------------------
29   Brian             
97   Bob               
3    Johnny McLongname 
nil

Installing Clojure with Emacs and SLIME: so easy, yet so hard

Wed, 14 Apr 2010 16:02:55 -0700

How do you install Clojure with SLIME/Emacs support? The answer is either really easy or really hard, depending.

Easy!

If you're new to Clojure and want to get it working fast, you are in luck. Go to swank-clojure's home on github and follow the directions to install it via ELPA. This is very easy and fast and it works.

If you want more documentation, there is (thankfully) a single, central, official-ish location where documentation for installing Clojure is slowly but surely accumulating. You can read about it at http://www.assembla.com/wiki/show/clojure/Getting_Started. This site includes instructions for Emacs, Netbeans, Eclipse, Maven, and everything else under the sun. This is a great resource, and attempts to solve the problem of out-of-date Clojure documentation. (This problem is natural given Clojure's young age and rapid growth, but it's painful for people trying to get started, I bet.)

Or is it...

This is all great. But if you want to do anything non-standard, you run into a bit of a wall. How do you install Clojure+SLIME on Windows? I find that sometimes it works, but sometimes things mysteriously fail, and I have no idea where even to begin to debug it, because there's so much automation going on in the background nowadays. What if you want to build a .emacs.d directory you can simply copy from one system to another and have everything work? (You can do this with most Emacs libs.)

Or how do you install and use a bleeding-edge Clojure or clojure-contrib tree with SLIME? There's good reason to want to, because it has some awesome new features, e.g. deftypes and protocols and rest-parameter destructuring. But most of the "official" methods of installing things give you stable versions of everything.

And then, you generally end up with a bunch of opaque .jar files, auto-downloaded all over your system, which isn't helpful if you want to hack on anything yourself. Most of these jars end up in ~/.m2 nowadays, I think? Used to be ~/.swank-clojure or ~/.clojure or something? And ~/.emacs.d/elpa gets stuff too? Color me continually confused.

The automated build tools we have nowadays (ELPA, Lein, Maven) are awesome when they work, but extremely painful when they fail or when you don't know which knobs to turn when you want to step slightly outside of the mainstream. This is incidental complexity if I ever saw it. While it's far more manual labor, I find it to be so much simpler to download things myself, build everything myself, put it all somewhere on my system myself, and configure Emacs to tell it where to find everything.

Well, this site is as close as I can find to good up-to-date documentation, and what I did to get set up was almost identical to this. But it will likely stop working in a month or two. There's also a recent thread on the Clojure mailing list which has some more tips (and valid complaints).

I will push my Emacs setup to my github repo once I get the kinks worked out. But if you want to get bleeding-edge everything working yourself, all I have to say is God help you, because it's largely unsupported. Be prepared to hack and patch until it works. I'm looking forward to the day when most of the kinks are worked out and things settle down a bit.

Getting list of referers out of Apache logs

Sun, 21 Feb 2010 12:46:34 -0800

I use Google Analytics, but it has a noticeable lag in updating its information. When my site is being hammered, I'd like to see where all the traffic is coming from. It'd also be nice to see how many hits my RSS feed is getting, and how many images and static files are being direct-linked, which Google Analytics currently isn't tracking for me at all.

So this script will look in my Apache logs and print referers for some URL, thanks to ApacheLogRegex:

#!/usr/bin/ruby

require 'apachelogregex'

raise "USAGE: #{$0} log_filename desired_url" unless ARGV[0] and ARGV[1]

format = '%v:%p %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"'
parser = ApacheLogRegex.new(format)
pat = Regexp.new(ARGV[1])
refs = {}

File.readlines(ARGV[0]).each do |line|
  x = parser.parse(line)
  if pat.match(x["%r"])
    r = x["%{Referer}i"]
    refs[r] = (refs[r] || 0) + 1
  end
end
refs.sort_by{|k,v| -v}.each do |ref,count|
  puts "%s: %s" % [count,ref]
end

I used to use awstats for this, but it was too heavyweight and a hassle to set up and keep running. Google Analytics is a no-brainer to use, even though the accuracy isn't as good as parsing Apache logs. At least I get an idea of which of my blatherings people are most interested in.