briancarper.net (λ)

Shuffle lines in Vim

Thu, 07 Jul 2011 14:07:41 -0700

In a pinch, I needed to randomize the order of a few thousand lines of plain text. In Linux you can just pipe the file through sort, even right inside Vim:

:%!sort -R

But I was stuck on Windows. And I don't know how to randomize a file in native Vim script. But doing it in Ruby is pretty easy, and luckily, Vim has awesome Ruby support. Tne minutes' work and a few peeks at :h ruby and we have a successful, working kludge:

function! ShuffleLines()
ruby << EOF
    buf = VIM::Buffer.current
    firstnum =  VIM::evaluate('a:firstline')
    lastnum = VIM::evaluate('a:lastline')
    lines = []
    firstnum.upto(lastnum) do |lnum|
      lines << buf[lnum]
    end
    lines.shuffle!
    firstnum.upto(lastnum) do |lnum|
      buf[lnum] = lines[lnum-firstnum]
    end
EOF
endfunction

2011-07-07 23:32 - Edited to remove a superfluous line.

2011-07-09 21:33 - Wrong parameter for sort, oops.

Keyword Arguments: Ruby, Clojure, Common Lisp

Fri, 24 Jun 2011 17:22:48 -0700

And suddenly I return to blogging, rising from the ashes like some kind of zombie phoenix. Turns out writing a book is a good absorber of time, like some sort of heavy-duty temporal paper towel. Now that I've gotten the terrible similes out of my system, let's talk about keyword arguments, one of my favorite features in any language that supports them.

Ruby, Clojure, and Common Lisp are all languages I enjoy to some degree, and they all have keyword arguments. Let's explore how keyword args differ in those languages.

Why keyword arguments?

Why are keyword arguments good?

You can omit arguments.
You can supply arguments in an arbitrary order.
Arguments are labeled, so you know what argument means what.

Positional arguments require mentally lining up the 7th argument in your function call with the 7th argument in the function signature, and so on. Keyword arguments become more and more attractive the more arguments you have in your function signature.

Keyword arguments trade a bit of verbosity for added explicitness, clarity and reduced mental burden. (Kind of like Lisps do overall. Fancy that.)

Ruby

Ruby doesn't have special support for keyword arguments. But Ruby likes its hashes, so you can just pass one in as an argument to a function. As some syntax sugar, if you pass a "flat" list of :key => val pairs, Ruby slurps them all together and stuffs them into a hash for you.

def foo(arg)
  p arg
end

foo({:x => 123})                  # => {:x=>123}
foo(:x => 123)                    # => {:x=>123}

With even more added sugar, you can leave off the parens in Ruby function calls. So this is pretty common in Ruby:

foo :x => 123                     # => {:x=>123}

How nice and punctuation-less. But then things get ugly. What about this?

foo {:x => 123}

That won't even compile. Ruby thinks {:x => 123} is a code block, and a bare key/value pair isn't valid syntax as the first thing in a code block. You need the parens. A bit unfortunate, but it gets worse...

def bar(arg1,arg2)
  puts "#{arg1} #{arg2}"
end

bar :x => 123, :y => 456          # Runtime error
bar {:x => 123}, {:y => 456}      # Won't compile

In the first example, all of the key/value pairs are slurped into one hash and end up in arg1. There's nothing left for arg2, so you get a "wrong number of arguments" exception. The second example won't even compile, of course, because again Ruby thinks {:x => 123} must be a code block with invalid syntax.

It gets worse if you change the argument list for bar slightly...

def bar2(arg1 = {}, arg2 = {})
  puts "#{arg1} #{arg2}"
end

This way, you don't even have to supply any arguments. This is nice, if all of your argument are optional.

bar2    # => {} {}

Now suppose you want arg1 to be {:x => 123}, and arg2 to be {:y => 456}. You might naively try this:

bar2 :x => 123, :y => 456         # => {:x=>123, :y=>456} {}

Oops, it all got dumped into arg1, and now instead of a missing argument error, arg2 silently ends up with a default, empty hash. You have to explicitly pass in an empty value for arg1 so that everything is slurped into arg2.

bar2 {}, :x => 123, :y => 456     # => WRONG!  Ruby thinks {} is a code block again.

bar2({}, :x => 123, :y => 456)    # => {} {:x=>123, :y=>456}

So much for syntax sugar. You might think you'd be unlikely to find this kind of thing in the wild, but in Ruby on Rails for example, there are quite a few functions whose argument lists look exactly like this. One signature for link_to is:

link_to(body, url_options = {}, html_options = {})

So...

link_to "foo", :controller => :x                          # OK
link_to "foo", :controller => :x, :class => "css_class"   # WRONG! 
link_to "foo", {:controller => :x}, :class => "css_class" # OK

What about support for default arguments? We might want to say that if you didn't pass in an :x argument, we want it to have some default value. You might think this would work:

def baz(x = {:x => 123})
   p x
end

But you would be sadly mistaken.

baz                               # => {:x=>123}
baz :y => 456                     # => {:y=>456} ... oops

Ruby doesn't merge your keyword arguments into the map in the parameter list. It uses that map if you don't supply any arguments, otherwise your map replaces the default entirely. So to get default arguments, you need something like

def baz2(args = {})
  args = {:x => 123}.merge(args)
  p args
end

baz2                              # => {:x=>123}
baz2 :x => 555                    # => {:x=>555}
baz2 :y => 456                    # => {:x=>123, :y=>456}

Kind of messy, but that's OK.

One last subtle ambiguity in Ruby is determining whether someone passed a nil argument for a keyword explicitly, or omitted a keyword entirely. It might make a difference in some circumstances.

def quux(args={})
  p args.include? :x
  args = {:x => nil}.merge(args)
  p args
end

quux                              # true, {:x=>nil}
quux :x => nil                    # false, {:x=>nil}

Pretty messy, but such is life.

Clojure

Part of the fun of Lisps is lack of ambiguity. Everything is spelled out in all its parenthesized glory. In Clojure, when you call a function like (foo :x 123), Clojure requires you to specify what you want to happen with those arguments.

It used to be that Clojure didn't have much support for keyword args at all. Clojure did have support for allowing variable numbers of arguments to functions though. So in the beginning, people used to slurp all of their arguments together into a list, and then turn it into a map inside the function.

user> (defn foo [& args]
        (let [args (apply hash-map args)]
          (prn args)))
#'user/foo
user> (foo)
{}
user> (foo :x 123)
{:x 123}
user> (foo :x 123 :y 456)
{:y 456, :x 123}

That worked. It still works today. But nowadays there's a better way. Why not slurp your arguments directly into a hash-map?

user> (defn foo [& {:as args}]
        (prn args))
#'user/foo
user> (foo)
nil
user> (foo :x 123)
{:x 123}
user> (foo :x 123 :y 456)
{:y 456, :x 123}

This is an example of destructuring. In this case, all of our arguments are slurped into a list (thanks to &), then this list is matched against our destructuring pattern, in this case {:as args}. :as says to take everything in the list, turn it into a map and give the resulting map the name args.

This isn't a special feature of defn. Destructuring works anywhere you're setting up bindings, for example in let, for, doseq etc. Like so:

user> (let [{:as args} (list :x 123 :y 456)] args)
{:y 456, :x 123}

It just so happens that & creates the list for you, out of the arguments you pass.

Destructuring does lots more than that though. We can immediately pull out the values for keywords we care about, so they'll be bound to names in our function body.

user> (defn foo [& {:keys [x y z]}]
        (prn x y z))
#'user/foo

user> (foo :z 123 :x 456)
456 nil 123

It's certainly a bit more verbose than Ruby in the function signature, but it lacks ambiguity and it saves you some repetition in the function body.

Clojure's approach also has the benefit of specifying directly in the function signature which keywords you're expecting. In a smart editor, like Emacs, you get an indicator of what kinds of keywords you should be passing in. See at the bottom?

This is also available at the Clojure REPL via the doc function.

user> (doc foo)
-------------------------
user/foo
([& {:keys [a b c]}])
  nil
nil

There's no better documentation than live, built-in documentation. There's nothing more distracting when programming than context shifts, and having to dig into a web browser to check a function signature is a huge mental page fault.

What about default values? Sure. You can use :or to specify defaults for some or all of your keywords. This works much more like I'd expect, compared to Ruby.

user> (defn foo [& {:keys [x y z] :or {x 1 y 2 z 3}}]
        (prn x y z))
#'user/foo

user> (foo :y 555)
1 555 3

What about determining whether the user passed nil for a keyword or whether they omitted the keyword entirely? For that, you have to resort to testing the argument map for the existence of the key, which isn't fun, but at least it's possible.

user> (defn foo [& {:keys [x y z] 
                    :or {x 1 y 2 z 3}
                    :as args}]
        (prn x y z)
        (prn args)
        (doseq [k [:x :y :z]]
          (println "Contains" k "=>" (contains? args k))))
#'user/foo

user> (foo :x 1 :y 2)
1 2 3
{:x 1, :y 2}
Contains :x => true
Contains :y => true
Contains :z => false

Why do I keep mentioning this? See Common Lisp below.

In any case, our parameter list is becoming huge and unweildy. The first time I saw sample Clojure code like this, I almost did a spit-take. But after a bit of getting used to, I'm finding that this syntax is pretty comfortable.

The simplest case may not be as concise as Ruby, but aside from lack of ambiguity, the benefit of Clojure's approach is being able to safely do powerful (and borderline insane) things, like nested destructuring. And this works everywhere you're setting up a binding.

(defn ow-my-eyes [a & {[b {:keys [x y]}
                        & {[g h] :r
                           :keys [p q]
                           :or {q 123}}]
                       :x}]
  (prn a b g h p q x y))



user> (ow-my-eyes 444 :x [1 {:x 555 :y 666} :p 3 :r [888 999]])
444 1 888 999 3 123 555 666

If you wrote code like that in real life, you'd likely be defenestrated, but at least you know you can do it. And destructuring in Clojure could likely be extended even further in the future, if someone came up with a use case for something that isn't supported.

Official documentation for all of this is here.

Common Lisp

When it comes to keyword arguments, Common Lisp supports mostly everything that Clojure does, and some things it doesn't. CL was likely a big inspiration for Clojure's destructuring. I highly recommend reading Practical Common Lisp to learn more about CL keyword arguments and list destructuring. (Read the rest of the book while you're at it.)

Keyword arguments in CL look like this:

> (defun foo (&key x (y 123) (z 456 z-supplied-p))
    (pprint (list x y z z-supplied-p)))
FOO

> (foo)
(NIL 123 456 NIL)

> (foo :z nil)             ; note, z-supplied-p tells us whether z was omitted or not
(NIL 123 NIL T)

> (foo :z 555 :x 666 :y 777)
(666 777 555 T)

> (foo :x 123)
(123 123 456 NIL)

> (foo :x 555)
(555 123 456 NIL)

Yeah, there's direct support for distinguishing supplied keys with nil values, and un-supplied keys. That's pretty nice.

Common Lisp also supports insane things like having keyword arguments' default values be a function of other arguments in the parameter list.

> (defun bar (&key (x 123) (y (+ x 1000)))
    (pprint (list x y)))
BAR

> (bar)
(123 1123)

> (bar :x 5)
(5 1005)

> (bar :x 5 :y 6)
(5 6)

Clojure destructuring can't do this by default, so you'd have to resort to a let in the function body.

(defn bar [& {:keys [x y] :or {x 123}}]
  (let [y (or y (+ x 1000))]
    (prn x y)

There's nothing stopping Clojure's destructuring from being patched to support this, of course. But this isn't a feature I've ever found myself wanting particularly badly.

As for how to destructure CL keyword arguments into a vector or hash-map, CL doesn't directly support doing it in function parameter lists like Clojure does. (Though there's nothing stopping someone from throwing together a reader macro to let CL do this, of course.) This isn't really surprising, because CL loves its cons cells, while Clojure embraces maps (and vectors and sets etc.), offering them default reader syntax and lots of other built-in support.

So there you have it. Keyword arguments. Use them. Love them.

Lessons:

Sugar can be bad for your health.
Ambiguous, bad. Explicit, good.

org-mode is awesome

Thu, 20 Jan 2011 16:14:44 -0800

I've seen org-mode for Emacs mentioned very frequently around the interwebs, so it went into my mental queue of topics to learn. It finally bubbled to the top this week, so I took a look.

Organizer? Nah.

As an organizer/calendar, well, I doubt I'll need it. Enforced use of MS Outlook is mandated by work. My Post-it-notes-all-over-my-desk method of organization will also continue to serve me well.

There are some nice agenda-related shortcuts that are probably worth using though, like typing C-c . to enter a datestamp, like <2011-01-20 Thu>. Then you can increment or decrement it one year/month/day at a time via S-up and S-down. I like this.

Plaintext editor? Yes!

As a plaintext outline and table editor... wow. org-mode rocks. Do you know how many hours of my life could have been saved by having a good ASCII table/bullet-list editor? org-mode lines everything up and keeps it all nice and neat for you.

You can also make plaintext check boxes and check/uncheck them. And you can insert hyperlinks and footnotes, and click them to open web pages or jump back and forth between footnote and reference.

There are ways to collapse and expand outlines, search for items and only display those items, and so on. The documentation for org-mode is very clear and took me less than an hour to read through. All-in-all a pleasant experience.

* Agenda
** Things to learn
1. [X] Clojure
2. [X] org-mode (see [fn:diagram1])
3. [ ] Haskell
4. [ ] Japanese
   1. [X] Hiragana
   2. [X] Katakana
   3. [ ] Kanji
5. [ ] The true meaning of friendship

* Footnotes
[fn:diagram1]

| Task                              | Annoyance (1-10) |
|-----------------------------------+------------------|
| Making ASCII tables by hand       |              9.5 |
| Making ASCII bullet lists by hand |              7.2 |
| Using org-mode                    |              0.4 |

It looks nice plastered into my blog, but you don't get a real idea of how many cool things you can do with it until you open it in Emacs and start shuffling items around, bumping them up/down a level in headlines, creating properly-numbered bullet items with one key, and seeing the columns in the table auto-resize as you type.

I also highly recommend putting (setq org-startup-indented t) into .emacs to make everything look pretty on-screen. It still saves as the simple plaintext above, but it looks like this in Emacs:

I can definitely see using org-mode for TODO files in some of my projects. (You can mark entries as TODO (just by typing TODO in front), and then toggle between TODO/DONE via C-c C-t.) I can also see using it as a general-purpose note-taker.

org-mode also has a mobile version for iPhone and Android, synced via WebDAV or Dropbox, so you can org-mode on your phone while you're driving to the grocery store¹. Again I don't really need this, but there it is.

The joy of plaintext

Plaintext is awesome.

It's the universal file format. It's readable and writeable by scripting langauges, terminals, text editors, IDEs, word processors, web browsers, even lowly humans.

Plaintext's one shortcoming is its lack of structure. It's just a bunch of letters. It doesn't have a color, it doesn't have a style, it doesn't line up into columns without a lot of effort. There's nothing stopping you from opening a parenthesized list and forgetting the closing paren.

Computers don't care about these problems, but humans are bad at producting plaintext by hand, and bad at editing it once it's produced. Our clumsy, stumpy fingers and inconsistent, chaotic brains can't handle the freedom.

Emacs (and Vim) are awesome because they let you do magical things to plaintext. They enforce structure. They provide shortcuts so you can get your plainext right the first time.

[ ] is just two braces and a space, but org-mode lets me hit C-c C-c and turn the space into an X. This may seem banal, hardly worth caring about, but add to this shortcut thousands upon thousands of others. Things like org-mode, or paredit, or all of Vim's built-in magic... it all adds up to something wonderful.

And best of all, you always still have the option of manually keyboarding over and typing that X between the braces yourself. It's still just plaintext underneath. So you end up with the best of both worlds.

I do not recommend using org-mode while driving, for public safety reasons. ↩

2010 in review

Wed, 05 Jan 2011 10:10:03 -0800

Another year down the drain. A good year, in the end.

2010 Geek Achievements

Wrote some code...
- cow-blog - The engine running this blog.
- oyako - Clojure ORM library.
- gaka - CSS compiler for Clojure
Finished a huge project for work, my first AJAX-y web app (in Rails). That was fun, albeit stressful.
Learned a lot of git.
Learned a lot of Clojure.
Learned a lot of Emacs.
Learned a lot of Javascript.
Learned a lot of PostgreSQL. It's good to be free of MySQL.
Switched to ZSH. This was a good switch.
Tried to learn a lot of Japanese, but kind of fizzled out at the end of the year.
Alllllllmost got a Clojure gold badge on Stack Overflow. I'll get it soon though. Not losing any sleep over it either way.
Read a lot of books. The best: probably Feynman's books of anecdotes.
Blogged a bit. Got an article in Hacker Monthly. Was flamed repeatedly. Learned a lot in the process.

2010 Non-Geek Achievements

Immigrated to Canada. A good move, without a doubt.
Lost 25ish lbs. :)
Learned how to cook better.
My most important achievement from 2010 is actually non-geek: I finally obtained a bit of an offline social life. This is not an easy task for one such as myself.
Continued to learn to appreciate good beer. Longwood Dunkelweizen, mmm.

2010 Failures

Did not blog enough.¹
Did not write enough code.¹
Missed the first Clojure Conj. Maybe next year.
Re-gained 10ish lbs. :( ²

Plans for 2011

Re-lose 25ish pounds. I'd like to reach the weight I had in college.
Finish my rewrite of oyako. I have ambitious plans for it, if I can just find the time.
Finish my rewrite of cow-blog to match oyako.
Keep working on the RPG my wife and I are creating (in Clojure).
Attend the next Clojure Conj, I hope.
Learn more Clojure.
Learn Haskell? Trying and failing to learn Haskell has become somewhat of a tradition, no sense stopping now.
Learn all 2000+ jouyou kanji by the end of the year.
Supar sekrit projekt. But I haven't signed the contract for it yet so I won't talk about it until I do.
Maintain social life at acceptable levels.
Buy a house.

I feel like I have solid plans for completing each of these things. Blogging more often and finishing oyako are high on my list of priorities. I expect 2011 to be my most productive year to date.

See also, non-geek achievement #4, "Obtained social life". ↩
See also non-geek achievement #3, "Learned how to cook better". ↩

Vim undo tree visualization

Mon, 18 Oct 2010 15:21:16 -0700

I wrote previously about an awsome plugin to give Emacs Vim-style undo trees.

Vim's undo trees are the best thing since sliced bread, but the interface for browsing through the tree is not pleasant. The Emacs undo-tree library has a way to visualize the tree and move through it with your keyboard, which solves this problem.

But now, thanks to Steve Losh, Vim has an undo-tree visualizer too. Delicious. Though it's still beta and promises to eat your babies, it seems to work pretty well. I think the diff view of the changes for the undo is a really good idea.

Thus continues the eternal Vim/Emacs arms race.

Productivity Booster

Fri, 08 Oct 2010 15:09:40 -0700

I came up with a great way to increase my productivity recently. You need a locally-running Apache server for this to work most effectively.

First you need to set up a redirect for 404 requests to localhost. On my system I determined that DocumentRoot is /srv/http, so I set this up in /etc/httpd/conf/httpd.conf:

<Directory "/srv/http">
    AllowOverride All
</Directory>

Next I edited /srv/http/.htaccess to redirect 404's to a main index page:

ErrorDocument 404 /index.html

Then I created /srv/http/index.html:

<html>
    <head>
        <title>GET BACK TO WORK, YOU HOBO</title>
        <style type="text/css">
            html {
                background: #f00;
            }
            h1 {
                color: #0f0;
                font-family: sans-serif;
                text-align: center;
                margin-top: 100px;
                font-size: 64pt;
            }
        </style>
    </head>
    <body>
        <h1>GET BACK TO WORK, YOU HOBO</h1>
    </body>
</html>

Almost done; the final step is to edit /etc/hosts:

127.0.0.1 reddit.com www.reddit.com slashdot.org news.ycombinator.com

Now I see this. Image hidden behind a link to spare the eyes of my readers.

If this doesn't work, you could try making the background color flash quickly between red and green, or add a background MIDI and some animated GIFs.

iPad? More like iAd. Vertisements.

Thu, 16 Sep 2010 18:43:35 -0700

Via Slashdot, it seems soon you may be able to subscribe to newspapers on the iPad in the near future.

Sure. Why pay $10 for a paper copy of something when you can pay the same $10 for a likely-DRM'ed copy that can only be read on a $500 portable computer?

In all honesty though, instant delivery, lack of clutter, "take it anywhere", being able to archive issues indefinitely, text search... those features might be worth the money, if it was a really good newspaper/magazine.

But wait, there's more.

The Cupertino company has agreed to provide an opt-in function for subscribers to allow Apple to share with publishers their information, which includes vital data that news organizations use to attract advertisers, industry sources say.

...

While the leap into the digital tablet market comes with short-term problems for newspapers, the iPad and future tablets will provide a new digital palette for publications to create sophisticated and lucrative ads, said Needham & Co. analyst Charles Wolf.

"I would say it's a risk, but I would argue it's a short-term risk," Wolf said. "If you can put animation and multimedia into ads, that will greatly enhance reader views. I am certain of that."

So as I understand it, first I buy a $500 gadget. Then I pay for a newspaper subscription. Then a bunch of companies want me to give them personal information about myself, so they can share it amongst themselves. And then I have to view ads.

Animated ads.

The only thing better would be if the iPad also woke you up at 4AM and tried to sell you life insurance. Maybe Apple would let me install a free ad-blocker script for my news reader though. It is my hardware, after all... pfffft, yeah, I could't keep going with a straight face.

And thus my desire to get an iPad, kind-of sort-of building over the past couple of months, once again flatlines.

Git info in your ZSH Prompt

Fri, 10 Sep 2010 13:31:37 -0700

Recently I discovered vcs_info recently. This nicely replaces the horrible hack I was using previously to show current Git status. vcs_info works with VCSes besides Git, and it handles some of the magic and keeps your .zshrc clean, so those are nice benefits.

I used some Unicode to display colored circles. Green if there are staged changes, yellow if there are unstaged changes, and red if there are new untracked-yet-unignored files. Below is a picture.

I like this because I'm constantly forgetting to git add newly-created files. Then I have to add them and amend my commit, and so on. I like a prompt that reminds me that new files showed up that need to be added or ignored.

Code:

autoload -Uz vcs_info

zstyle ':vcs_info:*' stagedstr '%F{28}●'
zstyle ':vcs_info:*' unstagedstr '%F{11}●'
zstyle ':vcs_info:*' check-for-changes true
zstyle ':vcs_info:(sv[nk]|bzr):*' branchformat '%b%F{1}:%F{11}%r'
zstyle ':vcs_info:*' enable git svn
precmd () {
    if [[ -z $(git ls-files --other --exclude-standard 2> /dev/null) ]] {
        zstyle ':vcs_info:*' formats ' [%F{green}%b%c%u%F{blue}]'
    } else {
        zstyle ':vcs_info:*' formats ' [%F{green}%b%c%u%F{red}●%F{blue}]'
    }

    vcs_info
}

setopt prompt_subst
PROMPT='%F{blue}%n@%m %c${vcs_info_msg_0_}%F{blue} %(?/%F{blue}/%F{red})%% %{$reset_color%}'

Picture:

Limitations

As you can see in the screenshot, when you have a brand new Git repo (no commits yet), vcs_info fails to show you that there are files staged. It works OK after you have at least one commit though.

vcs_info doesn't (yet?) handle showing untracked files. So I hacked a function to support it.

Finding a good Unicode symbol that displays nicely in monospace font was annoying. If I ever change fonts, I'll likely have to pick a new symbol. It also doesn't display too well in a real tty. Or over SSH when using Putty. So I may have to scrap the stoplights and use plus-signs or something. Sigh.

Vim :ruby and :rubydo scope

Tue, 31 Aug 2010 10:40:51 -0700

Note to self. In old Vim (tested in 7.2.320), I could do this:

:ruby x='foo'
:rubydo $_=x

Now every line in the file says foo. But in Vim 7.3 I get an error:

NameError: undefined local variable or method `x' for main:Object

The scoping rules for Ruby in Vim must have changed somewhere along the line. I was abusing this feature to do some handy things, so this is sad.

A workaround is to use global variables in Ruby instead. So this still works:

:ruby $x='foo'
:rubydo $_=$x

Phew.

Emacs undo trees

Tue, 17 Aug 2010 14:06:33 -0700

I've said it before: undo in Emacs is horrible. On the other hand, undo in Vim is awesome.

But this is true no longer. Now there are undo trees for Emacs! Yes, this news is so important I had to italicize and bold it. It's like Emacs has been punching me in the face for years, and today I got it to stop. I never thought I'd see the day.

And it works great too. You can even view the tree visually and navigate it with the cursor keys, which is a step up on what Vim offers out of the box.

In other news, Vim 7.3 is out and it now has persistent undo across reloads. It's like an arms race, and gleeful hackers reap the benefits.

Let's draw some pixels

Tue, 10 Aug 2010 11:19:42 -0700

I've been getting into pixel art a lot lately. It appeals to me on a lot of levels.

The coder in me likes it because it's so precise. Every pixel is placed just so. The color palette is limited to a dozen colors. Building a drawing out of such limited means reminds me of building programs out of primitives. There are design patterns in pixel art: dithering, manual anti-aliasing. There are abstractions that work and abstractions that don't. There's a lot of goofing around with RGB values and transparency settings; it's perhaps the most deeply computer-based art form you could come up with, and as a deeply computer-based human, I really like it.

The gamer in me is still partly stuck in the early 90's, so it's a huge injection of nostalgia to look at pixel art. NES- and SNES-era games had a charm that is unmatched by anything since. And I don't think that's entirely nostalgia talking; I still play old games and they're still so much fun. And the art in a lot of those games was just darned good. If you stop and look at it really carefully, and start to get an understanding of how it was made, you can't help but be impressed.

The "artist" in me (if there is such a thing in my brain somewhere) is blown away by some of the things good pixel artists can produce. Go look at foolstown.com and try not to slobber. Some of this stuff just looks amazing. Not "good for a pixel drawing", but good on a level anyone could appreciate.

Pixel doodles are also good practice for the RPG my wife and I are still ever-so-slowly creating. Creating art and music for a game are turning out to be much harder work than programming it.

In any case, I drew a cow standing beside a tree. And I made a new pixel art page to house my admittedly still-amateurish drawings.

Review: What Do You Care What Other People Think?

Fri, 06 Aug 2010 22:12:01 -0700

I recently reviewed Surely You're Joking, Mr. Feynman!. It was good enough that I had to get the sequel.

What Do You Care What Other People Think? is another collection of stories and anecdotes written by and/or about Richard Feynman. A bit in contrast to the first book, rather than a chronological series of anecdotes, this book focuses on a couple of main topics.

Feynman discusses his first wife in some detail. Of particular interest, he describes his and his wife's brutal devotion to honesty in their relationship, even in the face of highly unpleasant truths (terminal disease, in this case). It's the honesty of a scientist, carried into "everyday" life. This was bittersweet for me to read, because the story has a sad ending.

There is also a short series of letters from Feynman to others, where he discusses the silliness of pomp and circumstance, e.g. his foibles and breaches of protocol when meeting some king or other. As someone who hates ceremony, I got a huge kick out of these.

A large part of the book is devoted to discussing the Presidential Commission which investigated the cause of the Challenger shuttle disaster. Feynman's full report is included in the book as well.

As someone interested in astronomy and space flight (and who isn't interested in those?) I found this fascinating. There's a lot of behind-the-scenes stuff. Engineers are painted in a good light, managers and politicians not so much. (Software engineers come out looking especially good, which made me feel (unjustifiably) good about myself by proxy.) There are some diagrams and a lot of technical discussion of the shuttle. Not so much that it drowns the narrative, but enough that I'm probably going to spend the next week reading Wikipedia on the subject now.

Feynman explains his simple methods at getting to the truth in the investigation. Go talk to the guys who put things together. Get your hands on some O-ring rubber and test its resistance to temperature yourself in a glass of ice water. Cut to the heart of the matter. It's good stuff.

Ultimately, as you know if you've read the report, Feynman rips NASA apart, showing that they were fooling themselves into believing the shuttle was safer than it really was. The last sentence of the report says everything: "Nature cannot be fooled."

The last section of the book discusses the value of science. More specifically, Feynman discusses the value of doubt. I very much liked how the chapter ends:

It is our responsibility as scientists , knowing the great progress which comes from a satisfactory philosophy of ignorance, the great progress which is the fruit of freedom of thought, to proclaim the value of this freedom; to teach how doubt is not to be feared but welcomed and discussed; and to demand this freedom as our duty to all coming generations.

If there's one trait I had to pick to separate good people from bad, it would be the ability to admit being wrong. And if I had to separate the good from the excellent, it would be not just the ability to admit being wrong, but the eagerness to be proved wrong.

There's a certain kind of devotion to the truth that not many people achieve, and maybe not many people even want to achieve. There's comfort in thinking that you know things. It's very tempting. I think it's probably partly why most people are religious. I suspect it's a big reason why so many people are so stubbornly wrong about so many things in general. I suspect this comfort is an enormous source of suffering in the world.

But there's another kind of comfort that people miss out on. It's the comfort of knowing that although you're probably wrong about a lot of things, you're trying your hardest to be right. You pay the price of being aware of your own state of ignorance, but you can rest a bit easier knowing that you're maybe, hopefully, inching towards the truth. I never heard the word "freedom" used to describe this feeling before, as Feynman does above, but it fits.

That's why I like reading about Feynman and reading Feynman's words. He seemed to live this philosophy as well as anyone could hope to.

X automation with xte

Fri, 06 Aug 2010 13:03:01 -0700

I learned today (via a great blog post) about xte. This program lets you simulate X Windows mouse and keyboard events from the commandline. How much more awesome can you get?

Hans illustrates how to integrate xbindkeys and xte to make KDE4 effects activate. I wanted the KDE4 "Desktop Grid" to appear when I press a mouse button (because my new mouse has a lot of buttons to spare), so this is exactly what I was looking for.

xte is the kind of glue that makes Linux awesome. KDE lets you set global keyboard shortcuts for lots of things. xbindkeys lets you assign shell commands to mouse buttons. And xte ties the two together. Possibly none of the programmers on these three tools knew about the others, but they interact perfectly to let you do anything you want.

You may be thinking, "If you want to work with KDE from the commandline, why not use DBUS?" That's what I tried to do first. But I can't for the life of me figure it out. There's some indication that we might be able to do this somday, like so:

qdbus org.kde.kglobalaccel /component/kwin org.kde.kglobalaccel.Component.invokeShortcut ShowDesktopGrid

Or maybe it's already in the latest version of KDE and I haven't upgraded yet. Either way.

By the way: could DBUS possibly have a more verbose or cryptic interface? I was hunting through the available DBUS commands looking for something that would show the Desktop Grid, and I ended up having to scan through lists of crap like this:

~ % qdbus org.kde.kwin /KWin                         
method Q_NOREPLY void org.kde.KWin.cascadeDesktop()
method void org.kde.KWin.circulateDesktopApplications()
method bool org.kde.KWin.compositingActive()
signal void org.kde.KWin.compositingToggled(bool active)
method int org.kde.KWin.currentDesktop()
method QList<int> org.kde.KWin.decorationSupportedColors()
method void org.kde.KWin.doNotManage(QString name)
method Q_NOREPLY void org.kde.KWin.killWindow()
method QStringList org.kde.KWin.listOfEffects()
method void org.kde.KWin.loadEffect(QString name)
method QStringList org.kde.KWin.loadedEffects()
method void org.kde.KWin.nextDesktop()
method void org.kde.KWin.previousDesktop()
method Q_NOREPLY void org.kde.KWin.reconfigure()
method void org.kde.KWin.reconfigureEffect(QString name)
method void org.kde.KWin.refresh()
signal void org.kde.KWin.reinitCompositing()
signal void org.kde.KWin.reloadConfig()
method bool org.kde.KWin.setCurrentDesktop(int desktop)
method void org.kde.KWin.showWindowMenuAt(qlonglong winId, int x, int y)
method Q_NOREPLY void org.kde.KWin.toggleCompositing()
method void org.kde.KWin.toggleEffect(QString name)
method Q_NOREPLY void org.kde.KWin.unclutterDesktop()
method void org.kde.KWin.unloadEffect(QString name)
method bool org.kde.KWin.waitForCompositingSetup()
method QDBusVariant org.freedesktop.DBus.Properties.Get(QString interface_name, QString property_name)
method QVariantMap org.freedesktop.DBus.Properties.GetAll(QString interface_name)
method void org.freedesktop.DBus.Properties.Set(QString interface_name, QString property_name, QDBusVariant value)
method QString org.freedesktop.DBus.Introspectable.Introspect()

This is line noise to me.

gaka 0.2.0

Thu, 29 Jul 2010 13:59:22 -0700

Per many commenters' suggestions and thanks to code from Steve Purcell, you can now use maps for CSS attributes in gaka.

user> (println (gaka/css [:a {:color :red}]))
a {
  color: red;}

This looks more like vanilla CSS thanks to the curlies, which is nice. You just have to keep in mind that your key/value pairs could end up being printed in random order, and order is significant¹ in CSS.

It just so happens that maps are implemented in Clojure right now such that if they only have a few entries (16 key/value pairs), the order will be preserved, because you get a PersistentArrayMap instead of a PersistentHashMap. But it's highly dangerous to rely on such a thing. It could change at any time in the future.

In any case, you can also mix and match maps, lists and "flat" keyvals. They'll all be flattened That can help preserve attribute order in those cases where you need to.

user> (println (gaka/css [:a :color "red" {:padding 0} (list :margin 0)]))
a {
  color: red;
  padding: 0;
  margin: 0;}

I've also enhanced "mixins" a bit further. You can now mixin entire tags as well as attributes. Or a combination of both. Say you want a mixin that means "Make my element have no padding, and make links within the element be red":

user> (println (gaka/css [:div.foo mixin :margin 0]
                         [:div.bar mixin]))
div.foo {
  padding: 0;
  margin: 0;}

  div.foo a {
    color: red;}

div.bar {
  padding: 0;}

  div.bar a {
    color: red;}

You can get gaka from github or Clojars.

Order is only significant in cases where you're doing things like padding: 0; padding-left: 1px. This is arguably bad CSS style, but it's valid, and it's also possible you'll have this kind of thing if you're generating CSS procedurally. But most of the time, order is not significant. e.g. it doesn't matter if you set text color first and background color second, or vice versa. So maybe this isn't so much of a problem in practice. ↩

Footnotes

Tue, 27 Jul 2010 12:30:57 -0700

Did you ever notice how footnotes make your writing seem more important¹ somehow?

Maybe one reason is that "real" books use footnotes. At a glance, it looks like I have references² backing up everything I say. In reality, I don't, but the connotation carries through somehow³. Now my blog seems scholarly and authoritative.

And if you're like me, you can't resist clicking footnotes to see what they refer to⁴. According to my estimates, by utilizing footnotes, in one fell swoop I have decreased my readers' average reading efficiency by 73%.

In any case, I've added experimental, rudimentary support for footnotes to cow-blog.

I'm loosely copying the syntax from Markdown Extra for this. Markdown is great, except when it isn't. The standard doesn't have support for some useful extensions. I use Showdown for Markdown support, and I'm probably going to work on adding more features of Markdown Extra to Showdown in the near future.

I just dread actually doing it. Showdown (like Markdown itself) is implemented as a series of hackish regex transformations of blobs of text. It's not a proper grammar. Implementing more of Markdown Extra means more regex blobbing. It's brittle and fragile and even getting incomplete support for footnotes was less than enjoyable. But at the same time I find myself wanting to do things that Markdown can't so, so I may have to bite the bullet.

(If there's a Showdown Extra out there already, drop me a URL. It'd be most appreciated. But I couldn't find one.)

In reality nothing I say is important. ↩
Does my inner dialog count as a reference? ↩
Via telepathy. ↩
See? ↩

Review: Surely You're Joking, Mr. Feynman!

Mon, 26 Jul 2010 16:19:21 -0700

I'm probably the last person on earth to read "Surely You're Joking, Mr. Feynman!", a collection of stories and anecdotes from the life of Richard Feynman. But better late than never.

I'll keep this short. Feynman was the kind of nerd every nerd wishes he was, in one way or another. He was socially awkward. He was blunt and tactless. He felt out of place much of the time. And it was surprising to see how often Feynman expressed feelings of inadequacy. He wrote about being highly intimidated when talking in front of big names in physics, and jealous of the math abilities of some other people.

But none of that stopped him from having an exciting life. In fact he turned these traits to his advantage. He had romantic success, often via very highly unconventional means (e.g. walking up to girls and asking them for sex outright; they often said yes). His bluntness was seen as an admirable trait: where others might be intimidated by a famous physicist, Feynman would give his honest opinion, and that was often appreciated.

And Feynman went far outside his comfort zone. He did a stint in biology, even though he knew nothing of biology at the time. He went to Brazil and joined a samba band. He sold drawings and paintings for a while. He played drums for a ballet.

I picked up at least three lessons from this book.

Try new things, even if you suck at them. Life is boring if you stick to what you're good at.
Be intellectually honest. Brutally so. It's the only way to do good science, and arguably the only way to live a good life. I've always believed this, and Feynman hammers the point home well.
Even the best and brightest of us feel insecure at times. You shouldn't let it stop you.

On top of all of that, reading of Feynman's time at Los Alamos working on The Bomb was a fascinating piece of history. I highly recommend this book.

And watch as much of Feynman on Youtube as you can find. It never ceases to be fascinating.

Review: Logitech Performance MX

Mon, 26 Jul 2010 13:51:27 -0700

I have a long list of companies I won't buy from, due to horrible customer service experiences or shoddy merchandise. On the other hand, my "Must buy from this company" list is awfully short.

Logitech is one company I'm generally OK buying from. (For now...) I go through computer mouses¹ pretty fast, so I end up buying a new one every 4-5 years. Logitech hasn't failed me thus far.

In my opinion, the most important parts of your computer are the ones you interact with: Keyboard, mouse, monitor. I'd rather have a slow computer with a good mouse and keyboard than a fast computer with cheap peripherals. I already have a wonderful keyboard, and I like to have a nice mouse to match.

My latest mouse is the Logitech Performance MX.

How much can you really improve upon the concept of a mouse? It's a round thing you slide around and click. Well, if you believe the marketing on the box, it seems there are at least three areas for improvement:

Add more buttons.
Make it work on more surfaces.
Gimmicks.

None of the above features matter much to me. What I care about:

Is it comfortable to use all day?
Is it going to last me 4 or 5 years?

Let's hit each of these points.

More buttons

The Performance MX has a lot of buttons. If you believe xev, there are somewhere between 10 and 13 (counting the scroll wheel etc.).

I don't need all of these. I like to have a thumb button for middle-clicking, but this mouse has four of them. What am I going to use all of the extras for? Maybe I'll have one open Emacs.

The browser forward and back thumb buttons work out-of-the-box in Linux, which surprised me. I'm used to having to dork around with xmodmap or xbindkeys. Linux has come a long way.

Then there's a thumb button that's supposed to open Expose on OS X or your app-switcher of choice in your OS. It's in a place that's impossible to press consistently with your thumb, unless you're a contortionist. And it does nothing in Linux by default. Similarly there's a vanishingly small "zoom" button near your thumb. I can't even find this button without looking at the mouse, because it's too small.

There's also horizontal scroll, if you can manage to tilt the wheel to either side. My fingers lack the dexterity to do this easily, and the button is awfully stiff, so I'll never use this.

But hey, better too many buttons than too few.

Surfaces

The marketers wrote "This mouse works on glass!" all over the box, so this must be a selling point of the mouse. But my Logitech G500 also worked on glass, once I upgraded the firmware (via Logitech's own website), so I'm not sure why this feature is being pushed so hard.

Gimmicks

On the Performance MX, you can click this little switch and disengage part of the mechanism that makes the scroll wheel "click". This lets you spin the wheel hard and it'll spin freely forever until you stop it manually, or until its inertia runs out.

Do I really need, and I quote, hyper-fast scrolling? The scroll speed of my mouse wheel has never struck me as a productivity bottleneck. Now that I've played with it, sure, it's mildly interesting. If you scroll at just the right speed, you can go all the way from the top of a Reddit thread to the bottom, while still skimming the text. But I don't need this feature.

But the levels to which marketroids will stoop to hype up a product never ceases to amaze me. The name itself is ridiculous. Performance MX. Who comes up with this?

Comfort

My previous mouse was a G500, and it was really quite big. A bit too big. (And there were these removable weights you could use to adjust the weight of the mouse (the "gimmick" feature of that mouse), which were worthless.)

The MX is bigger, but it has more curves and that helps my hand sit on it better. There's a very large thumb rest, which I like. If you have small hands, you might have problems with this mouse. I like the feel so far.

Will it last?

This mouse feels very solid. It has a nice heft to it. It has metal bits (or sturdy metal-like plastic) in places other mouses have cheap plastic and rubber.

The wheel on my G500 wore down very fast. I was some kind of rubbery plastic, which became rounded off, nicked and worn-down a lot over the years. The wheel on the MX feels like some kind of metal, with nice ridges on the top to provide friction. I'm a bit concerned that the wheel disengagement switch might wear down over time though. The more moving parts, the more opportunity for something to break.

The pads on the bottom are huge: a good 1 inch x 1/8 inch. I love this. Using this on my Icemat mouse pad is very comfy. It slides effortlessly, but leaves me with plenty of control.

Battery

I normally prefer corded mouses. This one is rechargeable. Most rechargeable mouses I've used had some kind of stupid dock you had to jam the mouse into. Not only does it take up desk space, it also wears holes into your mouse over time from being inserted and removed from the dock. The battery connectors also wear down fast, and then it doesn't charge right unless you fiddle with it.

The MX can be charged while you use it, via a USB cable, which is a really good idea. It comes with a short cable, and a long extender cable if you need it. The cable looks like it has a non-standard connector though, which is a bad idea. Why not use mini-USB?

The charger cable is awfully heavy though. I wouldn't want to use this thing while it's plugged in for more than a short time.

I don't know what the battery life is like, because it hasn't run out yet, after a couple days. That's good enough for me.

Price

$120. That's a lot. But taken over 4-5 years, it's not that bad.

And the surgery I'd have to have done on my hands after crippling them via a cheap mouse over the next 40 years would be way more than $120.

Should you buy this?

Sure. I like this mouse so far. It works, and it feels like it'll last a while. Props to Logitech for making good products.

Yes, mouses. ↩

Clojure syntax highlighting via SyntaxHighlighter

Wed, 21 Jul 2010 15:17:42 -0700

How do you syntax-highlight Clojure code for display on a website? The best way I can find is SyntaxHighlighter.

Daniel Gómez wrote a brush to give SyntaxHighlighter Clojure support. I tweaked it a bit myself and integrated it into cow-blog. I also converted my favorite color scheme to a SyntaxHighlighter theme. So when I write this code:

(defn- ip
  "Given a request, return the IP.  Looks for an x-forwarded-for
  header, falls back to :remote-addr on the request."
  [request]
  (or (get-in request [:headers "x-forwarded-for"])
      (request :remote-addr)))

You should see something like this:

...unless you're reading this via RSS, or in a browser without Javascript enabled, in which case you'll see plain, depressing black and white. But that's one nice thing about SyntaxHighlighter. It degrades nicely.

One bad thing about SyntaxHighlighter is that it doesn't play nicely with Markdown. Or rather, Markdown isn't powerful enough to let you specify the class of any markdown-generated HTML tags. If you want the <pre class="brush: clojure"> that SyntaxHighlighter requires, you have to write out the HTML by hand. But I hacked Showdown a bit to let me specify classes more easily, so I can avoid having to do that.

The code for all of this is on github with the rest of my blog.

ANTLR via Clojure

Fri, 16 Jul 2010 17:41:04 -0700

ANTLR is a parser-generator for Java. Can you use it from Clojure? Sure. Would you want to? Maybe.

Here's how to do it, start to finish.

For the impatient among you, all of the code below is on github.

git clone git://github.com/briancarper/clojure-antlr-example.git

Setup

I'm going to use leiningen for this project. Let's make a new project called antlr-example.

$ lein new antlr-example

Now edit project.clj to tell lein to fetch ANTLR (and Swank, since I use Emacs). ANTLR is available in Maven Central, so leiningen can grab it for us.

(defproject antlr-example "1.0.0-SNAPSHOT"
  :description "ANTLR Clojure example"
  :dependencies [[org.clojure/clojure "1.2.0-beta1"]
                 [org.clojure/clojure-contrib "1.2.0-beta1"]
                 [org.antlr/antlr "3.2"]]
  :dev-dependencies [[swank-clojure "1.3.0-SNAPSHOT"]])

Then a simple lein deps will download all of the jars and all of ANTLR's dependencies for you. Handy. You end up with this:

antlr-example
├── classes
├── lib
│   ├── antlr-2.7.7.jar
│   ├── antlr-3.2.jar
│   ├── antlr-runtime-3.2.jar
│   ├── clojure-1.2.0-beta1.jar
│   ├── clojure-contrib-1.2.0-beta1.jar
│   ├── stringtemplate-3.2.jar
│   └── swank-clojure-1.3.0-20100502.112537-1.jar
├── project.clj
├── README
├── src
│   └── antlr_example
│       └── core.clj
└── test
    └── antlr_example
        └── core_test.clj

Interactive development? Kind of...

One weakness of ANTLR via Clojure is that development is no longer REPL-based. The ANTLR workflow is essentially a Java workflow:

Write a grammar
Compile the grammar into Java source code
Compile the Java source into Java .class files

Until and unless someone writes a Clojure backend for ANTLR, so that ANTLR can directly produce Clojure source code (and it seems like it should be possible to do so), you're back to a write/compile/debug cycle. Joy!

This also means restarting your REPL every time you alter and recompile your grammar.

But the good thing is that there's ANTLR Works, a free GUI for writing ANTLR grammars. ANTLR Works has an interactive interpreter, which is nice, and it has a compiler/debugger, which is better. These let you test your grammar as you write it, by trying inputs and seeing the resulting AST in graphical form, which is as good as you could hope for. This is actually even a bit nicer than a plaintext REPL.

Plus, it has a nice editor for ANTLR code. Emacs was freaking out over the indentation on a few grammars I tried.

So a decent workflow might be to write and debug your grammar in ANTLR Works, then fire up Clojure afterwards to consume the parser.

A simple grammar

We're going to use a classic textbook example, simple arithmetic expressions, as a proof of concept.

For Java (hence Clojure) to find this code, it has to be named properly and it has to be put into the proper directory on CLASSPATH. I'm going to create the grammar file src/antlr_example/Expr.g. (My grammar file, Java source, and Clojure source will all be jumbled together in src. You can easily do it differently if this offends your sensibilities.)

The first line in the grammar names the grammar (and the classes we'll consume from Clojure):

grammar Expr;

Clojure being a s-exp based language, it might be nice to have ANTLR generate an AST. ANTLR has good support for this:

options {
    output=AST;
    ASTLabelType=CommonTree;
}

Helper tokens for the grammar:

tokens {
    PLUS = '+';
    MINUS = '-';
    MULT = '*';
    DIV = '/';
}

These next lines are important. We want to generate classes antlr_example.ExprLexer and antlr_example.ExprParser, so we have to set our code up in the proper package, antlr_example.

This code includes both a parser and lexer, so you have to set the package for both in ANTLR.

@header        {package antlr_example;}
@lexer::header {package antlr_example;}

And then the grammar itself, which is simple:

expr: term ((PLUS | MINUS)^ term)* ;
term: factor ((MULT | DIV)^ factor)* ;
factor: INT ;

INT :   '0'..'9'+ ;
WS  :   ( ' ' | '\t' | '\r' | '\n')+ {$channel=HIDDEN;} ;

One thing to note is (PLUS | MINUS)^; the ^ here tells ANTLR to treat (PLUS | MINUS) as the parent of the node created for this rule. This means the term on either side will be children of this node. We do the same for MULT and DIV. This will give us a nice tree. Otherwise you end up with a flat list of tokens.

{$channel=HIDDEN;} for the whitespace rule tells ANTLR to ignore whitespace, also useful.

If you use ANTLR Works with this grammar, on the input 1 + 2 * 3 + 4, you can see that it works OK.

That was an awful lot of boilerplate and setup and ugly syntax though. See how much Clojure spoils you? But for a simple grammar, it's not that much code.

For a longer or more complex grammar, you might end up having to embed inline Java code into your ANTLR grammar. But again, that's not the end of the world.

Compile everything

Once you write your grammar (presumably in ANTLR Works), you can generate the Java code and compile it thusly (run from your project's base directory):

$ java -cp 'lib/*' org.antlr.Tool src/antlr_example/Expr.g
$ javac -cp 'lib/*' -d classes src/antlr_example/Expr{Lexer,Parser}.java

The first command should generate src/antlr_example/ExprLexer.java and src/antlr_example/ExprParser.java.

The second command should spew tons of class files into classes/. Leiningen and other tools generally include classes/ on your CLASSPATH, so that's a good place for them. Again putting them here is a fairly arbitrary decision on my part. You can put the source and class files anywhere, as long as the class files end up on your CLASSPATH when you start Clojure.

If you're really so lazy that you can't run these two commands, you could use ANTLR Works' built-in commands for compiling your grammar. The only way I could find to invoke it was by starting a Debug session, which compiled my code as a side-effect. And it spews files into places I don't want them, so I like the command line version better.

You could probably also set up a leiningen task for this if you're re-running these commands a lot, or configure your build tool of choice to do the same. I didn't bother.

Now (re)start your REPL and let's write some Clojure.

Clojure code

It's pretty easy to consume an ANTLR parser from Clojure. I'm putting the code below into src/antlr_example/core.clj

First import the classes. You probably need some ANTLR helper classes to set up the parser; you also need the classes you just generated.

(ns antlr-example.core
  (:import (org.antlr.runtime ANTLRStringStream
                              CommonTokenStream)
           (antlr_example ExprLexer ExprParser)))

Here's a function to parse a string using our new parser class:

(defn parse-expr [s]
  (let [lexer (ExprLexer. (ANTLRStringStream. s))
        tokens (CommonTokenStream. lexer)
        parser (ExprParser. tokens)]
    (.expr parser)))

.expr is the name of the top-level rule we want to invoke from our grammar. The rest is boilerplate; just plug in the proper class names.

So let's see what we've got.

user> (in-ns 'antlr-example.core)
#<Namespace antlr-example.core>
antlr-example.core> (parse-expr "1 + 2 * 3 + 4")
#<expr_return antlr_example.ExprParser$expr_return@3755e508>

Er, OK. A good way to inspect Java objects is via bean, so let's view the guts of this object:

antlr-example.core> (bean *1)
{:tree #<CommonTree +>,
 :template nil,
 :stop #<CommonToken [@12,12:12='4',<8>,1:12]>,
 :start #<CommonToken [@0,0:0='1',<8>,1:0]>,
 :class antlr_example.ExprParser$expr_return}

I see. We set up our grammar above to generate an AST, so :tree on the bean will give us that. This translates to .getTree on the Java object. So we can edit our function to call this for us, since we always want to have the tree.

(defn parse-expr [s]
  (let [lexer (ExprLexer. (ANTLRStringStream. s))
        tokens (CommonTokenStream. lexer)
        parser (ExprParser. tokens)]
    (.getTree (.expr parser))))

Now:

antlr-example.core> (parse-expr "1 + 2 * 3 + 4")
#<CommonTree +>

OK, it appears we have one node of the tree, the root node. Let's peak inside:

antlr-example.core> (bean *1)
{:children #<ArrayList [+, 4]>,
 :childIndex -1,
 :parent nil,
 :text "+",
 :nil false,
 :token #<CommonToken [@10,10:10='+',<4>,1:10]>,
 :class org.antlr.runtime.tree.CommonTree,
 :ancestors nil,
 :tokenStartIndex 0,
 :type 4,
 :childCount 2,
 :charPositionInLine 10,
 :tokenStopIndex 12,
 :line 1}

Looks like our children are in :children on the bean, which translates to .getChildren on the Java object. And we have .getChildCount to count the children, and .getText to get the text of the current node. (We could've learned all of this by reading the javadocs for ANTLR too, but what fun is that?)

Since we're dealing with a tree, we can use tree-seq in Clojure to get a flat list of all the tokens in our text.

tree-seq takes three arguments. First a function that returns true if the node has children, false otherwise. Second a function that returns the children for a node that has children. Third the root node of our tree.

That'll give us a seq of tree nodes. So finally we call .getText on all the resulting nodes in the list, to turn node objects into strings.

Easy enough:

(defn node-seq [x]
  (map #(.getText %)
   (tree-seq #(not (zero? (.getChildCount %)))
             #(.getChildren %)
             x)))

antlr-example.core> (node-seq (parse-expr "1 + 2 * 3 + 4"))
("+" "+" "1" "*" "2" "3" "4")

But that's no good. We lost our tree structure.

We'd rather have something like nested vectors or lists. It's easy enough to roll something by hand. This should do it:

(defn AST [node]
  (if (zero? (.getChildCount node))
    (.getText node)
    (let [children (map AST (.getChildren node))
          txt (.getText node)]
      (if txt
        (cons txt children)
        children))))

antlr-example.core> (AST (parse-expr "1 + 2 * 3 + 4"))
("+" ("+" "1" ("*" "2" "3")) "4")

That's better, but it's a list of strings. These strings all happen to be literal representations of Clojure objects, so a call to read-string in the proper places should give us something we can work with:

(defn AST [node]
  (if (zero? (.getChildCount node))
    (read-string (.getText node))
    (let [children (map AST (.getChildren node))
          txt (read-string (.getText node))]
      (if txt
        (apply list txt children)
        children))))

antlr-example.core> (AST (parse-expr "1 + 2 * 3 + 4"))
(+ (+ 1 (* 2 3)) 4)

Hey look, now we have something we can evaluate.

antlr-example.core> (eval *1)
11

Yeah I kind of planned that ahead. If you had a more complex grammar, you might not get away with something quite this easy.

Conclusion

The advantage of ANTLR is that it's mature, widely used, actively developed, and well-documented. (There's a whole book about ANTLR, The Definitive ANTLR Reference by Terence Parr.) There's also a lot of tooling available for it, not just ANTLR Works, but plugins for other Java IDEs.

The disadvantage is that it's a Java library, and as always, there will be some friction when consuming it from Clojure. But it's not that bad.

For pure-Clojure parser generator alternatives, there are fnparse and clojure-pg, neither of which I've tried much.

Clementine: A triumph of Free Software

Tue, 13 Jul 2010 11:48:51 -0700

Ages ago, in the long-forgotten days of 2008, there was Amarok 1.4. And it was good. Then KDE4 came along and Amarok was rewritten, reshaped, becoming something... different. Something unsettling. Something not altogether pleasant.

Fear not. Today we have Clementine.

I consider Clementine a triumph of Free Software. A great project fell off the rails, so someone else picked up the pieces, forked it and kept the spirit alive.

Features present

Clementine embodies everything good about Amarok 1.4, in a shiny Qt4 package. The layout is eminently pleasant to use. It uses the classic "spreadsheet" playlist view that saw so much success in Amarok 1.4. If you care about cramming as much information about your music as possible onto the screen, this is as good as it gets. It's boring, and that's a good thing. It gets the job done.

Like Amaork, 1.4, in Clementine you can very quickly drill into your music collection, filter it, view recently added tracks, group songs by artist or album or year or genre or a combination of those things. Clementine also handles all of the edge cases correctly: it lists albums with Various Artists exactly how I'd want (exactly like Amarok 1.4). It correctly handles songs with non-Latin tag text.

Clementine detects additions and changes to my music collection instantly, without the massive scan-lags on startup that plague some other music players. Clementine doesn't bat an eye at my 7,000 song collection. There's no MySQL integration, but I don't need it. Clementine's SQLite backend supposedly handles 300k songs without much problem, which is good enough for me.

Clementine has Last.FM integration. It has three different styles of desktop notification. It has visualizations. It handles USB devices. It understands reply gain. It has cross-fading. It has an equalizer. It has a transcoder. It has a cover manager.

I'm tired of listing features. Let's just say it has every useful feature you'd ever want. And if you don't need a feature, it stays out of your way.

And for a program under such active development, it's rock solid. I have yet to see a crash. And speaking of active development, if you follow the activity in Clementine's SVN repo, you will find that this program is updated almost daily. How the devs find the time, I don't know, but I'm grateful. This program has gone from non-existent to awesome in record time.

Clementine can use gstreamer, so it even works cross-platform. I fired it up on Windows 7 the other day and I was amazed at how good it looked and felt. It supposedly also works on OS X.

Clementine doesn't cook your breakfast for you, but that might be in the works.

How to make a good UI

A perfect example of the polish of Clementine's UI: Tagging. How do you tag a whole album worth of music at once? You can select some songs and right click and go into a dialog, like most music players allow.

Or:

Edit a tag for a single song (inline) by clicking the field. Let's say you edit Artist.
Select multiple songs in your playlist. (Click and drag, CTRL-click, Shift-click, CTRL-A, whatever.)
Right click the Artist tag in the song you edited, select Set Artist to "XXXXX", and now all the songs you selected will be updated.

This is the kind of UI innovation that I like. It's simple, it's useful, and it's predictable. You can get things done without going through dialog windows, without a million clicks, without spending a minute scratching your head figuring things out.

(Meanwhile Amarok 2 is busy getting rid of the Stop button and making the volume control circular.)

Features missing

Admittedly, Clementine is missing a couple of features I wouldn't mind having. You can't skin or theme Clementine. You can't rate songs. You can't display song lyrics. You can't "queue" songs. But oh well. I can live without these features because the rest of the program is so darned good. For all I know, these features might pop up next week. I wouldn't be surprised.

The Clementine devs seem to be very friendly and responsive to feature requests and feedback, which is also great.

Clementine is also missing a few features/bloat that I'm glad to see NOT ported from Amarok. Wikipedia integration? Good riddance.

I would pay money for this program.

In November 2009 I had this to say:

(Anyone out there reading this, if you port Amarok 1.4 to Qt4 intact, I will pay you. Seriously. I will pay you money.)

The offer still stands. I will pay money for Clementine. I'm still waiting for a Donate link so I can do so. (Clementine devs, are you reading this?)

Why do I care about this so much? Because I have music playing whenever I'm using this computer, and when you add up work plus free time, I'm at this computer 8-10 hours per day. Music keeps me sane during multi-hour debug sessions. Music is an integral part of my life, and a music app is an integral part of playing music.

It's very important to me that the programs and tools I use all day are comfortable. Otherwise I become cranky. If you were a carpenter, would you want to use a hammer with a wobbly handle all day? I'm a programmer, and I want to use comfortable computer programs.

Clementine is very comfortable.

Vim and plaintext data files

Mon, 12 Jul 2010 10:42:48 -0700

I use Vim to work with plaintext datasets. Here are some habits and code snippets I've picked up which make data files a bit easier to deal with.

Tab-delimited files

If you have a file full of tab-separated values, the columns may not line up very well due to the variable display-length of tabs. You'll often end up with something that looks like this:

foo bar
longvaluehere    quux

But things will line up if you set tabstop to a value greater than the longest column.

:setlocal tabstop=16

Then it'll look like this:

foo             bar
longvaluehere   quux

Now you can use visual-block mode to add and remove columns easily.

If you're working with TSV files, you'll want to set the listchars option in such a way that tabs are displayed specially. I use this, stolen/adapted from here:

:set listchars=eol:\ ,tab:»-,trail:·,precedes:…,extends:…,nbsp:‗

Remember also that you can insert a literal tab into your file no matter what you have expandtab set to, by hitting CTRL-v first.

Those things are just about all I need to work with TSV files comfortably.

Dealing with long lines

Sometimes I'll run some data through a statistics program and it'll tell me "Invalid character at line 576, column 9438". How do you jump to that location in your data? Easily enough in normal mode:

576G9438|

G jumps you to a line, | jumps you to a column. A good mnemonic to remember these: "g = goto line" and "| looks like a column".

Suppose (like me) you turn scrollbars off in your GVim window, or you work from a terminal. How do you scroll horizontally? With zL and zH. But those are hard to type and harder to remember, so I map them to CTRL+Shift+Right and Left.

nnoremap <C-S-Right> zL
nnoremap <C-S-Left> zH

If you have a lot of long lines, it behooves you to set up a mapping to turn soft line-wrapping on and off quickly:

nnoremap <Leader>w :setlocal nowrap!<CR>

Now you can jump back and forth between wrapped and unwrapped views of your data via \w.

Diffs on long lines

When using vimdiff, ]c and [c will jump you to lines that contain differences. But if your lines are thousands of characters long, it doesn't always help to know that two lines are different: you want to know where they differ. ]c puts you at the first column in the line, which isn't helpful.

This function will jump you to the column where the difference between two lines starts:

function! IsDiff(col)
    let hlID = diff_hlID(".", a:col)
    return hlID == 24
endfunction

function! FindDiffOnLine()
    let c = 1
    while c < col("$")
        if IsDiff(c)
            call cursor(".", c)
            return
        endif
        let c += 1
    endwhile
endfunction

This seems fragile, so if you know a better way please leave a comment and let me know.

Diff a buffer against itself

Sometimes I have files with lots of lines of data, and some lines might be duplicate. It's really easy to get rid of the duplicate lines:

:sort u

But what if I want to see what lines were just removed? I don't know of a built-in way to do it, so I use this simple function:

function! MarkDuplicateLines()
    let x = {}
    let count_dupes = 0
    for lnum in range(1, line('$'))
        let line = getline(lnum)
        if has_key(x, line)
            exe lnum . 'norm I *****'
            let count_dupes += 1
        else
            let x[line] = 1
        endif
    endfor
    echomsg count_dupes . " dupe(s) found"
endfunction

That'll put a bunch of asterisks at the beginning of every line that's a duplicate of a previous line.

Fix broken punctuation

Ever had to work with textual data that someone else sent you in MS Word or Excel? I have. It hurts. If you copy/paste from an MS document into a plaintext file, you'll probably end up with a bunch of funky unreadable or undisplayable characters.

This function fixes the most common Word-spawned "smart" punctuation characters that you're likely to run across:

function! FixInvisiblePunctuation()
    silent! %s/\%u2018/'/g
    silent! %s/\%u2019/'/g
    silent! %s/\%u2026/.../g
    silent! %s/\%uf0e0/->/g
    silent! %s/\%u0092/'/g
    silent! %s/\%u2013/--/g
    silent! %s/\%u2014/--/g
    silent! %s/\%u201C/"/g
    silent! %s/\%u201D/"/g
    silent! %s/\%u0052\%u20ac\%u2122/'/g
    silent! %s/\%ua0/ /g
    retab
endfunction

This function was built up during years of choking on special characters that crept up in my data, so I'm sure I'll be adding more to it in the future.

\%u#### here is a regex escape to represent a character as a hexidecimal Unicode codepoint.

Many of these characters show up as blank (whitespace) if you lack the font to display them. If you ever run across a character you can't see and you want to inspect it, put the cursor on it and do

:ascii

That's a bad name for the command, since it works on Unicode characters too. That'll give you the numeric code you can use in a regex to replace them all with something sane.

Dell: the aftermath

Thu, 08 Jul 2010 11:11:28 -0700

In a previous post I outlined the ways in which Dell's customer service sucks. I finally got my computer yesterday, a Studio XPS 9000. Here are my first impressions.

The bad

This computer weighs so much I almost hurt my back lifting it. I thought computers were supposed to be getting smaller and lighter?
The HD indicator light is tiny and on the top of the case. I can't see it with my computer under my desk.

The optical drive is behind one of those stupid plastic flap door things. So there isn't even an indicator light for the DVD drive. I'm seriously considering taking a screwdriver the case to fix this.
It didn't come with a Windows install disk or a driver disk. It only has a recovery partition on the HD.

I found an order form which I think will get me my disks. In the mail. Seriously, Dell? Seriously? Why not come to my house and kick me in the balls while you're at it?

The recovery partition doesn't help you worth a crap if you want to do things like repartition your drive to put Linux on it. Windows7's sucky partition shrinking app wouldn't shrink it lower than 500GB.
The Dell recovery program is called DataSafe or something, and when you use it, it tries to upsell you like crazy to get a "pro" version that has a bunch of useless backup features. Uggggghhhhh.
The side of the case is white. In 10 years it's either going to be yellow with age, or scuffed up beyond hope. Kind of ugly, but I don't care much. The front of the case looks OK though. Black with red highlights. About as good as I could expect.
It came pre-installed with some crappy "Dell Dock" knockoff of Apple's Dock. Worthless and instantly uninstalled.

This thing caused the desktop icons to be hidden by default. Who would possibly want to do this in Windows? I can image everyone and their grandmother being awfully confused by the missing icons.

When quitting this dock, it said "Undo is not possible". I love a program that has no going back once you quit.
I wanted to find drivers for the wireless card that came with the Dell. So I went to Dell's support site and typed in the tag number on my computer. It gave me a link to drivers for the wrong card. I had to google all over the place to find the right ones. Way to go.

Dell's website is a labyrinth full of outdated information and dead pages in general.
The instructions I got with the computer reference Vista. I don't have Vista.
There's a "Windows inside" logo on the case. It will be removed shortly. They leave an awful lot of glue behind.

The good?

The i7 is about as fast as I had hoped. It only took a half-dozen cores and 12 GB of RAM to let me watch full-screen flash videos on Youtube. I feel so modern.
The inside of the case is OK. There are a lot of hard drive bays and lots of extra screws. It should be easy to expand if I need to.
It came with bloatware and crapware, but actually far less than I was dreading. And most of it was trying to sell you Dell crap.

In the olden days you'd get a hundred links to AOL and other 3rd-party crap. I saw a link to Skype and the obligatory nag to buy an anti-virus subscription (fat chance), but not much else.
Dell delivered the computer 2 days past the original estimated delivery date. So in spite of all the bullcrap and phone-jockeying I had to go through for billing, I can't complain about how fast it got here. Two days late isn't bad.

I've heard rumors that these computers are built in Malaysia, and mine was definitely shipped from the US (per the Purolator tracking site). So I'm surprised they can get these things delivered as fast as they can given that it was shipped halfway around the world first, and had to go through Customs at least once coming from the US to Canada.

Purolator was the only shipping option for Canada. I would've preferred to rush it. But maybe that's not possible given that it's coming from the US. Oh well.
It runs pretty quiet, given how huge the fans are. We'll see how hot it gets once I start putting some load on it.
It came with a DVI to VGA adapater and a DVI to HDMI adapter. I thought that was a nice touch, though it could be that they come standard with any nVidia card nowadays.
Works OK with Linux. It took 20 minutes to set up. (Not counting wiping the Windows partition and re-installing on a smaller partition from my own copy, minus the crapware. That took over an hour.) Sound, video and wireless work out of the box in Linux. All 12 GB of RAM are usable, given a 64 bit OS. (I discovered this the fun way, by unthinkingly trying a 32 bit OS first.)
It didn't burst into flames (yet).
It has a peanut tray on the top. Or MP3-player tray, I guess. But I really want to put peanuts in there.

Brian, you're stupid!

So why did I get a Dell? Because I had good experience with them in the past, at home and at work. Given, that experience was 5 years in the past, and a lot can change. And I'm new to Canada, and relatively unaware of what options exist here.

The other (main) reason was that they were far, far cheaper than going through newegg.ca to get the same hardware. But I guess you get what you pay for. Caveat emptor.

I wouldn't recommend Dell to anyone else, given how chaotic the whole buying process was. Too much uncertainty, too much room for mistakes.

Dave asked in my previous post why I didn't just a computer myself, like I had in the past. I said I didn't have time, but what I meant wasn't build time, which should be an hour or two max. I meant research time. Trying to match up compatible hardware, trying to find the best prices on all the components, checking for Linux compatibility, this takes forever and a half. I don't have hours / days to dork around with this any more.

On the other hand I can just google "xps 9000 linux" and see instantly what problems people had. I can be semi-confident that the hardware would all be compatible. And that did work out OK.

And the last reason I got Dell is that unfortunately I need Windows for work and gaming. Blarg. Paying the Windows tax to Dell is bad enough, let alone buying one off the shelf for $6,000 or however much they cost nowadays.

Sorry for the downtime

Thu, 08 Jul 2010 08:29:37 -0700

This isn't the kind of email I like to see when I wake up:

Our backend monitoring system has detected an error on the host where your Linode resides which could lead to a failure condition. In order to protect your Linode, we have scheduled an emergency migration to a different host which will commence shortly. Please note that there is currently no issue with your Linode - this is a proactive measure we are taking to avoid an issue in the future.

We apologize for any inconvenience this may cause. Should you have any questions or concerns, please do not hesitate to reply to this ticket.

The server rebooted and my sites didn't come back up, so there was some downtime. Sorry about that.

In which border-radius is abused

Mon, 05 Jul 2010 15:54:24 -0700

I threw together a new blog layout today. I scaled back the level of cows a bit (just a bit, don't worry!) Criticism / feedback welcome. (IE-related feedback should be dropped off in the circular file by my desk.)

In what is surely a prelude to the future of the internets, I'm abusing border-radius pretty heavily in this layout. border-radius is likely to become the new marquee HTML tag or text-shadow CSS attribute: Maybe an OK idea at first, but then everyone uses it so much it makes your eyes bleed.

So I figured I'd best get my border-radiusing in early while it's still cool. IE8 users, you still get pointy corners. Sucks to be you.

Also, if you have any ideas for features I should implement for cow-blog, please let me know. I've been crawling the internet looking at blogs for ideas of things to implement and features to steal, but I'm running out of ideas. It does everything I want now, but I'm not a reader.

Dell sucks

Fri, 02 Jul 2010 10:57:41 -0700

Why did I order a computer from Dell? I guess I had a good opinion from 6 years ago when I last bought something from them.

Let's count the ways in which their customer service has failed me. (And my computer isn't even here yet.)

As documented, their website couldn't process my credit card without a phone call.
After a week of my computer being "in production", I started getting more phone calls from an unidentified phone number that Google told me was Dell. Fearing another billing problem, I called back. And I was told "Thanks for calling, but our order tracking system is down. And we're all going home. Call back tomorrow morning.".

If only Dell had some means to acquire reliable computer systems on which to build their order tracking database.
I called the next day and was told my order was fine. I was also told (per script, I'm certain) that I could check my order status on Dell's website. Which of course I knew. I know it costs the company money every time someone calls, and they try to strongly discourage calls for that reason, but their script made it sound like I was an imbecile.

I found it quite condescending. I dislike these canned scripts pander to the lowest common denominator of customer. They should be happy to take my call. I just spend upwards of a thousand dollars on their crap.
Turns out the phone calls I was getting were from someone trying to give me "free internet from Shaw or Telus for 3 months", and I was eligible because I bought a Dell computer. So I was being telemarketed before my computer even got here.

I said I already had internet service, and they said "Oh, too bad, it's for new customers only." I do not appreciate this.
I got an email saying my order shipped. Joy! 20 minutes later I got an email saying my order was delayed, and if it didn't ship in 5 days I should call. What?

It really did ship though, I have a tracking number. Why the contradictory emails?

All of my phone dealings with Dell were via some offshored far-eastern country, judging by the accents of the phone reps. I have nothing against this in principle; I'm not a xenophobe. But the phone connection is always so static-filled and laggy that it really puts a damper on communication.

My computer isn't here yet, and I just hope to God it works and doesn't break in a month. I kind of wish this article had come out a week earlier.

That'll teach me for trying to save time, I guess. Next time I'll build my own system from scratch. Dell goes onto my List of Companies Not to Buy From in the Future (LCNBFF), along with Westinghouse and oh so many others.