briancarper.net (λ)

gaka 0.2.0

Thu, 29 Jul 2010 13:59:22 -0700

Per many commenters' suggestions and thanks to code from Steve Purcell, you can now use maps for CSS attributes in gaka.

user> (println (gaka/css [:a {:color :red}]))
a {
  color: red;}

This looks more like vanilla CSS thanks to the curlies, which is nice. You just have to keep in mind that your key/value pairs could end up being printed in random order, and order is significant¹ in CSS.

It just so happens that maps are implemented in Clojure right now such that if they only have a few entries (16 key/value pairs), the order will be preserved, because you get a PersistentArrayMap instead of a PersistentHashMap. But it's highly dangerous to rely on such a thing. It could change at any time in the future.

In any case, you can also mix and match maps, lists and "flat" keyvals. They'll all be flattened That can help preserve attribute order in those cases where you need to.

user> (println (gaka/css [:a :color "red" {:padding 0} (list :margin 0)]))
a {
  color: red;
  padding: 0;
  margin: 0;}

I've also enhanced "mixins" a bit further. You can now mixin entire tags as well as attributes. Or a combination of both. Say you want a mixin that means "Make my element have no padding, and make links within the element be red":

user> (println (gaka/css [:div.foo mixin :margin 0]
                         [:div.bar mixin]))
div.foo {
  padding: 0;
  margin: 0;}

  div.foo a {
    color: red;}

div.bar {
  padding: 0;}

  div.bar a {
    color: red;}

You can get gaka from github or Clojars.

Order is only significant in cases where you're doing things like padding: 0; padding-left: 1px. This is arguably bad CSS style, but it's valid, and it's also possible you'll have this kind of thing if you're generating CSS procedurally. But most of the time, order is not significant. e.g. it doesn't matter if you set text color first and background color second, or vice versa. So maybe this isn't so much of a problem in practice. ↩

Footnotes

Tue, 27 Jul 2010 12:30:57 -0700

Did you ever notice how footnotes make your writing seem more important¹ somehow?

Maybe one reason is that "real" books use footnotes. At a glance, it looks like I have references² backing up everything I say. In reality, I don't, but the connotation carries through somehow³. Now my blog seems scholarly and authoritative.

And if you're like me, you can't resist clicking footnotes to see what they refer to⁴. According to my estimates, by utilizing footnotes, in one fell swoop I have decreased my readers' average reading efficiency by 73%.

In any case, I've added experimental, rudimentary support for footnotes to cow-blog.

I'm loosely copying the syntax from Markdown Extra for this. Markdown is great, except when it isn't. The standard doesn't have support for some useful extensions. I use Showdown for Markdown support, and I'm probably going to work on adding more features of Markdown Extra to Showdown in the near future.

I just dread actually doing it. Showdown (like Markdown itself) is implemented as a series of hackish regex transformations of blobs of text. It's not a proper grammar. Implementing more of Markdown Extra means more regex blobbing. It's brittle and fragile and even getting incomplete support for footnotes was less than enjoyable. But at the same time I find myself wanting to do things that Markdown can't so, so I may have to bite the bullet.

(If there's a Showdown Extra out there already, drop me a URL. It'd be most appreciated. But I couldn't find one.)

In reality nothing I say is important. ↩
Does my inner dialog count as a reference? ↩
Via telepathy. ↩
See? ↩

Review: Surely You're Joking, Mr. Feynman!

Mon, 26 Jul 2010 16:19:21 -0700

I'm probably the last person on earth to read "Surely You're Joking, Mr. Feynman!", a collection of stories and anecdotes from the life of Richard Feynman. But better late than never.

I'll keep this short. Feynman was the kind of nerd every nerd wishes he was, in one way or another. He was socially awkward. He was blunt and tactless. He felt out of place much of the time. And it was surprising to see how often Feynman expressed feelings of inadequacy. He wrote about being highly intimidated when talking in front of big names in physics, and jealous of the math abilities of some other people.

But none of that stopped him from having an exciting life. In fact he turned these traits to his advantage. He had romantic success, often via very highly unconventional means (e.g. walking up to girls and asking them for sex outright; they often said yes). His bluntness was seen as an admirable trait: where others might be intimidated by a famous physicist, Feynman would give his honest opinion, and that was often appreciated.

And Feynman went far outside his comfort zone. He did a stint in biology, even though he knew nothing of biology at the time. He went to Brazil and joined a samba band. He sold drawings and paintings for a while. He played drums for a ballet.

I picked up at least three lessons from this book.

Try new things, even if you suck at them. Life is boring if you stick to what you're good at.
Be intellectually honest. Brutally so. It's the only way to do good science, and arguably the only way to live a good life. I've always believed this, and Feynman hammers the point home well.
Even the best and brightest of us feel insecure at times. You shouldn't let it stop you.

On top of all of that, reading of Feynman's time at Los Alamos working on The Bomb was a fascinating piece of history. I highly recommend this book.

And watch as much of Feynman on Youtube as you can find. It never ceases to be fascinating.

Review: Logitech Performance MX

Mon, 26 Jul 2010 13:51:27 -0700

I have a long list of companies I won't buy from, due to horrible customer service experiences or shoddy merchandise. On the other hand, my "Must buy from this company" list is awfully short.

Logitech is one company I'm generally OK buying from. (For now...) I go through computer mouses¹ pretty fast, so I end up buying a new one every 4-5 years. Logitech hasn't failed me thus far.

In my opinion, the most important parts of your computer are the ones you interact with: Keyboard, mouse, monitor. I'd rather have a slow computer with a good mouse and keyboard than a fast computer with cheap peripherals. I already have a wonderful keyboard, and I like to have a nice mouse to match.

My latest mouse is the Logitech Performance MX.

How much can you really improve upon the concept of a mouse? It's a round thing you slide around and click. Well, if you believe the marketing on the box, it seems there are at least three areas for improvement:

Add more buttons.
Make it work on more surfaces.
Gimmicks.

None of the above features matter much to me. What I care about:

Is it comfortable to use all day?
Is it going to last me 4 or 5 years?

Let's hit each of these points.

More buttons

The Performance MX has a lot of buttons. If you believe xev, there are somewhere between 10 and 13 (counting the scroll wheel etc.).

I don't need all of these. I like to have a thumb button for middle-clicking, but this mouse has four of them. What am I going to use all of the extras for? Maybe I'll have one open Emacs.

The browser forward and back thumb buttons work out-of-the-box in Linux, which surprised me. I'm used to having to dork around with xmodmap or xbindkeys. Linux has come a long way.

Then there's a thumb button that's supposed to open Expose on OS X or your app-switcher of choice in your OS. It's in a place that's impossible to press consistently with your thumb, unless you're a contortionist. And it does nothing in Linux by default. Similarly there's a vanishingly small "zoom" button near your thumb. I can't even find this button without looking at the mouse, because it's too small.

There's also horizontal scroll, if you can manage to tilt the wheel to either side. My fingers lack the dexterity to do this easily, and the button is awfully stiff, so I'll never use this.

But hey, better too many buttons than too few.

Surfaces

The marketers wrote "This mouse works on glass!" all over the box, so this must be a selling point of the mouse. But my Logitech G500 also worked on glass, once I upgraded the firmware (via Logitech's own website), so I'm not sure why this feature is being pushed so hard.

Gimmicks

On the Performance MX, you can click this little switch and disengage part of the mechanism that makes the scroll wheel "click". This lets you spin the wheel hard and it'll spin freely forever until you stop it manually, or until its inertia runs out.

Do I really need, and I quote, hyper-fast scrolling? The scroll speed of my mouse wheel has never struck me as a productivity bottleneck. Now that I've played with it, sure, it's mildly interesting. If you scroll at just the right speed, you can go all the way from the top of a Reddit thread to the bottom, while still skimming the text. But I don't need this feature.

But the levels to which marketroids will stoop to hype up a product never ceases to amaze me. The name itself is ridiculous. Performance MX. Who comes up with this?

Comfort

My previous mouse was a G500, and it was really quite big. A bit too big. (And there were these removable weights you could use to adjust the weight of the mouse (the "gimmick" feature of that mouse), which were worthless.)

The MX is bigger, but it has more curves and that helps my hand sit on it better. There's a very large thumb rest, which I like. If you have small hands, you might have problems with this mouse. I like the feel so far.

Will it last?

This mouse feels very solid. It has a nice heft to it. It has metal bits (or sturdy metal-like plastic) in places other mouses have cheap plastic and rubber.

The wheel on my G500 wore down very fast. I was some kind of rubbery plastic, which became rounded off, nicked and worn-down a lot over the years. The wheel on the MX feels like some kind of metal, with nice ridges on the top to provide friction. I'm a bit concerned that the wheel disengagement switch might wear down over time though. The more moving parts, the more opportunity for something to break.

The pads on the bottom are huge: a good 1 inch x 1/8 inch. I love this. Using this on my Icemat mouse pad is very comfy. It slides effortlessly, but leaves me with plenty of control.

Battery

I normally prefer corded mouses. This one is rechargeable. Most rechargeable mouses I've used had some kind of stupid dock you had to jam the mouse into. Not only does it take up desk space, it also wears holes into your mouse over time from being inserted and removed from the dock. The battery connectors also wear down fast, and then it doesn't charge right unless you fiddle with it.

The MX can be charged while you use it, via a USB cable, which is a really good idea. It comes with a short cable, and a long extender cable if you need it. The cable looks like it has a non-standard connector though, which is a bad idea. Why not use mini-USB?

The charger cable is awfully heavy though. I wouldn't want to use this thing while it's plugged in for more than a short time.

I don't know what the battery life is like, because it hasn't run out yet, after a couple days. That's good enough for me.

Price

$120. That's a lot. But taken over 4-5 years, it's not that bad.

And the surgery I'd have to have done on my hands after crippling them via a cheap mouse over the next 40 years would be way more than $120.

Should you buy this?

Sure. I like this mouse so far. It works, and it feels like it'll last a while. Props to Logitech for making good products.

Yes, mouses. ↩

Clojure syntax highlighting via SyntaxHighlighter

Wed, 21 Jul 2010 15:17:42 -0700

How do you syntax-highlight Clojure code for display on a website? The best way I can find is SyntaxHighlighter.

Daniel Gómez wrote a brush to give SyntaxHighlighter Clojure support. I tweaked it a bit myself and integrated it into cow-blog. I also converted my favorite color scheme to a SyntaxHighlighter theme. So when I write this code:

(defn- ip
  "Given a request, return the IP.  Looks for an x-forwarded-for
  header, falls back to :remote-addr on the request."
  [request]
  (or (get-in request [:headers "x-forwarded-for"])
      (request :remote-addr)))

You should see something like this:

...unless you're reading this via RSS, or in a browser without Javascript enabled, in which case you'll see plain, depressing black and white. But that's one nice thing about SyntaxHighlighter. It degrades nicely.

One bad thing about SyntaxHighlighter is that it doesn't play nicely with Markdown. Or rather, Markdown isn't powerful enough to let you specify the class of any markdown-generated HTML tags. If you want the <pre class="brush: clojure"> that SyntaxHighlighter requires, you have to write out the HTML by hand. But I hacked Showdown a bit to let me specify classes more easily, so I can avoid having to do that.

The code for all of this is on github with the rest of my blog.

ANTLR via Clojure

Fri, 16 Jul 2010 17:41:04 -0700

ANTLR is a parser-generator for Java. Can you use it from Clojure? Sure. Would you want to? Maybe.

Here's how to do it, start to finish.

For the impatient among you, all of the code below is on github.

git clone git://github.com/briancarper/clojure-antlr-example.git

Setup

I'm going to use leiningen for this project. Let's make a new project called antlr-example.

$ lein new antlr-example

Now edit project.clj to tell lein to fetch ANTLR (and Swank, since I use Emacs). ANTLR is available in Maven Central, so leiningen can grab it for us.

(defproject antlr-example "1.0.0-SNAPSHOT"
  :description "ANTLR Clojure example"
  :dependencies [[org.clojure/clojure "1.2.0-beta1"]
                 [org.clojure/clojure-contrib "1.2.0-beta1"]
                 [org.antlr/antlr "3.2"]]
  :dev-dependencies [[swank-clojure "1.3.0-SNAPSHOT"]])

Then a simple lein deps will download all of the jars and all of ANTLR's dependencies for you. Handy. You end up with this:

antlr-example
├── classes
├── lib
│   ├── antlr-2.7.7.jar
│   ├── antlr-3.2.jar
│   ├── antlr-runtime-3.2.jar
│   ├── clojure-1.2.0-beta1.jar
│   ├── clojure-contrib-1.2.0-beta1.jar
│   ├── stringtemplate-3.2.jar
│   └── swank-clojure-1.3.0-20100502.112537-1.jar
├── project.clj
├── README
├── src
│   └── antlr_example
│       └── core.clj
└── test
    └── antlr_example
        └── core_test.clj

Interactive development? Kind of...

One weakness of ANTLR via Clojure is that development is no longer REPL-based. The ANTLR workflow is essentially a Java workflow:

Write a grammar
Compile the grammar into Java source code
Compile the Java source into Java .class files

Until and unless someone writes a Clojure backend for ANTLR, so that ANTLR can directly produce Clojure source code (and it seems like it should be possible to do so), you're back to a write/compile/debug cycle. Joy!

This also means restarting your REPL every time you alter and recompile your grammar.

But the good thing is that there's ANTLR Works, a free GUI for writing ANTLR grammars. ANTLR Works has an interactive interpreter, which is nice, and it has a compiler/debugger, which is better. These let you test your grammar as you write it, by trying inputs and seeing the resulting AST in graphical form, which is as good as you could hope for. This is actually even a bit nicer than a plaintext REPL.

Plus, it has a nice editor for ANTLR code. Emacs was freaking out over the indentation on a few grammars I tried.

So a decent workflow might be to write and debug your grammar in ANTLR Works, then fire up Clojure afterwards to consume the parser.

A simple grammar

We're going to use a classic textbook example, simple arithmetic expressions, as a proof of concept.

For Java (hence Clojure) to find this code, it has to be named properly and it has to be put into the proper directory on CLASSPATH. I'm going to create the grammar file src/antlr_example/Expr.g. (My grammar file, Java source, and Clojure source will all be jumbled together in src. You can easily do it differently if this offends your sensibilities.)

The first line in the grammar names the grammar (and the classes we'll consume from Clojure):

grammar Expr;

Clojure being a s-exp based language, it might be nice to have ANTLR generate an AST. ANTLR has good support for this:

options {
    output=AST;
    ASTLabelType=CommonTree;
}

Helper tokens for the grammar:

tokens {
    PLUS = '+';
    MINUS = '-';
    MULT = '*';
    DIV = '/';
}

These next lines are important. We want to generate classes antlr_example.ExprLexer and antlr_example.ExprParser, so we have to set our code up in the proper package, antlr_example.

This code includes both a parser and lexer, so you have to set the package for both in ANTLR.

@header        {package antlr_example;}
@lexer::header {package antlr_example;}

And then the grammar itself, which is simple:

expr: term ((PLUS | MINUS)^ term)* ;
term: factor ((MULT | DIV)^ factor)* ;
factor: INT ;

INT :   '0'..'9'+ ;
WS  :   ( ' ' | '\t' | '\r' | '\n')+ {$channel=HIDDEN;} ;

One thing to note is (PLUS | MINUS)^; the ^ here tells ANTLR to treat (PLUS | MINUS) as the parent of the node created for this rule. This means the term on either side will be children of this node. We do the same for MULT and DIV. This will give us a nice tree. Otherwise you end up with a flat list of tokens.

{$channel=HIDDEN;} for the whitespace rule tells ANTLR to ignore whitespace, also useful.

If you use ANTLR Works with this grammar, on the input 1 + 2 * 3 + 4, you can see that it works OK.

That was an awful lot of boilerplate and setup and ugly syntax though. See how much Clojure spoils you? But for a simple grammar, it's not that much code.

For a longer or more complex grammar, you might end up having to embed inline Java code into your ANTLR grammar. But again, that's not the end of the world.

Compile everything

Once you write your grammar (presumably in ANTLR Works), you can generate the Java code and compile it thusly (run from your project's base directory):

$ java -cp 'lib/*' org.antlr.Tool src/antlr_example/Expr.g
$ javac -cp 'lib/*' -d classes src/antlr_example/Expr{Lexer,Parser}.java

The first command should generate src/antlr_example/ExprLexer.java and src/antlr_example/ExprParser.java.

The second command should spew tons of class files into classes/. Leiningen and other tools generally include classes/ on your CLASSPATH, so that's a good place for them. Again putting them here is a fairly arbitrary decision on my part. You can put the source and class files anywhere, as long as the class files end up on your CLASSPATH when you start Clojure.

If you're really so lazy that you can't run these two commands, you could use ANTLR Works' built-in commands for compiling your grammar. The only way I could find to invoke it was by starting a Debug session, which compiled my code as a side-effect. And it spews files into places I don't want them, so I like the command line version better.

You could probably also set up a leiningen task for this if you're re-running these commands a lot, or configure your build tool of choice to do the same. I didn't bother.

Now (re)start your REPL and let's write some Clojure.

Clojure code

It's pretty easy to consume an ANTLR parser from Clojure. I'm putting the code below into src/antlr_example/core.clj

First import the classes. You probably need some ANTLR helper classes to set up the parser; you also need the classes you just generated.

(ns antlr-example.core
  (:import (org.antlr.runtime ANTLRStringStream
                              CommonTokenStream)
           (antlr_example ExprLexer ExprParser)))

Here's a function to parse a string using our new parser class:

(defn parse-expr [s]
  (let [lexer (ExprLexer. (ANTLRStringStream. s))
        tokens (CommonTokenStream. lexer)
        parser (ExprParser. tokens)]
    (.expr parser)))

.expr is the name of the top-level rule we want to invoke from our grammar. The rest is boilerplate; just plug in the proper class names.

So let's see what we've got.

user> (in-ns 'antlr-example.core)
#<Namespace antlr-example.core>
antlr-example.core> (parse-expr "1 + 2 * 3 + 4")
#<expr_return antlr_example.ExprParser$expr_return@3755e508>

Er, OK. A good way to inspect Java objects is via bean, so let's view the guts of this object:

antlr-example.core> (bean *1)
{:tree #<CommonTree +>,
 :template nil,
 :stop #<CommonToken [@12,12:12='4',<8>,1:12]>,
 :start #<CommonToken [@0,0:0='1',<8>,1:0]>,
 :class antlr_example.ExprParser$expr_return}

I see. We set up our grammar above to generate an AST, so :tree on the bean will give us that. This translates to .getTree on the Java object. So we can edit our function to call this for us, since we always want to have the tree.

(defn parse-expr [s]
  (let [lexer (ExprLexer. (ANTLRStringStream. s))
        tokens (CommonTokenStream. lexer)
        parser (ExprParser. tokens)]
    (.getTree (.expr parser))))

Now:

antlr-example.core> (parse-expr "1 + 2 * 3 + 4")
#<CommonTree +>

OK, it appears we have one node of the tree, the root node. Let's peak inside:

antlr-example.core> (bean *1)
{:children #<ArrayList [+, 4]>,
 :childIndex -1,
 :parent nil,
 :text "+",
 :nil false,
 :token #<CommonToken [@10,10:10='+',<4>,1:10]>,
 :class org.antlr.runtime.tree.CommonTree,
 :ancestors nil,
 :tokenStartIndex 0,
 :type 4,
 :childCount 2,
 :charPositionInLine 10,
 :tokenStopIndex 12,
 :line 1}

Looks like our children are in :children on the bean, which translates to .getChildren on the Java object. And we have .getChildCount to count the children, and .getText to get the text of the current node. (We could've learned all of this by reading the javadocs for ANTLR too, but what fun is that?)

Since we're dealing with a tree, we can use tree-seq in Clojure to get a flat list of all the tokens in our text.

tree-seq takes three arguments. First a function that returns true if the node has children, false otherwise. Second a function that returns the children for a node that has children. Third the root node of our tree.

That'll give us a seq of tree nodes. So finally we call .getText on all the resulting nodes in the list, to turn node objects into strings.

Easy enough:

(defn node-seq [x]
  (map #(.getText %)
   (tree-seq #(not (zero? (.getChildCount %)))
             #(.getChildren %)
             x)))

antlr-example.core> (node-seq (parse-expr "1 + 2 * 3 + 4"))
("+" "+" "1" "*" "2" "3" "4")

But that's no good. We lost our tree structure.

We'd rather have something like nested vectors or lists. It's easy enough to roll something by hand. This should do it:

(defn AST [node]
  (if (zero? (.getChildCount node))
    (.getText node)
    (let [children (map AST (.getChildren node))
          txt (.getText node)]
      (if txt
        (cons txt children)
        children))))

antlr-example.core> (AST (parse-expr "1 + 2 * 3 + 4"))
("+" ("+" "1" ("*" "2" "3")) "4")

That's better, but it's a list of strings. These strings all happen to be literal representations of Clojure objects, so a call to read-string in the proper places should give us something we can work with:

(defn AST [node]
  (if (zero? (.getChildCount node))
    (read-string (.getText node))
    (let [children (map AST (.getChildren node))
          txt (read-string (.getText node))]
      (if txt
        (apply list txt children)
        children))))

antlr-example.core> (AST (parse-expr "1 + 2 * 3 + 4"))
(+ (+ 1 (* 2 3)) 4)

Hey look, now we have something we can evaluate.

antlr-example.core> (eval *1)
11

Yeah I kind of planned that ahead. If you had a more complex grammar, you might not get away with something quite this easy.

Conclusion

The advantage of ANTLR is that it's mature, widely used, actively developed, and well-documented. (There's a whole book about ANTLR, The Definitive ANTLR Reference by Terence Parr.) There's also a lot of tooling available for it, not just ANTLR Works, but plugins for other Java IDEs.

The disadvantage is that it's a Java library, and as always, there will be some friction when consuming it from Clojure. But it's not that bad.

For pure-Clojure parser generator alternatives, there are fnparse and clojure-pg, neither of which I've tried much.

Clementine: A triumph of Free Software

Tue, 13 Jul 2010 11:48:51 -0700

Ages ago, in the long-forgotten days of 2008, there was Amarok 1.4. And it was good. Then KDE4 came along and Amarok was rewritten, reshaped, becoming something... different. Something unsettling. Something not altogether pleasant.

Fear not. Today we have Clementine.

I consider Clementine a triumph of Free Software. A great project fell off the rails, so someone else picked up the pieces, forked it and kept the spirit alive.

Features present

Clementine embodies everything good about Amarok 1.4, in a shiny Qt4 package. The layout is eminently pleasant to use. It uses the classic "spreadsheet" playlist view that saw so much success in Amarok 1.4. If you care about cramming as much information about your music as possible onto the screen, this is as good as it gets. It's boring, and that's a good thing. It gets the job done.

Like Amaork, 1.4, in Clementine you can very quickly drill into your music collection, filter it, view recently added tracks, group songs by artist or album or year or genre or a combination of those things. Clementine also handles all of the edge cases correctly: it lists albums with Various Artists exactly how I'd want (exactly like Amarok 1.4). It correctly handles songs with non-Latin tag text.

Clementine detects additions and changes to my music collection instantly, without the massive scan-lags on startup that plague some other music players. Clementine doesn't bat an eye at my 7,000 song collection. There's no MySQL integration, but I don't need it. Clementine's SQLite backend supposedly handles 300k songs without much problem, which is good enough for me.

Clementine has Last.FM integration. It has three different styles of desktop notification. It has visualizations. It handles USB devices. It understands reply gain. It has cross-fading. It has an equalizer. It has a transcoder. It has a cover manager.

I'm tired of listing features. Let's just say it has every useful feature you'd ever want. And if you don't need a feature, it stays out of your way.

And for a program under such active development, it's rock solid. I have yet to see a crash. And speaking of active development, if you follow the activity in Clementine's SVN repo, you will find that this program is updated almost daily. How the devs find the time, I don't know, but I'm grateful. This program has gone from non-existent to awesome in record time.

Clementine can use gstreamer, so it even works cross-platform. I fired it up on Windows 7 the other day and I was amazed at how good it looked and felt. It supposedly also works on OS X.

Clementine doesn't cook your breakfast for you, but that might be in the works.

How to make a good UI

A perfect example of the polish of Clementine's UI: Tagging. How do you tag a whole album worth of music at once? You can select some songs and right click and go into a dialog, like most music players allow.

Or:

Edit a tag for a single song (inline) by clicking the field. Let's say you edit Artist.
Select multiple songs in your playlist. (Click and drag, CTRL-click, Shift-click, CTRL-A, whatever.)
Right click the Artist tag in the song you edited, select Set Artist to "XXXXX", and now all the songs you selected will be updated.

This is the kind of UI innovation that I like. It's simple, it's useful, and it's predictable. You can get things done without going through dialog windows, without a million clicks, without spending a minute scratching your head figuring things out.

(Meanwhile Amarok 2 is busy getting rid of the Stop button and making the volume control circular.)

Features missing

Admittedly, Clementine is missing a couple of features I wouldn't mind having. You can't skin or theme Clementine. You can't rate songs. You can't display song lyrics. You can't "queue" songs. But oh well. I can live without these features because the rest of the program is so darned good. For all I know, these features might pop up next week. I wouldn't be surprised.

The Clementine devs seem to be very friendly and responsive to feature requests and feedback, which is also great.

Clementine is also missing a few features/bloat that I'm glad to see NOT ported from Amarok. Wikipedia integration? Good riddance.

I would pay money for this program.

In November 2009 I had this to say:

(Anyone out there reading this, if you port Amarok 1.4 to Qt4 intact, I will pay you. Seriously. I will pay you money.)

The offer still stands. I will pay money for Clementine. I'm still waiting for a Donate link so I can do so. (Clementine devs, are you reading this?)

Why do I care about this so much? Because I have music playing whenever I'm using this computer, and when you add up work plus free time, I'm at this computer 8-10 hours per day. Music keeps me sane during multi-hour debug sessions. Music is an integral part of my life, and a music app is an integral part of playing music.

It's very important to me that the programs and tools I use all day are comfortable. Otherwise I become cranky. If you were a carpenter, would you want to use a hammer with a wobbly handle all day? I'm a programmer, and I want to use comfortable computer programs.

Clementine is very comfortable.

Vim and plaintext data files

Mon, 12 Jul 2010 10:42:48 -0700

I use Vim to work with plaintext datasets. Here are some habits and code snippets I've picked up which make data files a bit easier to deal with.

Tab-delimited files

If you have a file full of tab-separated values, the columns may not line up very well due to the variable display-length of tabs. You'll often end up with something that looks like this:

foo bar
longvaluehere    quux

But things will line up if you set tabstop to a value greater than the longest column.

:setlocal tabstop=16

Then it'll look like this:

foo             bar
longvaluehere   quux

Now you can use visual-block mode to add and remove columns easily.

If you're working with TSV files, you'll want to set the listchars option in such a way that tabs are displayed specially. I use this, stolen/adapted from here:

:set listchars=eol:\ ,tab:»-,trail:·,precedes:…,extends:…,nbsp:‗

Remember also that you can insert a literal tab into your file no matter what you have expandtab set to, by hitting CTRL-v first.

Those things are just about all I need to work with TSV files comfortably.

Dealing with long lines

Sometimes I'll run some data through a statistics program and it'll tell me "Invalid character at line 576, column 9438". How do you jump to that location in your data? Easily enough in normal mode:

576G9438|

G jumps you to a line, | jumps you to a column. A good mnemonic to remember these: "g = goto line" and "| looks like a column".

Suppose (like me) you turn scrollbars off in your GVim window, or you work from a terminal. How do you scroll horizontally? With zL and zH. But those are hard to type and harder to remember, so I map them to CTRL+Shift+Right and Left.

nnoremap <C-S-Right> zL
nnoremap <C-S-Left> zH

If you have a lot of long lines, it behooves you to set up a mapping to turn soft line-wrapping on and off quickly:

nnoremap <Leader>w :setlocal nowrap!<CR>

Now you can jump back and forth between wrapped and unwrapped views of your data via \w.

Diffs on long lines

When using vimdiff, ]c and [c will jump you to lines that contain differences. But if your lines are thousands of characters long, it doesn't always help to know that two lines are different: you want to know where they differ. ]c puts you at the first column in the line, which isn't helpful.

This function will jump you to the column where the difference between two lines starts:

function! IsDiff(col)
    let hlID = diff_hlID(".", a:col)
    return hlID == 24
endfunction

function! FindDiffOnLine()
    let c = 1
    while c < col("$")
        if IsDiff(c)
            call cursor(".", c)
            return
        endif
        let c += 1
    endwhile
endfunction

This seems fragile, so if you know a better way please leave a comment and let me know.

Diff a buffer against itself

Sometimes I have files with lots of lines of data, and some lines might be duplicate. It's really easy to get rid of the duplicate lines:

:sort u

But what if I want to see what lines were just removed? I don't know of a built-in way to do it, so I use this simple function:

function! MarkDuplicateLines()
    let x = {}
    let count_dupes = 0
    for lnum in range(1, line('$'))
        let line = getline(lnum)
        if has_key(x, line)
            exe lnum . 'norm I *****'
            let count_dupes += 1
        else
            let x[line] = 1
        endif
    endfor
    echomsg count_dupes . " dupe(s) found"
endfunction

That'll put a bunch of asterisks at the beginning of every line that's a duplicate of a previous line.

Fix broken punctuation

Ever had to work with textual data that someone else sent you in MS Word or Excel? I have. It hurts. If you copy/paste from an MS document into a plaintext file, you'll probably end up with a bunch of funky unreadable or undisplayable characters.

This function fixes the most common Word-spawned "smart" punctuation characters that you're likely to run across:

function! FixInvisiblePunctuation()
    silent! %s/\%u2018/'/g
    silent! %s/\%u2019/'/g
    silent! %s/\%u2026/.../g
    silent! %s/\%uf0e0/->/g
    silent! %s/\%u0092/'/g
    silent! %s/\%u2013/--/g
    silent! %s/\%u2014/--/g
    silent! %s/\%u201C/"/g
    silent! %s/\%u201D/"/g
    silent! %s/\%u0052\%u20ac\%u2122/'/g
    silent! %s/\%ua0/ /g
    retab
endfunction

This function was built up during years of choking on special characters that crept up in my data, so I'm sure I'll be adding more to it in the future.

\%u#### here is a regex escape to represent a character as a hexidecimal Unicode codepoint.

Many of these characters show up as blank (whitespace) if you lack the font to display them. If you ever run across a character you can't see and you want to inspect it, put the cursor on it and do

:ascii

That's a bad name for the command, since it works on Unicode characters too. That'll give you the numeric code you can use in a regex to replace them all with something sane.

Dell: the aftermath

Thu, 08 Jul 2010 11:11:28 -0700

In a previous post I outlined the ways in which Dell's customer service sucks. I finally got my computer yesterday, a Studio XPS 9000. Here are my first impressions.

The bad

This computer weighs so much I almost hurt my back lifting it. I thought computers were supposed to be getting smaller and lighter?
The HD indicator light is tiny and on the top of the case. I can't see it with my computer under my desk.

The optical drive is behind one of those stupid plastic flap door things. So there isn't even an indicator light for the DVD drive. I'm seriously considering taking a screwdriver the case to fix this.
It didn't come with a Windows install disk or a driver disk. It only has a recovery partition on the HD.

I found an order form which I think will get me my disks. In the mail. Seriously, Dell? Seriously? Why not come to my house and kick me in the balls while you're at it?

The recovery partition doesn't help you worth a crap if you want to do things like repartition your drive to put Linux on it. Windows7's sucky partition shrinking app wouldn't shrink it lower than 500GB.
The Dell recovery program is called DataSafe or something, and when you use it, it tries to upsell you like crazy to get a "pro" version that has a bunch of useless backup features. Uggggghhhhh.
The side of the case is white. In 10 years it's either going to be yellow with age, or scuffed up beyond hope. Kind of ugly, but I don't care much. The front of the case looks OK though. Black with red highlights. About as good as I could expect.
It came pre-installed with some crappy "Dell Dock" knockoff of Apple's Dock. Worthless and instantly uninstalled.

This thing caused the desktop icons to be hidden by default. Who would possibly want to do this in Windows? I can image everyone and their grandmother being awfully confused by the missing icons.

When quitting this dock, it said "Undo is not possible". I love a program that has no going back once you quit.
I wanted to find drivers for the wireless card that came with the Dell. So I went to Dell's support site and typed in the tag number on my computer. It gave me a link to drivers for the wrong card. I had to google all over the place to find the right ones. Way to go.

Dell's website is a labyrinth full of outdated information and dead pages in general.
The instructions I got with the computer reference Vista. I don't have Vista.
There's a "Windows inside" logo on the case. It will be removed shortly. They leave an awful lot of glue behind.

The good?

The i7 is about as fast as I had hoped. It only took a half-dozen cores and 12 GB of RAM to let me watch full-screen flash videos on Youtube. I feel so modern.
The inside of the case is OK. There are a lot of hard drive bays and lots of extra screws. It should be easy to expand if I need to.
It came with bloatware and crapware, but actually far less than I was dreading. And most of it was trying to sell you Dell crap.

In the olden days you'd get a hundred links to AOL and other 3rd-party crap. I saw a link to Skype and the obligatory nag to buy an anti-virus subscription (fat chance), but not much else.
Dell delivered the computer 2 days past the original estimated delivery date. So in spite of all the bullcrap and phone-jockeying I had to go through for billing, I can't complain about how fast it got here. Two days late isn't bad.

I've heard rumors that these computers are built in Malaysia, and mine was definitely shipped from the US (per the Purolator tracking site). So I'm surprised they can get these things delivered as fast as they can given that it was shipped halfway around the world first, and had to go through Customs at least once coming from the US to Canada.

Purolator was the only shipping option for Canada. I would've preferred to rush it. But maybe that's not possible given that it's coming from the US. Oh well.
It runs pretty quiet, given how huge the fans are. We'll see how hot it gets once I start putting some load on it.
It came with a DVI to VGA adapater and a DVI to HDMI adapter. I thought that was a nice touch, though it could be that they come standard with any nVidia card nowadays.
Works OK with Linux. It took 20 minutes to set up. (Not counting wiping the Windows partition and re-installing on a smaller partition from my own copy, minus the crapware. That took over an hour.) Sound, video and wireless work out of the box in Linux. All 12 GB of RAM are usable, given a 64 bit OS. (I discovered this the fun way, by unthinkingly trying a 32 bit OS first.)
It didn't burst into flames (yet).
It has a peanut tray on the top. Or MP3-player tray, I guess. But I really want to put peanuts in there.

Brian, you're stupid!

So why did I get a Dell? Because I had good experience with them in the past, at home and at work. Given, that experience was 5 years in the past, and a lot can change. And I'm new to Canada, and relatively unaware of what options exist here.

The other (main) reason was that they were far, far cheaper than going through newegg.ca to get the same hardware. But I guess you get what you pay for. Caveat emptor.

I wouldn't recommend Dell to anyone else, given how chaotic the whole buying process was. Too much uncertainty, too much room for mistakes.

Dave asked in my previous post why I didn't just a computer myself, like I had in the past. I said I didn't have time, but what I meant wasn't build time, which should be an hour or two max. I meant research time. Trying to match up compatible hardware, trying to find the best prices on all the components, checking for Linux compatibility, this takes forever and a half. I don't have hours / days to dork around with this any more.

On the other hand I can just google "xps 9000 linux" and see instantly what problems people had. I can be semi-confident that the hardware would all be compatible. And that did work out OK.

And the last reason I got Dell is that unfortunately I need Windows for work and gaming. Blarg. Paying the Windows tax to Dell is bad enough, let alone buying one off the shelf for $6,000 or however much they cost nowadays.

Sorry for the downtime

Thu, 08 Jul 2010 08:29:37 -0700

This isn't the kind of email I like to see when I wake up:

Our backend monitoring system has detected an error on the host where your Linode resides which could lead to a failure condition. In order to protect your Linode, we have scheduled an emergency migration to a different host which will commence shortly. Please note that there is currently no issue with your Linode - this is a proactive measure we are taking to avoid an issue in the future.

We apologize for any inconvenience this may cause. Should you have any questions or concerns, please do not hesitate to reply to this ticket.

The server rebooted and my sites didn't come back up, so there was some downtime. Sorry about that.

In which border-radius is abused

Mon, 05 Jul 2010 15:54:24 -0700

I threw together a new blog layout today. I scaled back the level of cows a bit (just a bit, don't worry!) Criticism / feedback welcome. (IE-related feedback should be dropped off in the circular file by my desk.)

In what is surely a prelude to the future of the internets, I'm abusing border-radius pretty heavily in this layout. border-radius is likely to become the new marquee HTML tag or text-shadow CSS attribute: Maybe an OK idea at first, but then everyone uses it so much it makes your eyes bleed.

So I figured I'd best get my border-radiusing in early while it's still cool. IE8 users, you still get pointy corners. Sucks to be you.

Also, if you have any ideas for features I should implement for cow-blog, please let me know. I've been crawling the internet looking at blogs for ideas of things to implement and features to steal, but I'm running out of ideas. It does everything I want now, but I'm not a reader.

Dell sucks

Fri, 02 Jul 2010 10:57:41 -0700

Why did I order a computer from Dell? I guess I had a good opinion from 6 years ago when I last bought something from them.

Let's count the ways in which their customer service has failed me. (And my computer isn't even here yet.)

As documented, their website couldn't process my credit card without a phone call.
After a week of my computer being "in production", I started getting more phone calls from an unidentified phone number that Google told me was Dell. Fearing another billing problem, I called back. And I was told "Thanks for calling, but our order tracking system is down. And we're all going home. Call back tomorrow morning.".

If only Dell had some means to acquire reliable computer systems on which to build their order tracking database.
I called the next day and was told my order was fine. I was also told (per script, I'm certain) that I could check my order status on Dell's website. Which of course I knew. I know it costs the company money every time someone calls, and they try to strongly discourage calls for that reason, but their script made it sound like I was an imbecile.

I found it quite condescending. I dislike these canned scripts pander to the lowest common denominator of customer. They should be happy to take my call. I just spend upwards of a thousand dollars on their crap.
Turns out the phone calls I was getting were from someone trying to give me "free internet from Shaw or Telus for 3 months", and I was eligible because I bought a Dell computer. So I was being telemarketed before my computer even got here.

I said I already had internet service, and they said "Oh, too bad, it's for new customers only." I do not appreciate this.
I got an email saying my order shipped. Joy! 20 minutes later I got an email saying my order was delayed, and if it didn't ship in 5 days I should call. What?

It really did ship though, I have a tracking number. Why the contradictory emails?

All of my phone dealings with Dell were via some offshored far-eastern country, judging by the accents of the phone reps. I have nothing against this in principle; I'm not a xenophobe. But the phone connection is always so static-filled and laggy that it really puts a damper on communication.

My computer isn't here yet, and I just hope to God it works and doesn't break in a month. I kind of wish this article had come out a week earlier.

That'll teach me for trying to save time, I guess. Next time I'll build my own system from scratch. Dell goes onto my List of Companies Not to Buy From in the Future (LCNBFF), along with Westinghouse and oh so many others.

I am an edge case

Wed, 30 Jun 2010 10:03:20 -0700

I am an alien. An American who emigrated to Canada. This has resulted in a lot of fun and a bit of pain as I've managed to break the systems of many of the businesses I deal with.

As a programmer I can appreciate the importance (and sometimes difficulty) of handling edge cases. It's been an interesting experience living as an edge case myself.

H&R Block(heads)

Taxes are confusing enough when you don't have a wife from another country. The friendly folk at H&R Block had no idea how to handle my situation. Their computers demanded a Social Security Number for my wife, which she doesn't have, because she's Canadian. So they left it blank. (Leaving it blank was the consensus opinion of everyone at H&R Block, including the managers.)

To even be able to leave it blank, they had to print the forms, because the computer refused to submit my taxes electronically with a blank SSN. This should've been a red flag to me, looking back.

The correct thing to do was for her to get an ITIN from the US, which from what I know is a like an SSN that doesn't get you SS benefits or allow you to work in the US. It's just for tracking. But they didn't tell me that.

Moral of this story: That's the last time I'll ever use H&R Block. If I want to do my taxes wrong, I can do that myself for free.

IRS

The IRS refused to believe that my wife existed without a number to assign her, so they rejected my tax return and threatened to charge me tons of penalties. So I drove down to the friendly neighborhood IRS office. Surely they'd know how to fix this, right?

Wrong. The fellow at the IRS was familiar with filing taxes for Mexicans living in the US, but not for a "non-resident alien Canadian spouse" like mine.

The ITIN docs say that I need to submit a notarized copy of my wife's passport, to get her an ITIN. But notarized by whom? By a notary in Canada or the US? The IRS agent spent at least an hour reading his enormous IRS manual, looking up treaties and international law, trying to figure this out. Eventually he found a footnote scribbled into the margin of his book that said a Canadian notary was OK. So that's what I gave him.

He filled out the new tax forms himself, stapled on a photocopy of the page from his own IRS manual saying the Canada-notarized passport was OK, stamped everything all official-like, mailed it away himself, and... a month later it was rejected again. I needed to get a US notary to do it, or my wife had to drive to the freaking capital of her province to get it super-notarized or something. Nine months later and 3 more trips to the IRS office, I finally got it worked out.

As a US citizen, living in Canada, working for a US company, being paid by a Canadian payroll company, I live in constant fear of doing my taxes next spring.

Moral of this story: Even the IRS doesn't know their own tax code. Thanks again, US Government.

No credit

I have good credit in the US. I was able to get a car loan without a co-signer 6 years ago. I have credit cards.

In Canada, I don't exist. My credit score here is zero, as you might expect. I tried to buy a car, and they just flat-out wouldn't let me without a co-signer for a loan. I guess I can't blame them. (Except that I have money, they have something I want to buy, and we both ended up losing out.)

I went to the bank to open a checking (er, "chequing") account, and all hell broke loose. They wanted my business, but how could they justify giving an account to someone with "no credit"? Eventually the bank manager managed to run a US credit check on me (at least she said she did) and they let me open an account.

But I can't get a credit card here. Not even a $500-limit high-interest card like they give to kids in college. Not even after I had the bank write a letter of recommendation vouching for me. I have to wait a year and keep getting paychecks before I show up in the system. ("Paycheques"?) I'm still looking for other options in the meantime.

Moral of this story: Well, no moral. Sucks to be me, I guess.

Shopping

I tried to buy a computer online recently (from Dell, which I'm starting to regret). Not having a Canadian credit card, I used my US card. This is OK, my card works up here, with a small foreign transaction fee of %1-3.

But the website wouldn't take a US address as my billing address. I had to give a Canadian address. "Province" was a drop-down, mandatory field.

No matter, I'll just go to my bank's website and change my billing address to one in Canada. I checked with my bank before I moved, and they said it was no problem to keep the account even if I moved to another country. "We have lots of international customers!", said the teller.

Well, my bank's website won't let me specify a non-US billing address. "State" is a mandatory drop-down. Which is awesome. I emailed the bank and asked them to change my address for me, and they did. Now when I go to the "change my address" form on their website, half the fields are filled in and half are just blank. So if I ever use the form again, something will probably break.

Once I got on the phone with Dell, they were more than happy to take my US credit card as payment. Their online form just couldn't handle it.

Moral of this story: Text fields, not drop-downs.

An so on

When I fly across the border, I have to fill out a customs declaration form. There's a field asking "country of residence". Well, technically I am a resident of two countries. So I pick "US" when I fly down to the States, "Canada" when I fly back.

I tried to get a Costco membership in Canada, and they wanted a driver license. My license is from Oregon. Hilarity ensued, and they decided I didn't need one after all. Any time I need to give a driver license for any purpose, I end up breaking something.

Note that in every situation I've described, what I was trying to do was valid, and after some hassle, everything usually worked out OK. Computers just got in the way and slowed the process way down. And I'm not that much of an edge case. 250,000 people immigrate to Canada every year.

I suppose it might be better to optimize for the common cases, force people to pick their province from a drop-down. And then deal with the edge cases like mine manually later. Every text field is another opportunity for users to type in gibberish and chaos. But I wonder if the programmers actually thought about it this much, or if they were just being lazy.

I'm not really complaining. I don't expect the world to change to accommodate me. It's been more funny than annoying. But I do find it interesting to see the flaws in computer systems exposed. I get a certain sick satisfaction out of seeing people write "invalid" values into fields that I know are going to break someone's database down the line.

Goodbye Tokyo Cabinet, hello PostgreSQL?

Tue, 29 Jun 2010 11:32:58 -0700

The first version of this blog used MySQL; then I switched to Tokyo Cabinet. But now I've switched back to PostgreSQL. Here's why.

Why did I switch to TC to begin with?

There weren't any good ORM-type libraries for Clojure at the time (over a year ago). So there was a bit of an impedance mismatch trying to query and work with my data. In the DB I have separate tables for posts, comments, tags, categories. But 90% of the time I want to fetch a post and end up in Clojure with all related tags, comments etc. (A JOIN won't work here without a lot of work to un-mangle the results; you usually need multiple queries.)

With TC I could store anything in the DB, so I just dumped a post into the DB as serialized hash-maps with all the comments, tags, and categories as sub-keys. So querying was easy. (Or was it? More below.)
MySQL was really slow. This is largely because my queries were terrible, as I tried to solve the problem from #1 via brute force. TC on the other hand is fast.
I tried to solve #2 by using a Clojure ref as a cache. But tying the STM to a database's transaction system is as far as I know difficult or impossible right now (per many threads on the mailing list). I had a lot of potential race conditions, which (as far as I know) never bit me, but probably would've eventually. I had to deal with keeping the cache up-to-date as comments were posted and posts were added and deleted and renamed. Remember:

"There are only two hard problems in Computer Science: cache invalidation and naming things." --Phil Karlton

So why did I stop using TC?

I have no idea how to use a key/value store database properly. TC will take anything you dump into it, which is both a strength and a weakness.

There's a lot of crap you have to do by hand that a proper database does for you. Consider checking for null values, for example; I ended up with a lot of nils in my data because my validations weren't 100% foolproof, or because I imported data via code that didn't run the validations and I never noticed.. Or enforcing uniqueness of values; I had tag objects in the database with the same key but different values (due to capitalization differences), which screwed up a lot of stuff.

On the other hand, there's a lot of information about how to use RDBMS properly, and I have a lot of experience with it already. Constraints are easy to set up. Columns have types, which is nice. (Strange that I gravitate toward statically-type databases while I gravitate toward dynamically-typed programming languages.)
I have to compile and install Tokyo Cabinet by hand on my Linux distro. It's probably not worth distro maintainers to maintain a package that so few people use. MySQL and PostgreSQL have lots of people working on keeping them running OK on most Linux distros.
Some kinds of queries were still awkward in TC. "Give me post X" was great: I'd also get all the tags, categories etc. for free. But then how do you query to get all tags across all posts? Or all comments? Fetch all the posts and iterate over them, collecting their tags, then uniquify the resulting list? Not so pretty, and not so fast. So I was back to caching again, which still gave me nightmares about race conditions and dirty data.

So now why am I using an RDBMS again?

An RDBMS is exactly what I really need, if I could just query the thing concisely and get it to run fast. Thankfully there are some ORM-like libraries for Clojure in the works nowadays, already usable for a hobby project like this blog. There are clj-record, Carte, ClojureQL, and my own Oyako, and possibly others in the works.
For my tiny blog's database, Oyako gives me slightly slower performance than TC, but along the same order of magnitude, which is good enough.
Via Oyako I can (fairly concisely) fetch posts and get the associated tags, comments etc. But I can also easily fetch all tags, or all comments, since they're in their own tables. The "relational" part of RDBMS does come in handy sometimes.

Summary version

I switched to TC to begin with because I was using SQL wrong, and it was too slow and clumsy. Once I figured out how to use SQL correctly, it was a no-brainer to go back.

Introducing Gaka

Mon, 28 Jun 2010 17:59:26 -0700

The CSS for my blog is now being generated via gaka, a CSS-generating library I wrote this afternoon. It's extremely simple, but it got the job done for me. I turned around 600 lines of CSS into around 250 lines of Clojure without much effort. It looks like this:

user> (require '(gaka [core :as gaka]))
nil
user> (def rules [:div#foo
                  :margin "0px"
                  [:span.bar
                   :color "black"
                   :font-weight "bold"
                   [:a:hover
                    :text-decoration "none"]]])
#'user/rules
user> (println (gaka/css rules))
div#foo {
  margin: 0px;}

  div#foo span.bar {
    color: black;
    font-weight: bold;}

    div#foo span.bar a:hover {
      text-decoration: none;}

Gaka is partly inspired by Sass, which I found very pleasant to work with recently. And it's partly inspired by Hiccup, which is a delicious way to generate HTML in Clojure.

There's more info and more examples on github.

New blog engine up and running

Wed, 23 Jun 2010 15:48:24 -0700

Well, my new blog is up and running. Sorry for the temporary lack of cows in my layout. I'm dogfood-testing the blog engine in a fairly vanilla state until I work out some of the bugs. This layout is based upon barecity, a minimalist Wordpress theme that I adapted easily enough to my blog.

As a bonus, I applied a dirty hack to my RSS feed that I think should help avoid screwing up people's RSS readers with duplicate entries.

I'll write again soon with some info about the blog engine and some things I learned writing it.

(As mentioned previously, here's the code.)

Breaking links is easy to do

Tue, 22 Jun 2010 23:23:27 -0700

I apologize in advance to everyone who subscribes to my blog's RSS feed, but this week your RSS reader is probably going to suddenly find 25 "new" posts from me.

My blog currently uses /blog/title as the URL scheme, with similar URLs for categories and tags etc. Soon, I'm probably going to change it to /blog/123/title, as part of the impending release of version 0.2 of my blog engine. (The code-in-progress is in a branch on github, for the daring and foolish among you.)

This way, I can change the title of a post without breaking everything. I have heretofore lacked this ability. It's easy to code, you just tell Compojure to ignore everything after the number in a route. Something like this:

(defroutes foo
  (GET ["/blog/:id:etc" :id #"\d+" :etc #"(/[^/]*)?"] [id]
    (pages/post-page id)))

It's only a few lines of code to change, but the ramifications are widespread. It'll instantly break every link to my blog, for example. At least it's pretty easy to set up a bunch of redirects in Compojure to avoid that. I think this'll work:

(require (blog [db :as db]
               [link :as link])
         (oyako [core :as oyako])
         (ring.util [response :as response]))

(defn redirect-post [name]
  (when-let [post (oyako/fetch-one db/posts :url name)]
    (response/redirect (link/url post))))

(defroutes redirect-routes
  (GET ["/blog/:name" :name #"[^/]+$"] [name]
    (redirect-post name)))

(Oyako here is the experimental ORM-like library I'm using to interface with PostgreSQL nowadays, having ditched Tokyo Cabinet.)

Changing my URL scheme is also going to mess up RSS though, because I (foolishly) used post URLs as the GUIDs in my RSS feed up to this point. This problem I don't know how to avoid. I might reduce the number of posts included in my feed temporarily, to limit the damage.

Vim regex - remove kind-of-matching lines

Sun, 20 Jun 2010 22:49:19 -0700

I have a file where every line starts with a number (followed by whitespace and a bunch of other stuff). Every number appears on either one or two lines, and if two, the second line always has a b after the number.

I need to delete every line for which there's a corresponding b line. But if there's no corresponding b line I want to leave the original line there.

Before:

123 foo bar
456 blarg
789 quux
123b foo baz
789b quux blurble

After:

123b foo baz
456 blarg
789b quux blurble

Except in my real file, I have a thousand lines and it'd take a year to do by hand. Vim to the rescue:

:sort
:%s/^\v((\d+).*\n)(\2b.*)/\3/

And that is why Vim is awesome. Can you think of a shorter way to do this, in Vim or Emacs?

Ads on license plates?

Sun, 20 Jun 2010 22:09:08 -0700

What if when your car stops at a red light, your license plate displays ad banners? What could possibly go wrong?

Quoth the person(?) who wrote this bill:

"We're just trying to find creative ways of generating additional revenues," he said. "It's an exciting marriage of technology with need, and an opportunity to keep California in the forefront."

The forefront of annoying the hell out of people. Certainly what I need is more distractions on the road. I mean, what if there's a new brand of toothpaste and I didn't find out yet? Someone somewhere needs to earn a dime for telling me about it by any means necessary.

I'm just waiting for the first company to propose paying new parents a few hundred dollars to tattoo ads on their babies.

Printer spam: what could possibly go wrong?

Thu, 17 Jun 2010 10:59:57 -0700

As further evidence that there are no depths to which companies won't stoop when it comes to advertising, HP has come up with a great idea: Get people to hook their printers up to the internet and then spew advertisements out of their printers.

Well, it's a win-win situation for the companies doing the advertising: Not only will people see your ads, they'll pay for the ink and paper to print them. Maybe not such a great situation for the end-user though.

And then there are the privacy implications of targeting ads based on geolocating the IP address of the printer. Which I find a bit disturbing, but I guess advertisers already do that with online ads. But wait, there's more:

Ads can also be targeted based on a user's behavior as well as the content, said Vyomesh Joshi, head of the HP's Imaging and Printing Group.

Looking at what I'm printing so you can try to sell me things? Just a bit creepy.

Most troubling to me is the intrusiveness of the whole thing. They're taking control of a physical object in my house and using it against me. May as well kidnap my cat and train him to spell out "BUY PEPSI" in his cat litter.

Quote some slimeball at HP:

"What we discovered is that people were not bothered by it [an advertisement]," Nigro said. "Part of it I think our belief is you're used to it. You're used to seeing things with ads."

Translation: "We know this is a really horrible idea, but if people are complacent enough to sit there and take it without complaint, what's stopping us?"

He's right though, people are used to it.

I guess TV, radio, internet, phones, product placement in movies and games, print media, billboards and the postal service just aren't enough. Clearly what the world really needs is another ad-delivery mechanism.

Emacs creating zombie buffers

Mon, 14 Jun 2010 15:44:18 -0700

Note to self:

Using bleeding-edge swank-clojure and Slime, Emacs was creating buffers that didn't point to any file, every time C-c C-k wanted to display warnings. This was really quite annoying as it resulted in a mess of empty buffers being created. This fixed it:

(setq slime-highlight-compiler-notes nil)

Clojure, from a Ruby perspective

Wed, 09 Jun 2010 13:22:58 -0700

Fogus' recent article "clojure.rb" speculates about why there seem to be so many Ruby users adopting Clojure. As a Ruby user who adopted Clojure, I figured I'd write about my experiences.

What do Ruby and Clojure have in common, that would attract a Rubyist to Clojure? A lot. Obviously, this is somewhat subjective and I don't expect anyone else to agree, but this is what did it for me.

Semantic consistency

In Ruby, everything is an object. It makes it simple to write code without worrying much about what kind of thing you have. foo.some_method(1,2,3) will generally work for any foo.

In Clojure, everything is not an object, specifically because it inherits primitives from Java land (though this doesn't hurt much in the kind of everyday use I put Clojure to). But also because Clojure by design doesn't even attempt to be object-oriented.

Clojure does have abstractions though. For example there's an abstraction that says "this thing can be called like a function". And then you can treat any callable-thing as a function without worrying about what it is.

In Ruby, a lot of things are Enumerable, which means you can do foo.each{} and other similar things for a lot of different types of foo. Clojure has something similar with its seq abstraction. Similarly, most Clojure data types and many Java ones are seq-able, and most built-in core functions can iterate over the guts of various things using seqs. This includes regex matches, strings, directories of files, and so on.

Ruby-style OOP brings a lot of complexity and baggage which Clojure avoids by not being OOP. For example (ignoring Java for the moment), in Clojure land you don't have to worry about a member being public/private/protected, and there are few times when you have to worry about inheritance and class hierarchies. (And there's the whole thread-safety thing.) In Clojure data is stupid and immutable, and functions are just things that take input and give output, usually side-effect free. The separation is clean and this results in programs that are very easy to reason about.

Another example of consistency: Expressions. In both Ruby and Clojure, everything has a value. Things that are "statements" in other languages are instead expressions that return something. This alone makes a lot of programs just a little bit better/easier to write.

Aesthetics

Like Ruby, Clojure code tends to be terse and expressive.

Ruby reads like poetry, because it's mostly words and not so much punctuation. In Ruby, you have none of Perl's sigils, very little of C's semi-colon line-endings and curly-delimited blocks. Especially when you start omitting optional parentheses, it has a very minimalistic vibe which is appealing to many.

    def foo(bar)
      bar.each do |x|
        puts x
      end
    end

Clojure's s-expressions are another story, of course. Some love them, some hate them. Personally, I love them. The tired old trope about Lispers not paying attention to parentheses is true; after a while they blend into the background.

    (defn foo [bar]
      (doseq [x bar]
        (prn x)))

What I see:

    defn foo [bar]
      doseq [x bar]
        prn x

Clojure has the same minimalistic feel to it, in my eyes. It also helps that so many Clojure function names are short and concise. And being able to use punctuation like ? and ! in variable/function names (true? and false? are function names), hyphens instead of underscores... these things help Clojure read smoothly.

But along with aesthetics, in Clojure you get the benefits of s-exps: no order of operations to deal with, and absolute consistency. Everything is (function param1 param2 param3). Look at how many syntax rules you have to memorize for the Ruby code above to make sense. Dot means method call, blocks are do/end delimited and have those weird pipes in there, etc.

Literals and syntax sugar

Ruby has literal syntax for many types. This includes:

:symbols
[arrays]
{:hash => :maps}
/regex/
do block end and {block}

This really is a big deal. I'm spoiled and I can't use a language without these things nowadays. It saves a ton of typing and it makes those things stand out in the code, making it both easier to write and to read. I use those structures all the time in every program I write, so they should have a terse representation.

Clojure has literal support for the same types as Ruby, and they even look mostly the same (with #"regex" being a change I can live with). And then it also has (among others):

#{sets}
#(function-literals %)
'(quoted forms)
`(quasi-quoted ~forms)

Is this a contradiction of my last point? What happened to s-expressions and consistency? Well, in Clojure, the reader shortcuts are just sugar that reduce to s-exps. You can avoid all use of that sugar if you hate it. (hash-map :key "val"), (vector 1 2 3), (quote foo) etc.

But more importantly, let's draw a(n arbitrary) distinction between "good" syntax and "bad" syntax.

Clojure's reader-macro sugar makes your code shorter, but doesn't change the structure of your code. Take a function call (f x y z), and you can always substitute a vector or regex literal or quoted form into it. (f [vector] #{set} #"regex"). The syntax sugar is very local, very self-contained. It doesn't leak into the surrounding code. And of course you can combine them in nearly arbitrary ways: '[quoted vector], {:hash-map-containing-a #{:set 'of #(functions)}}. This is "good" syntax.

Compare this to things like the x ? y : z construct, or heredocs. These things are not as orthogonal. They not only mean something on the "inside", they also influence and interact with the code before and after them, thanks to precedence rules and special parsing rules. Can you stick a heredoc in the middle of a function call? Maybe (I don't even know), but have fun with the indentation and line-breaks if so. When should you use do/end and when should you use {} for blocks? When do you need parens around your ternary if-then-else construct and when don't you? When do you need to use and and when &&? That's the "bad" kind of syntax. Sometimes, maybe even most of the time, it makes your code shorter, but there are a lot of rules to memorize and you never know when you'll be bitten.

Clojure largely avoids the "bad" syntax while taking advantage of the "good". Reader macros make your code shorter and visually easier to scan, but they rarely require you to do backflips to get your code to compile or run properly.

First-class functions

Ruby's blocks and yield and friends let you deal with first-class functions. This is a huge step in the Lisp direction already, and it's one of the things that makes Ruby great. But there are limits. Blocks use funky, special syntax, and in idiomatic Ruby, you will pass around only one block per method. There's the whole lambda and proc mess, and then there are methods-as-objects which are different still. And a lot of Ruby just calls .send on an object and passes in a method name as a symbol.

Being a Lisp, Clojure takes this a bit further. First-class functions are ingrained in nearly everything you do in Clojure. And they are easy to define and easy to call. Define f via defn (to make it top-level), fn (for a local function), or #() (sugar for fn), and then call it like (f).

Clojure also takes advantage of some functional-programming mainstays like partial and complement and comp(osition). We're not in full-blown Haskell territory, but it's a lot more FP than idiomatic Ruby.

And hash-maps, vectors, sets, keywords, and symbols are also callable as functions in Clojure. ({:foo 1} :foo) => 1. Many things can be treated as functions.

Metaprogramming

In Ruby you can mess with the innards of any class you want. There are facilities for defining methods dynamically, opening and inspecting classes at runtime, catch-all handlers for undefined methods, and all kinds of other dark magic. But again there are limits... Ruby needs to make use of eval to get certain things done. And monkey-patching is a shotgun aimed at your foot.

Well, if you like metaprogramming, Lisp macros are top of the line. You can abstract away boilerplate with a vengeance. Macros are the ultimate application of DRY.

Clojure doesn't deal much with classes, so there isn't much of that kind of introspection, but the Lisp principle of code-as-data enables a kind of introspection that you won't find in Ruby. The line between compile-time and run-time is very blurry, which enables all kinds of magic.

Java itself does offer Ruby-style reflection and such, if you need it, but you won't often, while in Clojure land.

Multimethods (and soon, protocols and defrecord) let you avoid monkey-patching and get some of the same kinds of "extend a class" things done in a saner and safer way.

So?

Fogus suspects:

Ruby programmers being the adventurous lot to begin with, are not satisfied with “halfway to Lisp”. Instead, they want it all.

This is true in my case. I like Ruby largely insofar as it borrowed and adapted many great features of Lisp. It only makes sense that I would like Clojure, which takes most of those things one step further. Clojure in particular, as a "modern" Lisp with vaguely Ruby-like syntax in certain places, is an obvious choice.

On top of that, Clojure is fast, thanks to the JVM. Ruby has JRuby too, but vanilla Ruby is not known for its speed. Clojure integrates with a REPL in a way that Ruby really doesn't, making interactive development enjoyable. Clojure is a compiled language, which has benefits for deployment. And again, there's the whole thread-safety thing. Clojure is awesome for writing sane, safe multi-threaded programs. These things are rather appealing.

I do still use Ruby though. Ruby is great for scripting, Clojure not so much, thanks to the JVM startup time, among other things. Ruby can be banged out quickly in any editor, but Clojure isn't much fun to edit in any editor that lacks good paren-matching support and REPL integration.

Rubygems offers dead-simple install of a ton of libraries, whereas Clojure is still working out the details of a standard build tool and install tool. Ruby has a library for anything, and while Clojure can use Java libraries, Java libraries tend to be huge and feature-rich, sometimes too huge for one-off tasks where a small Ruby library is a perfect fit.

Screenshot June 2010

Tue, 08 Jun 2010 12:27:44 -0700

I haven't posted one of these in a while. I've been in an 8-bit kind of mindset for a while:

What I actually stare at for 8 hours every day:

KDE4, Buuf icons, QtCurve, wallpaper is from somewhere on the internets.

Emacs isn't for everyone

Tue, 08 Jun 2010 11:26:15 -0700

Chas Emerick recently posted the results of his State of Clojure survey. It turns out that the (self-selected) group of Clojure-using respondents happen to prefer Emacs as their IDE of choice, eclipsing all other editors by a large margin.

Chas then has this to say:

I continue to maintain that broad acceptance and usage of Clojure will require that there be top-notch development environments for it that mere mortals can use and not be intimidated by...and IMO, while emacs is hugely capable, I think it falls down badly on a number of counts related to usability, community/ecosystem, and interoperability.

As an avid, die-hard Vim and Emacs user for life, I'm going to agree.

Mere mortals?

Emacs isn't difficult to learn. Not in the sense of requiring skill or cleverness. It is however extremely painful to learn. I think there's a difference.

The key word is tedium. Learning Emacs is a long process of rote memorization and repetition of commands until they become muscle memory. If you're smart enough to write programs, you can learn Emacs. You just have to keep dumping time into the task until you become comfortable.

Until you're comfortable, you face the unpleasant task of un-learning all of your habits and forming new ones. And you're trying to do this at the same time you're undertaking another, even harder task: writing programs. And if you're a new Clojurist, and you're learning Emacs and Clojure from scratch at the same time, well, get the headache medication ready.

As a programmer and someone who sits in front of a computer 12+ hours a day, I consider myself pretty flexible and capable of picking up a new user interface. As someone who had been using Vim for years prior to trying Emacs, I considered myself more than capable of learning even a strange and foreign interface. I'd done it once before.

But learning Emacs still hurt. Oh how it hurt. I blogged while I was learning it, and you can see my pain firsthand. I sometimes hear people say "I tried Emacs for a whole month and I still couldn't get it". Well, it took me over a year to be able to sit down at Emacs and use it fluidly for long periods of time without tripping over the editor.

To be fair, I'm talking here about using Emacs as a programming environment. Using Emacs as a Notepad replacement could be learned in short order. C-x C-f, C-x C-s, or use the menus, there you go. Using it comfortably as a full-fledged IDE is significantly harder and requires you to touch (and master) many more features. Syntax highlighting, tab-completion, directory traversal and cwd issues, enabling line numbers, version-control integration, build tool integration, Emacs' funky regex syntax for search/replace, Emacs' bizarre kill rings and undo rings, the list goes on. These things are very flexible in Emacs, which is a great thing, but it's also an impediment to learning how to configure and use them. There's no getting around the time investment.

And it's not just a matter of learning some new keyboard shortcuts. There's a new vocabulary to learn. You don't open files, you visit them. What's a buffer? What's a window? (Not what you think it is.) What's a point? What's a mark? Kill? Yank? "Apropos"? Huh? C-c M-o means what exactly? My keyboard doesn't have a Meta key. Yeah, you can use CUA mode and get your modernized Copy/Cut/Paste shortcuts back, but that's the tip of the iceberg. It's hard even to know where to begin looking for help.

Yeah, Emacs came first, before our more common and more modern conventions were established, and that explains why it's so different. That doesn't change the fact that Emacs today is a strange beast.

Community and ecosystem

Personally I find the Emacs community to be a pretty nice bunch. In the highest tradition of hackerdom and open source software, Emacs users seem to be eager and willing to share their elisp snippets and bend over backwards to help other people learn the editor. I got lots of help when I was struggling and learning Emacs.

The Emacs wiki is an awesome resource. The official documentation is so complete (and so long) that it leaves me speechless sometimes. And there are a million 3rd-party scripts for it. Whatever you want Emacs to do is generally a short google away.

If there's anything wrong with the Emacs community, it'd be people who take Emacs evangelism overboard. The answer to "I don't want to have to use Emacs to use your language" can't be "Be quiet and learn more Emacs", or "If you're too dumb to learn Emacs, go away". In some communities there is certainly some of that. But thankfully I don't see it much in the Clojure community. Let's hope it stays that way.

Interoperability

Once someone spends the time to write a suitable amount of elisp, Emacs can interoperate with anything. I think so many people use SLIME for Clojure development precisely because it interoperates so darned well with Lisps. SLIME is amazing. You probably can't beat Paredit either, and Emacs' flexibility is precisely what makes things like Paredit possible.

The problem is the amount of time you have to spend to get that interoperability set up and to learn how to use it. After two years of using Emacs and Clojure together, every once in a while I still find myself bashing my face on my desk trying to get the latest SLIME or swank to work just right, or trying to get a broken key binding fixed, or tweaking some other aspect of Emacs that's driving me crazy. One day, curly braces stopped being recognized as matched pairs by Paredit. Why? No idea; I fixed it, but it was a half hour of wasted time.

Emacs is good at integrating with Git too. So good that there are four or five different Emacs-Git libraries, each with a different interface and feature set. I gave up eventually and went back to using the command line. (You can embed a shell / command line right in Emacs. There are three or four different libraries to do that too.)

The wealth of options of ways to do things in Emacs is simultaneously a good thing, overwhelming and confusing. If all you want is something that works and gets out of your way, too many options can be worse than one option, even if that one option isn't entirely ideal.

Emacs' Java interop, I know nothing about. Almost certainly, Emacs can come close to a modern Java IDE for fancy features like tab-completion and document lookups and project management. But how long is it going to take you to figure out that tab-completion is called hippie-expand in Emacs? That and a million other surprises await you.

What's my point?

There was a pithy quote floating around on Twitter a while back (I think quoting Rich Hickey):

One possible way to deal with being unfamiliar with something is to become familiar with it.

That's true, and you could say that of Emacs. I strongly believe that when it comes to computers, there's no such thing as "intuitive". There's stuff you've already spent a lot of time getting used to, and there's stuff you haven't.

But certain things require more of a time investment than others. Could I learn Clojure if all the keywords were in Russian or Chinese instead of my native English? Sure, but it'd take me a long time. I'd certainly have to have a good reason to attempt it.

I learned Emacs partly because it was hard. I saw it as a challenge. It was fun, yet painful, but more pain, more glory. Mastering it makes me feel like I've accomplished something. I'd encourage other people to learn Emacs and Vim too. I think the benefits of knowing them outweigh the cost and time investment of learning them.

But I didn't learn Emacs with the goal of being productive. I learned it for the same reason some people build cars in their garages, while most people just buy one and drive it to and from work every day. I learned Emacs because I love programming and I love playing with toys, and Vim or Emacs are as nice a toy as I could ask for. (I love programming enough to form strong opinions and write huge blog posts about text editors.) For me, productivity was a beneficial side-effect.

There are only so many hours in a day. There are a lot of other challenges to conquer, some of which offer more tangible benefits than Emacs mastery would get you. Mastering an arcane text editor isn't necessarily going to be on the top of the list of everyone's goals in life, especially when there are other editors that are easier to use and give you a significant subset of what Emacs would give you. We have to pick our battles.

So I understand when people say they don't want to learn Emacs. I think maybe so many Clojurists use Emacs right now because we're still in the early adopter stage. If you're using Clojure today, you're probably pretty enthusiastic about programming. You're likely invested enough to be willing to burn the required time to learn Emacs.

If Clojure becomes "big", there are going to be a lot of casual users. A casual user of Clojure isn't going to learn Emacs. They're going to silently move on to another language. And I really think that new blood is vital to the strength of a community and necessary for the continued healthy existence of a programming language.

So Clojure does need alternatives. I'll stick with Emacs myself, but there should be practical alternatives. I'd encourage the Clojure community to continue to support and enjoy Emacs, but don't push it too hard.

Emacs: Yank lines as lines

Fri, 21 May 2010 12:07:13 -0700

One thing nice about Vim is manipulating whole lines at a time. dd deletes a line (including trailing newline), regardless of where the cursor is on the line. Then, p puts that line (with its newline) as a new line after the current line, and P puts it above the current line, again regardless of where your cursor is at the moment. (It also jumps the cursor to the beginning of the text you just inserted, which is nice.)

Emacs has kill-whole-line (C-S-Backspace) which is like Vim's dd. But I didn't find an equivalent of p and P. So here's my version:

(defun yank-with-newline ()
  "Yank, appending a newline if the yanked text doesn't end with one."
  (yank)
  (when (not (string-match "\n$" (current-kill 0)))
    (newline-and-indent)))

(defun yank-as-line-above ()
  "Yank text as a new line above the current line.

Also moves point to the beginning of the text you just yanked."
  (interactive)
  (let ((lnum (line-number-at-pos (point))))
    (beginning-of-line)
    (yank-with-newline)
    (goto-line lnum)))

(defun yank-as-line-below ()
  "Yank text as a new line below the current line.

Also moves point to the beginning of the text you just yanked."
  (interactive)
  (let* ((lnum (line-number-at-pos (point)))
         (lnum (if (eobp) lnum (1+ lnum))))
    (if (and (eobp) (not (bolp)))
        (newline-and-indent)
      (forward-line 1))
    (yank-with-newline)
    (goto-line lnum)))

(global-set-key "\M-P" 'yank-as-line-above)
(global-set-key "\M-p" 'yank-as-line-below)

Just one more step along the path to Vimmify my Emacs setup. Emacs has some weird edge cases because you can move the cursor one "line" past the last real line in the file. But I think I worked out something comfortable for myself.

PS: I've written about this before, but if you use C-S-Backspace a lot in Emacs on Linux, I highly recommend putting this into your X11 config:

Option "DontZap" "True"

It's really easy to mix up C-S-Backspace and C-M-Backspace (the latter of which kills your X server). It's not fun to mix those up. Not fun at all.

PPS: This thread on Stack Overflow has some Emacs equivalents of Vim's o and O which are pretty nice too.