briancarper.net (λ) (Tag: Ruby)

Shuffle lines in Vim

Thu, 07 Jul 2011 14:07:41 -0700

In a pinch, I needed to randomize the order of a few thousand lines of plain text. In Linux you can just pipe the file through sort, even right inside Vim:

:%!sort -R

But I was stuck on Windows. And I don't know how to randomize a file in native Vim script. But doing it in Ruby is pretty easy, and luckily, Vim has awesome Ruby support. Tne minutes' work and a few peeks at :h ruby and we have a successful, working kludge:

function! ShuffleLines()
ruby << EOF
    buf = VIM::Buffer.current
    firstnum =  VIM::evaluate('a:firstline')
    lastnum = VIM::evaluate('a:lastline')
    lines = []
    firstnum.upto(lastnum) do |lnum|
      lines << buf[lnum]
    end
    lines.shuffle!
    firstnum.upto(lastnum) do |lnum|
      buf[lnum] = lines[lnum-firstnum]
    end
EOF
endfunction

2011-07-07 23:32 - Edited to remove a superfluous line.

2011-07-09 21:33 - Wrong parameter for sort, oops.

Keyword Arguments: Ruby, Clojure, Common Lisp

Fri, 24 Jun 2011 17:22:48 -0700

And suddenly I return to blogging, rising from the ashes like some kind of zombie phoenix. Turns out writing a book is a good absorber of time, like some sort of heavy-duty temporal paper towel. Now that I've gotten the terrible similes out of my system, let's talk about keyword arguments, one of my favorite features in any language that supports them.

Ruby, Clojure, and Common Lisp are all languages I enjoy to some degree, and they all have keyword arguments. Let's explore how keyword args differ in those languages.

Why keyword arguments?

Why are keyword arguments good?

You can omit arguments.
You can supply arguments in an arbitrary order.
Arguments are labeled, so you know what argument means what.

Positional arguments require mentally lining up the 7th argument in your function call with the 7th argument in the function signature, and so on. Keyword arguments become more and more attractive the more arguments you have in your function signature.

Keyword arguments trade a bit of verbosity for added explicitness, clarity and reduced mental burden. (Kind of like Lisps do overall. Fancy that.)

Ruby

Ruby doesn't have special support for keyword arguments. But Ruby likes its hashes, so you can just pass one in as an argument to a function. As some syntax sugar, if you pass a "flat" list of :key => val pairs, Ruby slurps them all together and stuffs them into a hash for you.

def foo(arg)
  p arg
end

foo({:x => 123})                  # => {:x=>123}
foo(:x => 123)                    # => {:x=>123}

With even more added sugar, you can leave off the parens in Ruby function calls. So this is pretty common in Ruby:

foo :x => 123                     # => {:x=>123}

How nice and punctuation-less. But then things get ugly. What about this?

foo {:x => 123}

That won't even compile. Ruby thinks {:x => 123} is a code block, and a bare key/value pair isn't valid syntax as the first thing in a code block. You need the parens. A bit unfortunate, but it gets worse...

def bar(arg1,arg2)
  puts "#{arg1} #{arg2}"
end

bar :x => 123, :y => 456          # Runtime error
bar {:x => 123}, {:y => 456}      # Won't compile

In the first example, all of the key/value pairs are slurped into one hash and end up in arg1. There's nothing left for arg2, so you get a "wrong number of arguments" exception. The second example won't even compile, of course, because again Ruby thinks {:x => 123} must be a code block with invalid syntax.

It gets worse if you change the argument list for bar slightly...

def bar2(arg1 = {}, arg2 = {})
  puts "#{arg1} #{arg2}"
end

This way, you don't even have to supply any arguments. This is nice, if all of your argument are optional.

bar2    # => {} {}

Now suppose you want arg1 to be {:x => 123}, and arg2 to be {:y => 456}. You might naively try this:

bar2 :x => 123, :y => 456         # => {:x=>123, :y=>456} {}

Oops, it all got dumped into arg1, and now instead of a missing argument error, arg2 silently ends up with a default, empty hash. You have to explicitly pass in an empty value for arg1 so that everything is slurped into arg2.

bar2 {}, :x => 123, :y => 456     # => WRONG!  Ruby thinks {} is a code block again.

bar2({}, :x => 123, :y => 456)    # => {} {:x=>123, :y=>456}

So much for syntax sugar. You might think you'd be unlikely to find this kind of thing in the wild, but in Ruby on Rails for example, there are quite a few functions whose argument lists look exactly like this. One signature for link_to is:

link_to(body, url_options = {}, html_options = {})

So...

link_to "foo", :controller => :x                          # OK
link_to "foo", :controller => :x, :class => "css_class"   # WRONG! 
link_to "foo", {:controller => :x}, :class => "css_class" # OK

What about support for default arguments? We might want to say that if you didn't pass in an :x argument, we want it to have some default value. You might think this would work:

def baz(x = {:x => 123})
   p x
end

But you would be sadly mistaken.

baz                               # => {:x=>123}
baz :y => 456                     # => {:y=>456} ... oops

Ruby doesn't merge your keyword arguments into the map in the parameter list. It uses that map if you don't supply any arguments, otherwise your map replaces the default entirely. So to get default arguments, you need something like

def baz2(args = {})
  args = {:x => 123}.merge(args)
  p args
end

baz2                              # => {:x=>123}
baz2 :x => 555                    # => {:x=>555}
baz2 :y => 456                    # => {:x=>123, :y=>456}

Kind of messy, but that's OK.

One last subtle ambiguity in Ruby is determining whether someone passed a nil argument for a keyword explicitly, or omitted a keyword entirely. It might make a difference in some circumstances.

def quux(args={})
  p args.include? :x
  args = {:x => nil}.merge(args)
  p args
end

quux                              # true, {:x=>nil}
quux :x => nil                    # false, {:x=>nil}

Pretty messy, but such is life.

Clojure

Part of the fun of Lisps is lack of ambiguity. Everything is spelled out in all its parenthesized glory. In Clojure, when you call a function like (foo :x 123), Clojure requires you to specify what you want to happen with those arguments.

It used to be that Clojure didn't have much support for keyword args at all. Clojure did have support for allowing variable numbers of arguments to functions though. So in the beginning, people used to slurp all of their arguments together into a list, and then turn it into a map inside the function.

user> (defn foo [& args]
        (let [args (apply hash-map args)]
          (prn args)))
#'user/foo
user> (foo)
{}
user> (foo :x 123)
{:x 123}
user> (foo :x 123 :y 456)
{:y 456, :x 123}

That worked. It still works today. But nowadays there's a better way. Why not slurp your arguments directly into a hash-map?

user> (defn foo [& {:as args}]
        (prn args))
#'user/foo
user> (foo)
nil
user> (foo :x 123)
{:x 123}
user> (foo :x 123 :y 456)
{:y 456, :x 123}

This is an example of destructuring. In this case, all of our arguments are slurped into a list (thanks to &), then this list is matched against our destructuring pattern, in this case {:as args}. :as says to take everything in the list, turn it into a map and give the resulting map the name args.

This isn't a special feature of defn. Destructuring works anywhere you're setting up bindings, for example in let, for, doseq etc. Like so:

user> (let [{:as args} (list :x 123 :y 456)] args)
{:y 456, :x 123}

It just so happens that & creates the list for you, out of the arguments you pass.

Destructuring does lots more than that though. We can immediately pull out the values for keywords we care about, so they'll be bound to names in our function body.

user> (defn foo [& {:keys [x y z]}]
        (prn x y z))
#'user/foo

user> (foo :z 123 :x 456)
456 nil 123

It's certainly a bit more verbose than Ruby in the function signature, but it lacks ambiguity and it saves you some repetition in the function body.

Clojure's approach also has the benefit of specifying directly in the function signature which keywords you're expecting. In a smart editor, like Emacs, you get an indicator of what kinds of keywords you should be passing in. See at the bottom?

This is also available at the Clojure REPL via the doc function.

user> (doc foo)
-------------------------
user/foo
([& {:keys [a b c]}])
  nil
nil

There's no better documentation than live, built-in documentation. There's nothing more distracting when programming than context shifts, and having to dig into a web browser to check a function signature is a huge mental page fault.

What about default values? Sure. You can use :or to specify defaults for some or all of your keywords. This works much more like I'd expect, compared to Ruby.

user> (defn foo [& {:keys [x y z] :or {x 1 y 2 z 3}}]
        (prn x y z))
#'user/foo

user> (foo :y 555)
1 555 3

What about determining whether the user passed nil for a keyword or whether they omitted the keyword entirely? For that, you have to resort to testing the argument map for the existence of the key, which isn't fun, but at least it's possible.

user> (defn foo [& {:keys [x y z] 
                    :or {x 1 y 2 z 3}
                    :as args}]
        (prn x y z)
        (prn args)
        (doseq [k [:x :y :z]]
          (println "Contains" k "=>" (contains? args k))))
#'user/foo

user> (foo :x 1 :y 2)
1 2 3
{:x 1, :y 2}
Contains :x => true
Contains :y => true
Contains :z => false

Why do I keep mentioning this? See Common Lisp below.

In any case, our parameter list is becoming huge and unweildy. The first time I saw sample Clojure code like this, I almost did a spit-take. But after a bit of getting used to, I'm finding that this syntax is pretty comfortable.

The simplest case may not be as concise as Ruby, but aside from lack of ambiguity, the benefit of Clojure's approach is being able to safely do powerful (and borderline insane) things, like nested destructuring. And this works everywhere you're setting up a binding.

(defn ow-my-eyes [a & {[b {:keys [x y]}
                        & {[g h] :r
                           :keys [p q]
                           :or {q 123}}]
                       :x}]
  (prn a b g h p q x y))



user> (ow-my-eyes 444 :x [1 {:x 555 :y 666} :p 3 :r [888 999]])
444 1 888 999 3 123 555 666

If you wrote code like that in real life, you'd likely be defenestrated, but at least you know you can do it. And destructuring in Clojure could likely be extended even further in the future, if someone came up with a use case for something that isn't supported.

Official documentation for all of this is here.

Common Lisp

When it comes to keyword arguments, Common Lisp supports mostly everything that Clojure does, and some things it doesn't. CL was likely a big inspiration for Clojure's destructuring. I highly recommend reading Practical Common Lisp to learn more about CL keyword arguments and list destructuring. (Read the rest of the book while you're at it.)

Keyword arguments in CL look like this:

> (defun foo (&key x (y 123) (z 456 z-supplied-p))
    (pprint (list x y z z-supplied-p)))
FOO

> (foo)
(NIL 123 456 NIL)

> (foo :z nil)             ; note, z-supplied-p tells us whether z was omitted or not
(NIL 123 NIL T)

> (foo :z 555 :x 666 :y 777)
(666 777 555 T)

> (foo :x 123)
(123 123 456 NIL)

> (foo :x 555)
(555 123 456 NIL)

Yeah, there's direct support for distinguishing supplied keys with nil values, and un-supplied keys. That's pretty nice.

Common Lisp also supports insane things like having keyword arguments' default values be a function of other arguments in the parameter list.

> (defun bar (&key (x 123) (y (+ x 1000)))
    (pprint (list x y)))
BAR

> (bar)
(123 1123)

> (bar :x 5)
(5 1005)

> (bar :x 5 :y 6)
(5 6)

Clojure destructuring can't do this by default, so you'd have to resort to a let in the function body.

(defn bar [& {:keys [x y] :or {x 123}}]
  (let [y (or y (+ x 1000))]
    (prn x y)

There's nothing stopping Clojure's destructuring from being patched to support this, of course. But this isn't a feature I've ever found myself wanting particularly badly.

As for how to destructure CL keyword arguments into a vector or hash-map, CL doesn't directly support doing it in function parameter lists like Clojure does. (Though there's nothing stopping someone from throwing together a reader macro to let CL do this, of course.) This isn't really surprising, because CL loves its cons cells, while Clojure embraces maps (and vectors and sets etc.), offering them default reader syntax and lots of other built-in support.

So there you have it. Keyword arguments. Use them. Love them.

Lessons:

Sugar can be bad for your health.
Ambiguous, bad. Explicit, good.

Vim :ruby and :rubydo scope

Tue, 31 Aug 2010 10:40:51 -0700

Note to self. In old Vim (tested in 7.2.320), I could do this:

:ruby x='foo'
:rubydo $_=x

Now every line in the file says foo. But in Vim 7.3 I get an error:

NameError: undefined local variable or method `x' for main:Object

The scoping rules for Ruby in Vim must have changed somewhere along the line. I was abusing this feature to do some handy things, so this is sad.

A workaround is to use global variables in Ruby instead. So this still works:

:ruby $x='foo'
:rubydo $_=$x

Phew.

Clojure, from a Ruby perspective

Wed, 09 Jun 2010 13:22:58 -0700

Fogus' recent article "clojure.rb" speculates about why there seem to be so many Ruby users adopting Clojure. As a Ruby user who adopted Clojure, I figured I'd write about my experiences.

What do Ruby and Clojure have in common, that would attract a Rubyist to Clojure? A lot. Obviously, this is somewhat subjective and I don't expect anyone else to agree, but this is what did it for me.

Semantic consistency

In Ruby, everything is an object. It makes it simple to write code without worrying much about what kind of thing you have. foo.some_method(1,2,3) will generally work for any foo.

In Clojure, everything is not an object, specifically because it inherits primitives from Java land (though this doesn't hurt much in the kind of everyday use I put Clojure to). But also because Clojure by design doesn't even attempt to be object-oriented.

Clojure does have abstractions though. For example there's an abstraction that says "this thing can be called like a function". And then you can treat any callable-thing as a function without worrying about what it is.

In Ruby, a lot of things are Enumerable, which means you can do foo.each{} and other similar things for a lot of different types of foo. Clojure has something similar with its seq abstraction. Similarly, most Clojure data types and many Java ones are seq-able, and most built-in core functions can iterate over the guts of various things using seqs. This includes regex matches, strings, directories of files, and so on.

Ruby-style OOP brings a lot of complexity and baggage which Clojure avoids by not being OOP. For example (ignoring Java for the moment), in Clojure land you don't have to worry about a member being public/private/protected, and there are few times when you have to worry about inheritance and class hierarchies. (And there's the whole thread-safety thing.) In Clojure data is stupid and immutable, and functions are just things that take input and give output, usually side-effect free. The separation is clean and this results in programs that are very easy to reason about.

Another example of consistency: Expressions. In both Ruby and Clojure, everything has a value. Things that are "statements" in other languages are instead expressions that return something. This alone makes a lot of programs just a little bit better/easier to write.

Aesthetics

Like Ruby, Clojure code tends to be terse and expressive.

Ruby reads like poetry, because it's mostly words and not so much punctuation. In Ruby, you have none of Perl's sigils, very little of C's semi-colon line-endings and curly-delimited blocks. Especially when you start omitting optional parentheses, it has a very minimalistic vibe which is appealing to many.

    def foo(bar)
      bar.each do |x|
        puts x
      end
    end

Clojure's s-expressions are another story, of course. Some love them, some hate them. Personally, I love them. The tired old trope about Lispers not paying attention to parentheses is true; after a while they blend into the background.

    (defn foo [bar]
      (doseq [x bar]
        (prn x)))

What I see:

    defn foo [bar]
      doseq [x bar]
        prn x

Clojure has the same minimalistic feel to it, in my eyes. It also helps that so many Clojure function names are short and concise. And being able to use punctuation like ? and ! in variable/function names (true? and false? are function names), hyphens instead of underscores... these things help Clojure read smoothly.

But along with aesthetics, in Clojure you get the benefits of s-exps: no order of operations to deal with, and absolute consistency. Everything is (function param1 param2 param3). Look at how many syntax rules you have to memorize for the Ruby code above to make sense. Dot means method call, blocks are do/end delimited and have those weird pipes in there, etc.

Literals and syntax sugar

Ruby has literal syntax for many types. This includes:

:symbols
[arrays]
{:hash => :maps}
/regex/
do block end and {block}

This really is a big deal. I'm spoiled and I can't use a language without these things nowadays. It saves a ton of typing and it makes those things stand out in the code, making it both easier to write and to read. I use those structures all the time in every program I write, so they should have a terse representation.

Clojure has literal support for the same types as Ruby, and they even look mostly the same (with #"regex" being a change I can live with). And then it also has (among others):

#{sets}
#(function-literals %)
'(quoted forms)
`(quasi-quoted ~forms)

Is this a contradiction of my last point? What happened to s-expressions and consistency? Well, in Clojure, the reader shortcuts are just sugar that reduce to s-exps. You can avoid all use of that sugar if you hate it. (hash-map :key "val"), (vector 1 2 3), (quote foo) etc.

But more importantly, let's draw a(n arbitrary) distinction between "good" syntax and "bad" syntax.

Clojure's reader-macro sugar makes your code shorter, but doesn't change the structure of your code. Take a function call (f x y z), and you can always substitute a vector or regex literal or quoted form into it. (f [vector] #{set} #"regex"). The syntax sugar is very local, very self-contained. It doesn't leak into the surrounding code. And of course you can combine them in nearly arbitrary ways: '[quoted vector], {:hash-map-containing-a #{:set 'of #(functions)}}. This is "good" syntax.

Compare this to things like the x ? y : z construct, or heredocs. These things are not as orthogonal. They not only mean something on the "inside", they also influence and interact with the code before and after them, thanks to precedence rules and special parsing rules. Can you stick a heredoc in the middle of a function call? Maybe (I don't even know), but have fun with the indentation and line-breaks if so. When should you use do/end and when should you use {} for blocks? When do you need parens around your ternary if-then-else construct and when don't you? When do you need to use and and when &&? That's the "bad" kind of syntax. Sometimes, maybe even most of the time, it makes your code shorter, but there are a lot of rules to memorize and you never know when you'll be bitten.

Clojure largely avoids the "bad" syntax while taking advantage of the "good". Reader macros make your code shorter and visually easier to scan, but they rarely require you to do backflips to get your code to compile or run properly.

First-class functions

Ruby's blocks and yield and friends let you deal with first-class functions. This is a huge step in the Lisp direction already, and it's one of the things that makes Ruby great. But there are limits. Blocks use funky, special syntax, and in idiomatic Ruby, you will pass around only one block per method. There's the whole lambda and proc mess, and then there are methods-as-objects which are different still. And a lot of Ruby just calls .send on an object and passes in a method name as a symbol.

Being a Lisp, Clojure takes this a bit further. First-class functions are ingrained in nearly everything you do in Clojure. And they are easy to define and easy to call. Define f via defn (to make it top-level), fn (for a local function), or #() (sugar for fn), and then call it like (f).

Clojure also takes advantage of some functional-programming mainstays like partial and complement and comp(osition). We're not in full-blown Haskell territory, but it's a lot more FP than idiomatic Ruby.

And hash-maps, vectors, sets, keywords, and symbols are also callable as functions in Clojure. ({:foo 1} :foo) => 1. Many things can be treated as functions.

Metaprogramming

In Ruby you can mess with the innards of any class you want. There are facilities for defining methods dynamically, opening and inspecting classes at runtime, catch-all handlers for undefined methods, and all kinds of other dark magic. But again there are limits... Ruby needs to make use of eval to get certain things done. And monkey-patching is a shotgun aimed at your foot.

Well, if you like metaprogramming, Lisp macros are top of the line. You can abstract away boilerplate with a vengeance. Macros are the ultimate application of DRY.

Clojure doesn't deal much with classes, so there isn't much of that kind of introspection, but the Lisp principle of code-as-data enables a kind of introspection that you won't find in Ruby. The line between compile-time and run-time is very blurry, which enables all kinds of magic.

Java itself does offer Ruby-style reflection and such, if you need it, but you won't often, while in Clojure land.

Multimethods (and soon, protocols and defrecord) let you avoid monkey-patching and get some of the same kinds of "extend a class" things done in a saner and safer way.

So?

Fogus suspects:

Ruby programmers being the adventurous lot to begin with, are not satisfied with “halfway to Lisp”. Instead, they want it all.

This is true in my case. I like Ruby largely insofar as it borrowed and adapted many great features of Lisp. It only makes sense that I would like Clojure, which takes most of those things one step further. Clojure in particular, as a "modern" Lisp with vaguely Ruby-like syntax in certain places, is an obvious choice.

On top of that, Clojure is fast, thanks to the JVM. Ruby has JRuby too, but vanilla Ruby is not known for its speed. Clojure integrates with a REPL in a way that Ruby really doesn't, making interactive development enjoyable. Clojure is a compiled language, which has benefits for deployment. And again, there's the whole thread-safety thing. Clojure is awesome for writing sane, safe multi-threaded programs. These things are rather appealing.

I do still use Ruby though. Ruby is great for scripting, Clojure not so much, thanks to the JVM startup time, among other things. Ruby can be banged out quickly in any editor, but Clojure isn't much fun to edit in any editor that lacks good paren-matching support and REPL integration.

Rubygems offers dead-simple install of a ton of libraries, whereas Clojure is still working out the details of a standard build tool and install tool. Ruby has a library for anything, and while Clojure can use Java libraries, Java libraries tend to be huge and feature-rich, sometimes too huge for one-off tasks where a small Ruby library is a perfect fit.

Getting list of referers out of Apache logs

Sun, 21 Feb 2010 12:46:34 -0800

I use Google Analytics, but it has a noticeable lag in updating its information. When my site is being hammered, I'd like to see where all the traffic is coming from. It'd also be nice to see how many hits my RSS feed is getting, and how many images and static files are being direct-linked, which Google Analytics currently isn't tracking for me at all.

So this script will look in my Apache logs and print referers for some URL, thanks to ApacheLogRegex:

#!/usr/bin/ruby

require 'apachelogregex'

raise "USAGE: #{$0} log_filename desired_url" unless ARGV[0] and ARGV[1]

format = '%v:%p %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"'
parser = ApacheLogRegex.new(format)
pat = Regexp.new(ARGV[1])
refs = {}

File.readlines(ARGV[0]).each do |line|
  x = parser.parse(line)
  if pat.match(x["%r"])
    r = x["%{Referer}i"]
    refs[r] = (refs[r] || 0) + 1
  end
end
refs.sort_by{|k,v| -v}.each do |ref,count|
  puts "%s: %s" % [count,ref]
end

I used to use awstats for this, but it was too heavyweight and a hassle to set up and keep running. Google Analytics is a no-brainer to use, even though the accuracy isn't as good as parsing Apache logs. At least I get an idea of which of my blatherings people are most interested in.

Clojure ORM-ish stuff

Tue, 03 Nov 2009 18:21:03 -0800

Suppose I have this:

user> (def foo [{:id 1 :foo 123} {:id 2 :foo 456}])
#'user/foo
user> (def bar [{:foo_id 1 :bar 111} {:foo_id 1 :bar 222}])
#'user/bar

What I want is to "join" foo and bar so that each item in foo ends up with a sub-list of bars based on matching key fields.

In real life, these lists-of-hash-maps are coming out of a database via clojure.contrib.sql, so this is something I actually want to do pretty often. This is also vaguely similar to what you get out of a Rails-like ORM, where you end up with an object that has lists of sub-objects anywhere you have a one-to-many relationship.

Here's how I end up doing this in Clojure:

(defn one-to-many
  ([xs name ys f]
      (for [x xs :let [ys (filter (partial f x) ys)]]
        (assoc x name ys))))

Now I can do this:

user> (pprint (one-to-many foo :bars bar #(= (:id %1) (:foo_id %2))))
({:bars ({:foo_id 1, :bar 111} {:foo_id 1, :bar 222}), :id 1, :foo 123}
 {:bars (), :id 2, :foo 456})

And if I define a helper function:

(defn key=
  ([xkey ykey]
     #(= (xkey %1) (ykey %2))))

Then I can write it more concisely:

user> (pprint (one-to-many foo :bars bar (key= :id :foo_id)))
;; same as above

And if I have another "table" of data like this:

user> (def baz [{:foo_id 1 :baz 555} {:foo_id 2 :baz 999}])
#'user/baz

Then I can join them all like this:

user> (pprint (-> foo
                  (one-to-many :bars bar (key= :id :foo_id))
                  (one-to-many :bazzes baz (key= :id :foo_id))))
({:bazzes ({:foo_id 1, :baz 555}),
  :bars ({:foo_id 1, :bar 111} {:foo_id 1, :bar 222}),
  :id 1,
  :foo 123}
 {:bazzes ({:foo_id 2, :baz 999}), :bars (), :id 2, :foo 456})

This is pretty concise. It may be possible to do it in an even more concise way, (if so, do share). If I was willing to adhere to some Rails-y naming convention for my table names and for the id fields in my tables, I could make this shorter by not having to specify the names of the id fields, but I don't want to go there. It's trivial to write similar functions for a one-to-one relationship, or to use a join-table to "join" two tables with a many-to-many relationship.

I am happily surprised sometimes by how simple it is to roll my own version of things that previously seemed like dark magic. I used Rails for a long time and it seemed like a crapload of code must have gone into making the ORM work. But four lines of code gets me 75% of what I ever needed Rails' ORM for.

This may be more thanks to me opening my eyes a bit than to Clojure being awesome, but either way, I'll take it.

Happy 2nd Birthday Clojure

Sat, 17 Oct 2009 23:24:56 -0700

Clojure is two years old and it's looking good. Clojure development has been a bit quiet lately but that's because lots of big changes are apparently being worked on behind the scenes, for example rewriting Clojure in Clojure (and enhancing the Java-Clojure interop along the way, to help make this more possible). Meanwhile clojure-contrib continues to grow and the community continues to be vibrant.

I've been putting Clojure to good use at work in data munging and reporting. I've got data in an MS SQL Server database on one box (not by choice, I assure you) and a mysql database on another box, and then there's a bunch of data files in wide variety of other formats floating around the network. I use Clojure to query and compare it all.

Thanks to JDBC and clojure.contrib.sql it's easy to slurp data from a DB into a bunch of Clojure hash-maps. Thanks to clojure.contrib.duck-streams and Clojure's good regex support, the same is true of data files in general.

In the past I'd have written some enormous SQL queries to generate reports, but it's so much easier to use Clojure's wide array of sequence-manipulation functions to manipulate hash-maps. Doing what I want is rarely more difficult than mapping some transformation over the data, filtering out the data I want, and then formatting it nicely (which is easy thanks to Common Lisp-style formatting).

And once I notice patterns in how I'm using those things, I write a few functions and macros to make it more concise. Consicion is one area where Lisps cannot be beaten (short of APL). For example, give me all data from the mysql db "data4" which is collected at "Site1" and happened before 2008, and group it by the person who collected it:

user> (group-by :collector 
                (filter #(and (= (:site %) "Site1")
                              (date-before? (:date %) 
                                            (date-from-string "2008-01-01")))
                        (mysql-data :data4)))

date-from-string isn't a standard function but here's how easy it is to write it, thanks to the JVM:

(defn date-from-string [s]
  (.parse (java.text.SimpleDateFormat. "yyyy-mm-DD") s))

That's pretty much how my data-querying looks. To some that probably looks terrifying, but thanks to Emacs+Paredit it's a few handy keystrokes to type auto-complete and manipulate and automatically pretty-format such things, and thanks to a suitably large dose of Lisp brand Kool-Aid I find it very natural and comfortable to read at this point.

Then I can (csv ...) it or (plaintext-table ...) it or whatever. I replaced thousands of lines of Ruby and SQL queries with a few hundred lines of Clojure this way, and the Clojure version does more and does it better.

One of the things I like about Clojure is that it's such a small language, you can be reasonably sure you know the whole language (or at least have some passing familiarity with all parts of it) once you've read the docs and the API a few times. This is in contrast to languages like Common Lisp for example where the standard is thick enough to be considered a deadly weapon. Java lurks underneath but you can get away with ignoring it almost entirely, until those rare times you need it.

Alan Perlis said:

It is better to have 100 functions operate on one data structure than 10 functions on 10 data structures

This is true, but Clojure takes it further: It's even better to have a lot of functions that act on one abstraction of a bunch of data structures. Clojure gives you a bunch of data structures that can all be accessed under the same seq abstraction, and a bunch of functions that work on that abstraction, and the end result is more than the sum of the parts, because everything is interchangeable in lovely ways. I bounce data between sets and hash-maps and vectors without even thinking about it half the time.

That's just one reason of many that I love Clojure nowadays. I get crap done with it, quickly and easily and with surprising amounts of fun. Thanks Clojure and Clojure devs for making my life better.

Now I have two problems

Fri, 25 Sep 2009 23:02:14 -0700

I'm converting one of my websites from Ruby on Rails to Clojure in my spare time. I stupidly put a bunch of RoR-style links inline into certain bits of plaintext content, so in my DB there are a bunch of text fields with <%= link_to ... %> in the middle.

It was easy to fix with a regex though:

(defn clean [txt]
  (re-gsub #"<%=\s*link_to\s+(\"[^\"]+\"|'[^']+')\s*(?:,\s*'([^']+)'\s*)?(?:,\s*image_path\(['\"]([^'\"]+)['\"]\)\s*)?(?:,\s*:controller\s*=>\s*(?::(\S+)|['\"]([^\"']+)['\"])\s*)?(?:,\s*:action\s*=>\s*(?::(\S+)|['\"]([^\"']+)['\"])\s*)?(?:,\s*:id\s*=>\s*(?:(\d+)|:(\S+)|['\"]([^\"']+)['\"])\s*)?\s*%>"
           (fn [[_ s & parts]] (let [href (str-join "/" (filter identity parts))]
                           (str "<a href=\"/" href "\">" (re-gsub #"^[\"']|[\"']$" "" s) "</a>")))
           txt))

And by easy, I mean not easy.

Note to self, try something other than a regex next time.

Note to self, don't bury some framework's funky-syntax DSL in the middle of plaintext content. Next time use HTML or do the conversion from DSL to HTML early rather than late.

Silly how two years ago I thought I'd be using Ruby for that site forever.

Plamsa + Ruby = Ouch

Tue, 08 Sep 2009 23:34:17 -0700

I wrote my first KDE4 plasmoid the other day. I can't release it because it's essentially a clone of something you aren't allowed to copy (maybe I can replace him with a penguin and release it that way though).

But I need to rewrite it first anyways, because I did it using the Ruby bindings for Qt4 and Plasma, and wow it's painful. It has a 50/50 shot of even initializing at any given point. When it does initialize, it has about a 1 on 8 chance of immediately crashing Plasma. And some things I just can't get to work at all, e.g. setting a default size or resizing the applet programmatically; X-Plasma-DefaultSize in the metadata is supposed to do it but it does nothing. And it's not just my system (using KDE 4.3), because I tried it on a Kubuntu machine using stable KDE 4.2 and had the same problems.

The other snag is that the documentation of the Plasma API is buried so deep on the KDE site that I don't even know how I found it. Here it is for those who care (and for my own future reference). I hit lots of dead links on the KDE site on the way there.

Next step is to rewrite the plasmoid in Python or C++ I guess.

Practicality: PHP vs. Lisp?

Mon, 22 Sep 2008 01:17:25 -0700

Eric at LispCast wrote an article about why PHP is so ridiculously dominant as a web language, when arguably more powerful languages like Common Lisp linger in obscurity.

I think the answer is pretty easy. In real life, practicality usually trumps everything else. Most programmers aren't paid to revolutionize the world of computer science. Most programmers are code monkeys, or to put it more nicely, they're craftsmen who build things that other people pay them to create. The code is a tool to help people do a job. The code is not an end in itself.

In real life, here's a typical situation. You have to make a website for your employer that collects survey data from various people out in the world, in a way that no current off-the-shelf program quite does correctly. If you could buy a program to do it that'd be ideal, but you can't find a good one, so you decide to write one from scratch. The data collection is time-sensitive and absolutely must start by X date. The interface is a web page, and people are going to pointy-clicky their way through, and type some numbers, that's it; the backend just doesn't matter. For your server, someone dug an old dusty desktop machine out of a closet and threw Linux on there for you and gave you an SSH account. Oh right, and this project isn't your only job. It's one of many things you're trying to juggle in a 40-hour work week.

One option is to write it in Common Lisp. You can start by going on a quest for a web server. Don't even think about mod_lisp, would be my advice, based on past experience. Hunchentoot is good, or you can pay a fortune for one of the commercial Lisps. If you want you could also look for a web framework; there are many to choose from, each more esoteric, poorly documented and nearly impossible to install than the last. Then you get to hunt for a Lisp implementation that actually runs those frameworks. Then you get to try to install it and all of your libraries on your Linux server, and on the Windows desktop machine you have to use as a workstation. Good luck.

Once you manage to get Emacs and SLIME going (I'm assuming you already know Emacs intimately, because if you don't, you already lose) you get to start writing your app. Collecting data and moving it around and putting it into a database and exporting it to various statistics packages is common, so you'd do well to start looking for some libraries to help you out with such things. In the Common Lisp world you're likely not to find what you need, or if you're lucky, you'll find what you need in the form of undocumented abandonware. So you can just fix or write those libraries yourself, because Lisp makes writing libraries from scratch easy! Not as easy as downloading one that's already been written and debugged and matured, but anyways. Then you can also roll your own method of deploying your app to your server and keeping it running 24/7, which isn't quite so easy. If you like, you can try explaining your hand-rolled system to the team of sysadmins in another department who keep your server machine running.

Don't bet on anyone in your office being able to help you with writing code, because no one knows Lisp. Might not want to mention to your boss that if you're run over by a bus tomorrow, it's going to be impossible to hire someone to replace you, because no one will be able to read what you wrote. When your boss asks why it's taking you so long, you can mention that the YAML parser you had to write from scratch to interact with a bunch of legacy stuff is super cool and a lovely piece of Lisp code, even if it did take you a week to write and debug given your other workload.

Be sure to wave to your deadline as it goes whooshing by. If you're a genius, maybe you managed to do all of the above and still had time to roll out a 5-layer-deep Domain Specific Language to solve all of your problems so well it brings tears to your eye. But most of us aren't geniuses, especially on a tight deadline.

Another option is to use PHP. Apache is everywhere. MySQL is one simple apt-get away. PHP works with no effort. You can download a single-click-install WAMP stack nowadays. PHP libraries for everything are everywhere and free and mature because thousands of people already use them. The PHP official documentation is ridiculously thorough, with community participation at the bottom of every page. Google any question you can imagine and you come up with a million answers because the community is huge. Or walk down the hall and ask anyone who's ever done web programming.

The language is stupid, but stupid means easy to learn. You can learn PHP in a day or two if you're familiar with any other language. You can write PHP code in any editor or environment you want. Emacs? Vim? Notepad? nano? Who cares? Whatever floats your boat. Being a stupid language also means that everyone knows it. If you jump ship, your boss can throw together a "PHP coder wanted" ad and replace you in short order.

And what do you lose? You have to use a butt-ugly horrid language, but the price you pay in headaches and swallowed bile is more than offset by the practical gains. PHP is overly verbose and terribly inconsistent and lacks powerful methods of abstraction and proper closures and easy-to-use meta-programming goodness and Lisp-macro syntactic wonders; in that sense it's not a very powerful language. Your web framework in PHP probably isn't continuation-based, it probably doesn't compile your s-expression HTML tree into assembler code before rendering it.

But PHP is probably the most powerful language around for many jobs if you judge by the one and only measure that counts for many people: wall clock time from "Here, do this" to "Yay, I'm done, it's not the prettiest thing in the world but it works".

The above situation was one I experienced at work, and I did choose PHP right from the start, and I did get it done quickly, and it was apparently not too bad because everyone likes the website. No one witnessed the pain of writing all that PHP code, but that pain doesn't matter to anyone but the code monkey.

If I had to do it over again I might pick Ruby, but certainly never Lisp. I hate PHP more than almost anything (maybe with the exception of Java) but I still use it when it's called for. An old rusty wobbly-headed crooked-handled hammer is the best tool for the job if it's right next to you and you only need to pound in a couple of nails.

Perl6 features borrowed from Lisp

Tue, 09 Sep 2008 15:28:57 -0700

Via PerlMonks I found a couple of articles discussing in good detail some of the new features of Perl6.

Perl6 steals even more things from Common Lisp than Perl5 did: it has multimethods / multiple dispatch for example, which is a huge plus. Via this interview with Damian Conway we learn that Perl6 will also have named, optional, and "rest" parameters to subs, just like in CL. That's also a good thing; CL's parameter-passing styles are nice, and it's awesome how you can combine them. Certainly better than Perl5 (but everything is better than Perl5). There's also apparently special Perl6 syntax for applying functions to lists and currying functions, and weird Capture objects to explicitly deal with multiple-value returns from subs. Good stuff.

Perl6 is also apparently taking first-class functional objects to an extreme; blocks, subs, and methods are all objects and there are all kinds of metaprogramming hooks to screw around with them. This is one area where Ruby is just a little bit lacking: functions and methods aren't quite first-class enough in Ruby. Most people seem to pass around symbols / names of methods rather than pass around methods as objects themslves. Anonymous blocks are used liberally but mostly via yield, limiting you to one block per method and largely hiding away the block objects themselves.

I'm honestly a bit excited about Perl6, but largely as a curiosity or new toy to play with. It is kind of interesting how languages keep creeping more and more toward Common Lisp. If Perl is a nicer-looking Common Lisp which I can edit properly in Vim, it'll be almost a dream come true; I hate Emacs and Common Lisp tends to be butt-ugly. (Not talking about the parens, mostly about the verbosity and cruft and inconsistencies. Larry Wall famously said that Common Lisp looks like (paraphrased) "oatmeal with toenail clippings mixed in". Perl is certainly at the other extreme.)

http://rakudo.org/ is a good site for keeping up on Perl6 news. It's pretty active. Here's hoping we see a real release of Perl6 someyear.

Vim + screen + REPL = win

Sat, 06 Sep 2008 00:36:48 -0700

Via the Clojure wiki I found a great page describing how you can use GNU screen and some Vim magic to let Vim play nicely with an interactive commandline program like a Common Lisp REPL, Ruby's irb, or Python's, well, python.

That page is a very stripped-down and simpler version of what Limp does for Vim+Lisp. But Jonathan Palardy's version has the benefit of being so simple that you can set it up yourself manually in a second or two. I still have never gotten Limp to work quite right and I don't have the time to debug a big mess of Vim script.

The idea is to start up a named screen session via e.g. screen -S foo -t bar, then start an irb session (or whatever) in there, and then in Vim you can simply yank some text into a named register and send it off to screen via a system call. Download Jonathan's code and see.

It's not a full-blown SLIME; it doesn't have tab-completion or weird interactive debugging windows or such bullcrap. It doesn't capture the output of your command and feed it back into your Vim buffer. But hey, it's pretty good for something you can throw together in 2 minutes, and it works.

So there goes my last reason to ever use Emacs. Good riddance, I must say.

Honestly, Emacs just frustrates the living hell out of me. Oh how I tried to like it. I really did. I've used it on and off constantly over the past year. I have Emacs shortcuts written all over the whiteboard in my office. But its braindead window management, its terrible broken undo/redo system, its finger-crippling key-chord combos, its lack of features I need (like line numbering), its reliance on broken 3rd-party elisp hack scripts for things Vim has built in (like line numbering!), its ugly fonts and GUI elements, and so on and so forth. Vim is such a joy in comparison.

Making Java not suck

Mon, 07 Jul 2008 21:45:32 -0700

There are some good things about Java. The virtual machine has been refined for quite some time. The garbage collector is likely to perform well. The standard library has gone through many iterations and is very encompassing and complete and amazingly well-documented. The community is enormous. The language is as cross-platform as you could reasonably expect any huge program to be. It has nice GUI frameworks (which nowadays even look native on Windows and Linux, if you use SWT), a good threading library, good socket libraries, and all the things I wish Ruby or Common Lisp had.

The one unignorably bad thing about Java is that you have to write it in Java. It's next to impossible to write Java by hand, and it's still a whole lot of pain even if you use one of the massive Java IDEs that trick you into not noticing the pain. The language is way too verbose. The syntax is busy and full of mandatory brackets and parens and punctuation and bullcrap. The demand that you catch every conceivable exception is tiresome. The ability to abstract all of those things away isn't present. The package / import scheme is way too much typing for any human being. No Lisp-style macros, no easy-to-use anonymous functions, clunky iterators, primitive looping constructs. Everything is forced into a Object Oriented mindset even if it doesn't fit well. And so on.

But the good thing is that nowadays you don't have to write Java in Java. You can write Java in Ruby using JRuby, or write Java in Lisp using Clojure. I gave both of these a try in the past week or so, and both are awesome. You can write the bulk of your program in a nice powerful fun-to-write language, and call out to Java to handle the GUI bits or whatever Java is good at handling. You can write tasty Ruby or Lisp abstractions that hide the horrible mess that is Java's syntax. (There's also Jython, if you swing that way.) You'd think it'd be more effort to write Ruby code that translates to Java than just to write plain old Java, but Ruby is so much better than Java that it actually ends up being easier. For me anyways.

It seems like the lines between programming languages is awfully blurry nowadays. Most languages have some way to interface directly with C code. There are GTK and QT bindings for everything under the sun. We have people writing Lisp interpreters in Javascript, Python interpreters in Lisp, and so on. You have all these VMs (JVM and .NET and Parrot (if it ever gets done)) which let you write the same program in whatever language you feel like. And most of them are cross-platform to some degree or other.

It's an interesting trend which makes a lot of practical sense. No language is great at everything, so why bother limiting yourself to one language per program? Especially with computers being so fast today, we can get away with layering language on top of language. It's a nice situation, if you want to get programs done as fast and with as little effort as humanly possible.

Wish list

Tue, 17 Jun 2008 23:47:43 -0700

What's the Common Lisp version of Perlmonks or Ruby-forum? I have yet to find it.

comp.lang.lisp is largely crap. 50% of the traffic on that list is spam about shoes and fake watches. The other half is equally split between:

People debating tiny, silly semantic points of the Common Lisp Hyperspec.
People stuck in the 70's or 80's, talking about the good old days, ruminating about Lisp history.
Flame wars.
New people asking for help. Some get good honest advice and helpful answers, many are flamed and ridiculed into next week if they even hint that they dislike the parentheses.

The Common Lisp community (if you can call it that) is a bunch of really smart guys, but they all live isolated in hermit shacks up in the mountains and they spend their time doing magic tricks with Lisp that few people ever see, and if you wander too close they throw rocks at you.

What's the Common Lisp equivalent of perldoc or rdoc? We have the Hyperspec. It's an impressive document, but it's a bunch of painful HTML that looks like it was created in the early 90's, probably because it was. It reads like a dusty, dry, technical document probably because it is. What it's not, is friendly or easily readable.

Perl has CPAN, Ruby has rubygems, what does Lisp have? Either a hand-rolled system definition script, or if you're lucky an ASDF install file. ASDF is the semi-standard Lisp way of installing libraries, except that it doesn't quite work in Windows, it doesn't check dependencies or handle different versions of a package very well, and it doesn't work the same on all Lisp implementations. Many people in the so-called community think it's not very good.

The fellow running Lispcast makes another good point. Where can you download Lisp? It's not obvious.

You could say "OK Brian, good idea, now get to work!" The problem is that even if I had the time or willpower, I'm not the smartest guy in the world. I honestly don't think I could design and run and maintain a CPAN. And even if I did, would anyone use it? But I do know that there ARE plenty of smart, enthusiastic people using Lisp. Yet high-quality friendly code is largely not being produced.

Peter Christensen wrote about "langauge snobs" and the importance of community. One point made is that some really ugly, horrific languages have been extremely successful simply because they've been accessible and fun. An example given is the scripting language in Second Life, which has over 2.5 billion lines of code written in by tens of thousands of amateurs and has accurately modeled a realistic 3D environment with thousands of users at any given time. All in an ugly language some guy invented AND implemented in one week. The developers admit that the language is total crap, but it doesn't matter. 1) It has very good and accessible documentation, 2) it has a very newbie-friendly community, and 3) and it's easy to pick up, throw together some code and get immediate results. Three things Common Lisp lacks.

This is something I've said myself many times: an active, supportive, enthusiastic community is essential for the health of any programming language. Common Lisp simply doesn't have one and it's a shame.

I still secretly hope that Clojure or NewLisp or Arc turn out to be a huge success. They are the kinds of things Lisp needs today.

Email woes

Fri, 18 Apr 2008 20:49:16 -0700

I own my own domain (or five) and one of the good things about that is having nearly infinitely many email accounts if you want them. So I tend to make up a new account for every site I register at. This leads to amusing things like getting an email from a marketing firm asking me to complete a survey for an airline "who wants to remain STRICTLY ANONYMOUS". Sent of course to UNITED@briancarper.net. Oops.

Because of laziness I set up a catchall account on my domain so every email sent to anything @briancarper.net would be sent to me. This was such a horribly bad idea, I'm unsure how I lasted for a couple of years this way. I was getting about a few hundred spam emails per day. Amazingly spamassassin + Thuderbird's junk mail filter caught almost every single one of them to the point where I hardly even noticed. Spam filters can be bad in the same way pain killers can be bad. They don't solve a problem, they only mask the pain so you can ignore the problem.

So I decided to stop using a catchall. Problem is that I already have around a hundred email addresses I've used for various message boards and companies and friends and family, and there's no way I'm going around to change them all. So I decided to just get a list of them all and set up a big list of postfix aliases for now.

So, I downloaded my whole email account in mbox format and wrote a Ruby script to crawl it and make a list of all the email accounts I've ever received mail from. Thank you Linux mailserver for storing email sanely in plaintext. Luckily for me, I haven't deleted any emails from my server since 2005; so my generated list of emails is likely to be pretty complete. It pays to be obsessive sometimes.

Even a braindead brute-force Ruby script is fast enough to do this. Took a minute or two to scan 200MB of plaintext.

#!/usr/bin/ruby
require 'find'
found = {}
Find.find(ARGV[0]) do |fn|
  next unless File.file? fn
  File.read(fn).scan(/[A-Za-z0-9_-]+@briancarper.net/) do |email|
    next if found[email]
    found[email] = true
    puts email
  end
end

Debugging (awesomely)

Sat, 12 Apr 2008 12:56:38 -0700

The simple act of jamming lots of print statements into your code to output values to help with debugging turns out to be tedious once you do it a billion times. One problem is in Ruby code that looks like this contrived example:

arr.sort.map{|el| "'#{el}'"}.join("\n")

If you wanted to inspect your object right in the middle, say after the sort but before the map, how could you? Like this maybe:

tmp = arr.sort
p tmp
tmp.map{|el| "'#{el}'"}.join("\n")

Ugly. Ruby 1.9 apparently has Object#tap, which has been widely used in the community for a while anyways I believe but will now be a standard method. It has the simple definition:

def tap
  yield self
  self
end

So you can do

arr.sort.tap{|o| p o}.map{|el| "'#{el}'"}.join("\n")

Just makes your life a bit easier. (Though tap can be used for more than printing things obviously.) Ruby's ability to mess with the innards of standard classes makes this possible. In languages which lack this power, you couldn't do this so easily.

Then you have Common Lisp, which arguably takes that kind of power to another level. So you can do something awesome like this and print intermediate results the whole way down the call chain without even having to edit your original code at all. The author mentions that it won't work with certain macros and special forms, but it's still awesome and useful even given its limitations. How could you do this in Ruby?

Perl 6

Mon, 07 Apr 2008 23:14:25 -0700

I found this Perl6 imaginary timeline hilarious. Sadly we're in 2008 and I don't think that blue line is quite as high as it was projected to be. Though it appears that development is still slowly chugging along. I still check up on Perl 6 once every six months or so. You can actually download perl6 (called rakudo?) and run it nowadays.

It's been a long time since I've used Perl for anything. If Perl 6 ever sees the light of day in a big way, I'm sure I will hunt down the Perl 6 bandwagon and chain myself to the back of it. Perl was my first religion.

I don't want to consider myself fickle, rather curious and eager to try and learn new things. But honestly, fickle may be closer to the truth. Already my Lisp enthusiasm is starting to wear off, largely from the impracticality of writing anything in Lisp (in spite of good book titles to the contrary). I've written a bunch of things in the past few months, but I never even considered Common Lisp for any of them. Mostly because I was being paid to write them, and I don't think people want to pay me to screw around with something that's going to take me 10x as long as using a language I know already.

I believe the main way I came to know and love Perl and later Ruby was through all the little 5-minute throwaway scripts I wrote to get my job(s) done. Those little scripts led to bigger scripts, which led to even bigger scripts. Emotionally it's satisfying to actually solve a problem in a language, even a small problem. Not so easy to do in Lisp; it doesn't seem to lend itself well to scripting. Lisp is nice if you've come up with the perfect abstraction for your program and want to implement it exactly like you're thinking of it. Most of the time, by the time I think up the perfect abstraction, I'd already be done if I'd written it in straightforward Ruby to begin with.

And, to rant briefly, in Ruby I wouldn't have to write my own function to fetch all the keys of a hash, or print a formatted date string, or any of the other countless little holes in Lisp that are nicely filled in Ruby by all the available libraries. I've said it before and I'll say it again: a large active community is one of the biggest necessities for a healthy programming langauge. It really doesn't seem like many people in the Lisp community give a crap about getting lots of new people to use Lisp. Or if so, I never hear much about it, compared with other communities. There's no reason a language can't be really great and powerful, and still accessible to the masses. That's one of Ruby's strengths. You don't have to read a few hundred pages of hyperspec and learn 50 years of history and spend a week and half setting up an arcane SLIME/Emacs environment to start writing Ruby.

But yeah, Perl 6. I want it to succeed, for entirely emotional reasons. I want a new toy.

C++, blech

Tue, 01 Apr 2008 22:15:36 -0700

I decided to try a GUI app. I decided to use QT4. I then quickly remembered why I don't ever use C++ for anything. The amount of bullcrap you have to go through just to get a simple program going isn't worth the bother for the kinds of programs I write. I spent an hour on this and I haven't even started writing any of the actual program yet. Just setting up boilerplate garbage, using qmake to make the project file used to make the Makefile used to compile my program someday if I ever actually progress far enough to write it. But no, the compiler couldn't find some of Qt's headers, so I get to hunt them down and pray. Nope, not worth my time. You know it's a bad sign when you can copy/paste a Qt tutorial right from the Qt docs and it won't compile.

I've always detested C++. I thought maybe I'd start liking it more as I learned more about how computers and compilers and interpreters work, but no, actually I like C++ less and less. I read some books on C++ a while back and it seemed nice at the time, but reading about a language and using it yourself are two very different things. I find myself disliking statically typed languages in general in ever-increasing amounts. And languages without garbage collectors seem archaic to me. And any language that requires you to learn multiple new languages just to be able to configure and use the tools used to build your program is ridiculous. I have a hard enough time getting my logic right, let alone having to hand-hold the computer while it laboriously and clumsily stumbles through my code. Ruby and Lisp have spoiled me.

And God help me if I ever want to run it on someone else's machine. Or a different OS. I started looking into how to compile Qt apps in Windows and now I shudder to think what I almost got myself into.

Interpreted languages are as close as anyone's going to get to "Write once, run anywhere". Step 1: Install Ruby. Step 2: Write Ruby code. Step 3: Run it. Short of places the OS leaks through like pathnames or system calls, your program will work. End. Underneath all of that, someone still had to figure out all the low-level crap. But someone smarter than me figured that out, and had to figure it out ONCE, and then it's done with and I can take advantage. Worst thing that can happen is I get some missing Ruby files and throw them in a directory somewhere Ruby can find them.

Programming after all is all about abstraction. Abstraction layers help us deal with complexity. Ruby is an abstraction layer over a mess of C code and compiled and linked binaries and libraries that I never have to look at. That's the way I likes it.

I was going to use Ruby's QT bindings for this app, but I want it to be multithreaded, which probably rules Ruby right out. Sad. Maybe Python has good Qt4 bindings. Horrifying though the thought of writing Python may be, at least the syntax is more stomach-able than C++.

Blah blah blah

Sat, 22 Mar 2008 09:25:47 -0700

My Westinghouse L2410NM LCD monitor arrived at Westinghouse's factory, according to UPS tracking, on Thursday at 10AM. Thus the clock starts. In one week I'll begin periodic phone calls to Westinghouse requesting updates on the status of my RMA. I know that every phone call you make to a call center costs the company money; it's in Westinghouse's best interests as well as my own to give me good information and satisfy my curiosity.
Yet another reason Vista sucks: trying to get VPN to work. Took me a week and the help of many people from my company's IT dept. before I even got close. The Control Panel UI philosophy of Vista seems to have been to take the XP Control Panel and scatter the options to the four winds; then build an insultingly dumbed-down GUI that has lots and lots of links to all those scattered pieces. Any kind of network configuration takes twice as long and twice as much navigation through a maze of windows and tabs and icons and links than it did in XP. And XP wasn't all that great to begin with.
My Ruby on Rails project for work is turning out well. Deploying Rails is a bit easier than the last time I had to do it over a year ago. Apache's mod_proxy + Mongrel makes things pretty easy. Getting it to work with SSL is also doable; as per these lengthy instructions you have to put
```
RequestHeader set X_FORWARDED_PROTO 'https'
```
in your Apache config (this line requires Apache's mod_headers).
Ruby on Ubuntu / Debian still sucks. I don't like how they break standard Ruby up into many parts and different packages. You kind of expect Ruby to come with rdoc and ri, not to have to install them separately.

(Read the whole crappy story of Westinghouse's dishonesty and horrible customer service: The beginning, Update 1, Update 2, Update 3, Update 4, Update 5, Update 6, Update 7, Update 8, Update 9, VICTORY, aftermath.)

Ruby on Rails migrations

Wed, 12 Mar 2008 23:15:21 -0700

I started a new RoR project recently, and for the first time tried using migrations. They're mildly acceptable yet I still find myself not caring much for them.

The benefits of migrations as I understand it (and why those benefits don't matter to me) are:

You get to "version control" your database schema.

But if I want version control, I'll use a real version control system. A bunch of files in a directory sorted by a numeric prefix is rudimentary compared with Bazaar or SVN or anything else.

Migrations also require you to write all the code to "rollback" your changes yourself by hand, which is tedious. Every time I write an add_column I have to write a corresponding remove_column too. Many times I've gotten the add part right, and typoed the remove part, so the migration was broken, and I didn't realize until much later. Many times I wrote a remove part that I never ever had a need to use.

Because migrations are a single big long linear path, rolling way back to some point in history will obliterate a lot of changes you may want to keep that were made after that point; the older a migration is, the less likely you'll ever run it again.

Maybe I should be unit testing my migrations, you may say. I say what's the point? I'm adding a column to a DB. Do I really need to test that? Unit testing is fine, but there's a point beyond which it becomes silly. Especially for a web app, especially for something so trivial. * You can write DB stuff in Ruby rather than writing SQL manually.

This saves you a little bit of effort. But if you know SQL, it's not THAT hard to write an ALTER TABLE query. In fact given the whole script/generate migration SomeLongName step and then opening and editing and saving and testing and uploading and running the migration file, it's actually far faster for me to drop into a mysql prompt and do it that way.

There are also some things you can't do with Rails migration methods, and you need raw SQL. In which case you end up writing SQL anyways. Depending what you're doing, this may never happen or it may happen frequently enough to be annoying. * You can deploy your DB changes by uploading the migrations and running rake.

This is a good thing, but you can also save some SQL into a text file and run it by piping it into mysql from the command line, or importing it via phpmyadmin, or various other methods. * Migrations are database-agnostic, whereas SQL is usually tied to a single database.

My first reaction is, so what? What are the chances that I'm going to change my DB engine? Not very likely in the kinds of things I work on. My second reaction is that the kind of queries you write to create and edit tables are so generic that they're likely to work in most or all DB engines that Ruby supports anyways, or will work after minor edits.

Migrations do some magic things like automatically setting up an id field for your models etc.

This is the only real benefit to migrations that I can see: a small amount of time savings and convenience and integration with the rest of the tangled mess of RoR.

What I'm trying right now is writing all my table creation code in SQL, putting it all into .sql files, and putting those files under version control in Bazaar. This is better version control than migrations offer. And my table definitions are completely intact in a single file each, which is easily browsable and editable. (I think Rails lets you export schemas too, but it's not as nice as having them in files to look at and edit directly in my opinion.) If I want to see differences in my schema over time, I can bzr diff the files that contain them. If I want to take a single table and reproduce it in some DB somewhere, I can run that single .sql file. If I want to deploy, I rsync the files to my server and pipe them into a mysql prompt.

The good thing about RoR is that I can completely ignore migrations if I don't like them. And for my large legacy RoR app, I am ignoring them. However I plan to stick with them on this new app as far as I can; maybe more benefits will reveal themselves.

Confidence

Tue, 11 Mar 2008 16:23:55 -0700

One of my flaws when it comes to coding is lack of confidence, or you could call it lack of imagination.

At work we have some data files which are some funky binary format, readable only by some special program. This program has no command line interface. On a whim I thought hey, wouldn't it be nice if I could write my own tool to read these data files, then I could make a command line tool and it'd be scriptable. My first reaction was "Impossible! I don't have a spec for that file." That's a typical reaction on my part.

But when I got a hex editor and looked at the file, it turned out to be super-easy to parse. There were field names in ASCII, then some garbage bytes, then a byte indicating the length of the value, then the value, then a null byte. Records in the file were separated by a certain string of eight bytes. There's a bunch of other garbage in there which may or may not turn out to be significant, but it's not that hard to guess. I ended up writing a naive Ruby parser for these files and it was only around 20 lines of code.

A good strategy that I need to work on adopting is just to assume I can do things until it's been proved that I can't. A lot of problems that look hard are really easy given a good algorithm or a good approach to writing your code. On the other hand a lot of problems that seem easy on the surface turn out to be NP-hard, so the opposite is true too. Programming simply is not an intuitive exercise to my brain, which means the best solution is to try to avoid relying on my intuition.

RSS feed (Common Lisp)

Mon, 18 Feb 2008 18:26:58 -0800

I made an RSS 2.0 feed for my origami gallery the other day. Thanks to CL-WHO this is trivial. Here's the complete code:

;; RSS
(defun rss ()
  (cl-who:with-html-output-to-string (s nil :prologue "<?xml version=\"1.0\" encoding=\"ISO-8859-1\" ?>")
    (:rss :version "2.0" :|xmlns:atom| "http://www.w3.org/2005/Atom"
          (:channel (:title "An Origami Gallery")
                    (:link "http://origamigallery.net")
                    (:|atom:link| :href "http://origamigallery.net/feed" :rel "self" :type "application/rss+xml")
                    (:description "A photo gallery.  Of origami models.")
                    (loop for model in (all-models)
                          do (htm (:item (:title (str (fullname model)))
                                         (:link (str (absolute-url model)))
                                         (:guid  (str (absolute-url model)))
                                         (:description (str (clean-remarks (remarks model)))))))))))

There are other Common Lisp libraries to make RSS feeds, which may have made this even easier, but I like CL-WHO.

Interesting about this code is how short it is. CL-WHO is an example of embedding another language in Common Lisp (e.g. a domain specific language), in this case, XML or HTML.

XML in particular happens to map very nicely onto s-expressions, allowing nested tags and attribute/value pairs and whatnot. Except that s-exps are far less verbose than XML, far nicer to type, and far easier to read in my opinion. One quirk here is that CL-WHO uses keyword symbols for XML tag names (CL keyword symbols begin with a colon), and XML uses colons for namespaces. To make a keyword symbol with a colon in the middle in CL, you have to surround it in pipes. No big deal though.

This kind of thing isn't entirely possible or quite as elegant in most other languages. (Ruby does pretty good though.) In PHP, for example, you can mix up PHP and HTML content, but the PHP interpreter doesn't parse the HTML or do anything interesting with it. It just treats it as a bunch of strings; embedding PHP in HTML is just a shortcut for string concatenation.

You could write a library in PHP to generate XML code (and I'm sure such libraries exist) but they're either going to have some kind of functional / OO interface, which will make it very verbose and probably clunky to use in comparison to CL, or else it's going to be something like Smarty which actually isn't PHP at all but rather a separate language with its own parser and interpreter or compiler that itself happens to be written in PHP. What you could not easily do is write PHP code that turns other PHP code into an XML document.

But that's largely what CL-WHO does. Because CL-WHO parses the s-exps that represent my XML data, if I forget a closing paren, Lisp will yell at me, so my XML is guaranteed to be well-formed. CL-WHO doesn't verify that I'm using proper RSS tags in the proper places or anything but it could easily be extended to do that. See PCL for an HTML interpreter/compiler which can verify that you only use valid HTML tags, for example.

CL-WHO also lets me change whether my XML attribute values are in single or double quotes, or whether I want to allow empty tags (as in old HTML) or require them all to be closed, and other such things. CL-WHO can nicely indent my XML, or generate it unindented.

And since this is all just Common Lisp, you see that I can embed a loop right in the middle of the XML s-exps, or do conditionals, or I could write a function that generates some or all these s-exps for me programatically. Or any number of other things. I have all of Common Lisp at my fingertips to create these s-exps.

See also "The Nature of Lisp" which is a nice long discussion of how Lisp is a better XML than XML. (And I do mean looong. I haven't read the whole article myself, but enough to know it's pretty good.)

Lisp ramblings

Fri, 15 Feb 2008 00:09:06 -0800

Two posts in one, just because I can.

Rambling #1: One thing that makes a language like Ruby easy to read is that foo.bar always has the same meaning: object.method.

In Common Lisp you don't have that. One of CL's strengths is that everything looks the same, it's all s-expressions. However a cost of this is that you can't tell just by looking whether (foo bar) is:

A special operator foo, that does something with bar (maybe assigning it a value, maybe conditionally evaluating it, who knows).
A system-defined function call foo, where bar is evaluated and the result is passed in as an argument, and which you can't redefine because it's in the COMMON-LISP system package.
A user-defined function call foo, where bar is evaluated and the result is passed in as an argument, and which you can re-define if you want.
A method call foo, where bar is possibly an object used to dispatch the method call (or maybe not).
A macro foo, which can produce code that does practically anything, or nothing at all, with bar.
Other?

Along the same lines, even if you know for sure that (foo bar baz) is a method call, you can't tell by looking whether bar, or baz, or both, are objects which determine how the method call is dispatched. You have to look it up in documentation or inspect the method somehow. As far as I know. (EDIT: Slime has lots of tools for doing this kind of inspection., as was pointed out to me by Ivar.)

This is a good feature of Common Lisp, in terms of flexibility and power; multimethods are great, the consistency and lack of syntax and precedence is great in many ways. But it can be a bad thing in terms of easy readability. It's a bit of a tradeoff.

Ruby code like arr.reverse.sort.join(',') just flows right along. And it's highly predictable. I know reverse is a method call, and it's defined in whatever class arr is an instance of (or a mixin of that class, or a meta-class I guess), and it returns something that has a sort method. And I can redefine any of those things.

Common Lisp non-working pseudo-code: (join (sort (reverse arr)) ","). Not as much fun to read, and oops, in this case SORT is a system-defined standard function, so if I want to write this I'll have to go with MY-SORT or SORT2 to avoid collisions (which you see a lot of in Lisp examples and literature).

Rambling #2: Way back in 2006, I decided to learn Common Lisp because I'd heard such nice things about it. I ended up abandoning that project very quickly (only to try again late in 2007, and stick with it this time). One reason (among many many others) that I gave up was the fact that it took me a good hour or two even to figure out how/where to download and install Common Lisp.

I remember going to a terminal and searching the Portage tree for packages matching the name "common lisp". Zero results. OK, how about just "lisp". One result I got back was clisp, which at that time, based on the name, I assumed was "just another name for the Common Lisp interpreter". However a search of the Gentoo forums revealed that a lot of people were using SBCL, or CMUCL, or many others. And those things were "Common Lisp" too? How confusing.

Little did I know that there is no THE Common Lisp. There are a bunch of implementations of the Common Lisp standard (or varying parts of the standard). To a newbie, this is somewhat confusing. Which one should you pick? What the heck is the difference? If it's a standard, does it even matter which you pick? So before you can even start writing code, you get to treat yourself to a history lesson to learn why there are all these different implementations, and then go on a fact-finding mission to try to find an implementation that's right for you.

It wouldn't matter, if the standard was complete enough that it covered all of what you'd want to do. But the standard is old, and arguably out-of-date. And it doesn't cover things like threads, sockets, POSIX utilities, etc. that you're really likely to use. And if something isn't in the standard, you can bet good money that every implementation does that thing in its own way, or not at all. A lot of sufficiently complex code in Common Lisp will demand that you go to great lengths to get it to run on different implementations, judging by some of the library code I've read.

Consider CL-FAD. A function DELETE-DIRECTORY-AND-FILES is used to (you guessed it) recursively delete a directory and its files. Here are the guts of that function:

(cond ((directory-pathname-p file)
      #+:lispworks (lw:delete-directory file)
      #+:cmu (multiple-value-bind (ok err-number)
                 (unix:unix-rmdir (namestring (truename file)))
               (unless ok
                 (error "Error number ~A when trying to delete ~A"
                        err-number file)))
      #+:scl (multiple-value-bind (ok errno)
                 (unix:unix-rmdir (ext:unix-namestring (truename file)))
               (unless ok
                 (error "~@<Error deleting ~S: ~A~@:>"
                        file (unix:get-unix-error-msg errno))))
      #+:sbcl (sb-posix:rmdir file)
      #+:clisp (ext:delete-dir file)
      #+:openmcl (ccl:delete-directory file)
      #+:cormanlisp (win32:delete-directory file)
      #+:ecl (si:rmdir file)
      #+(or :abcl :digitool) (delete-file file))
     (t (delete-file file)))

What's going on there? All those #+ and #- lines are kind of like the Common Lisp equivalent of C preprocessor macros. #+ and #- hide (or include) a line of code from the view of the compiler, depending on which implementation you're using. Ugh.

So in this case, you have what is a pretty simple function (1-5 lines of code, in each implementation), but you get to have the fun of writing it TEN TIMES, all woven together like spaghetti. There are many other such compatibility layers.

Kudos to people who go to the effort of writing portable code like this, but I don't see why it should be necessary. This strikes me as a senseless and wasteful duplication of effort. Not just on the part of the people writing code to run on all these implementations, but also on the part of the people working on the implementations themselves. Why not join forces and make one big mature official Common Lisp for everyone to use? What benefit is there to having so many implementations? Does the benefit outweigh the amount of work someone has to do to write portable code to share with the community? Maybe there's a good reason for it that I'm not aware of. But it seems to me like it defeats the purpose of having a standard in the first place.

On the other hand if I want to write some Perl, I do an emerge perl (substituting your package manager of choice) and then I have Perl. The same exact Perl (barring version differences, and slight differences between Perls on different OSes) that every other Perler in the world has. If I write a Perl script, any other person with Perl could reasonably expect to be able to run it. Same with PHP, Python, Ruby, and many other languages. (People are working on separate interpreters for Ruby, e.g. JRuby but from what I know, they're going to great lengths to make it run EXACTLY like the "official" Ruby, even down to copying bugs to maintain compatibility.)

I know now that there are things like Lisp in a Box that give you an easily-installable Common Lisp environment. Which is really nice for learning purposes. I wish I'd have found it back in 2006 (if it existed at that time). But it only delays the mess you're going to have to handle someday if you want to write portable code, or if you want to use anything other than CLISP or Allegro.

I have an urge to start porting some of my favorite Ruby gems to Common Lisp. But I am not going to install 10 CL implementations just so I can re-write my code 10 times to get it working on all the Lisps in the world. Very likely my code would be ruthlessly SBCL-specific.

(This is one reason I'm kind of hoping Arc takes off. Hopefully there would only be one Arc implementation.)

PAIP, review one of probably many

Wed, 06 Feb 2008 01:12:18 -0800

I got through the first four chapters of PAIP yesterday and today. It's good so far. I had reservations about spending $80 on a book, but it's 950 pages and almost all of it is good stuff, from what I can see at a glance. Little space is wasted on introduction material or tutorials for people who know nothing about programming. He dives right into the good stuff in chapter four.

Already I've picked up a few nice bits of info, interesting standard functions / macros like TRACE which is good for debugging. And SUBLIS which takes a list of (key . value) sublists and replaces all the keys with all the values, in some other list. Common Lisp is such a freaking huge language, in terms of its standard library of functions. Good thing or bad thing? It may be a bad thing if not for the fact that you load Lisp once and it persists forever.

Norvig makes an interesting point about the REPL. What do most programs need to do? Take input from the user, do something with it, and give output back to the user.

To get input, your program can read from command-line args, or prompt to STDIN, or take it out of text fields in a QT app, or read it from a config file. But it's generally always strings. So you get some strings, and then what? You have to figure out what the strings say, and either turn the strings into something usable or use the strings to figure out what action to perform. So you may have a dedicated library to parse command-line args and a dispatch table to map them to function calls. Or you may match the strings against a regex and depending what the strings look like, turn them into your language's representation of an integer or float using some "parseInt" or "parseFloat" functions. Or if your strings are XML, you may parse them into some big funky XML structure, then access bits of that structure to figure out what to do.

In Lisp on the other hand, you happen already to have a powerful reader/parser at your disposal: the REPL itself. It already knows how to read string representations of lists, numbers, pathnames, and many other things, and it can turn them into Lisp objects for you with no effort on your part. If you were using Ruby or Perl, short of ~~evil~~ eval, you wouldn't get anything close to that capability in your program.

What's more, the REPL knows how to read string representations of Lisp code (new functions and function calls etc.), and evaluate the code and end up with Lisp objects. It would be like if you wrote a C program which prompted you for input, and the input you gave could be a string representing a new C program that when run would produce a string that contained your input (or produce in-memory C structures your C program could use). So the main program would call GCC and compile and run your input to get results and use those results as its final input. Except that still wouldn't be nearly as powerful as what Lisp is doing.

So if you can figure out how to shape your input into a form that the Lisp reader can read, you're set. And it so happens sexps are a great representation for a great many things. Maybe this is part of why Lispers seem to worship the REPL and shun compiling their apps into command-line, standalone executables? Maybe this is part of what's meant by the oft-quoted-to-death Greenspun's 10th Rule?

Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp.

Much more PAIP to follow. It should be enjoyable. There's nothing like a good book.

PAIP; Lisp Lisp Lisp

Thu, 31 Jan 2008 02:31:48 -0800

I ordered PAIP today. Good computer books are so expensive. It hurts me to spend $80 on a book. But there are many worse ways to spend $80 than on something which contains so much knowledge, I guess. I got a $25 gift card for a book store for Christmas, so may as well put it to good use. I've read good reviews of PAIP, so I hope it lives up to them. I had only one AI course in college, and it was taught in C (ugh) by a grad student, and I learned little to nothing from it. It'll be nice to revisit the subject. (And learn more Common Lisp, of course.)

In other non-news, my Common Lisp-powered website is still up and running after a few days, no crashes or spontaneous hard drive reformats or anything, so that's good. Deploying it was interesting. Actually compared to the one Ruby on Rails site I deployed, Hunchentoot was far easier to deploy in many ways.

When you deploy Rails you have the problem that Rails is a beast of a framework, and Ruby is slow as a dog to begin with. So there are all kinds of silly nasty ugly hacks to get it to run with acceptable performance. Personally for my Rails site I have to run three Mongrel servers and use Apache's mod_proxy_balancer to send traffic to them. Some other people use fastcgi or such CGI-caching scripts, which I never could get working well. Just to run Rails apps at all, there are all kinds of Apache mod_rewrite hacks going on to get routes to work. Etc. etc. etc. It's a huge mess and I pray my Rails server never dies because I sure don't want to have to go through to pain of re-deploying it now that it's up and running.

With Hunchentoot on the other hand, you start Lisp once and it runs forever. No Apache to screw with means happy times are had by all. In fact my Hunchentoot-run website does exactly what Rails routes do, all via a single function that parses the request URI and dispatches based on it. With no .htaccess wizardry or cryptic Rails config files anywhere in sight.

Every request to Hunchentoot is apparently handled in a Lisp thread, so concurrency issues are generally all taken care of for you, no worries. In fact since you have Lisp running forever you can get away with crazy things like using Lisp objects as a sort of cache for data instead of hitting your database for every request, which I imagine probably works wonders for performance. I'll never know for sure, because my site will never ever be popular enough to put it to the test, but one can dream.

There were a few snags deploying my Hunchentoot server, none of which were the fault of Lisp, more the fault(?) of Linux. One was that I wanted to bind Hunchentoot to port 80. As you may know, only root has the right to bind a program to a "privileged" port like 80. I certainly did not want to have a Lisp image permanently running on my server as root, for security reasons. Thankfully Debian has a drop-in solution for this in the form of authbind, which lets normal users bind programs to privileged ports.

The second problem is that Common Lisp via SBCL is a sort of interactive program (see also, the REPL), and isn't really made to be run as a daemon, at least not the way I usually run it. There are many solutions to this however, and the solution I chose was detachtty, which does exactly what the name says. As the linked site describes, it's like a very stripped-down sort of GNU Screen.

Then there's the issue of how I can log into that running Lisp image from my home computer to run commands. I could easily SSH to the server, and use attachtty (which comes with detachtty) to attach to SBCL directly and type commands there. But SBCL's built-in REPL is a bit primitive. SLIME would be nicer. The solution here is to start Swank on the server. then use Slime running on my local machine to connect to the remote Swank.

This is, again, extremely simple. In a previous entry I linked to a movie that describes in great detail how this can be done, start to finish. You can see another more thorough example of how to do it here. Essentially you run SWANK:CREATE-SERVER in the remote SBCL; Swank binds only to localhost on the server, so to use it from home, you then have to make an SSH tunnel. Local Emacs can slime-connect to the remote Swank via the tunnel.

So that's it. authbind starts detachtty, which runs SBCL, which --loads a little 5-line Lisp script that imports some libraries and my photo blog source code, then starts Swank, and then starts my server. A little init script to make this happen by default every time the server reboots, and there's nothing more to be done. There's no real need to restart the server from now on, since I can log into the REPL and run commands and update the running program on-the-fly. But if I had to restart it or redeploy it entirely, it would be no problem.

This strikes me as an example of one of the great things about Linux. Many small tools that all do a single job very well. A tool to let non-root users bind port 80, a tool to let you run any program as a daemon, the glue to make those things work together with SBCL, and a great text editor to talk to SBCL after it's running. It all works beautifully, and there are no mysteries anywhere in the process start to finish.