This is a read-only archive!

Am I too dumb for Common Lisp?

Yesterday was a horrible day for me writing Common Lisp code. After writing today's blog post, I remembered reading this and realized I was echoing the same sentiment Larry Wall put so elegantly:

By policy, LISP has never really catered to mere mortals.

The thing I notice about Common Lisp is that people who use it tend to be geniuses by my standards. Read Planet Lisp for example. Or the Common Lisp mailing lists. There you find posts like (to paraphrase), "I just wrote a 12-line macro that allows recursive lazy evaluation of infinite series!" (EDIT: Apparently this post was a joke. Phew.)

As I mentioned before, according to someone who interviewed me for my last job, he often had difficulty even finding a programmer who knew what the word "recursion" means. How many programmers in the world know what lazy evaluation is? How many programmers in the world have a need to use infinite series? How many people in the world would be able to understand that macro, let alone write it, let alone use it for anything useful, ever? I know what recursion is, I know how it's useful, I know what lazy evaluation is, but I do not understand that macro nor can I think of any even remotely possible situation where I'd in my wildest dreams consider using it for anything.

It seems that this is the kind of programmer the Lisp universe caters to: The most power-tastic of power user imaginable. The level of assumed knowledge is enormous. Assumed knowledge about Common Lisp itself, and assumed knowledge about what I'd call advanced topics in computer science and software design. Read the documentation for Elephant as a good example of a tool designed for someone far beyond my capabilities. I tried and mightily, fantastically failed to get Elephant to work for my needs. Oh how I failed. It was a failure of truly epic proportions.

With Common Lisp it's not so much a learning curve as it is a sheer cliff face in the middle of the road. Documentation is sparse, and where it does exist, it's terse and extremely technical and bare-bones. Tools and libraries have gaps and quirks and it's just plain expected that you'll be able to easily fill the gaps and fix the quirks yourself. Libraries tend not to be the kinds of things you can download and use and get away with not knowing the details of how they work. Instead of an easy-to-use straightforward library, you often end up holding the remote control to a maniacal super-powerful flame-throwing sword-weilding death machine which can just as easily burn your house to the ground as it can solve your problem for you.

Meanwhile there are people like me who can barely get Common Lisp's object system to work right. I can just barely operate the text editor that I need to use to write Lisp code in the first place. Here I am still struggling to write a simple photo blog in Common Lisp. It's been well over a month since I started. I have something that almost works but I'm embarrassed to look at the code and certain parts of it I just can't get to work right. I feel like it could be so much better if I knew how to use this language properly. I could go ask for help, and sound like a moron. I'd feel like someone posting "What's a command line?" to a Perl mailing list. There's a certain amount of R-ingTFM that you have to accept as necessary. But never have I felt this frustrated or unable to figure out what the heck I'm doing.

It's like my goal is to build a box. Ruby or Perl or PHP give me rudimentary tools like "hammers" and "nails". Common Lisp gives me a drawer full of electrical components that an engineer could use to build a box-building robot capable of constructing boxes of arbitrary sizes from arbitrary materials. But all I want to do is build a freaking box. Sometimes I wonder what the heck I'm doing here.

January 20, 2008 @ 8:57 PM PST
Cateogory: Programming
Tags: Lisp


Ivar Refsdal
Quoth Ivar Refsdal on January 21, 2008 @ 4:12 AM PST

Well. I'll reply to all of your last three posts here :-)

First, thanks for the link. You might also want to check out this. Perhaps you'd also like paredit.el? I think that's really nice.

I've read a bit of SICP. Part 1 of the book. You should check out PAIP as well. I think that book puts it more simply: "Use the most natural notation available to solve a problem, then worry about writing an interpreter for that notation later." That's wise words in my opinion and it shifted my thinking about programming. Norvig even quoted me. It's funny how NONE (if I remember correctly) of the previous programming books I've read have even tried to teach you to think like that. I've only read the first two chapters (or three?), but it gave a lot of insight even so. (I was a bit thrown of by the recursion stuff, so I read "The Little Schemer", a very "user-friendly" book.)

"I can't seem to break my thinking or my design into clean, separated layers." Yes, I have the same problem. A lot. However, I do believe for web stuff Weblocks is a big step in the right direction. With widgets maintaining their own state. That's good design. (I'm not good at Weblocks at the moment though.)

Elephant. Well. My guess is... Since it seems (I don't know for sure) like a young piece of software, it's probably not going to have a lot of good docs. But like it states, if you're familiar with ...., then Elephant should look similar. So perhaps you should go with more mature tools, which probably also have better documentation, and then back to Elephant.

Perhaps making a photo blog was a too big shot? I've never made anything "big" in Lisp. Lisp, I guess, isn't exactly specifically designed for making web pages. So again, that means (in my opinion) that you have to struggle more than in some other environment which is more mature. Have you seen Lispcast? He talks a bit about Hunchentoot and all that -- I've only seen episode one.

As for that macro... No, I don't fully get it. (However, another tip, you can expand macros with C-c RET with the cursor on the right form.) I think that could make it a lot more clear as to what's going on.

More later... Got to go.

Ivar Refsdal
Quoth Ivar Refsdal on January 21, 2008 @ 7:31 AM PST

Well, I "debunked" the macro. It wasn't that bad (depending on how you see it). The naming of the variable _ was the most confusing I think..

Check out html (crappy) or text. I hope that makes some sense.

Let me know if you don't understand something / something is unclear.

Ivar Refsdal
Quoth Ivar Refsdal on January 21, 2008 @ 9:11 PM PST

What did you understand? The biggest macro or the lexical closure thing?

Here's a reduced version of the code.

If the new ---> function (not macro) looks funny to you, you should read The Little Schemer for sure. (It would have been looking funny to me before I read that book anyway..)

Nikodemus Siivola
Quoth Nikodemus Siivola on January 22, 2008 @ 2:24 AM PST

Uh. I hope that fact that so many people seem to have taken my entry seriously tells more about my lack of skills as a humorist, then it does about the state of programming blogs...

The arrow macros are intentionally hard to read: they parody a certain style of programming that values terseness over readability and long-term maintainability. Terseness and conciseness do not imply expressiveness.

The secondary parody is over the tendency of some people to use trivial operations over small sets to demonstrate benefits of a given approach for non-trivial operations over large sets.

Finally: "the utter clarity functional programming delivers". The just like code that depends on global side-effects is hard to understand, code that depends on intricate flow of values can be hard to understand. Saying "it's functional" is not an excuse of impenetrable code: sometimes aesthetics and clarity of expression are at odds, and one must find a balance that does not unduly stress future generations that must live with the code we write today.

sigh I suppose I better post this on my own blog as well. Disclaimers are for wussies...

Quoth Brian on January 22, 2008 @ 5:00 AM PST

Heh. That's awesome. Now I don't feel so bad for being terrified. There's a fine line between humor and horror, and I guess you found it.

I'm new to this world so the problem was probably more with me than with your joke, don't worry.

Ivar Refsdal
Quoth Ivar Refsdal on January 22, 2008 @ 8:58 PM PST

Ack. The joke's one me I guess! Hrm.

Nikodemus: I'm very happy you posted that though. HOPEFULLY it will make me more critical to the code I do read. Your piece was really the first Lisp code I've tried to understand that wasn't in some book along with a good explanation.

Ian Eslick
Quoth Ian Eslick on March 27, 2009 @ 1:28 AM PDT

I'm sorry to hear you found Elephant difficult to use! I'm now the primary developer/maintainer and spent a solid month and a half on an earlier release trying to build a complete, comprehensible manual for the system. However, the user is always right; if what I wrote was unclear then I'd love to talk offline about what could be done to improve the clarity of the manual and flatten out the necessary learning curve.

Some of the difficulty may be that it presumes an understanding of OODBs in general and CLOS slot semantics in particular. There is much that can be done to make Elephant more bullet proof and useable.

Anyway, please don't feel bad about where you are. There are numerous new users, including myself, who have similar experiences. I started to learn and use Common Lisp seriously about five years ago and still feel like there is much I don't understand yet. My first major web application took over 9 months to pull together (although weblocks was still pretty immature when I started using it). Even after all this time I'm still learning things about how to think better about programs written in Lisp.

This state of things is both bad and good. Bad in that it's hard to understand programs written by experts, but good because you are using a language that makes it possible to write such programs. I now find it very hard/frustrating to use other languages; it feels so confining. Even Python was annoying enough that I scrapped my Django site and started over in Lisp - still not sure if that was the right idea or not!

By the way, have you read 'Practical Common Lisp' by Siebel (PCL)? It's the best intro to practical uses of Common Lisp out there. SICP and others are wonderful for teaching you to think in lispy ways, but I believe PCL enables you to do practical work quickly and there are some concepts I never really got my head around with that he clarified for me (like restarts and some aspects of CLOS method combinations).

At the Intl. Lisp Conference this year we had extensive discussions about how to smooth out the learning curve and simplify setup, library use, etc for new users. Hopefully things will improve over the coming year.

We need better conceptual models, LispBox-like distributions, library packaging, seriously improved documentation and for heaven's sake get a graphics designer to help build a Lisp portal that isn't so embarrassingly ugly!

What do you think is needed? I think there are numerous people very interesting in how to make life easier for newcomers and it won't happen unless they speak up!

Quoth Brian on March 27, 2009 @ 3:56 AM PDT

@Ian: Thanks for the offer of help. This blog post was from last year; I have read PCL and SICP and On Lisp and PAIP and others in the meantime. I have gotten a better handle on Lisp at this point and I generally use Clojure for most things nowadays.

At the time I tried Elephant the documentation reads like a computer science textbook. It is thorough and complete but it exposes a lot of complex ideas. An example:

Nested aggregates are stored in one buffer. If you store an set of objects in a hash table you try to store a hash table, all of those objects will get stored in one large binary buffer with the hash keys. This is true for all other aggregates that can store type T (cons, array, standard object, etc).

Aside from the fact that it's in broken English, I don't know what this means. What's an aggregate, in this context? What's a buffer, in this context? What's "type T"? What's a "standard object"? I have vague ideas but the specifics elude me. And this is the tutorial! I don't have time to learn a new vocabulary for every library I use.

In terms of documentation, my suggestions:

  1. Dumb it down a few notches.
  2. More examples (clear, easy, practical examples) would be good. There are some good examples but they're spread over a chapter's worth of reading. Compile all of those together and make them the focus of the documentation. "Here's how you put something in, here's how you get it back out. Here are the limitations of the library. Here's an FAQ. Here are pitfalls new users of the library will encounter, and ways to work around them. If you want to know how or why this all works, go read this document I've hidden out of your way." Emphasis on practicality rather than theory.
  3. Expose complexity and theory for people who want it, but keep it out of the way otherwise.

Just remember that most users are in a hurry, they have their mind on their own projects, and they want something to work with minimal time investment. If I'm trying to make a website, I don't care about btrees. I want a black box that I stick data into and it works. There comes a point where the time investment to learn your library exceeds the time investment to use a different library or write one myself.

It could be that I and people like me are just not the target audience for your library, that's fine too. But Common Lisp has a lot going for it and I'd love to see it succeed and attract new blood.

(Elephant looks like an awesome library, kudos for all your hard work.)

Ian Eslick
Quoth Ian Eslick on March 27, 2009 @ 2:40 PM PDT

@Brian Thanks for the detailed explanation.

I would definitely like the library to be easy to use more generally. In part the manual was a response to common questions I've gotten. Some of the language and conceptual issues are a side effect of the fact that it was written in stolen minutes while I was the fulltime caretaker for infant twin girls. :)

I think it hadn't occurred to me (too close to the subject) to include a more gentle introduction to the lisp data model, notions of serialization, the lisp type system, etc. All of which are assumed in the snippet you presented. This, plus additional assumptions about CS background, definitely tilt it towards a certain audience. Perhaps a true 'first tutorial' that walks through a practical application would be a nice, separate complement to the manual...

I think this phenomenon you describe is a real issue with all of common lisp. Those who know it well enough to develop rich libraries have often forgotten what is hard or non-obvious. Edi Wiesz (hunchentoot, cl-ppcre, etc) seems to be able to span those two worlds nicely.

Part of the challenge with Elephant, which isn't a challenge for Clojure due to its design, is that there is no 100% clean way to integrate persistence with the assumptions underlying common lisp. The result is that if you try to do something that seems obvious you get bugs that are hard to reason about. That paragraph was trying to capture the essence of the problem, but definitely assumes too much background knowledge.

I am trying to add some data structures that are simple interfaces such as ptable and pset in addition to defpclass so people use these instead of hashes and lists.

If you ever run across a good data storage library or persistence solution with a tutorial you think is good, please forward it along. I'm easy to find online.

Cheers, Ian