briancarper.net (λ) (Tag: PCL)

Wish list

Tue, 17 Jun 2008 23:47:43 -0700

What's the Common Lisp version of Perlmonks or Ruby-forum? I have yet to find it.

comp.lang.lisp is largely crap. 50% of the traffic on that list is spam about shoes and fake watches. The other half is equally split between:

People debating tiny, silly semantic points of the Common Lisp Hyperspec.
People stuck in the 70's or 80's, talking about the good old days, ruminating about Lisp history.
Flame wars.
New people asking for help. Some get good honest advice and helpful answers, many are flamed and ridiculed into next week if they even hint that they dislike the parentheses.

The Common Lisp community (if you can call it that) is a bunch of really smart guys, but they all live isolated in hermit shacks up in the mountains and they spend their time doing magic tricks with Lisp that few people ever see, and if you wander too close they throw rocks at you.

What's the Common Lisp equivalent of perldoc or rdoc? We have the Hyperspec. It's an impressive document, but it's a bunch of painful HTML that looks like it was created in the early 90's, probably because it was. It reads like a dusty, dry, technical document probably because it is. What it's not, is friendly or easily readable.

Perl has CPAN, Ruby has rubygems, what does Lisp have? Either a hand-rolled system definition script, or if you're lucky an ASDF install file. ASDF is the semi-standard Lisp way of installing libraries, except that it doesn't quite work in Windows, it doesn't check dependencies or handle different versions of a package very well, and it doesn't work the same on all Lisp implementations. Many people in the so-called community think it's not very good.

The fellow running Lispcast makes another good point. Where can you download Lisp? It's not obvious.

You could say "OK Brian, good idea, now get to work!" The problem is that even if I had the time or willpower, I'm not the smartest guy in the world. I honestly don't think I could design and run and maintain a CPAN. And even if I did, would anyone use it? But I do know that there ARE plenty of smart, enthusiastic people using Lisp. Yet high-quality friendly code is largely not being produced.

Peter Christensen wrote about "langauge snobs" and the importance of community. One point made is that some really ugly, horrific languages have been extremely successful simply because they've been accessible and fun. An example given is the scripting language in Second Life, which has over 2.5 billion lines of code written in by tens of thousands of amateurs and has accurately modeled a realistic 3D environment with thousands of users at any given time. All in an ugly language some guy invented AND implemented in one week. The developers admit that the language is total crap, but it doesn't matter. 1) It has very good and accessible documentation, 2) it has a very newbie-friendly community, and 3) and it's easy to pick up, throw together some code and get immediate results. Three things Common Lisp lacks.

This is something I've said myself many times: an active, supportive, enthusiastic community is essential for the health of any programming language. Common Lisp simply doesn't have one and it's a shame.

I still secretly hope that Clojure or NewLisp or Arc turn out to be a huge success. They are the kinds of things Lisp needs today.

Perl 6

Mon, 07 Apr 2008 23:14:25 -0700

I found this Perl6 imaginary timeline hilarious. Sadly we're in 2008 and I don't think that blue line is quite as high as it was projected to be. Though it appears that development is still slowly chugging along. I still check up on Perl 6 once every six months or so. You can actually download perl6 (called rakudo?) and run it nowadays.

It's been a long time since I've used Perl for anything. If Perl 6 ever sees the light of day in a big way, I'm sure I will hunt down the Perl 6 bandwagon and chain myself to the back of it. Perl was my first religion.

I don't want to consider myself fickle, rather curious and eager to try and learn new things. But honestly, fickle may be closer to the truth. Already my Lisp enthusiasm is starting to wear off, largely from the impracticality of writing anything in Lisp (in spite of good book titles to the contrary). I've written a bunch of things in the past few months, but I never even considered Common Lisp for any of them. Mostly because I was being paid to write them, and I don't think people want to pay me to screw around with something that's going to take me 10x as long as using a language I know already.

I believe the main way I came to know and love Perl and later Ruby was through all the little 5-minute throwaway scripts I wrote to get my job(s) done. Those little scripts led to bigger scripts, which led to even bigger scripts. Emotionally it's satisfying to actually solve a problem in a language, even a small problem. Not so easy to do in Lisp; it doesn't seem to lend itself well to scripting. Lisp is nice if you've come up with the perfect abstraction for your program and want to implement it exactly like you're thinking of it. Most of the time, by the time I think up the perfect abstraction, I'd already be done if I'd written it in straightforward Ruby to begin with.

And, to rant briefly, in Ruby I wouldn't have to write my own function to fetch all the keys of a hash, or print a formatted date string, or any of the other countless little holes in Lisp that are nicely filled in Ruby by all the available libraries. I've said it before and I'll say it again: a large active community is one of the biggest necessities for a healthy programming langauge. It really doesn't seem like many people in the Lisp community give a crap about getting lots of new people to use Lisp. Or if so, I never hear much about it, compared with other communities. There's no reason a language can't be really great and powerful, and still accessible to the masses. That's one of Ruby's strengths. You don't have to read a few hundred pages of hyperspec and learn 50 years of history and spend a week and half setting up an arcane SLIME/Emacs environment to start writing Ruby.

But yeah, Perl 6. I want it to succeed, for entirely emotional reasons. I want a new toy.

PAIP review

Mon, 10 Mar 2008 17:21:35 -0700

It's been a while since I acquired PAIP. I can't say I've read all of it over the past couple months, but I've read most of it.

The initial chapters that give an intro to Common Lisp seem largely useless to me. There are better books to introduce Common Lisp (especially now that we have PCL). Norvig admits as much himself, and thankfully doesn't devote much of the book to intro material.

Once you get past those, this book is densely packed with information. Just tracing through and understanding the source code would probably take me a couple of months, let alone grasping all the concepts the source code is trying to exemplify. It's a wonder to me that one person can produce this much material. But the downside is that it's a bit difficult to read at times; it's kind of dry and reads like a text book. In spite of that, I was glued to this book for quite some time; the information inside is engaging enough to make up for the lack of presentation.

Some of the examples of code in the book really are quite startling. For example Norvig writes a pattern-matcher that implements some subset of Perlish regular expressions, and it only takes a couple pages of code. Later in another couple pages of code, he implements a variant of Prolog. At one point he writes some pattern-matchers that can parse a surprisingly large subset of the English language. But a lot of other examples of code weren't quite so interesting to me personally; a lot of the material is dedicated to tree and graph search algorithms, which brought to mind long boring lectures from my college days. (As a side note, it's very sad how much of the history of AI can be reduced to "clever graph search algorithms". I don't know much about AI today but I hope it's advanced a bit past that.)

This book is from 1992 and some of it does feel a bit dated. This book was apparently written just after Common Lisp was standardized. Object oriented programming was apparently relatively new back then, and Norvig glosses over CLOS. Most of the code is written in functional style. However Common Lisp today still doesn't force OO on you by any means, and a lot of Lispers today apparently still don't care much for OO, so the code is more relevant than you'd imagine.

And as you can't help realizing after reading an old CS book or two, CS as a science hasn't seen all that many breakthroughs in the short couple of decades it's been around. It's really quite sad in a way that a book written over 15 years ago can still be so relevant; we have all these huge advances in computer hardware, yet it still feels like we're in the Stone Age of computer programming.

Not being an Artifical Intelligence person myself, I can't judge how well this book serves as a guide to AI. But Norvig's stated goal was in large part to show the history of AI, and he does that very well. (This was history back in 1992 when the book was written, so it's even older now, but still interesting.) He walks through implementing a lot of famous AI programs from the past, and explains their limitations and how they can be improved. Want to write ELISA or other chat-bots? He does that. Want to write an Othello-playing program? He does that too.

One of the best things about this book is that Norvig's code is just about the Lispiest code I've seen. He does many things that take full advantage of all the nooks and crannies of Common Lisp and wouldn't make much sense in other languages. This is a good book to help you think in Lisp.

So I recommend this book. Though I'm unsure I'll ever manage to absorb most of the information in it.

Arc

Sat, 09 Feb 2008 03:30:02 -0800

There's apparently some excitement over Arc which is a new Lisp being designed from scratch by Paul Graham.

There are a lot of varied opinions in the Lisp world about this, I guess. Many people are saying Arc is no different than a hundred other failed attempts to make a new Lisp over the past 5 decades. On comp.lang.lisp Arc has been thoroughly derided (though it seems like almost everything is thoroughly derided on that list).

On the other hand, here we have something new in the Lisp world (everyone likes something new), and it has a big name behind it (everyone likes a smart and charismatic leader). I think those two things alone give it a shot. If there's anything PHP or Java or any number of other horrendous languages can teach us, it's that whether it deserves it on technical grounds or not, a big enough community can make anything a success. For some reason I can't help but think of the mobs of people begging drobbins to fork Gentoo. Hmm. Maybe if I forked my blog I'd double the amount of people reading it. (Oh right. Charismatic leader. Darnit. :( )

In all seriousness, I don't know enough to judge Arc or any of the other Lisp spinoffs (e.g. Clojure looks neat). I barely know Common Lisp to begin with. I've never tried Arc and don't necessarily plan to any time soon, due to lack of time if nothing else. But I do know that there are plenty of things about CL that I dislike, and can't help but think mostly everyone else also dislikes. Yet these things stay the same, for decades and decades.

(For example pathnames. If Common Lisp pathnames weren't so broken, it wouldn't have taken a whole chapter of PCL to write a wrapper that makes them usable. Yeah there are good historical reasons for the way pathnames are done in CL. But I don't care. I live in the present. If I'm transported back to 1980, I'll want to have 1980's style pathname libraries. And maybe in 1980, a function called ENOUGH-NAMESTRING meant something to someone? But what I need today is 2008 style pathnames. There are other things in CL that are just as crusty and awkward, and "historical reasons" are always touted as the excuse. These kinds of things kill me. Even if I fix them in my own code, chances are I'm going to have to read someone else's code where these things aren't fixed, and so it crusts up the whole community. Arc has a chance to fix all those things. Whether it'll take advantage or not is yet to be seen.)

I really do think that a large part of what determines the success or failure of a language is the amount of people using it. There's no doubt that if more people were using Common Lisp, all Common Lisp users would benefit. Even if most people are dumb newbies, dumb newbies eventually turn into worthwhile contributors if they stick around long enough. And even dumb newbies have a good idea now and then. It only takes one good idea. The community can take that one good idea, ignore all the other stuff, and everyone benefits. And even if a person has no good breakthrough ideas, there's a lot of grunt-work that needs to be done to keep a language healthy. Little libraries that aren't hard to write but still make everyone's life easier once they're written. Documentation. Community support, helping others with troubleshooting, introducing new people into the fold. These things are important. I'd love to see a Lisp with a community like Ruby'.s

Paul Graham's site says:

The Arc community is very newbie-friendly, because all the users are newbies to some extent.

And that is very refreshing, and maybe it's enough to get a critical mass going. But who knows.

Whether Arc dies a horrible death or is the next big thing, it's kind of exciting to be present at the birth of a language. I tend not to find out about things until they're already big enough to get everyone's attention. It's fun to watch Arc being born.

Does syntax matter?

Tue, 18 Dec 2007 23:19:31 -0800

Peter Seibel of Practical Common Lisp fame gave a talk to some group or other about syntax and whether it matters. His answer is (of course) that it does and it doesn't. You can watch videos of the talk if you'd like. I found it very interesting.

Obviously I'm on a Lisp kick lately. I tend to get really excited about new languages I learn. My first infatuation was Perl. Then Ruby, which is a better Perl than Perl in many ways. Then Lisp, which may be a better Ruby than Ruby in certain ways that matter.

I heard said once that a sign of intelligence is being able to entertain an idea without accepting it. I think I like to take it one step further. I think a sign of intelligence is being willing to try on an idea like a pair of shoes, and walk around in it for a while to see how it feels.

Clearly some ideas (like some shoes) are so filthy and nasty that I won't go near them. But other ideas seem bad to me only because of my habits and prejudices. It's a fundamental fact of the universe that I'm wrong in some of my beliefs (probably many of them) at the moment. Well not really a fundamental fact, more like a statistical likelihood so great that it may as well be a fact. The problem is that everything I believe seems right to me (otherwise I wouldn't believe it).

In spite of this problem, the only way to survive in the world is for me to keep acting as though what I believe is right, until it's shown to be otherwise. Luckily this attitude acts as a good test of whether I'm right or not, as long as I'm always open to noticing the things that point out that I'm wrong.

But on top of that, I like to throw some random mutation into the mix. I like to take things I strongly believe and test them by trying something very different. I like to put my tried-and-true Vim aside and try Emacs for a while. I like to turn away from my comfortable Ruby and give Lisp a go. This is inevitably painful. But it's a nice kind of painful.

Sometimes it works out and I change my mind, which is great. Sometimes I don't change my mind, which is also great. I was very heavy into C++ for a while; now not so much. But at least now I have good reasons not to like it. Same with Java. At this point I'm not sure I'm ever going back there, but who knows. I may. Any theory worth testing is worth re-testing.

But Lisp is a nice pair of shoes to try on. It's sufficiently different from ALGOLish languages that it tests many beliefs many people probably take for granted. Like that syntax matters.

Even more Lisp

Fri, 30 Nov 2007 00:58:05 -0800

Having read and re-read Practical Common Lisp probably four or five times at this point if you add it all up, and having skimmed Graham's On Lisp enough to realize I need to come back to it later, I decided to move on to Structure and Interpreation of Computer Programs, which is a book available free online.

Side note: It's so wonderfully delicious that so many amazing books are available free online, isn't it? I'm almost in disbelief sometimes at how amazing that is. Such knowledge and secrets, not only easily accessible, but easily accessible for free. As I did with PCL, I'll probably end up buying one or both of the other books I mentioned above just because I like to have a hard copy of things to read in bed, but in the meantime the online version is great.

If that wasn't enough, you can also download (for free) videos of lectures covering this material, given by two guys who authored SICP. I'm only up to lecture 2b, but it's proving very informative to say the least.

SICP is a book apparently used in CS classes at MIT, and it deals with abstractions, the different kinds, and particularly functions and procedures as first-class objects and functional programming. The language used throughout is Scheme, which is different enough from Common Lisp to trip me up already. Although not being all that comfortable with CL at this point maybe also be a blessing, because if it turns out I like Scheme more I can make the switch without having to break any ingrained habits.

I've been watching the lectures and translating the Scheme they give as examples into CL as I watch, as a means of trying to familiarize myself with both languages. A while back I read about the Lisp₁ vs. Lisp₂ distinction but didn't pay it much mind. Then I read about it again on a Lisp mailing list last week, but didn't pay it any mind. Well today in the course of my Scheme -> CL translations, in a very roundabout manner, this distinction bit me and caused me no end of trouble. Which is funny because I still never saw it coming, even having read about it in two different places and two different context. But now at least I can say understand the distinction in a very tangible way. Just goes to show the only real way to learn a language is by writing code in it.

(You can read much more about Lisp₁ vs. Lisp₂ here. For those who don't want to read, and for my own future reference: Lisp₁ vs. Lisp₂ refers to whether there's a single namespace for functions and variables, or two separate namespaces for those things. Scheme has one namespace, CL standardized on two. This results in CL being much more verbose when calling lambdas and passing them around between functions, but also arguably results in CL being more suited to using macros without having to worry about name collisions. That's my probably limited understanding at this point. )

On Lisp (review)

Sat, 24 Nov 2007 22:09:32 -0800

Paul Graham's On Lisp is available in PDF form (and Postscript) for free download, so I thought I'd give it a read. It's an older book (early 90's) and you can tell. It has a lot of good knowledge if you can make it past some of the things that obscure the more relevant parts.

A lot of the code given in the book favors efficiency over simplicity and readability, which is probably a product of the book being 15 years old and written by someone who's clearly been programming for a lot longer than that. It's something I notice a lot in older books. Of course there was a good reason for it: you had to worry about efficiency if you wanted your program to finish before next year.

Given today's computers and the kind of programming I tend to do personally, I don't care about efficiency beyond making my code "fast enough". Very rarely does this ever mean manipulating my code to make it tail-recursive so my compiler can optimize away some of my function calls, which Graham devotes a lot of effort to in the book. Someone once said something about "premature optimization" which I'm sure everyone is familiar with and may apply here.

(But I have a ton of respect for people who programmed computers back when every clock cycle and every byte of RAM was precious. I'm not sure how I could've ever managed in that kind of world. Sometimes it seems like you had to be a genius to get anything done back then. Maybe necessity was the mother of invention though.)

Common Lisp was apparently just being standardized or being heavily revised when this book was written, judging by much of what Graham writes. A lot of stuff I read in Practical Common Lisp that Seibel glossed over as being "something the old-school Lispers used to do" is in full-effect in Graham's book, and a lot of what Graham mentions as "new" is probably very solidly standard nowadays. Graham uses lambdas for EVERYTHING: data storage, algorithm optimization, sometimes even just to introduce a new lexical scope. And our primitive friends the CAR and CDR and their many cousins make plenty guest appearances in this book, to the detriment of my poor brain.

Graham also glosses over a lot of the code in his book and leaves it to speak for itself, making this probably a book intended for far more advanced Lispers than I. For an intro to Lisp I still highly recommend Seibel's book; his code and prose are so transparent and easy to understand that it's a joy to read. Graham's book reads more like a math textbook.

If you can get past all of those things, there's clearly a lot to learn from the book. It's really amazing some of the things you can pull off in Lisp, and Graham gives a great demonstration. Many of the design principles that Graham lists as goals for his new language Arc are explained really well in this book, so it's worth a read if you're interested in that. It's funny that a book about Lisp often blurs into a book about programming language design, probably because as Graham says, when you write Lisp you often build the language up to your program at the same time you build your program down to the language.

Lisp vs. Smarty

Fri, 16 Nov 2007 21:28:40 -0800

I started writing a post called "Lisp vs. Smarty" two nights ago but lost my train of thought before I could finish. Lately I see lots of blog entries about PHP and in particular this one by Sean Potter about Smarty made me want to finish typing this.

In Practical Common Lisp, in two chapters (30 and 31) and a few hundred lines of code, Peter Seibel produces a library he calls FOO that can compile (or interpret) s-expressions into HTML. FOO is essentially an interpreter/compiler for a little mini HTML-sexp language, which can compile the HTML-sexp language into Lisp code (that can be run later as a Lisp script) or can use the HTML-sexp code to produce HTML output directly. The book is there on his site for free in its entirety, feel free to read those chapters.

This FOO library is immensely powerful, and the things that make FOO more powerful than Smarty are the things that make Lisp more powerful than PHP in general. Personally I found it a great example to help me understand some of the differences between Lisp and non-Lisp languages.

Both Smarty and FOO are libraries that have at least two purposes: To produce dynamic HTML based on parameterized input (i.e. feed it values that are inserted into the final HTML) and to produce HTML without having to write a thousand static files manually by using control and looping constructs and by allowing you to have re-usable modular templates.

So, you can embed Lisp code into FOO to act as simple text filters. You can refer to Lisp variables in FOO scripts. You can use control structures like if/then/else, or looping constructs, or what have you. That's nothing different than Smarty can do though. Smarty is good at those things.

But since this is Lisp, you can easily write macros that simplify, abstract, or otherwise help you write the FOO HTML-sexps. In other words, you can write Lisp code that largely writes your HTML-sexps for you.

With Smarty, you're much more limited in what kinds of abstractions are available to you. You're probably going to write the Smarty templates mostly by hand. You're usually not going to do the equivalent of Lisp and write PHP code that writes your Smarty templates. If you do, it's going to be in certain limited ways Otherwise it would probably be more verbose and more of a pain than writing the Smarty templates yourself. After all, the point of Smarty is to help you NOT to have to write PHP. Trying to programatically produce Smarty templates using PHP is a mess largely because there's no easy way for PHP to understand Smarty templates other than as strings. Using PHP to produce Smarty and using PHP to produce HTML are equally nasty problems to try to solve.

So instead you often use Smarty to "write" Smarty. But not really. You write template files, split them up, and then you import various bits and pieces of some template files into the insides of other template files. You can throw Smarty looping and control constructs around your Smarty include statements, so this all works pretty well. But the Smarty template language is obviously rather restricted in what tools it gives you for this kind of meta-templating. And templates are about the finest granularity you're going to get.

And note, the Smarty templates surely aren't going to write the HTML tags for you. They aren't going to auto-close your open HTML tags, letting you choose how to do it based on your preference for HTML or XHTML. Smarty templates aren't going to make sure you put quotes around your attribute lists. They aren't going to (optionally) properly indent your HTML code. FOO does all of these things and more. FOO can also be used to make "templates" much like Smarty, but it gives you far greater control than that.

So let's talk extensibility. Smarty gives you hooks you can use to write plugins that do various things. The Smarty plugin documentation is around 15 pages of documentation largely telling you what you CAN'T do. Smarty plugin files for example must be named certain things, and put in certain folders. Your tags must look and act in certain ways. When your functions are called, they receive parameters X, Y, and Z, which Smarty provides, and then you must return exactly Q, which Smarty expects from you.

There are a limited number of very specific KINDS of functions you can write. You can write a function to filter some text to change it into other text. You can write a function that gets a big string full of raw un-parsed template text and returns a big string back after you do something to it. You can write a function that gets a big string full of already-parsed template text and return back a big string of text after you do something to it. Etc. Key word here: BIG STRINGS.

But let's say for example that the {if}{else}{/if} construct was missing from Smarty, and you wanted to write it. Could you do it? Smarty gives you no hook to write something like that. So you could try to use what Smarty dose give you. You could make up three tags ( {if}, {else}, and {/if} ). And then you could maybe write some functions that take big strings of the template text that's in between those tags, and parse it yourself. Maybe you could write (in PHP) a sort of state machine, global enough to last between calls to the custom functions you wrote. Would it handle an arbitrary number of {else}'s per {if}? Would it work recursively? That might take some more work.

If you're resorting to such a thing, you're almost re-inventing Smarty yourself. So hey, you could just go and edit the Smarty compiler PHP code directly to add this functionality.

I just looked at the Smarty_Compiler.class.php file that comes with Smarty. Oh the horror. Inside are line after line of regular expressions designed to take tokenize Smarty template text. Once it's been tokenized, there's an enormous switch/case statement to deal with the allowed tags. Some tags are translated directly into PHP code, which is output as a bunch of our friend, the plain old string, later to be presumably eval'ed. Some tags rely on other tags, so there's a stack to which tags are pushed and popped as the file is parsed. It really is a full-fledged compiler: A "Smarty template language"-to-PHP compiler, written in nice slow PHP.

I tried to track down the code responsible for implementing the {if}{else}{/if} construct in this Smarty compiler file. The 40+ lines of swtich/case that handle the {if}-related tags make a couple calls to, among other things, a function called _compile_if_tag. This one function is 163 lines long. Line count isn't a good example of complexity, of course. So here are just a few of those 163 lines:

/* Tokenize args for 'if' tag. */
preg_match_all('~(?>
    ' . $this->_obj_call_regexp . '(?:' . $this->_mod_regexp . '*)? | # valid object call
    ' . $this->_var_regexp . '(?:' . $this->_mod_regexp . '*)?    | # var or quoted string
    \-?0[xX][0-9a-fA-F]+|\-?\d+(?:\.\d+)?|\.\d+|!==|===|==|!=|<>|<<|>>|<=|>=|\&\&|\|\||\(|\)|,|\!|\^|=|\&|\~|<|>|\||\%|\+|\-|\/|\*|\@    | # valid non-word token
    \b\w+\b                                                        | # valid word token
    \S+                                                           # anything else
)~x', $tag_args, $match);

...

if(preg_match('~^' . $this->_func_regexp . '$~', $token) ) {
        // function call
        if($this->security &&
           !in_array($token, $this->security_settings['IF_FUNCS'])) {
            $this->_syntax_error("(secure mode) '$token' not allowed in if statement", E_USER_ERROR, __FILE__, __LINE__);
        }
} elseif(preg_match('~^' . $this->_var_regexp . '$~', $token) && (strpos('+-*/^%&|', substr($token, -1)) === false) && isset($tokens[$i+1]) && $tokens[$i+1] == '(') {
    // variable function call
    $this->_syntax_error("variable function call '$token' not allowed in if statement", E_USER_ERROR, __FILE__, __LINE__);                      
} elseif(preg_match('~^' . $this->_obj_call_regexp . '|' . $this->_var_regexp . '(?:' . $this->_mod_regexp . '*)$~', $token)) {
    // object or variable
    $token = $this->_parse_var_props($token);
} elseif(is_numeric($token)) {
    // number, skip it
} else {
    $this->_syntax_error("unidentified token '$token'", E_USER_ERROR, __FILE__, __LINE__);
}

And so on, and so forth.

Suffice it to say, in the Smarty compiler, the code that implements {if}{else}{/if} is at least a couple hundred lines of code that looks like the above. It would take me quite a bit of time and effort even to figure out what that code does, let alone write it myself and debug it.

Now, that code is part of the Smarty compiler itself, written by the people who wrote Smarty, i.e. the people in the world who know Smarty best. If I was trying to write my OWN code that gave me a {if}{else}{/if} structure in Smarty, my code would by necessity be FAR MORE complex than that. If written as a plugin, my code wouldn't have direct access to the guts of the Smarty compiler. Instead my code would have the very limited access that Smarty provides to plugins and custom functions. Even if I tried to hack the compiler msyelf, my code would be written by me, who certainly doesn't have the expert knowledge necessary or understand the inner workings of Smarty well enough to get this code working without a good deal of effort.

So let's contrast this with FOO. If an if-then-else sort of construct was missing from FOO, could you write it yourself? It turns out you could. Peter Seibel calls this a "trivial example" and mentions it in passing. It turns out you could add this construct to the FOO mini HTML-sexp language using two lines of Lisp.

Using TWO LINES.

TWO LINES!

(define-html-macro :if (test then else)
  `(if ,test (html ,then) (html ,else)))

There's around 50 lines of Lisp that puts everything in place to you allow you to write "plugins" like the above or far more complex than the above. 50 lines of Lisp that let you pretty much arbitrarily extend the HTML-sexp mini language itself, to add new tags including entirely new control constructs, that the HTML-sexp compiler understands.

Lisp already has a perfectly fine if-then-else construct after all, as does PHP. Somewhere in the PHP compiler itself, there's logic (maybe written in C? Maybe assembler?) that implements if-then-else. Why can't we use that compiler code? Why did we have to write our own Smarty compiler to do it all for us?

Rather than write a new templating language, and then write a tokenizer and lexer and parser for that language, and then slurp up a bunch of strings and translate it all back to a plain old PHP if-then-else, which is then transformed by the PHP compiler itself into native code that implements if-then-else... in FOO, we use Lisp. Lisp can already read s-expressions and parse them properly, and give us access to the result as native Lisp objects. Lisp has if-then-else, which we can use directly. That's all we need. Aside from the wonderful simplicity of it all, imagine the difference in performance.

The key is that the HTML-sexp language is s-expressions. Lisp itself is s-expressions. The compiler that translates the FOO code into Lisp is s-expressions. The code that will run the FOO compiler is s-expressions. The macro code that helps you abstract all of the above is s-expressions. The macros that help you write THOSE macros are s-expressions. Everything is s-expressions. And Lisp is VERY GOOD at reading, writing, and arbitrarily manipulating s-expressions. Which means that Lisp is VERY GOOD at reading, writing, and arbitrarily manipulating mini-languages like FOO, mini-language-interpreters and compilers, Lisp code, Lisp code that runs Lisp code, and Lisp code that writes Lisp code.

One guy wrote FOO, and then very nicely explained start-to-finish every single line of code that makes it work, in two short chapters of a beginners' Lisp book. Compare this with thousands of lines of PHP written by a team of programmers over the course of apparently six years, to produce a library that at the end of the day does far less than FOO can do.

Lisp, part 2

Mon, 14 Aug 2006 00:22:47 -0700

Everyone I know who does Lisp says "Learn Lisp and it will change how you look at all other programming languages!" I don't know if that's true or not, but I think I'm starting to learn a bit.

Chapter 8 of Practical Common Lisp discusses macros. As has been pounded into my brain by the author, macros in Lisp aren't like macros in other languages. Nosirree. Nope. Not like them. I definitely won't forget that Lisp macros are different than macros in other languages. And you won't forget that either. If you read this book.

Macros in Lisp let you write code that writes code. Quoth one of my professors in college, "Code that writes code is some of the neatest code there is!". I'll agree with that. Writing a whole database or a whole test unit framework in Lisp in 50-odd lines is pretty impressive. Lisp doesn't even seem like that concise a language, compared to Ruby for example. But it's powerful. I'm reminded of Haskell, where everything is recursion. When you start layering recursion on top of recursion, things tend to branch out and get "big" very quickly; that "bigness" equals power and efficiency, if used properly. That seems to be one of Lisp's strengths; it lets you do that pretty easily.

Well, a lot of languages let you write "code that writes code". Perl has eval() which can take a custom-built string of code and execute it somewhat dynamically. The thing is though, eval sucks. If it dies, it dies at runtime, not compiletime. And you can only put things together via string manipulation. Lisp's code-building macros are more "built into the language", for lack of a better term.

Ruby lets you do some neat things too though. I found this article to be a pretty good description. Ruby lets you throw blocks of code around all over the place, and the more I learn of Lisp, the more I'm reminded of Ruby, or the more I think of ways I could be doing things Lisply in Ruby. Ruby has the added advantage of letting you do all kinds of horribly powerful things at runtime. Chopping up classes and methods and rebuilding them as you see fit, etc. The book hints that this is something Lisp lets you do too, since so much of Lisp is actually written IN Lisp.

To draw a nice contrast, today I was forced to do a lot of PHP programming. Oh sweet Jeebus. It burns. I hate PHP with a passion. Lisp is wonderfully consistent. The parentheses do get on my nerves, but there's never any doubt what your syntax should look like at a given time. It's always parentheses. PHP on the other hand is both very verbose, and very inconsistent, which is the worst possible combination. It's never predictable or discoverable. It's impossible to program PHP without PHP.net open in another window to constantly look crap up.

Any time I'm forced to use a traditional for() loop now, I start freaking out. That is something Ruby and Lisp both thankfully avoid. Ruby's iterators are my best friend. Looping through an array copying things to temp variables seems so barbaric now that I've seen alternatives. It's just one of the little things that gets to you; in general in PHP I feel like I'm trying to build a scaffold with pieces that can't be cut to the right size. Ruby gives me pieces that bend easily, and Lisp gives me really tiny pieces that fit together in all kinds of interesting ways.