How do you view and edit data in a mysql DB? Lots of ways.

There's always commandline mysql. This is how I do it around 50% of the time. But it could be better. ASCII-art table dumps are not the easiest things to read, and readline-based history and editing only gets you so far.

phpMyAdmin is what I grew up with. It's pretty good but it's way too heavyweight. It also has all kinds of funky Javascript, to the point where I actually use Greasemonkey to remove some of it. Auto-selecting query text when you try to edit it in particular drives me crazy. Auto-selecting ANYTHING on Linux, where selecting text usually equals clobbering the X clipboard, is a really really bad idea. Last I tried it's a one line of Javascript in Greasemonkey to fix this by the way:

document.getElementById('sqlquery').removeAttribute('onfocus');

But phpMyAdmin may have changed since last I used it, it's been a while.

There's the native mysql GUI, called mysql-gui-tools in Gentoo. It's a standalone app that's pretty good, but the Linux version is gimped up compared to the Windows version for some strange reason. In any case it seems to be discontinued or something. There's some new MySQL Workbench thing coming, which I'll probably try once it hits Gentoo, but it looks like overkill for my simple needs.

SQuirreL SQL is another cross-platform GUI app. It connects to lots of different kinds of DBs (pretty much any DB that Java can talk to), so at work where I have get my mysql server to talk to someone else's MS SQLServer (ugh) I use this. But it's very heavyweight and not the most enjoyable interface.

Last week I chanced upon something pretty good. Adminer is a single PHP file you throw on a server and there you go. It gives you something vaguely similar to phpMyAdmin but far more lightweight. There's very little Javascript messing with my query-writing and the styling is minimal and easy to read. I don't know how secure it is, so I don't plan to put it on any public servers, but on my test server doing web development it's good. I love bouncing back and forth between a the database in one browser tab and my website in another tab. This is what I use now.

Goodbye Wordpress

As I mentioned many times, I've been working on replacing Wordpress for my blogging needs. Wordpress has been pretty good for the past three years, but it's time to move on, for a bunch of reasons.

Primarily, the way Wordpress automatically mangles my text is annoying. For example, it turns newlines into paragraphs inconsistently (especially when it comes to pre/code blocks). This blog is mostly about programming, which means being able to post code without having my quotes turned into "smart" quotes and my --flags turned into long-dashes is kind of important. HTML is sometimes automatically escaped, and sometimes not. I can't count how many comments I've gotten where someone posted some code, then posted again to inform me that Wordpress ate the code for dinner. There are plugins to fix some of this, which break every time Wordpress releases a new version, and have never really worked that well for me.

Writing a theme for Wordpress means a mix of PHP and HTML and CSS, which is painful to read and even more painful to write. Aside from the considerable ugliness of PHP itself, there's a lot of weird magic involved with themes, based on naming conventions for files, weird fall-through behavior when certain theme files aren't present and so on. The Wordpress API is enormous and not fun to work with if you want to do something other than the standard Wordpressy kind of blog structure. Static pages aren't too much fun to work with in Wordpress either.

Lately I think I was getting hammered with spam partly because Wordpress is such an easy target. Askimet is nice but it wasn't catching enough lately; maybe 10-15 spams per week were slipping through. And there was always the chance that some widely-known exploit in Wordpress was going to leave my site susceptible to some roving bot.

And so on.

Hello Clojure

Why Clojure? Because it's awesome and fun and powerful and I wanted to learn it better.

Compojure is a web framework for Clojure that made a lot of this very easy. Coming here from a Ruby on Rails background, Compojure has a lot going for it in comparison. Compojure is lightweight and more low-level than Rails. For example Compojure doesn't enforce MVC on you, doesn't force a unit testing framework on you, and doesn't care how you access your data. Compojure just lets you route HTTP requests to Clojure functions based on the URL and request method (RESTfully: POST/GET/DELETE/PUT), and it gives you easy access to the request information, session, GET/POST parameters and cookies.

Under the hood it's all servlets and Jetty, both of which are solid, stable, well-tested, well-documented technologies. However, thankfully, all of that Java stuff is under the hood, and well under it. I didn't have to write a single line of Java or interact with single servlet directly. Everything (session, params, headers) is a Clojure hash-map from the perspective of my code.

Compojure also comes with a domain-specific language for writing HTML, which is similar to CL-WHO and myriad other Common Lisp HTML DSL's. All of which are awesome. I can't say enough how much nicer it is to write (or generate) structured s-exps than to write HTML by hand. More on that below.

Compojure doesn't come with any way to interact with a database, so I had to write one. clojure.contrib has an SQL lib which easily lets you interact with a MySQL database. (Clojure can talk to MySQL via MySQL's JDBC connector, of course.) I used clojure.contrib.sql to write a small (192 lines) library which slurps up a bunch of database tables into Clojure refs, and provides a few functions for basic CRUD operations so that any updates to the ref data is also transparently reflected in the database. The database is essentially only for keeping an on-disk cache of the data in case I need to restart the server. The average number of DB queries per page is zero; everything except posting/editing/deleting data just reads out of a Clojure ref.

With possibly multiple users posting data at once, it's nice to have Clojure's built-in concurrency support. Updating the data refs with new data is always safe from multiple threads simply by throwing a (dosync) around all of the write accesses. This was completely painless to write.

I decided I wanted to use Markdown for posting comments and authoring new pages. This was also very simple to do; I outlined how to get Markdown working in Java and Clojure, in a previous post. The real-time previews for comments are largely inspired by / ripped-off from Stack Overflow, implemented mostly using open-source Javascript libraries like Showdown, JQuery, TypeWatch and TextAreaResizer.

A Brief Comparison: Clojure vs. Wordpress

All of my code including the CRUD library, all of the HTML for the templates and layout, admin controls, and all the glue to put it together is 1,253 lines of code. Wordpress is somewhere over 78,000 lines of PHP depending what you count (doesn't include any themes or layout, but does include Wordpress features I didn't need and didn't implement). It's still a pretty nice reduction in code overall, any way you look at it.

As an example, in my old Wordpress site I had a plugin catcloud to generate a "tag cloud". This plugin itself is 226 lines of PHP, not bad. However, here's the Clojure code to generate a similar tag cloud (which you can see here currently):

(defn tag-cloud []
  (let [tags (sort-by #(.toLowerCase (:name (first %))) (all-tags-with-counts))
        counts (map second tags)
        max-count (apply max counts)
        min-count (apply min counts)
        min-size 90.0
        max-size 200.0
        color-fn (fn [val]
                   (let [b (min (- 255 (Math/round (* val 255))) 200)]
                     (str "rgb(" b "," b "," b ")")))
        tag-fn (fn [[tag c]]
                 (let [weight (/ (- (Math/log c) (Math/log min-count))
                                 (- (Math/log max-count) (Math/log min-count)))
                       size (+ min-size (Math/round (* weight
                                                       (- max-size min-size))))
                       color (color-fn (* weight 1.0))]
                   [:a {:href (:url tag)
                        :style (str "font-size: " size "%;" "color:" color)}
                    (:name tag)]))]
    (block nil
           [:h2 "Tags"]
           [:div.tag-cloud
            (apply html (interleave (map tag-fn tags)
                                    (repeat " ")))])))

This is 10 times less code, which is a good reduction in my opinion. Most of the code is the math to generate a weight logarithmically for each tag so they scale nicely. (all-tags-with-counts) fetches a seq of two-item pairs: the tags themselves (which are hash-maps) and a count of posts for each tag. There are two locally-defined functions in the let which generate the text color and the font size and HTML for each tag.

The vectors that look like [:h2 "Tags"] are input for Compojure's HTML-generating DSL; this would be transformed for example into <h2>Tags</h2>. (block ...) is a macro which wraps its content in HTML for the rounded borders of my layout. (Math/log ...) and friends are calls to standard Java math functions.

This whole function is less code than just the horrible boilerplate array declarations at the top of the Wordpress plugin:

$catcloud_field_data = array(
  array('name' => 'Minimum Font Size', 'option' => 'catcloud_min_font_size', 'size' => '4', 'maxlength' => '3',
       'default' => '9', 'note' => 'Used for the least frequent categories', 'validation' => '/^\d{1,3}(\.\d{1,3})?$/'),
  array('name' => 'Maximum Font Size', 'option' => 'catcloud_max_font_size', 'size' => '4', 'maxlength' => '3',
       'default' => '18', 'note' => 'Used for the most frequent categories', 'validation' => '/^\d{1,3}(\.\d{1,3})?$/'),
  array('name' => 'Font Face', 'option' => 'catcloud_font_face', 'size' => '15', 'maxlength' => '254',
       'default' => '', 'note' => 'Set an optional list of font faces', 'validation' => '/.*/'),
  array('name' => 'Font Units', 'option' => 'catcloud_font_units', 'size' => '3', 'maxlength' => '2',
       'default' => 'pt', 'note' => 'Choose one of em, pt, px or %', 'validation' => '/^(%|em|pt|px)$/'),
  array('name' => 'Color Start', 'option' => 'catcloud_color_start', 'size' => '7', 'maxlength' => '6',
       'default' => '0066CC', 'note' => 'For the least frequent categories. Use a hexadecimal RGB triplet. ie. 0066CC',
       'validation' => '/^[\dA-F]{6}$/i'),
  array('name' => 'Color End', 'option' => 'catcloud_color_end', 'size' => '7', 'maxlength' => '6',
       'default' => 'CC6600', 'note' => 'For the most frequent categories. Use a hexadecimal RGB triplet. ie. CC6600',
       'validation' => '/^[\dA-F]{6}$/i'),
  array('name' => 'Before Category', 'option' => 'catcloud_before', 'size' => '3', 'maxlength' => '20',
       'default' => '[', 'note' => 'Set the character(s) to display before category names', 'validation' => '/.*/'),
  array('name' => 'After Category', 'option' => 'catcloud_after', 'size' => '3', 'maxlength' => '20',
       'default' => ']', 'note' => 'Set the character(s) to display after category names', 'validation' => '/.*/'),
  array('name' => 'Show Top N Categories', 'option' => 'catcloud_top_n_cats', 'size' => '5', 'maxlength' => '3',
       'default' => '', 'note' => 'Show only the top N categories (where N is a number like 10 or 25 or whatever. Set to 0 or empty for no limit.',
       'validation' => '/^\d*$/'),
  array('name' => 'Excluded Categories', 'option' => 'catcloud_excluded_cats', 'size' => '15', 'maxlength' => '254',
       'default' => '', 'note' => 'A comma-separated list of category ids.',
       'validation' => '/^[\d, ]*$/'),
)

Ugh. As another example, here's the code that handles a POST request to add a new blog page:

(defn do-new-post []
  (check-login
   (let [post (add-post *params*)]
     (sync-tags post (:all-tags *params*))
     (redirect-to "/"))))

It does exactly what it says: Check to make sure the user is logged in, add the post based on the POST params, sync up the tags for that post and redirect to the front page. Lisp lets you say what you want very concisely, with a bare minimum of boilerplate.

How about speed? My Clojure code is actually generating HTML in the most brute-force and wasteful way possible. The HTML for each page is regenerated from scratch, via a cascade of a couple dozen function and macro calls, every time you load a page. But it's still pretty fast, a couple hundred milliseconds for most page requests. This is slightly faster than the Wordpress version of my site. If I ever have performance issues I can switch to another Clojure HTML library, like clj-html which uses the same vector-style syntax but pre-compiles the HTML.

How hard was it to set up on the server? Wordpress is pretty famous for being dirt-easy to deploy anywhere. My Clojure app by comparison was slightly more difficult, as you might expect, but it wasn't brain surgery. My server runs Debian. First I installed the JVM via apt, then I rsynced a bunch of jar's and clj files to the server, then I installed emacs and screen also via apt. Then I put two lines into an Apache config file to proxy-forward traffic to a local port where jetty would be listening. I started Emacs, did (require 'bcc.blog.server), did (bcc.blog.server/go) to start everything, and that's about it. Took about 15 minutes to set up from scratch. When I find a bug, I SSH in, re-attach to screen, fix it in Emacs, hit C-c C-c to recompile just the functions I need to update, and then detach from screen again.

I'm pretty pleased with this so far. It was fun to write and has all the features I used from Wordpress, plus more, and the building blocks are there to extend things if I imagine up a new feature I like.

Looks like my blog is still running today in spite of my predictions. Still waiting for the JVM to crash though, I know it's coming. I plan to post the source code for some of this once I'm sure it works.

Eric at LispCast wrote an article about why PHP is so ridiculously dominant as a web language, when arguably more powerful languages like Common Lisp linger in obscurity.

I think the answer is pretty easy. In real life, practicality usually trumps everything else. Most programmers aren't paid to revolutionize the world of computer science. Most programmers are code monkeys, or to put it more nicely, they're craftsmen who build things that other people pay them to create. The code is a tool to help people do a job. The code is not an end in itself.

In real life, here's a typical situation. You have to make a website for your employer that collects survey data from various people out in the world, in a way that no current off-the-shelf program quite does correctly. If you could buy a program to do it that'd be ideal, but you can't find a good one, so you decide to write one from scratch. The data collection is time-sensitive and absolutely must start by X date. The interface is a web page, and people are going to pointy-clicky their way through, and type some numbers, that's it; the backend just doesn't matter. For your server, someone dug an old dusty desktop machine out of a closet and threw Linux on there for you and gave you an SSH account. Oh right, and this project isn't your only job. It's one of many things you're trying to juggle in a 40-hour work week.

One option is to write it in Common Lisp. You can start by going on a quest for a web server. Don't even think about mod_lisp, would be my advice, based on past experience. Hunchentoot is good, or you can pay a fortune for one of the commercial Lisps. If you want you could also look for a web framework; there are many to choose from, each more esoteric, poorly documented and nearly impossible to install than the last. Then you get to hunt for a Lisp implementation that actually runs those frameworks. Then you get to try to install it and all of your libraries on your Linux server, and on the Windows desktop machine you have to use as a workstation. Good luck.

Once you manage to get Emacs and SLIME going (I'm assuming you already know Emacs intimately, because if you don't, you already lose) you get to start writing your app. Collecting data and moving it around and putting it into a database and exporting it to various statistics packages is common, so you'd do well to start looking for some libraries to help you out with such things. In the Common Lisp world you're likely not to find what you need, or if you're lucky, you'll find what you need in the form of undocumented abandonware. So you can just fix or write those libraries yourself, because Lisp makes writing libraries from scratch easy! Not as easy as downloading one that's already been written and debugged and matured, but anyways. Then you can also roll your own method of deploying your app to your server and keeping it running 24/7, which isn't quite so easy. If you like, you can try explaining your hand-rolled system to the team of sysadmins in another department who keep your server machine running.

Don't bet on anyone in your office being able to help you with writing code, because no one knows Lisp. Might not want to mention to your boss that if you're run over by a bus tomorrow, it's going to be impossible to hire someone to replace you, because no one will be able to read what you wrote. When your boss asks why it's taking you so long, you can mention that the YAML parser you had to write from scratch to interact with a bunch of legacy stuff is super cool and a lovely piece of Lisp code, even if it did take you a week to write and debug given your other workload.

Be sure to wave to your deadline as it goes whooshing by. If you're a genius, maybe you managed to do all of the above and still had time to roll out a 5-layer-deep Domain Specific Language to solve all of your problems so well it brings tears to your eye. But most of us aren't geniuses, especially on a tight deadline.

Another option is to use PHP. Apache is everywhere. MySQL is one simple apt-get away. PHP works with no effort. You can download a single-click-install WAMP stack nowadays. PHP libraries for everything are everywhere and free and mature because thousands of people already use them. The PHP official documentation is ridiculously thorough, with community participation at the bottom of every page. Google any question you can imagine and you come up with a million answers because the community is huge. Or walk down the hall and ask anyone who's ever done web programming.

The language is stupid, but stupid means easy to learn. You can learn PHP in a day or two if you're familiar with any other language. You can write PHP code in any editor or environment you want. Emacs? Vim? Notepad? nano? Who cares? Whatever floats your boat. Being a stupid language also means that everyone knows it. If you jump ship, your boss can throw together a "PHP coder wanted" ad and replace you in short order.

And what do you lose? You have to use a butt-ugly horrid language, but the price you pay in headaches and swallowed bile is more than offset by the practical gains. PHP is overly verbose and terribly inconsistent and lacks powerful methods of abstraction and proper closures and easy-to-use meta-programming goodness and Lisp-macro syntactic wonders; in that sense it's not a very powerful language. Your web framework in PHP probably isn't continuation-based, it probably doesn't compile your s-expression HTML tree into assembler code before rendering it.

But PHP is probably the most powerful language around for many jobs if you judge by the one and only measure that counts for many people: wall clock time from "Here, do this" to "Yay, I'm done, it's not the prettiest thing in the world but it works".

The above situation was one I experienced at work, and I did choose PHP right from the start, and I did get it done quickly, and it was apparently not too bad because everyone likes the website. No one witnessed the pain of writing all that PHP code, but that pain doesn't matter to anyone but the code monkey.

If I had to do it over again I might pick Ruby, but certainly never Lisp. I hate PHP more than almost anything (maybe with the exception of Java) but I still use it when it's called for. An old rusty wobbly-headed crooked-handled hammer is the best tool for the job if it's right next to you and you only need to pound in a couple of nails.

I made an RSS 2.0 feed for my origami gallery the other day. Thanks to CL-WHO this is trivial. Here's the complete code:

;; RSS
(defun rss ()
  (cl-who:with-html-output-to-string (s nil :prologue "<?xml version=\"1.0\" encoding=\"ISO-8859-1\" ?>")
    (:rss :version "2.0" :|xmlns:atom| "http://www.w3.org/2005/Atom"
          (:channel (:title "An Origami Gallery")
                    (:link "http://origamigallery.net")
                    (:|atom:link| :href "http://origamigallery.net/feed" :rel "self" :type "application/rss+xml")
                    (:description "A photo gallery.  Of origami models.")
                    (loop for model in (all-models)
                          do (htm (:item (:title (str (fullname model)))
                                         (:link (str (absolute-url model)))
                                         (:guid  (str (absolute-url model)))
                                         (:description (str (clean-remarks (remarks model)))))))))))

There are other Common Lisp libraries to make RSS feeds, which may have made this even easier, but I like CL-WHO.

Interesting about this code is how short it is. CL-WHO is an example of embedding another language in Common Lisp (e.g. a domain specific language), in this case, XML or HTML.

XML in particular happens to map very nicely onto s-expressions, allowing nested tags and attribute/value pairs and whatnot. Except that s-exps are far less verbose than XML, far nicer to type, and far easier to read in my opinion. One quirk here is that CL-WHO uses keyword symbols for XML tag names (CL keyword symbols begin with a colon), and XML uses colons for namespaces. To make a keyword symbol with a colon in the middle in CL, you have to surround it in pipes. No big deal though.

This kind of thing isn't entirely possible or quite as elegant in most other languages. (Ruby does pretty good though.) In PHP, for example, you can mix up PHP and HTML content, but the PHP interpreter doesn't parse the HTML or do anything interesting with it. It just treats it as a bunch of strings; embedding PHP in HTML is just a shortcut for string concatenation.

You could write a library in PHP to generate XML code (and I'm sure such libraries exist) but they're either going to have some kind of functional / OO interface, which will make it very verbose and probably clunky to use in comparison to CL, or else it's going to be something like Smarty which actually isn't PHP at all but rather a separate language with its own parser and interpreter or compiler that itself happens to be written in PHP. What you could not easily do is write PHP code that turns other PHP code into an XML document.

But that's largely what CL-WHO does. Because CL-WHO parses the s-exps that represent my XML data, if I forget a closing paren, Lisp will yell at me, so my XML is guaranteed to be well-formed. CL-WHO doesn't verify that I'm using proper RSS tags in the proper places or anything but it could easily be extended to do that. See PCL for an HTML interpreter/compiler which can verify that you only use valid HTML tags, for example.

CL-WHO also lets me change whether my XML attribute values are in single or double quotes, or whether I want to allow empty tags (as in old HTML) or require them all to be closed, and other such things. CL-WHO can nicely indent my XML, or generate it unindented.

And since this is all just Common Lisp, you see that I can embed a loop right in the middle of the XML s-exps, or do conditionals, or I could write a function that generates some or all these s-exps for me programatically. Or any number of other things. I have all of Common Lisp at my fingertips to create these s-exps.

See also "The Nature of Lisp" which is a nice long discussion of how Lisp is a better XML than XML. (And I do mean looong. I haven't read the whole article myself, but enough to know it's pretty good.)

My Common Lisp-powered origami photoblog is still up and running smoothly so far. (I posted more models, go look, shameless plugs and so on and so forth.) No major problems to report in the past couple weeks.

One huge benefit (in my opinion) of a site run on Common Lisp is the way you can solve the "admin control panel" problem. Most web site frameworks / blog engines / message boards have some control panel interface, for example the one I'm using to type this blog entry.

Why are control panels necessary? You could SSH to your server and run SQL queries directly to input your blog posts. One reason we don't do this is because it bypasses the logic of your website. We use a PHP form so that it can complain if you type a post with no title, or run a filter to convert your line-breaks to HTML tags, or do spell-checking, or whatever. It does all these things before entering your data into the DB, and then when the DB does need to be updated, it updates it in a consistent way. If multiple tables need to be updated, it doesn't forget to do them all. Etc. etc.

The problem I have with control panels is that due to limitations of web languages and how the internet works, these control panels must be run as web pages just like any other. Meaning they're open and accessible to the world if you visit the right URL.

To solve this we start heaping on the passwords. Above and beyond SQL client access and FTP access and shell account access, we make a brand new custom layer of user accounts and permissions. Only we implement it using .htaccess files and SQL hacks and fragile HTTP connections and cookies and sessions. And then we get to test how well we did by letting every script kiddie and crawler-bot in the world hammer on it at will.

I agonized about how to make a "control panel" for my photo blog. These things aren't easy to get right. Eventually, I realized it wasn't necessary. The REPL is my control panel. I can implement the necessary logic as simple Lisp functions. When I want to post a new model, I SSH to my server, fire up Emacs, connect to the Lisp running on my running server via local SLIME, and run an ADD-MODEL function. Simple. Lisp automatically timestamps my posts, and checks for empty-string model names, and all the other good stuff you'd want from a control panel. (Plus if it fails, I get a debugger.)

What about authentication? I get that for free. SSH is my authentication. Authentication is what SSH is made for and it's going to be far better at it than some web-page SQL-based hack-job I come up with. I don't even have to remember a password, since I have certificates set up for passwordless login. And there's no URL that points to my photo blog control panel, which is nice.

What about letting multiple people log into the "control panel"? Anyone you don't trust to SSH into a shell account usually shouldn't be trusted to log into a Wordpress or VBulletin control panel either in my opinion. Run Lisp as a non-privileged user (probably should be doing that anyways, I am) and run SSH in a chroot if you really care.

What about the typical web gallery feature of letting you upload an image and then having it automatically thumbnailed? I wrote a two line shell script that uses ImageMagick (which is likely what a PHP-run gallery would be using anyways) to thumbnail all my photos locally, then rsyncs them to my server. Why re-invent the wheel?

Couldn't Perl or Ruby or PHP do the same thing? Well, Perl/Ruby/PHP don't run persistently on the server. So it'd be a bit different. Couldn't you write some standalone scripts to run from the commandline to insert new posts into your blog? I suppose, but it surely wouldn't be as nice an interface as Slime in Emacs, unless you enjoy using bash as your text editor. And have fun with quoting / escaping. (EDIT: As some readers reminded me, yes they can run persistently via Apache hackery. And Rails has a "console". I stand corrected.)

The REPL obviously isn't a solution to every problem. If your authentication requirements are complex enough, you'll have to build something yourself. And it's a problem if your users don't know Lisp (and yeah, that right there kills it for 99.999998% of the world). Web-based control panels have the benefit of being "so easy anyone can use them".

But for my needs, and probably the needs of a great many websites run by one or two trusted Linux-savvy people who just need to be able to securely update the site once in a while, a few Linux tools + the REPL works beautifully.

I spent four hours working on my photo-blog in Common Lisp that I've mentioned earlier.

Why does Common Lisp hate me? After four hours, I'm not yet even to the point where I can reliably, successfully load all the libraries I need to load. How pathetic is that? I haven't even gotten to the point where I could, you know, actually write my program.

This article explains how to set up an ASDF system file. I'm using ASDF because you almost HAVE to use it if you're utilizing any 3rd-party libraries, because so many 3rd-party libraries use ASDF themselves and because I wouldn't know where to begin to write my own dependency-resolution scripts from scratch.

So my ASD file has all the right dependencies, so all the libraries load in the right order and Lisp knows they exist. Then I have a packages.lisp that defines my package. Then I have a main source file with all my code in it, which is IN-PACKAGE my package. This all appears to work.

As long as I fully-qualify every function that comes from another package, it works fine. Problem is that if I try to (:use :cl-who) in my package, to avoid having to endlessly repeat myself every time I use a function from CL-WHO, I get a flood of warnings saying:

undefined variable: PRINC

Funny, because PRINC is a built-in function in Common Lisp. And PRINC is defined in my package. And if I IN-PACKAGE to CL-WHO, it's accessible from there too.

Can I use ASDF and still have my package inherit functions from other packages? This is the great mystery which I have been completely unable to solve. Is this some compile-time vs. runtime mess that only exists in Common Lisp? Am I accidentally interning symbols and obliterating other packages' symbols? Am I just mentally retarded?

This is extremely frustrating. I don't care how wonderfully great your programming language is: if you have to arm-wrestle the thing for a week to get your environment set up, what's the point? How can something so simple as importing functions from other packages be so difficult?

I could've had everything done already if I'd have used Ruby. I could gem install what I need, and then require it and wow, it works. Hell, I could've had this done in PHP or Perl by now.

If I ever get CL working on my machine, then I have to worry about installing SBCL and all these libraries and mod_lisp and whatnot on my Debian server. God almighty, I don't even want to think about that.

In Ruby, the only things that are false are nil and false. This is a nice change over other languages where other things evaluate to false in boolean tests. Like C and many others, which considers the integer 0 to be false. So is 0.0 false? How about 1e-999999? Or there's Perl, which also considers an empty string to be false, and auto-casts between numbers and strings, which can make things fun. And of course Perl has Perlish insanity like 0E0, "zero but true".

However all languages pale in comparison with PHP. They provide these handy charts to help you figure out what's true and what's false.

That really sums up PHP well. You have three standard functions (isset, is_null, empty) with completely inconsistent names. Two equality test operators whose meaning completely changes between PHP versions. And then a couple of 12x12 grids of crap necessary to figure out how they behave.

"0" == 0? True.

0 == ""? True.

"0" == ""? False...

null == array()? True....?

Really makes you appreaciate Ruby.

Today I turned my 750+ lines of ugly PHP into 200 lines of much nicer PHP, and the shorter version works much better than the longer version. So I'm feeling happier with myself than I was yesterday. I did finally manage to make my parser recursive and remove all the special-case handling code. I had an interesting experience where I was diligently hacking away at my code, and then I noticed everything was actually working and had been working for quite some time without my noticing.

Still if I was a better programmer it wouldn't have taken me a month to figure this out.

My only worry now is that if anyone but me ever has to use or maintain these scripts, they're going to have to understand recursion and understand what I was going for here. That's a scary thought. One guy who interviewed me for a job a few years ago said that he asks a lot of people to explain recursion to him during job interviews, largely because a lot of programmers don't even know what the word means.

Part of what Ableson and Sussman keep saying in these lovely lecture videos is that a good way to solve a problem is to divide it into layers of abstraction, where one layer of abstraction gets its work done by using a "language" that you wrote in the abstraction layer immediately below it. They say that a good way to solve a problem is to write something powerful enough to solve the general KINDS of problems that you're trying to solve, because you're probably going to need something that powerful later anyways. In these videos they tend to say in a very clear, direct, rigorous way things that I've heard said many times in other places by other people.

In particular, in the videos they stress trying to keep your "custom languages" written in such a way that they can be used recursively. My grand PHP project that I've been working on for some time at work is disturbingly similar to some of the examples they use in those lectures; namely, my project seems to be a good example of how NOT to do things.

My project is a set of scripts to build forms to conduct an interview. It's basically an enormous HTML form, but the form is split into pages, and the pages contain questions, which are groupings of HTML controls and text labels and Javscript and other things. To borrow language from SICP: questions are really my primitives. In addition to questions there are "groups" of questions that behave like single questions in certain circumstances but not others. And some questions have "sub-questions" that depends on them in hard-to-handle ways. So in SICP terminology: Questions can be combined by forming groups, or forming question/sub-question pairs, or otherwise just by listing them sequentially. The groupings can be into rows, or into columns, or into both, or into other kinds of patterns.

The program works by having someone write the survey in YAML, which is a very easy plaintext-ish way to write config files. Then my scripts parse the YAML, build my question objects and render them. The end. What I have though is essentially a custom language, with YAML as its syntax, used for building arbitrary surveys. It works fine and I was able to write our 20-page interview in this survey-language very easily. Ableson and Sussman would hopefully be proud (if highly disgusted by the details of my implementation in PHP).

When I was designing these scripts I do remember having the thought of whether I should make the questions in my language behave fully recursively. In other words I could simply make EVERY question have the ability to store references to one or more sub-questions. Then there would be only one kind of question, which may or may not have sub-questions. When I render a question, I'd just have it render itself and then render all its children recursively.

I gave that a try but gave up. In practice (so I thought at the time), questions should never really need to nest more than one or two levels deep, and the questions should only be combined into groups in a very limited couple of ways. So really, a fully recursive language wasn't necessary. So I wrote my language such that grouping questions together was a special case handled with extra code. Likewise sub-questions for normal questions were a special case handled with even more extra code. This worked OK.

If only I'd have seen these videos a month ago. This week I came upon a situation where the survey-building language I came up with isn't powerful enough to handle a certain type of grouping construct which I need. My language isn't arbitrary enough to let me do what I need to do. To make matter worse, looking at my code now, I realized that all the special-case code required to handle all the things I thought wouldn't be a problem has accumulated far beyond what I'd have ever expected.

Live and learn, I suppose. I think I was wandering down the right path, even if I fell off into the ditch before I reached the end of it.

Some other guy built a similar survey system for a different group at work a while back, and I had a chance to look at his code. That code consisted of one long listing of intermingled PHP, SQL and HTML, hard-coded throughtout to work specifically for the survey the group was conducting. I wanted to use those scripts myself but they were essentially unusable because they were so non-generic. I at least did better than that.

I started writing a post called "Lisp vs. Smarty" two nights ago but lost my train of thought before I could finish. Lately I see lots of blog entries about PHP and in particular this one by Sean Potter about Smarty made me want to finish typing this.

In Practical Common Lisp, in two chapters (30 and 31) and a few hundred lines of code, Peter Seibel produces a library he calls FOO that can compile (or interpret) s-expressions into HTML. FOO is essentially an interpreter/compiler for a little mini HTML-sexp language, which can compile the HTML-sexp language into Lisp code (that can be run later as a Lisp script) or can use the HTML-sexp code to produce HTML output directly. The book is there on his site for free in its entirety, feel free to read those chapters.

This FOO library is immensely powerful, and the things that make FOO more powerful than Smarty are the things that make Lisp more powerful than PHP in general. Personally I found it a great example to help me understand some of the differences between Lisp and non-Lisp languages.

Both Smarty and FOO are libraries that have at least two purposes: To produce dynamic HTML based on parameterized input (i.e. feed it values that are inserted into the final HTML) and to produce HTML without having to write a thousand static files manually by using control and looping constructs and by allowing you to have re-usable modular templates.

So, you can embed Lisp code into FOO to act as simple text filters. You can refer to Lisp variables in FOO scripts. You can use control structures like if/then/else, or looping constructs, or what have you. That's nothing different than Smarty can do though. Smarty is good at those things.

But since this is Lisp, you can easily write macros that simplify, abstract, or otherwise help you write the FOO HTML-sexps. In other words, you can write Lisp code that largely writes your HTML-sexps for you.

With Smarty, you're much more limited in what kinds of abstractions are available to you. You're probably going to write the Smarty templates mostly by hand. You're usually not going to do the equivalent of Lisp and write PHP code that writes your Smarty templates. If you do, it's going to be in certain limited ways Otherwise it would probably be more verbose and more of a pain than writing the Smarty templates yourself. After all, the point of Smarty is to help you NOT to have to write PHP. Trying to programatically produce Smarty templates using PHP is a mess largely because there's no easy way for PHP to understand Smarty templates other than as strings. Using PHP to produce Smarty and using PHP to produce HTML are equally nasty problems to try to solve.

So instead you often use Smarty to "write" Smarty. But not really. You write template files, split them up, and then you import various bits and pieces of some template files into the insides of other template files. You can throw Smarty looping and control constructs around your Smarty include statements, so this all works pretty well. But the Smarty template language is obviously rather restricted in what tools it gives you for this kind of meta-templating. And templates are about the finest granularity you're going to get.

And note, the Smarty templates surely aren't going to write the HTML tags for you. They aren't going to auto-close your open HTML tags, letting you choose how to do it based on your preference for HTML or XHTML. Smarty templates aren't going to make sure you put quotes around your attribute lists. They aren't going to (optionally) properly indent your HTML code. FOO does all of these things and more. FOO can also be used to make "templates" much like Smarty, but it gives you far greater control than that.

So let's talk extensibility. Smarty gives you hooks you can use to write plugins that do various things. The Smarty plugin documentation is around 15 pages of documentation largely telling you what you CAN'T do. Smarty plugin files for example must be named certain things, and put in certain folders. Your tags must look and act in certain ways. When your functions are called, they receive parameters X, Y, and Z, which Smarty provides, and then you must return exactly Q, which Smarty expects from you.

There are a limited number of very specific KINDS of functions you can write. You can write a function to filter some text to change it into other text. You can write a function that gets a big string full of raw un-parsed template text and returns a big string back after you do something to it. You can write a function that gets a big string full of already-parsed template text and return back a big string of text after you do something to it. Etc. Key word here: BIG STRINGS.

But let's say for example that the {if}{else}{/if} construct was missing from Smarty, and you wanted to write it. Could you do it? Smarty gives you no hook to write something like that. So you could try to use what Smarty dose give you. You could make up three tags ( {if}, {else}, and {/if} ). And then you could maybe write some functions that take big strings of the template text that's in between those tags, and parse it yourself. Maybe you could write (in PHP) a sort of state machine, global enough to last between calls to the custom functions you wrote. Would it handle an arbitrary number of {else}'s per {if}? Would it work recursively? That might take some more work.

If you're resorting to such a thing, you're almost re-inventing Smarty yourself. So hey, you could just go and edit the Smarty compiler PHP code directly to add this functionality.

I just looked at the Smarty_Compiler.class.php file that comes with Smarty. Oh the horror. Inside are line after line of regular expressions designed to take tokenize Smarty template text. Once it's been tokenized, there's an enormous switch/case statement to deal with the allowed tags. Some tags are translated directly into PHP code, which is output as a bunch of our friend, the plain old string, later to be presumably eval'ed. Some tags rely on other tags, so there's a stack to which tags are pushed and popped as the file is parsed. It really is a full-fledged compiler: A "Smarty template language"-to-PHP compiler, written in nice slow PHP.

I tried to track down the code responsible for implementing the {if}{else}{/if} construct in this Smarty compiler file. The 40+ lines of swtich/case that handle the {if}-related tags make a couple calls to, among other things, a function called _compile_if_tag. This one function is 163 lines long. Line count isn't a good example of complexity, of course. So here are just a few of those 163 lines:

/* Tokenize args for 'if' tag. */
preg_match_all('~(?>
    ' . $this->_obj_call_regexp . '(?:' . $this->_mod_regexp . '*)? | # valid object call
    ' . $this->_var_regexp . '(?:' . $this->_mod_regexp . '*)?    | # var or quoted string
    \-?0[xX][0-9a-fA-F]+|\-?\d+(?:\.\d+)?|\.\d+|!==|===|==|!=|<>|<<|>>|<=|>=|\&\&|\|\||\(|\)|,|\!|\^|=|\&|\~|<|>|\||\%|\+|\-|\/|\*|\@    | # valid non-word token
    \b\w+\b                                                        | # valid word token
    \S+                                                           # anything else
)~x', $tag_args, $match);

...

if(preg_match('~^' . $this->_func_regexp . '$~', $token) ) {
        // function call
        if($this->security &&
           !in_array($token, $this->security_settings['IF_FUNCS'])) {
            $this->_syntax_error("(secure mode) '$token' not allowed in if statement", E_USER_ERROR, __FILE__, __LINE__);
        }
} elseif(preg_match('~^' . $this->_var_regexp . '$~', $token) && (strpos('+-*/^%&|', substr($token, -1)) === false) && isset($tokens[$i+1]) && $tokens[$i+1] == '(') {
    // variable function call
    $this->_syntax_error("variable function call '$token' not allowed in if statement", E_USER_ERROR, __FILE__, __LINE__);                      
} elseif(preg_match('~^' . $this->_obj_call_regexp . '|' . $this->_var_regexp . '(?:' . $this->_mod_regexp . '*)$~', $token)) {
    // object or variable
    $token = $this->_parse_var_props($token);
} elseif(is_numeric($token)) {
    // number, skip it
} else {
    $this->_syntax_error("unidentified token '$token'", E_USER_ERROR, __FILE__, __LINE__);
}

And so on, and so forth.

Suffice it to say, in the Smarty compiler, the code that implements {if}{else}{/if} is at least a couple hundred lines of code that looks like the above. It would take me quite a bit of time and effort even to figure out what that code does, let alone write it myself and debug it.

Now, that code is part of the Smarty compiler itself, written by the people who wrote Smarty, i.e. the people in the world who know Smarty best. If I was trying to write my OWN code that gave me a {if}{else}{/if} structure in Smarty, my code would by necessity be FAR MORE complex than that. If written as a plugin, my code wouldn't have direct access to the guts of the Smarty compiler. Instead my code would have the very limited access that Smarty provides to plugins and custom functions. Even if I tried to hack the compiler msyelf, my code would be written by me, who certainly doesn't have the expert knowledge necessary or understand the inner workings of Smarty well enough to get this code working without a good deal of effort.

So let's contrast this with FOO. If an if-then-else sort of construct was missing from FOO, could you write it yourself? It turns out you could. Peter Seibel calls this a "trivial example" and mentions it in passing. It turns out you could add this construct to the FOO mini HTML-sexp language using two lines of Lisp.

Using TWO LINES.

TWO LINES!

(define-html-macro :if (test then else)
  `(if ,test (html ,then) (html ,else)))

There's around 50 lines of Lisp that puts everything in place to you allow you to write "plugins" like the above or far more complex than the above. 50 lines of Lisp that let you pretty much arbitrarily extend the HTML-sexp mini language itself, to add new tags including entirely new control constructs, that the HTML-sexp compiler understands.

Lisp already has a perfectly fine if-then-else construct after all, as does PHP. Somewhere in the PHP compiler itself, there's logic (maybe written in C? Maybe assembler?) that implements if-then-else. Why can't we use that compiler code? Why did we have to write our own Smarty compiler to do it all for us?

Rather than write a new templating language, and then write a tokenizer and lexer and parser for that language, and then slurp up a bunch of strings and translate it all back to a plain old PHP if-then-else, which is then transformed by the PHP compiler itself into native code that implements if-then-else... in FOO, we use Lisp. Lisp can already read s-expressions and parse them properly, and give us access to the result as native Lisp objects. Lisp has if-then-else, which we can use directly. That's all we need. Aside from the wonderful simplicity of it all, imagine the difference in performance.

The key is that the HTML-sexp language is s-expressions. Lisp itself is s-expressions. The compiler that translates the FOO code into Lisp is s-expressions. The code that will run the FOO compiler is s-expressions. The macro code that helps you abstract all of the above is s-expressions. The macros that help you write THOSE macros are s-expressions. Everything is s-expressions. And Lisp is VERY GOOD at reading, writing, and arbitrarily manipulating s-expressions. Which means that Lisp is VERY GOOD at reading, writing, and arbitrarily manipulating mini-languages like FOO, mini-language-interpreters and compilers, Lisp code, Lisp code that runs Lisp code, and Lisp code that writes Lisp code.

One guy wrote FOO, and then very nicely explained start-to-finish every single line of code that makes it work, in two short chapters of a beginners' Lisp book. Compare this with thousands of lines of PHP written by a team of programmers over the course of apparently six years, to produce a library that at the end of the day does far less than FOO can do.