This is a read-only archive!

Lisp vs. Smarty

I started writing a post called "Lisp vs. Smarty" two nights ago but lost my train of thought before I could finish. Lately I see lots of blog entries about PHP and in particular this one by Sean Potter about Smarty made me want to finish typing this.

In Practical Common Lisp, in two chapters (30 and 31) and a few hundred lines of code, Peter Seibel produces a library he calls FOO that can compile (or interpret) s-expressions into HTML. FOO is essentially an interpreter/compiler for a little mini HTML-sexp language, which can compile the HTML-sexp language into Lisp code (that can be run later as a Lisp script) or can use the HTML-sexp code to produce HTML output directly. The book is there on his site for free in its entirety, feel free to read those chapters.

This FOO library is immensely powerful, and the things that make FOO more powerful than Smarty are the things that make Lisp more powerful than PHP in general. Personally I found it a great example to help me understand some of the differences between Lisp and non-Lisp languages.

Both Smarty and FOO are libraries that have at least two purposes: To produce dynamic HTML based on parameterized input (i.e. feed it values that are inserted into the final HTML) and to produce HTML without having to write a thousand static files manually by using control and looping constructs and by allowing you to have re-usable modular templates.

So, you can embed Lisp code into FOO to act as simple text filters. You can refer to Lisp variables in FOO scripts. You can use control structures like if/then/else, or looping constructs, or what have you. That's nothing different than Smarty can do though. Smarty is good at those things.

But since this is Lisp, you can easily write macros that simplify, abstract, or otherwise help you write the FOO HTML-sexps. In other words, you can write Lisp code that largely writes your HTML-sexps for you.

With Smarty, you're much more limited in what kinds of abstractions are available to you. You're probably going to write the Smarty templates mostly by hand. You're usually not going to do the equivalent of Lisp and write PHP code that writes your Smarty templates. If you do, it's going to be in certain limited ways Otherwise it would probably be more verbose and more of a pain than writing the Smarty templates yourself. After all, the point of Smarty is to help you NOT to have to write PHP. Trying to programatically produce Smarty templates using PHP is a mess largely because there's no easy way for PHP to understand Smarty templates other than as strings. Using PHP to produce Smarty and using PHP to produce HTML are equally nasty problems to try to solve.

So instead you often use Smarty to "write" Smarty. But not really. You write template files, split them up, and then you import various bits and pieces of some template files into the insides of other template files. You can throw Smarty looping and control constructs around your Smarty include statements, so this all works pretty well. But the Smarty template language is obviously rather restricted in what tools it gives you for this kind of meta-templating. And templates are about the finest granularity you're going to get.

And note, the Smarty templates surely aren't going to write the HTML tags for you. They aren't going to auto-close your open HTML tags, letting you choose how to do it based on your preference for HTML or XHTML. Smarty templates aren't going to make sure you put quotes around your attribute lists. They aren't going to (optionally) properly indent your HTML code. FOO does all of these things and more. FOO can also be used to make "templates" much like Smarty, but it gives you far greater control than that.

So let's talk extensibility. Smarty gives you hooks you can use to write plugins that do various things. The Smarty plugin documentation is around 15 pages of documentation largely telling you what you CAN'T do. Smarty plugin files for example must be named certain things, and put in certain folders. Your tags must look and act in certain ways. When your functions are called, they receive parameters X, Y, and Z, which Smarty provides, and then you must return exactly Q, which Smarty expects from you.

There are a limited number of very specific KINDS of functions you can write. You can write a function to filter some text to change it into other text. You can write a function that gets a big string full of raw un-parsed template text and returns a big string back after you do something to it. You can write a function that gets a big string full of already-parsed template text and return back a big string of text after you do something to it. Etc. Key word here: BIG STRINGS.

But let's say for example that the {if}{else}{/if} construct was missing from Smarty, and you wanted to write it. Could you do it? Smarty gives you no hook to write something like that. So you could try to use what Smarty dose give you. You could make up three tags ( {if}, {else}, and {/if} ). And then you could maybe write some functions that take big strings of the template text that's in between those tags, and parse it yourself. Maybe you could write (in PHP) a sort of state machine, global enough to last between calls to the custom functions you wrote. Would it handle an arbitrary number of {else}'s per {if}? Would it work recursively? That might take some more work.

If you're resorting to such a thing, you're almost re-inventing Smarty yourself. So hey, you could just go and edit the Smarty compiler PHP code directly to add this functionality.

I just looked at the Smarty_Compiler.class.php file that comes with Smarty. Oh the horror. Inside are line after line of regular expressions designed to take tokenize Smarty template text. Once it's been tokenized, there's an enormous switch/case statement to deal with the allowed tags. Some tags are translated directly into PHP code, which is output as a bunch of our friend, the plain old string, later to be presumably eval'ed. Some tags rely on other tags, so there's a stack to which tags are pushed and popped as the file is parsed. It really is a full-fledged compiler: A "Smarty template language"-to-PHP compiler, written in nice slow PHP.

I tried to track down the code responsible for implementing the {if}{else}{/if} construct in this Smarty compiler file. The 40+ lines of swtich/case that handle the {if}-related tags make a couple calls to, among other things, a function called _compile_if_tag. This one function is 163 lines long. Line count isn't a good example of complexity, of course. So here are just a few of those 163 lines:

/* Tokenize args for 'if' tag. */
preg_match_all('~(?>
    ' . $this->_obj_call_regexp . '(?:' . $this->_mod_regexp . '*)? | # valid object call
    ' . $this->_var_regexp . '(?:' . $this->_mod_regexp . '*)?    | # var or quoted string
    \-?0[xX][0-9a-fA-F]+|\-?\d+(?:\.\d+)?|\.\d+|!==|===|==|!=|<>|<<|>>|<=|>=|\&\&|\|\||\(|\)|,|\!|\^|=|\&|\~|<|>|\||\%|\+|\-|\/|\*|\@    | # valid non-word token
    \b\w+\b                                                        | # valid word token
    \S+                                                           # anything else
)~x', $tag_args, $match);

...

if(preg_match('~^' . $this->_func_regexp . '$~', $token) ) {
        // function call
        if($this->security &&
           !in_array($token, $this->security_settings['IF_FUNCS'])) {
            $this->_syntax_error("(secure mode) '$token' not allowed in if statement", E_USER_ERROR, __FILE__, __LINE__);
        }
} elseif(preg_match('~^' . $this->_var_regexp . '$~', $token) && (strpos('+-*/^%&|', substr($token, -1)) === false) && isset($tokens[$i+1]) && $tokens[$i+1] == '(') {
    // variable function call
    $this->_syntax_error("variable function call '$token' not allowed in if statement", E_USER_ERROR, __FILE__, __LINE__);                      
} elseif(preg_match('~^' . $this->_obj_call_regexp . '|' . $this->_var_regexp . '(?:' . $this->_mod_regexp . '*)$~', $token)) {
    // object or variable
    $token = $this->_parse_var_props($token);
} elseif(is_numeric($token)) {
    // number, skip it
} else {
    $this->_syntax_error("unidentified token '$token'", E_USER_ERROR, __FILE__, __LINE__);
}

And so on, and so forth.

Suffice it to say, in the Smarty compiler, the code that implements {if}{else}{/if} is at least a couple hundred lines of code that looks like the above. It would take me quite a bit of time and effort even to figure out what that code does, let alone write it myself and debug it.

Now, that code is part of the Smarty compiler itself, written by the people who wrote Smarty, i.e. the people in the world who know Smarty best. If I was trying to write my OWN code that gave me a {if}{else}{/if} structure in Smarty, my code would by necessity be FAR MORE complex than that. If written as a plugin, my code wouldn't have direct access to the guts of the Smarty compiler. Instead my code would have the very limited access that Smarty provides to plugins and custom functions. Even if I tried to hack the compiler msyelf, my code would be written by me, who certainly doesn't have the expert knowledge necessary or understand the inner workings of Smarty well enough to get this code working without a good deal of effort.

So let's contrast this with FOO. If an if-then-else sort of construct was missing from FOO, could you write it yourself? It turns out you could. Peter Seibel calls this a "trivial example" and mentions it in passing. It turns out you could add this construct to the FOO mini HTML-sexp language using two lines of Lisp.

Using TWO LINES.

TWO LINES!

(define-html-macro :if (test then else)
  `(if ,test (html ,then) (html ,else)))

There's around 50 lines of Lisp that puts everything in place to you allow you to write "plugins" like the above or far more complex than the above. 50 lines of Lisp that let you pretty much arbitrarily extend the HTML-sexp mini language itself, to add new tags including entirely new control constructs, that the HTML-sexp compiler understands.

Lisp already has a perfectly fine if-then-else construct after all, as does PHP. Somewhere in the PHP compiler itself, there's logic (maybe written in C? Maybe assembler?) that implements if-then-else. Why can't we use that compiler code? Why did we have to write our own Smarty compiler to do it all for us?

Rather than write a new templating language, and then write a tokenizer and lexer and parser for that language, and then slurp up a bunch of strings and translate it all back to a plain old PHP if-then-else, which is then transformed by the PHP compiler itself into native code that implements if-then-else... in FOO, we use Lisp. Lisp can already read s-expressions and parse them properly, and give us access to the result as native Lisp objects. Lisp has if-then-else, which we can use directly. That's all we need. Aside from the wonderful simplicity of it all, imagine the difference in performance.

The key is that the HTML-sexp language is s-expressions. Lisp itself is s-expressions. The compiler that translates the FOO code into Lisp is s-expressions. The code that will run the FOO compiler is s-expressions. The macro code that helps you abstract all of the above is s-expressions. The macros that help you write THOSE macros are s-expressions. Everything is s-expressions. And Lisp is VERY GOOD at reading, writing, and arbitrarily manipulating s-expressions. Which means that Lisp is VERY GOOD at reading, writing, and arbitrarily manipulating mini-languages like FOO, mini-language-interpreters and compilers, Lisp code, Lisp code that runs Lisp code, and Lisp code that writes Lisp code.

One guy wrote FOO, and then very nicely explained start-to-finish every single line of code that makes it work, in two short chapters of a beginners' Lisp book. Compare this with thousands of lines of PHP written by a team of programmers over the course of apparently six years, to produce a library that at the end of the day does far less than FOO can do.

November 16, 2007 @ 1:28 PM PST
Cateogory: Programming
Tags: PCL, Lisp, PHP, Smarty

1 Comment

James
Quoth James on March 12, 2012 @ 10:15 AM PDT

Kinda mind-blowing, isn't it? I keep butting my head against the Lisp wall, one day I'll eventually break through, and in the words of 'Ben' Kenobi - "become more powerful than you can imagine"! Lisp, a language for a more elegant time.