This is a read-only archive!

One benefit of a Lisp-driven website

My Common Lisp-powered origami photoblog is still up and running smoothly so far. (I posted more models, go look, shameless plugs and so on and so forth.) No major problems to report in the past couple weeks.

One huge benefit (in my opinion) of a site run on Common Lisp is the way you can solve the "admin control panel" problem. Most web site frameworks / blog engines / message boards have some control panel interface, for example the one I'm using to type this blog entry.

Why are control panels necessary? You could SSH to your server and run SQL queries directly to input your blog posts. One reason we don't do this is because it bypasses the logic of your website. We use a PHP form so that it can complain if you type a post with no title, or run a filter to convert your line-breaks to HTML tags, or do spell-checking, or whatever. It does all these things before entering your data into the DB, and then when the DB does need to be updated, it updates it in a consistent way. If multiple tables need to be updated, it doesn't forget to do them all. Etc. etc.

The problem I have with control panels is that due to limitations of web languages and how the internet works, these control panels must be run as web pages just like any other. Meaning they're open and accessible to the world if you visit the right URL.

To solve this we start heaping on the passwords. Above and beyond SQL client access and FTP access and shell account access, we make a brand new custom layer of user accounts and permissions. Only we implement it using .htaccess files and SQL hacks and fragile HTTP connections and cookies and sessions. And then we get to test how well we did by letting every script kiddie and crawler-bot in the world hammer on it at will.

I agonized about how to make a "control panel" for my photo blog. These things aren't easy to get right. Eventually, I realized it wasn't necessary. The REPL is my control panel. I can implement the necessary logic as simple Lisp functions. When I want to post a new model, I SSH to my server, fire up Emacs, connect to the Lisp running on my running server via local SLIME, and run an ADD-MODEL function. Simple. Lisp automatically timestamps my posts, and checks for empty-string model names, and all the other good stuff you'd want from a control panel. (Plus if it fails, I get a debugger.)

What about authentication? I get that for free. SSH is my authentication. Authentication is what SSH is made for and it's going to be far better at it than some web-page SQL-based hack-job I come up with. I don't even have to remember a password, since I have certificates set up for passwordless login. And there's no URL that points to my photo blog control panel, which is nice.

What about letting multiple people log into the "control panel"? Anyone you don't trust to SSH into a shell account usually shouldn't be trusted to log into a Wordpress or VBulletin control panel either in my opinion. Run Lisp as a non-privileged user (probably should be doing that anyways, I am) and run SSH in a chroot if you really care.

What about the typical web gallery feature of letting you upload an image and then having it automatically thumbnailed? I wrote a two line shell script that uses ImageMagick (which is likely what a PHP-run gallery would be using anyways) to thumbnail all my photos locally, then rsyncs them to my server. Why re-invent the wheel?

Couldn't Perl or Ruby or PHP do the same thing? Well, Perl/Ruby/PHP don't run persistently on the server. So it'd be a bit different. Couldn't you write some standalone scripts to run from the commandline to insert new posts into your blog? I suppose, but it surely wouldn't be as nice an interface as Slime in Emacs, unless you enjoy using bash as your text editor. And have fun with quoting / escaping. (EDIT: As some readers reminded me, yes they can run persistently via Apache hackery. And Rails has a "console". I stand corrected.)

The REPL obviously isn't a solution to every problem. If your authentication requirements are complex enough, you'll have to build something yourself. And it's a problem if your users don't know Lisp (and yeah, that right there kills it for 99.999998% of the world). Web-based control panels have the benefit of being "so easy anyone can use them".

But for my needs, and probably the needs of a great many websites run by one or two trusted Linux-savvy people who just need to be able to securely update the site once in a while, a few Linux tools + the REPL works beautifully.

February 11, 2008 @ 2:06 PM PST
Cateogory: Programming

26 Comments

Joey
Quoth Joey on February 11, 2008 @ 5:45 PM PST

I like your gallery, except that the order "jumps" oddly in switching pictures. I know they're ordered 0-n, but by taking the picture I'm on out of the list, the pictures jump around. Leave all the thumbnails available all the time, and either don't indicate which one I'm on (I think I can tell) or dim/point at/highlight the picture I'm on.

Brian
Quoth Brian on February 11, 2008 @ 6:24 PM PST

What it's supposed to do is always show the picture you're currently on in the middle of the row of thumbnails. Except when you hit the beginning or end of the list of thumbnails, in which case it "stops" scrolling the list (so it always shows five thumbnails minimum). Problem is that there are only six pictures in the gallery total right now so it's kind of pointless. When there are many more pictures it might make more sense. It actually does highlight the picture you're currently on, in white, but it's not very noticeable.

But yeah it's confusing. I thought a long time about what algorithm to use for that row of thumbnails (sadly) and probably didn't come up with the best one. I may just resort to showing a random selection down there, it'd be better than confusing everyone. Thanks for letting me know.

Michael Jung
Quoth Michael Jung on February 11, 2008 @ 6:33 PM PST

I hope sometime I get as far as you already. Really appreciate reading your blog. Am trying to learn Lisp myself now for more than a year but have to interrupt so often due to my job (freelancer developing software for M$ Win ...). Moved working env to Mac OS X recently, office server is running Gentoo already for more than 5 years now.

Jon
Quoth Jon on February 11, 2008 @ 11:05 PM PST

"Couldn't Perl or Ruby or PHP do the same thing? Well, Perl/Ruby/PHP don't run persistently on the server."

In the case of ruby on rails, it does run persistently on the server (with fcgi) and you can open a console to do what you described (script/console). Same for python/django (manage.py shell).

numerodix
Quoth numerodix on February 12, 2008 @ 12:04 AM PST

So this lisp you have running is some kind of daemon that accepts programmatic requests as well as serving up pages?

Masklinn
Quoth Masklinn on February 12, 2008 @ 1:08 AM PST

re Well, Perl/Ruby/PHP don't run persistently on the server.

I fail to see why you'd need to connect to an "always running" process to do what you're stating. Django provides a Python shell hooking straight into the project, so it's possible to do exactly what you're saying very easily. Since all the state of the application is in the DB and the shell allows the user to edit the db's content (going through the project's models and everything, not hitting the db straight), it works perfectly.

Also, Django has a very nifty feature: it can generate an administration interface for the user, just so you don't have to go through SSH.

And I think that latter feature more than makes up for the lack of Slime.

Sp3w » Blog Archive » Linkage 2007.02.12
Quoth Sp3w » Blog Archive » Linkage 2007.02.12 on February 12, 2008 @ 1:44 AM PST

[...] A Lisp-driven website [...]

tum
Quoth tum on February 12, 2008 @ 2:21 AM PST

Your idea with the ssh/emacs backend is fine as long as technical users (like you and me :) administer the site, but you need to have a web-gui to give control to a non-technical site-admin (content admin = client).

jrockway
Quoth jrockway on February 12, 2008 @ 4:22 AM PST

tum: not all software is designed for non-technical users. I wrote a blog engine that only lets you post by placing GPG-signed text files in a certain directory. It has plenty of users.

Brian
Quoth Brian on February 12, 2008 @ 4:34 AM PST

numerodix: Yeah, exactly.

Masklinn: All the state in my site isn't in the database, though I didn't say much about that here. I don't actually have to use a database at all. The database is only for persistence in case I shut down Lisp and want to start over with the same data later. There are a lot of persistence libraries that don't use an SQL database at all. Most of the state in my site is just variables and functions initialized in the running Lisp.

Jon: Interesting. I've only ever done Rails apps using a bunch of Mongrels.

nothingmuch
Quoth nothingmuch on February 12, 2008 @ 5:33 AM PST

Well, Perl/Ruby/PHP don't run persistently on the server

This claim is preposterous.

These three language can all run either persistently or not so, in a web environment or any other environment, and they can all implement RPC features.

Almost any language that has been around long enough has fastcgi bindings, apache modules, or libraries for writing a micro webserver (Mongrel is a well known example). The only requirement is socket support.

While SLIME is indeed remarkable, the fact that you can operate your blog has nothing to do with LISP's superiority.

Furthermore I find it amusing that you think a two step upload process is better than "reinventing the wheel". Such a wheel can be easily borrowed form other places, or even just installed on your system as part of a photoblog that wasn't reinvented.

Your claims are unresearched and your conceptual understanding seems lacking too, so this all sounds like zealotry to me.

To summarize, LISP is just another language. It's got cool features, and is supported by great tools, but that doesn't mean that other languages are inherently flawed by comparison.

Masklinn
Quoth Masklinn on February 12, 2008 @ 5:42 AM PST

re: Masklinn: All the state in my site isn?t in the database, though I didn?t say much about that here. I don?t actually have to use a database at all. The database is only for persistence in case I shut down Lisp and want to start over with the same data later. There are a lot of persistence libraries that don?t use an SQL database at all. Most of the state in my site is just variables and functions initialized in the running Lisp.

Would work that way with smalltalk images, but the result's the same: I have no problem using an REPL to update a Django or Rails application, going through all the models check and everything, never hitting the datastore directly and always ensuring complete coherence of the application's data and state

Brian
Quoth Brian on February 12, 2008 @ 6:12 AM PST

nothingmuch: Calm down. Everything is going to be OK.

Yes Lisp is just another language, I never claimed otherwise. No need to bash straw men. No I don't think other languages are "inherently flawed". I use Ruby and Perl all the time. I have a site that runs on Rails.

My "conceptual understanding" or lack thereof is not evident based on one blog post, unless you're a mind reader. I'm aware of mod_perl and fastcgi etc. I still feel that Lisp is somewhat different from those things. My choice of words was poor. I've corrected my post. Your use of ad hominem however is unnecessary and silly.

To summarize, Lisp lets you do neat things that are somewhat different from the way other languages do things.

Masklinn: I suppose it can be pretty similar. I'll have to look into Django, I've heard good things about it.

jim Thompson
Quoth jim Thompson on February 12, 2008 @ 6:50 AM PST

While Brian didn't provide any support for his assertions, its NOT true that "Lisp is just another language".

Lisp is the godfather of computer languages.

giles bowkett
Quoth giles bowkett on February 12, 2008 @ 8:53 AM PST

Ruby on Rails has had this since at least 2006, probably 2005 or earlier.

A friend of mine coded a Ruby library which additionally enables you to open the REPL over TCP/IP from within a running application.

(We call the REPL a console, however. Crazy Rubyists and their impenetrable jargon!)

Brian
Quoth Brian on February 12, 2008 @ 9:09 AM PST

Now that I look more closely at Rails' "console", it is much more similar to a Lisp REPL than I thought. (Though not quite the same.) I stand corrected.

Common Lisp jargon is very thick, yes. It's quite annoying at times.

Jonathan Allen
Quoth Jonathan Allen on February 12, 2008 @ 10:11 AM PST

>You could SSH to your server and run SQL queries directly to input your blog posts. One reason we don't do this is because it bypasses the logic of your website.

> The REPL is my control panel. I can implement the necessary logic as simple Lisp functions.

Um, I fail to see how using SQL to bypass the normal logic of one application is a bad thing but using Lisp somehow makes it alright.

jim Thompson
Quoth jim Thompson on February 12, 2008 @ 10:51 AM PST

> Um, I fail to see how using SQL to bypass the normal logic of one > application is a bad thing but using Lisp somehow makes it alright.

With Lisp, its all (just) data. Sometimes you execute it.

Until you grok this, you don't understand why Lisp is different.

Jim

Brian
Quoth Brian on February 12, 2008 @ 10:56 AM PST

Lisp provides a higher-level interface just like a web control panel would. So e.g. I can refer to a post by its title rather than by its id in the database. Using Lisp isn't bypassing anything because Lisp is where the logic is implemented.

The same Lisp process also runs the web server itself in another thread. And it has all the low-level DB-interface stuff, if I wanted to stick my nose into it and screw something up. Etc. etc. It kind of does it all, top to bottom. But when I just need to make a new post I stick to the higher-level interface functions (that I wrote for myself).

jim Thompson
Quoth jim Thompson on February 12, 2008 @ 11:21 AM PST

Our policy of fixing bugs on the fly changed the relationship between customer support people and hackers. At most software companies, support people are underpaid human shields, and hackers are little copies of God the Father, creators of the world. Whatever the procedure for reporting bugs, it is likely to be one-directional: support people who hear about bugs fill out some form that eventually gets passed on (possibly via QA) to programmers, who put it on their list of things to do. It was very different at Viaweb. Within a minute of hearing about a bug from a customer, the support people could be standing next to a programmer hearing him say "Shit, you're right, it's a bug." It delighted the support people to hear that "you're right" from the hackers. They used to bring us bugs with the same expectant air as a cat bringing you a mouse it has just killed. It also made them more careful in judging the seriousness of a bug, because now their honor was on the line.

jim Thompson
Quoth jim Thompson on February 12, 2008 @ 11:22 AM PST

ya know, I had full attribution in the above (is Paul Graham, in "The Other Road Ahead".

http://www.paulgraham.com/road.html

Please fix your blog software.

jim

Masklinn
Quoth Masklinn on February 12, 2008 @ 8:38 PM PST

re: Um, I fail to see how using SQL to bypass the normal logic of one application is a bad thing but using Lisp somehow makes it alright.

His point was that by using Lisp he does not bypass the application logic. He just performs application logic calls directly in the REPL/console/interpreter instead of going through an admin interface (and thus having to create that interface in the first place).

Thus, all the consistency checks and automatic manglings and everything (such as automatically generating slugs from titles and stuff like that) is performed as and when it should, and the internal consistency of the application isn't at risk of being compromised (which would be the case by doing raw SQL queries, unless you're using a good db such as Postgres and actually thought when you created your DB models).

Chris Barts
Quoth Chris Barts on February 13, 2008 @ 1:04 AM PST

As per Alan Cox, "chroot is not and never has been a security tool." If you are using chroot as a security tool, you are misleading yourself with the sweetest poison known to sysadmins: A false sense of security.

aggieben
Quoth aggieben on February 18, 2008 @ 12:08 PM PST

Brian, can we see the sources to your origami site? I'm working on a webapp using Lisp, and I'd like to see how you did some things...

Brian
Quoth Brian on February 18, 2008 @ 12:13 PM PST

I may post it eventually. Right now it's just a mess though. Probably nothing in there is of much use to anyone. If you search for "lispcast" or "create a lisp blog" you get some good tutorials though. See for example this one and this one.

-= Linkage 2007.02.12 =-
Quoth -= Linkage 2007.02.12 =- on January 25, 2009 @ 11:38 PM PST

[...] A Lisp-driven website<br/> [...]