Via this page I learned of the existence of apachebench, which was installed on my system already (probably along with Apache2) as
/usr/sbin/ab. This tool lets you hammer a server and see how it handles concurrent requests etc. Long before I read this diatribe I knew that I didn't know crap about statistics and that benchmarks in the end are of marginal use to me, but apachebench lets me partially answer the questions I had about how SBCL+Hunchentoot handles concurrent requests.
I got mixed results. Hunchentoot happily gives birth to worker threads to handle all the requests, so that answers one question for me. But when running Hunchentoot behind mod_lisp and Apache, when I hit it with too many requests at once I get dropped into the debugger in the REPL. (Actually, I get dropped simultaneously into tens or hundreds of debuggers. Ouch.) And I get all kinds of errors spewed into my logs. Things about broken pipes, missing sockets, dead streams, unbound HUNCHENTOOT:REPLY variables, you name it.
One thing I looked at first was whether my packages were up to date. This brings me to a side rant: They were not, because many / most of the CL packages (even unstable) in the Portage tree are really really out of date. Because this is Gentoo, it's not hard to version-bump some new ebuilds into
/usr/local/portage myself, which I did, but I didn't realize the tree was lagging this much.
(There is (was?) a Gentoo "Common Lisp Portage Overlay project" linked from CLiki but it seems entirely abandoned since 2006. Apparently there was a lone dev (Matthew Kennedy) back in 2006 who was really pushing a lot of CL stuff for Gentoo but he quit in January 2007. But according to the gentoo-lisp mailing list, there's a newer git overlay repo that's serving the same purpose nowadays.
It's kind of sad how much detective-work I needed to do to track this information down. I'm sure the Gentoo Lisp-devs are doing the best they can though. If only I had the skill, knowledge, time, and ambition (in that order of importance) to help out myself.)
In any case, upgrading packages to current versions didn't help. Even loading a fresh SBCL image and loading hunchentoot and starting a server and serving just the default Hunchentoot "Hey you have no pages defined" page, I get a thread massacre and SLIME is debugger-bombed even if I simply hit refresh in my web browser a bunch of times quickly.
The good news is that if there's no human around to handle these thread errors / debugger sessions in the REPL, the threads quickly timeout and die off. This is probably what I'd want to have happen on a server I wasn't monitoring 24 hours a day. The bad news is that requests dying left and right isn't that great when you're a user waiting for a page to be displayed.
However the other good news is if I just use Hunchentoot by itself and ditch mod_lisp and Apache, I have no problems at all. Requests are handled quickly and even with lots and lots of concurrent requests in apachebench I get no thread-slaughter. So it seems likely that the problem is with mod_lisp, or with the communication between it and Apache and SBCL.
When it comes right down to it, why do I even need Apache? I don't really. My first thought was that Apache was hogging up port 80 on my machine and I don't want a Hunchentoot-driven site to have my URL always include some non-standard port, but then I remembered I pay for two IP addresses with my host anyways, so I can use one IP for my other, Apache-driven sites and dedicate the other IP to Hunchentoot.
If I can set that up, why use Apache? There's no real reason other than the one that bites me so often as I've been learning Lisp: It's what I'm used to, it's what I'm comfortable with. The idea that a hierarchy of web pages are actually all coming from a single Lisp script running in a single persistent Lisp image that's parsing and fiddling with request URI strings and calling functions appropriately, is very different from the naive filesystem directory tree full of HTML files or PHP scripts that I suppose I usually think of when I picture a web page. It's different, but also very nice in many ways.
And so the journey continues. We'll see what tomorrow has in store.