This is a read-only archive!

Depressing

In spite of my previous efforts I've still been getting memory exhaustion errors on my Lisp-driven website. (If you click this link, you'll notice that the site is down. Read on.)

It's not a lot of fun having to kill and restart SBCL every 2 days. So I decided to restart SBCL one last time and NOT load my website, or any of my own code at all. I just loaded Hunchentoot with its default "you haven't made a webpage yet" index page. I left it running for three days. Lo and behold, I still get memory exhaustion errors. And errors about hitting a limit on the number of threads that are allowed. And other errors I don't really even understand. Given the fact that I myself am probably the only visitor of this site, this doesn't make me feel well for what would happen if the site ever got more popular.

So yeah, this is depressing. Not to say there's anything wrong with Common Lisp, SBCL, or Hunchentoot. But probably it's safe to say that the combination of those three things plus a VPS host is not workable. At least not for the VPS host I use. Or at least, I don't know how to make it work, in spite of my best effort. I'm tempted to throw some money at the problem and get a VPS with more RAM. But I draw the line right about there. Why should such a thing ever be necessary? (And would it work, anyways?)

Of all the problems I thought I'd have running Common Lisp on my server, running out of memory wasn't one that I ever expected. Is this one of the risks you take running something that's so far out of "mainstream"? If more people used CL, probably more people would've had this problem and solved it already. I really do think that the ease of deploying a web framework is one of the biggest determining factors of whether people will use it, and CL fails horribly in this respect when compared with everything else I've ever used.

I'm debating whether to change hosts, upgrade my VPS on my current host to have more RAM, or just rewrite my site in PHP or Ruby. I'm leaning toward rewriting the site at the moment. PHP and RoR both work just fine on my server, and have for a year with not a single alert about memory exhaustion. It'd probably take me an evening or two to rewrite, compared with the month and a half it took me to do it in Common Lisp. I'm not sad I gave it a shot anyways. CL is a nice language.

March 02, 2008 @ 10:22 AM PST
Cateogory: Programming

11 Comments

HR
Quoth HR on March 02, 2008 @ 4:38 PM PST

Did you try putting it behind a proxy? Its something that requires virtually zero effort to try.

ProxyPass /hunchentoot http://127.0.0.1:3000/hunchentoot ProxyPassReverse /hunchentoot http://127.0.0.1:3000/hunchentoot

The theory being that it might serve as a buffer of sorts to hide some of the underlying problems / symptoms, until they can be worked out in detail.

michael
Quoth michael on March 02, 2008 @ 5:03 PM PST

Personally I am using old .cgi pages written in ruby. It already works a lot better than .php with the single exception that it takes a bit longer for them to load (but since I never have a lot of visitors anyway, this disadvantage is a non-issue for me, and the time saved by NOT using php is a lot as well. I feel that ruby as language empowers much more than php does so)

If you dont need all of RoR there is still ramaze and rack, rack may even go as far as be included in a future official ruby release (matz wants to find an alternative to the aging CGI in standard ruby)

Peter
Quoth Peter on March 02, 2008 @ 5:11 PM PST

I've started using Ruby recently for simple web sites. I had some reservations at the beginning, but in the end it works quite well. I am using Rack with CGI handler for a simple forum that is replacing PhpBB. I have a moderately visited site and so far all is good.

Ciaran McCreesh
Quoth Ciaran McCreesh on March 02, 2008 @ 5:58 PM PST

Your problem is that you're using a programming language where memory usage isn't guaranteed or controlled by the programmer. Haskell's probably the worst offender, but all of the common functional languages try to be (and have to be, since the programmer can't) clever with caching results and doing lazy single evaluation. The price you pay for having that all done for you is that you can't easily stop it from happening.

Marijn Haverbeke
Quoth Marijn Haverbeke on March 02, 2008 @ 6:26 PM PST

Where I work, we are using Apache, mod_lisp, Hunchentoot, and SBCL for a webservice that gets something like a million hits per day. We used to have some issues with threading, but never with running out of memory.

Sven Van Caekenberghe
Quoth Sven Van Caekenberghe on March 02, 2008 @ 6:51 PM PST

You must be doing something obviously wrong - or the combination of the tools you are using has some flaw. I would start by asking questions on the Hunchentoot mailing lists with much more detail. I have several common lisp web apps in production use, some of my lisp images have been running for over a year without restarting, so there certainly is no general memory problem.

David Schneider-Joseph
Quoth David Schneider-Joseph on March 02, 2008 @ 7:19 PM PST

I've implemented an SBCL back-end for a web site with hundreds of thousands of users. I also encountered heap exhaustion problems. Try running with, for example, "--dynamic-space-size 1024". That doubles what I believe is the default of 512 MB. Don't worry, you still won't use that much resident memory unless necessary.

David Schneider-Joseph
Quoth David Schneider-Joseph on March 02, 2008 @ 7:20 PM PST

(that's two dashes in "--dynamic-space-size". The comment formatting made it look like a single emdash.)

gwenhwyfaer
Quoth gwenhwyfaer on March 02, 2008 @ 11:19 PM PST

Which kind of VPS is it? I have a tiny VPS (64Mb, linux-vserver) that ran into memory exhaustion problems when I tried to run thttpd on it - and heaven knows, thttpd is not a memory-hog! I eventually reached the conclusion that it was something to do with select() or something related to it; I switched over to mini_httpd (which forks for each connection) and haven't had any problems. (All of these were compiled with dietlibc, incidentally.)

So Lisp's memory management might not be to blame; the way it's listening for connections might have more to do with it.

mattrepl
Quoth mattrepl on March 03, 2008 @ 12:47 AM PST

I use SBCL and Hunchentoot for two sites, both on different VPS providers (OCS Solutions and Slicehost) and have not had problems like you describe. Both have been running for months without restarts (very low traffic, though).

The only time I ran into a problem was when I wasn't properly using CLSQL between threads, the server would crash when stress-testing with httprof. I hope your problem is an easy fix.

sclv
Quoth sclv on March 03, 2008 @ 4:04 AM PST

a programming language where memory usage isn?t guaranteed or controlled by the programmer

This is just FUD. Haskell provides a great set of tools for dealing with laziness, strictness, and leaks. And furthermore, that's not even the problem you're describing, which sounds like a common issue of all modern, garbage-collected languages (i.e. anything not c++) -- garbage collection can't operate on things which there are still references to in memory because, well, because duh. And in any case, assuming your runtime garbage-collection is decent, it does and will have provable, controllable characteristics, assuming you have a clue what you're doing.

None of the issues you mention, by the way, have to do with "caching results" or "single lazy evaluation" whatever that means. Lazy evaluation of thunks, for one, does not involve being "clever" at all, but rather a fairly intuitive semantics. And the one issue that does arise with this, accumulating a long chain of thunks, leads only to stack issues, which are actually startlingly distinct from memory issues.