briancarper.net (λ) (Tag: Debian)

SBCLish host wanted

Mon, 03 Mar 2008 13:44:08 -0800

After sleeping on it, I think I'm going to switch hosts and try to keep going with SBCL and Hunchentoot.

If you're reading this and you know of a VPS unmanaged web host that's known to work well with SBCL, could you please comment and let me know what host you use, and what plan at that host, if it offers more than one plan, and what OS you prefer? So far I've heard Slicehost and OCS Solutions suggested. (Slicehost offers Gentoo. Do I dare?)

Note to all, my current host for my SBCL-driven site is FutureHosting. I'd advise people to avoid them if you plan to run SBCL. Their support and service is usually pretty good as long as you don't push it too far, but their servers are pretty wonky at times and after the comments from my last post, I wouldn't be surprised if a lot of my problems were due to some kind of VPS misconfiguration. Their support staff told me nothing more than "pay for more RAM" when I informed them of the problems I was having.

Yikes!

Wed, 20 Feb 2008 23:49:25 -0800

My origami website server kept hitting a limit on "privvmpages", which is a cap (by the VPS) on some aspect of system memory. SBCL died with some interesting notes about heap exhaustion. I already had to hack SBCL even to get it to start on my server. And now this. I'm running a VBulletin + a Rails app + my Lisp site but none of them are high-traffic and you'd think the server could handle it. I figured I must be doing something wrong.

Luckily lots of other people apparently were also doing it wrong. Google for "vps sbcl" and you'll find plenty of people having difficulty running it. Through googling I learned of the handy ROOM function which tells you some things about SBCL's current memory usage. And also SB-EXT:GC which in SBCL appears to invoke the garbage collector immediately.

Dynamic space usage is:   72,748,936 bytes.
Read-only space usage is:      1,912 bytes.
Static space usage is:         2,736 bytes.
Control stack usage is:        1,244 bytes.
Binding stack usage is:          336 bytes.
Control and binding stack usage is for the current thread only.
Garbage collection is currently enabled.
etc.

Is 72MB good or bad? I don't know, but I'd guess bad, given that my server was dying. So next I went through all of the packages my package depended on, and loaded them manually one by one to see how much usage was increasing for each. CL-WHO + CLSQL + CL-FAD + CL-PPCRE + HUNCHENTOOT were increasing memory usage by about 10MB total, apparently. My own package, which does nothing but run my website, was increasing it another 21MB. Ouch. Clearly I was doing something very wrong.

So then I started loading each of the functions in each of my own files one by one, in order, to try to account for those 21MB. This is easy to do in Slime via C-c C-c. After each I invoked the GC and checked my memory usage again. I noticed that one inconspicuous function was adding around 9MB to that number. It was a function where I was doing a glob of a bunch of filenames, then removing every filename that matched the string "thumbs". Here's the relevant part:

(remove-if #'(lambda (p) (cl-ppcre:scan "thumbs"
                                        (namestring p)))
           (cl-fad:list-directory (merge-pathnames (name model)
                                                   *photo-dir*)))

When I changed it to this, my memory usage went down by 9MB:

(let ((thumbs-regex (cl-ppcre:create-scanner "thumbs")))
  (remove-if #'(lambda (p) (cl-ppcre:scan thumbs-regex
                                          (namestring p)))
             (cl-fad:list-directory (merge-pathnames (name model)
                                                     *photo-dir*))))

Yikes! Still not sure what was going on there. Perhaps it would be best if I didn't rely so heavily on regular expressions to begin with.

I've heard said that Lisp can be dangerous because expensive things don't necessarily look expensive. Code that looks short might macro-expand into enormous amounts of code and you may never notice. Though I imagine the same can be true of any language, and normal function calls can just as easily hide a lot of complexity. I can't blame Lisp, I can only blame myself for not profiling my code better. Lesson learned to be more careful, I guess. I just hope this helps fix my memory exhaustion problems. Otherwise back to the drawing board.

PAIP; Lisp Lisp Lisp

Thu, 31 Jan 2008 02:31:48 -0800

I ordered PAIP today. Good computer books are so expensive. It hurts me to spend $80 on a book. But there are many worse ways to spend $80 than on something which contains so much knowledge, I guess. I got a $25 gift card for a book store for Christmas, so may as well put it to good use. I've read good reviews of PAIP, so I hope it lives up to them. I had only one AI course in college, and it was taught in C (ugh) by a grad student, and I learned little to nothing from it. It'll be nice to revisit the subject. (And learn more Common Lisp, of course.)

In other non-news, my Common Lisp-powered website is still up and running after a few days, no crashes or spontaneous hard drive reformats or anything, so that's good. Deploying it was interesting. Actually compared to the one Ruby on Rails site I deployed, Hunchentoot was far easier to deploy in many ways.

When you deploy Rails you have the problem that Rails is a beast of a framework, and Ruby is slow as a dog to begin with. So there are all kinds of silly nasty ugly hacks to get it to run with acceptable performance. Personally for my Rails site I have to run three Mongrel servers and use Apache's mod_proxy_balancer to send traffic to them. Some other people use fastcgi or such CGI-caching scripts, which I never could get working well. Just to run Rails apps at all, there are all kinds of Apache mod_rewrite hacks going on to get routes to work. Etc. etc. etc. It's a huge mess and I pray my Rails server never dies because I sure don't want to have to go through to pain of re-deploying it now that it's up and running.

With Hunchentoot on the other hand, you start Lisp once and it runs forever. No Apache to screw with means happy times are had by all. In fact my Hunchentoot-run website does exactly what Rails routes do, all via a single function that parses the request URI and dispatches based on it. With no .htaccess wizardry or cryptic Rails config files anywhere in sight.

Every request to Hunchentoot is apparently handled in a Lisp thread, so concurrency issues are generally all taken care of for you, no worries. In fact since you have Lisp running forever you can get away with crazy things like using Lisp objects as a sort of cache for data instead of hitting your database for every request, which I imagine probably works wonders for performance. I'll never know for sure, because my site will never ever be popular enough to put it to the test, but one can dream.

There were a few snags deploying my Hunchentoot server, none of which were the fault of Lisp, more the fault(?) of Linux. One was that I wanted to bind Hunchentoot to port 80. As you may know, only root has the right to bind a program to a "privileged" port like 80. I certainly did not want to have a Lisp image permanently running on my server as root, for security reasons. Thankfully Debian has a drop-in solution for this in the form of authbind, which lets normal users bind programs to privileged ports.

The second problem is that Common Lisp via SBCL is a sort of interactive program (see also, the REPL), and isn't really made to be run as a daemon, at least not the way I usually run it. There are many solutions to this however, and the solution I chose was detachtty, which does exactly what the name says. As the linked site describes, it's like a very stripped-down sort of GNU Screen.

Then there's the issue of how I can log into that running Lisp image from my home computer to run commands. I could easily SSH to the server, and use attachtty (which comes with detachtty) to attach to SBCL directly and type commands there. But SBCL's built-in REPL is a bit primitive. SLIME would be nicer. The solution here is to start Swank on the server. then use Slime running on my local machine to connect to the remote Swank.

This is, again, extremely simple. In a previous entry I linked to a movie that describes in great detail how this can be done, start to finish. You can see another more thorough example of how to do it here. Essentially you run SWANK:CREATE-SERVER in the remote SBCL; Swank binds only to localhost on the server, so to use it from home, you then have to make an SSH tunnel. Local Emacs can slime-connect to the remote Swank via the tunnel.

So that's it. authbind starts detachtty, which runs SBCL, which --loads a little 5-line Lisp script that imports some libraries and my photo blog source code, then starts Swank, and then starts my server. A little init script to make this happen by default every time the server reboots, and there's nothing more to be done. There's no real need to restart the server from now on, since I can log into the REPL and run commands and update the running program on-the-fly. But if I had to restart it or redeploy it entirely, it would be no problem.

This strikes me as an example of one of the great things about Linux. Many small tools that all do a single job very well. A tool to let non-root users bind port 80, a tool to let you run any program as a daemon, the glue to make those things work together with SBCL, and a great text editor to talk to SBCL after it's running. It all works beautifully, and there are no mysteries anywhere in the process start to finish.

SBCL on Debian (sucks)

Sat, 26 Jan 2008 20:44:01 -0800

In my oh-so-neverending quest to learn Lisp and make a webpage using it, today I actually bothered trying to install SBCL on my server, which is a VPS running Debian.

Failure. I got errors like this:

mmap: Cannot allocate memory
ensure_space: failed to validate 536870912 bytes at 0x09000000
(hint: Try "ulimit -a"; maybe you should increase memory limits.)

This is a well-known error involving CMUCL and SBCL and Debian (and some other distros), especiall when running from a VPS apparently. Lots and lots and lots and lots of people have this problem, seemingly. Reading through all of that, it's hard to figure out whether it's a bug in SBCL, a bug in Debian's package, or a bug in the kernel.

Given that I'm on a VPS on someone else's server, memory and other resource caps are set and I can't touch them, so a lot of the (probably unhealthy) hacks that some people list as solutions above don't work for me.

Fortunately there's a solution posted to one of the SBCL lists. Download the SBCL source, and edit src/compiler/x86/parms.lisp, find the section for your OS and change one of the lines to read:

(def!constant dynamic-space-end         #x10000000)

Then it's a matter of recompiling SBCL. You have to do this on a machine that already has a working SBCL (or other Common Lisp) installed. The install instructions on the website are not as complete as the INSTALL that comes in the source tarball. The included INSTALL doc has some important things you need to do if you want to compile SBCL with threading enabled. (Note: You probably do want threading enabled.) If I'd read that first it would've saved me 30 minutes worth of compiling an SBCL that I couldn't use for Hunchentoot due to no threading.

Here is yet another example of where a binary distro fails and fails hard. If this was Gentoo I could've tweaked an ebuild and thrown it into an overlay and taken care of most of this bookkeeping crap for me. With Debian I get to compile everything by hand and SFTP files around between servers and look up to make sure I have install prefixes set correctly and so on and so on. With Gentoo I wouldn't have forgotten to enable threads because I'd have a USE flag right in front of my face.

In any case the above mess at least got me a working SBCL that let me install and run Hunchentoot. Time will tell whether it's going to survive when I actually start hosting a web page on it, or whether reducing SBCL's memory allocation is going to starve and kill my program.

CentOS + CPanel sucks

Fri, 31 Aug 2007 18:47:31 -0700

Recently I switched one of my websites to a VPS server. It's so much better than shared hosting. Even if I do still share a physical machine with N other people, as a VPS I have full root access which makes things much better. I made the mistake of trying "managed" hosting first, which means the tech people at the hosting company help you set things up.

Bad idea. That plan was on CentOS 4.2 and used CPanel. CPanel doesn't even support Apache 2 yet. Let alone Ruby on Rails. They've apparently been promising Apache 2 support for months and have yet to deliver except in their experimental branches. The CPanel forums are littered with people complaining about it. Running a Rails app with any kind of acceptable speed demands some features of Apache that Apache 1.x does not have. Apache 1.x has fastcgi, but it sucks and judging by mailing lists and my own experiences, getting it to actually work is a miracle. The people at my hosting company surely did not succeed in doing it.

So I "downgraded" my hosting to one where I run everything myself (it's $15/month cheaper) with no CPanel, and got Debian 4, and now everything works fine. Took me one evening to get it all going. And now I don't have the useless overhead of CPanel to deal with. I run Apache 2.2 and a cluster of Mongrel servers for my Rails app and the speed is much better.

In addition to that, I had to set up EVERYTHING on my server. So I got my first taste of setting up a DNS server (bind) , which was a really good learning experience. I never knew how DNS worked at that level of detail until I set it all up myself. So I set it all up, then ran some tests, and found out I did it entirely wrong, so I redid it chrooted and disabled DNS recursion and all those bad things and now hopefully I won't be cracked within a week.

Then I had to set up sendmail. I accomplished this by glancing at the horrific configuration file and then immediately uninstalling it from the server. I don't need another mail server that badly. In the end, my opinion of CentOS + Cpanel is pretty low. The CentOS repository was severely lacking in a great many packages that I needed. I had to resort to compiling crap on my own, and it didn't work. Debian at least has mostly everything I want via apt-get. And CPanel is severely limiting if you want to do anything outside of the narrow bits of software it officially supports. It sucks hard for running Rails.

Layout update

Wed, 25 Oct 2006 23:54:18 -0700

I've made some fairly significant changes to my blog's layout. Most importantly, I've added a cow's bum. I've said it before and I'll say it again: You simply can't go wrong with cow-based web site layouts. I still have a bit more work to do (for another day) but I think it turned out OK.

This went smoothly largely thanks to Debian. I spent an hour or so today setting up my Debian box as a mirror of my blog. It really was simple and easy and most of all extremely fast. I had some weird problems though; for example apparently bzip2 isn't installed on Debian by default (???). That's a first. But hey, it installs in 5 seconds, so it's not a huge problem. Mmmm, non-source-based distro. Samba also was extremely easy to set up. I noticed that Debian's samba comes with a rather more sane default config file than Gentoo's, which is something I think I've noticed about Debian in general.

Debian?

Mon, 09 Oct 2006 11:32:03 -0700

So scrapped FreeBSD on my secondary computer. I figured I don't need to play with yet another source-based OS. I was a bit confused by the odd mixture of packages I had to compile myself, and packages I could install in binary form. There were also a large number of tools that could be used as an interface to installing packages.

So I went with Debian. It has a reputation for stability etc. right? We'll see. I'm also sort of testing the waters again for a potential jump-ship to Ubuntu. We'll see. The Debian install was mostly uneventful. I installed from a single DVD. There were two DVD ISOs available for download; I couldn't find anywhere that indicated the contents of either one, so I figured the second probably only had additional packages on it, or whatever. I guessed right, I imagine.

After booting from the DVD, the installer died after about a minute, complaining about a corrupt package on the DVD. I ran the "Verify CD contents" and it confirmed, corrupt disc or package on the disc. (bochs, I think.) I know the ISO was fine, I checked the md5sum. So I rebooted. Same DVD, second run, it worked fine. Some kind of ghost problem? Flaky DVD-R? Who knows.

I don't like that Debian installed an X server by default. I chose to do a "server" install, and I picked File, Print, Web, SSH server from the options. Somehow this still installed X. I was happy though at how easy it was to uninstall. apt-get remove xfree86 or something similar, and it removed all the dependencies. A nice change from what I'm used to. The whole install took maybe 45 minutes. When I boot, I get a ton and a half of kernel module errors though. That's the price you pay when you don't compile your own kernel, I guess. Everything still works, so I don't know how much it matters. For now.

A very minor observation: For some reason I like Debian's networking config file much more than Gentoo's.

Debian's /etc/networking/interfaces:

auto eth1
iface eth1 inet static
    address 192.168.15.7
    netmask 255.255.255.0
    gateway 192.168.15.1

Gentoo's /etc/conf.d/net:

config_eth0=("192.168.15.7 netmask 255.255.255.0 broadcast 192.168.15.255")
routes_eth0=(
    "default via 192.168.15.1"
)