This is a read-only archive!

Yikes!

My origami website server kept hitting a limit on "privvmpages", which is a cap (by the VPS) on some aspect of system memory. SBCL died with some interesting notes about heap exhaustion. I already had to hack SBCL even to get it to start on my server. And now this. I'm running a VBulletin + a Rails app + my Lisp site but none of them are high-traffic and you'd think the server could handle it. I figured I must be doing something wrong.

Luckily lots of other people apparently were also doing it wrong. Google for "vps sbcl" and you'll find plenty of people having difficulty running it. Through googling I learned of the handy ROOM function which tells you some things about SBCL's current memory usage. And also SB-EXT:GC which in SBCL appears to invoke the garbage collector immediately.

Dynamic space usage is:   72,748,936 bytes.
Read-only space usage is:      1,912 bytes.
Static space usage is:         2,736 bytes.
Control stack usage is:        1,244 bytes.
Binding stack usage is:          336 bytes.
Control and binding stack usage is for the current thread only.
Garbage collection is currently enabled.
etc.

Is 72MB good or bad? I don't know, but I'd guess bad, given that my server was dying. So next I went through all of the packages my package depended on, and loaded them manually one by one to see how much usage was increasing for each. CL-WHO + CLSQL + CL-FAD + CL-PPCRE + HUNCHENTOOT were increasing memory usage by about 10MB total, apparently. My own package, which does nothing but run my website, was increasing it another 21MB. Ouch. Clearly I was doing something very wrong.

So then I started loading each of the functions in each of my own files one by one, in order, to try to account for those 21MB. This is easy to do in Slime via C-c C-c. After each I invoked the GC and checked my memory usage again. I noticed that one inconspicuous function was adding around 9MB to that number. It was a function where I was doing a glob of a bunch of filenames, then removing every filename that matched the string "thumbs". Here's the relevant part:

(remove-if #'(lambda (p) (cl-ppcre:scan "thumbs"
                                        (namestring p)))
           (cl-fad:list-directory (merge-pathnames (name model)
                                                   *photo-dir*)))

When I changed it to this, my memory usage went down by 9MB:

(let ((thumbs-regex (cl-ppcre:create-scanner "thumbs")))
  (remove-if #'(lambda (p) (cl-ppcre:scan thumbs-regex
                                          (namestring p)))
             (cl-fad:list-directory (merge-pathnames (name model)
                                                     *photo-dir*))))

Yikes! Still not sure what was going on there. Perhaps it would be best if I didn't rely so heavily on regular expressions to begin with.

I've heard said that Lisp can be dangerous because expensive things don't necessarily look expensive. Code that looks short might macro-expand into enormous amounts of code and you may never notice. Though I imagine the same can be true of any language, and normal function calls can just as easily hide a lot of complexity. I can't blame Lisp, I can only blame myself for not profiling my code better. Lesson learned to be more careful, I guess. I just hope this helps fix my memory exhaustion problems. Otherwise back to the drawing board.

February 20, 2008 @ 3:49 PM PST
Cateogory: Programming

4 Comments

Zach Beane
Quoth Zach Beane on February 20, 2008 @ 8:02 PM PST

http://wigflip.com/ has been holding steady at 80-100M for almost a year. It generates about 10k graphics per day. It runs on a system with 1500M of memory, so I don't really care about the memory use...it's rock-solid.

Christopher GIroir
Quoth Christopher GIroir on February 20, 2008 @ 10:30 PM PST

I can't speak from experience, but all of the lisp books I've read all mention how this is a nice thing about lisp. How you can write everything easily, then go back and improve parts that need improving. Thanks for the post, it's nice to see how someone actually does this!

Adam Sloboda
Quoth Adam Sloboda on February 21, 2008 @ 12:30 AM PST

Well, I wonder how is it that my thought on first sight was "why he didn't generate scanner"... Then I saw tour next piece of code (and I'm not experienced Lisper).

Maybe it comes from my "C with a piece of assembler" background :)

Nikodemus Siivola
Quoth Nikodemus Siivola on March 05, 2008 @ 8:18 PM PST

Runtime memory exhaustion almost never has anything to do with the amount of space a specific piece of your application takes when you load it.

(let (list) (defun oops (x) (push x list)))

is not going to take almost any space, but call it often enough and you will run out of space.

What you want to do is to figure out which part of your application is consing and holding on to garbage. One thing you want to do is check if you have any global stores that grow without bounds (or just too much): global or closed over variables, slots in long-lived instances, etc. The other thing is to find out which bits are consing (more then you expect). SB-SPROF run in allocation profiling mode is good for this.

Finally, you shouldn't have to patch SBCL. --dynamic-space-size runtime option should Do The Right Thing for you. (But of course, if your VPS refuses to give you enough memory for your application to run... there is fairly little to be done about that except getting a better VPS.)