My origami website server kept hitting a limit on "privvmpages", which is a cap (by the VPS) on some aspect of system memory. SBCL died with some interesting notes about heap exhaustion. I already had to hack SBCL even to get it to start on my server. And now this. I'm running a VBulletin + a Rails app + my Lisp site but none of them are high-traffic and you'd think the server could handle it. I figured I must be doing something wrong.
Luckily lots of other people apparently were also doing it wrong. Google for "vps sbcl" and you'll find plenty of people having difficulty running it. Through googling I learned of the handy
ROOM function which tells you some things about SBCL's current memory usage. And also
SB-EXT:GC which in SBCL appears to invoke the garbage collector immediately.
Dynamic space usage is: 72,748,936 bytes. Read-only space usage is: 1,912 bytes. Static space usage is: 2,736 bytes. Control stack usage is: 1,244 bytes. Binding stack usage is: 336 bytes. Control and binding stack usage is for the current thread only. Garbage collection is currently enabled. etc.
Is 72MB good or bad? I don't know, but I'd guess bad, given that my server was dying. So next I went through all of the packages my package depended on, and loaded them manually one by one to see how much usage was increasing for each. CL-WHO + CLSQL + CL-FAD + CL-PPCRE + HUNCHENTOOT were increasing memory usage by about 10MB total, apparently. My own package, which does nothing but run my website, was increasing it another 21MB. Ouch. Clearly I was doing something very wrong.
So then I started loading each of the functions in each of my own files one by one, in order, to try to account for those 21MB. This is easy to do in Slime via
C-c C-c. After each I invoked the GC and checked my memory usage again. I noticed that one inconspicuous function was adding around 9MB to that number. It was a function where I was doing a glob of a bunch of filenames, then removing every filename that matched the string "thumbs". Here's the relevant part:
(remove-if #'(lambda (p) (cl-ppcre:scan "thumbs" (namestring p))) (cl-fad:list-directory (merge-pathnames (name model) *photo-dir*)))
When I changed it to this, my memory usage went down by 9MB:
(let ((thumbs-regex (cl-ppcre:create-scanner "thumbs"))) (remove-if #'(lambda (p) (cl-ppcre:scan thumbs-regex (namestring p))) (cl-fad:list-directory (merge-pathnames (name model) *photo-dir*))))
Yikes! Still not sure what was going on there. Perhaps it would be best if I didn't rely so heavily on regular expressions to begin with.
I've heard said that Lisp can be dangerous because expensive things don't necessarily look expensive. Code that looks short might macro-expand into enormous amounts of code and you may never notice. Though I imagine the same can be true of any language, and normal function calls can just as easily hide a lot of complexity. I can't blame Lisp, I can only blame myself for not profiling my code better. Lesson learned to be more careful, I guess. I just hope this helps fix my memory exhaustion problems. Otherwise back to the drawing board.