It's been nine months since I ditched Wordpress and moved to a blog system I wrote from scratch (in Clojure). This was a great move in so many ways. One of those ways is comment spam. My site is as popular now (or maybe slightly more popular now) as it was when I was running Wordpress, so I think comparing before and after is valid.

With Wordpress, every morning I'd do the ritual of deleting overnight spambot droppings. Typically I got between 1 and 5 every night. I had a default Wordpress install and all I used for spam filtering was Askimet. Askimet did a surprisingly good job, catching literally if not thousands of spams every week which otherwise would've been ruining my site. But inevitably some would still get through. And what's worse, there were a lot more false positives than I could tolerate.

Since I started counting with my new system, which is around 6 months, to the best of my knowledge I've gotten zero spambot-produced comments that made it through my filters. This is pleasant, to say the least.

The system I'm using is stupid. None of it is stuff I thought of myself, I got ideas from other lots of other blogs or articles I read, but the implementation is mine and it's not sophisticated. It would take a bot author a few seconds to work around it. But no one has bothered. Why bother writing a bot for my one-man blog, when you can write a bot for Wordpress and have it work on tens of thousands of blogs? And I can change my system to defeat the bots with a few lines of code just as easily as they can work around it.

So here's why I think it's working.

So far my silly anti-spam measures are working. Since last week I've had 1861 spam comment attempts, of which 0 were successful. 1857 of them didn't even alter the text my the captcha text field at all. Four of them inexplicably HTML-escaped the < into a <.

One feature I didn't implement from Wordpress is subscribing to comments via email. Sending an email from Java is possible but a little bit painful to implement. The Javamail API is a monster.

I do think it's useful to be able to know when someone responds to comment you left, but is spamming your inbox really the best way? I have to think there's a better way.

I did implement an RSS feed for each individual post's comments. And separate RSS feeds for all the tags on my blog, and all the categories. When RSS feeds are generated dynamically, why not? This is all of the code for the tag feeds:

(defn tag-rss [tagname]
  (if-let [tag (get-tag tagname)]
    (rss
        (str "briancarper.net Tag: " (:name tag))
        (str "http://briancarper.net/" (:url tag))
        "briancarper.net"
        (map rss-item (take 25 (all-posts-with-tag tag))))
    (error-404 )))

Plus the routing code:

(GET "/feed/tag/:name" (tag-rss (route :name)))

But I haven't uploaded the comment-feed feature because I don't know if it's overkill. Personally I am liberal with my RSS feeds, I just pop them into my Akregator and off I go. But I don't know if other people take their feeds more seriously, or what. RSS feeds can be a bit heavyweight. Maybe I should make a feed for all of my comments across all posts.

After I implemented that silly CAPTCHA yesterday, the spam was stopped. There's also a honeypot form field (it's hidden via CSS so humans don't know it's there, and if any bot POSTs text for that field, the data is rejected automatically). It's silly and easily defeated, yet it stopped all 262 spam attempts since yesterday. It looks like all the spam is for one site, but it's coming from a huge range of IPs. So it's probably a botnet. Thanks, MS Windows!

I rewrote my whole CRUD layer so that I could use it for more than one database at once, and then rewrote my gallery code to take advantage, and now two hours later I have my origami gallery back up and running. Both sites are running from the same JVM. I wonder how many sites I can have going at once before the server melts into a puddle of Java-inflicted goo.

  PID PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
11338 16   0  512m 128m  12m S    0  0.3   0:28.33 java

Good thing I have plenty of RAM on the server. From looking at before and after shots of the memory usage, 66 MB is the JVM itself, and 40MB more is Jetty and Compojure and my code and all the dependencies. Then the last ~20 MB or so is my database slurped into RAM. So I can probably fit another few tens of thousands of posts and comments in here before I have to worry much. The real test will be letting this thing run for a couple weeks and see how hard it leaks.

One fun thing about playing with Compojure is that it doesn't do much with HTTP headers for you, which is a good learning opportunity. RFC 2616 is rather helpful here.

For example I learned that if you don't set a Cache-Control or Expires header, your browser will happily re-fetch files over and over, which is a bit of performance hit. Static files that don't change often like images etc. can be set with a higher Expires value so they're cached.

Another thing to keep in mind (note to self) is that using mod_proxy to forward traffic to a local Jetty server means that the "remote IP" you get from (.getRemoteAddr request) will always be 127.0.0.1. If you want the user's real remote IP, you have to look in the X-Forwarded-For header (easily accessed as (:x-forwarded-for headers) in Compojure. Given that Identicons are generated from a hash of an IP address, this has resulted in some screwed up (wrongly identical) avatars for a bunch of people in posts for the past couple days. Oops. Not much I can do to fix that now.

In other non-news, I just the spam logging for the blog so I can see the kinds of things bots are doing to get around my feeble anti-spam measures. Sadly the spam seems to have stopped entirely, right after I set this up. How annoying.

I was in a rush to get this darn blog finally done, so I threw some stupid anti-spam measures on here. Namely, the comment form included 20 textareas, 19 of which were display: hidden and one of which was randomly the right one, and any text in the hidden ones would cause the comment posting to fail.

It only took a spam bot 48 hours to figure this out, I guess, because the last hour I've been hammered. So I implemented a CAPTCHA as another short-term holdover until I can code up something good. At least it immediately stopped this spam bot whose crap I've been deleting for the past hour.

Hopefully this isn't too intrusive. I think it fits the site fairly well, as you will probably agree once you see it.

I own my own domain (or five) and one of the good things about that is having nearly infinitely many email accounts if you want them. So I tend to make up a new account for every site I register at. This leads to amusing things like getting an email from a marketing firm asking me to complete a survey for an airline "who wants to remain STRICTLY ANONYMOUS". Sent of course to UNITED@briancarper.net. Oops.

Because of laziness I set up a catchall account on my domain so every email sent to anything @briancarper.net would be sent to me. This was such a horribly bad idea, I'm unsure how I lasted for a couple of years this way. I was getting about a few hundred spam emails per day. Amazingly spamassassin + Thuderbird's junk mail filter caught almost every single one of them to the point where I hardly even noticed. Spam filters can be bad in the same way pain killers can be bad. They don't solve a problem, they only mask the pain so you can ignore the problem.

So I decided to stop using a catchall. Problem is that I already have around a hundred email addresses I've used for various message boards and companies and friends and family, and there's no way I'm going around to change them all. So I decided to just get a list of them all and set up a big list of postfix aliases for now.

So, I downloaded my whole email account in mbox format and wrote a Ruby script to crawl it and make a list of all the email accounts I've ever received mail from. Thank you Linux mailserver for storing email sanely in plaintext. Luckily for me, I haven't deleted any emails from my server since 2005; so my generated list of emails is likely to be pretty complete. It pays to be obsessive sometimes.

Even a braindead brute-force Ruby script is fast enough to do this. Took a minute or two to scan 200MB of plaintext.

#!/usr/bin/ruby
require 'find'
found = {}
Find.find(ARGV[0]) do |fn|
  next unless File.file? fn
  File.read(fn).scan(/[A-Za-z0-9_-]+@briancarper.net/) do |email|
    next if found[email]
    found[email] = true
    puts email
  end
end