briancarper.net (λ) (Tag: Spam)

Ads on license plates?

Sun, 20 Jun 2010 22:09:08 -0700

What if when your car stops at a red light, your license plate displays ad banners? What could possibly go wrong?

Quoth the person(?) who wrote this bill:

"We're just trying to find creative ways of generating additional revenues," he said. "It's an exciting marriage of technology with need, and an opportunity to keep California in the forefront."

The forefront of annoying the hell out of people. Certainly what I need is more distractions on the road. I mean, what if there's a new brand of toothpaste and I didn't find out yet? Someone somewhere needs to earn a dime for telling me about it by any means necessary.

I'm just waiting for the first company to propose paying new parents a few hundred dollars to tattoo ads on their babies.

Printer spam: what could possibly go wrong?

Thu, 17 Jun 2010 10:59:57 -0700

As further evidence that there are no depths to which companies won't stoop when it comes to advertising, HP has come up with a great idea: Get people to hook their printers up to the internet and then spew advertisements out of their printers.

Well, it's a win-win situation for the companies doing the advertising: Not only will people see your ads, they'll pay for the ink and paper to print them. Maybe not such a great situation for the end-user though.

And then there are the privacy implications of targeting ads based on geolocating the IP address of the printer. Which I find a bit disturbing, but I guess advertisers already do that with online ads. But wait, there's more:

Ads can also be targeted based on a user's behavior as well as the content, said Vyomesh Joshi, head of the HP's Imaging and Printing Group.

Looking at what I'm printing so you can try to sell me things? Just a bit creepy.

Most troubling to me is the intrusiveness of the whole thing. They're taking control of a physical object in my house and using it against me. May as well kidnap my cat and train him to spell out "BUY PEPSI" in his cat litter.

Quote some slimeball at HP:

"What we discovered is that people were not bothered by it [an advertisement]," Nigro said. "Part of it I think our belief is you're used to it. You're used to seeing things with ads."

Translation: "We know this is a really horrible idea, but if people are complacent enough to sit there and take it without complaint, what's stopping us?"

He's right though, people are used to it.

I guess TV, radio, internet, phones, product placement in movies and games, print media, billboards and the postal service just aren't enough. Clearly what the world really needs is another ad-delivery mechanism.

Lame comment spam management that works

Sun, 06 Dec 2009 02:34:08 -0800

It's been nine months since I ditched Wordpress and moved to a blog system I wrote from scratch (in Clojure). This was a great move in so many ways. One of those ways is comment spam. My site is as popular now (or maybe slightly more popular now) as it was when I was running Wordpress, so I think comparing before and after is valid.

With Wordpress, every morning I'd do the ritual of deleting overnight spambot droppings. Typically I got between 1 and 5 every night. I had a default Wordpress install and all I used for spam filtering was Askimet. Askimet did a surprisingly good job, catching literally if not thousands of spams every week which otherwise would've been ruining my site. But inevitably some would still get through. And what's worse, there were a lot more false positives than I could tolerate.

Since I started counting with my new system, which is around 6 months, to the best of my knowledge I've gotten zero spambot-produced comments that made it through my filters. This is pleasant, to say the least.

The system I'm using is stupid. None of it is stuff I thought of myself, I got ideas from other lots of other blogs or articles I read, but the implementation is mine and it's not sophisticated. It would take a bot author a few seconds to work around it. But no one has bothered. Why bother writing a bot for my one-man blog, when you can write a bot for Wordpress and have it work on tens of thousands of blogs? And I can change my system to defeat the bots with a few lines of code just as easily as they can work around it.

So here's why I think it's working.

1. It's not Wordpress

Just by using something slightly different from Wordpress, I'm think I'm already ahead. For example if you have a blog where a form posts comments to /wp-comments-post.php, a bot doesn't even need to look at your site to spam you. They can blast your server with POST data at that URL in a format they already know Wordpress will accept. My site is all custom code, so everything is different enough that default bot attempts fail immediately.

I think this is the reason that only 1853 spam comments have even been POSTed at me in the last six months. That's an improvement of one or two orders of magnitude already.

2. Honeypot text field

So what about the comments that are actually POSTed? They are presumably the result of bots that parse sites' HTML looking for comment forms and try to POST data that satisfies the form.

So in my comment form I have a field called referer. A "How did you find my site?" kind of thing. In fact I don't care how you found this site, this field is a honeypot. The div containing this field is hidden via CSS.

div#referer-row {
    display: none;
}

So you shouldn't ever see it if you're a human using a browser. But bots parsing the HTML see this field, unless they also bother to parse my CSS and see that it's hidden, which would be expensive and apparently they don't do it.

If you put anything in this referer field, your comment will be rejected as spam. Simple enough.

Many blogs require you to fill in every field or else the comment is rejected, so it seems reasonable to expect most bots to fill in all of your form fields. (My blog actually requires you to fill in nothing but the comment text; author will be set to Anonymous Cow if you don't fill it in.)

In fact this seems to be the case; of 1853 spam comments since March, 1810 put something into this field. Most of the time it's a random string of letters. Not even a URL. Sometimes it's a couple words like "insurance quotes" or something about drugs or casinos.

The downside of this is if you're a human using a browser that doesn't understand CSS, you will see this field. Then if you type something into it and try to comment, it'll end up as spam. So Lynx users and time travelers from 1987 trying to leave me comments might be confused at first.

However as far as I can tell, no intelligible data has ever been entered into this field by a human, so I don't think it's a concern. Six times, the word "None" was entered, but I don't think this is a human because that's nonsense answer to "How did you find this site?". But you never know.

3. Lame static CAPTCHA

That leaves 43 spam comments that made it this far. My other anti-spam measure is a word you have to type. But it's always the same word, and the word is COWS. This CAPTCHA caught the remaining 43. It looks like this:

There's a normal text field with a default value of <= Type this word specified right in the HTML.

<input type="text" value="<= Type this word" name="test" id="test"/>

There are no other instructions besides "Type this word". I'm assuming either that commenters are familiar enough with CAPTCHAs to know what I want, or can figure it out using common sense. Given that my target audience is computer geeks and programmers, this should be a safe assumption. In fact I've had less than a dozen false positives in the past six months via people failing this; see below for details.

To post a comment, the value of this field must contain the word "COWS" somewhere in it, case-insensitively. Otherwise it's spam. Easy enough to implement.

If you have Javacsript enabled, clicking on this field will clear out the default value. If you unfocus the field without typing anything, Javascript will put the default value back in. This is only for convenience. If you don't have Javascript enabled, you have to highlight and backspace over the default text. I don't think this is a huge burden.

Of 1853 spam comments, here's the breakdown for what values end up in this field.

1131: "&lt;= Type this word"
691:  "<= Type this word"
21:   Random letters and numbers
6:    empty
2:    A bunch of URLs
2:    Human beings making typos, e.g. "COW" or "COS".

So most bots are too stupid even to remove the default value from the field, and none of them entered the correct value. 691 times the bot was somehow smart enough to un-escape < into <, which is interesting, but didn't help it defeat the filter. A lot of the random words look like they were made by Markov chains, e.g. 'fridwolfur' and 'lyndonvolk' and 'calbertdom'. If I need to write a childerns' poem I'll know where to look for ideas. One time a bot or spammer managed to type "vows" somehow, but this might or might not be coincidence.

I think this is better than a normal CAPTCHA because:

It's always the word "COWS" in a normal font, so it requires no thought or eye strain to figure out.
It's black and white, so hopefully people with minor vision problems and color blindness can see it.
It fits thematically with my blog layout (it's COWS and it has cow spots).
It's kind of silly, so hopefully people chuckle rather than become ticked off.

It's worse than a real CAPTCHA because it requires no effort to break. So it wouldn't work for Wordpress or VBulletin or something with a million users. But I wonder if every Wordpress had a single static word as a CAPTCHA, but a different word for every blog (generated at install-time maybe), would it work better or worse than the random mangled multi-color CAPTCHAs no one can read? Real randomly-generated CAPTCHAs don't work anyways; bots can already beat them via OCR or other means. A simple word would be less annoying for a human, to be sure.

The other downside is that this is not very accessible to the blind or other people using screen readers, or browsers without image support. This is unfortunate and I'm still trying to figure out how to get around this. Right now the ALT text for the CAPTCHA image is This says 'COWS'; I don't know if this is enough help for people in those situations.

Of course I'll never know how many people see my CAPTCHA and storm away in a rage without even trying to post a comment. But I've never heard a complaint. If this level of CAPTCHA ticks you off personally, please swallow your anger and leave me a comment here saying so, if you feel obliged; I'd love to hear it.

False positives

As best I can tell, there are no false positives from people filling in the honeypot field. But even as simple as it is, some people don't succeed at the CAPTCHA image. Either they typo it or they ignore it entirely.

I just checked and I counted around 6 comments by real humans where the CAPTCHA was ignored and the default <= Type this word ended up in the spam DB. 4 of those people re-posted their comment successfully immediately afterwards by filling in the CAPTCHA. I'm not sure I'm ever going to get much better than that.

Spam that makes it through

I have still gotten spam. Maybe a dozen or so in the past six months. It's all been in the form of a human typing a normal-looking and relevant comment, about open source software or BASH for example, but with a spammy URL buried in it, e.g. a link to a really dodgy-looking blog trying to sell something, or some scummy SEO site. It's either a human or a very sophisticated (or lucky) bot; the comment text in these is indistinguishable from a real comment other than the spam URLs. I have to delete these by hand.

But I was getting these with Wordpress too. No automted anti-spam system is going to defeat a human being, so I don't worry about it.

That's it

The moral of this story is that it doesn't take much to protect yourself from comment spam if you write the code yourself. As long as it's unique, you'll probably be fine.

The other moral is that you don't have to annoy the hell out of your users to filter spam effectively. I'm making the assumption here that my COWS method is not that annoying; tell me if I'm wrong.

I don't know how well this scales. Probably not so well. My blog isn't that highly trafficked. If my site were more popular it might be worse for me. But the improvement over Wordpress is unquestionable.

I've seen all kinds of complicated measures suggested elsewhere, like trying to predict if it's a bot by how many milliseconds it takes between page load and comment posting, or measuring keypress speed, or escaping the HTML of your forms and un-escaping it at loadtime it via Javascript, or setting and retrieving cookies and such. But a lot of this stuff seems fragile and if your browser doesn't suppoort Javascript or cookies (or your users block them), you're screwed. I block these things myself, so I expect visitors to do the same.

If everyone wrote their own blog engines, the world would be a slightly less spammy place. Or else we'd have much smarter bots.

Spam spam spam spam spammity spam

Sat, 17 Oct 2009 11:59:05 -0700

I woke up this morning to about 50 spam emails and some notifications from my host that my CPU usage was about 200% over the past four hours. Turns out spamd was going mental. Not sure what caused it but it seems to be working again after I restarted it.

One of the worst things about running your own mail server is spam. I don't much about how to do it properly. I have SpamAssassin running, I tweaked the settings and trained it well, and it works OK. Of 8,000 spams in the past week or two, I think only two made it through to my inbox. But I keep thinking there must be a better way.

For a while I tried greylisting. Greylisting means you pseudo-bounce every email you get, and force the mail server to resend it. Once it's resent, that server is added to a whitelist. The idea is that spam servers won't bother resending and genuine mail servers will.

I ran this way via Postgrey for a couple months. The good thing is that it works pretty much as advertised. I went from hundreds of spam emails per day, to fewer than a dozen. SpamAssassin caught all of those dozen and I never saw them. It was nice.

The problem with this, however, is twofold.

All mail from people you've never heard from before is delayed 5-10 minutes. This is very annoying in certain circumstances, e.g. registering for an account at a new message board or buying something from an online store you never used before. I'd rather like to see the receipt or user registration right away. So to get around this I had to go add them to a whitelist on the server every time, which was ridiculous.
Not all genuine mail servers bother resending after the temporary bounce, so you lose mail. You need only look in /etc/postgrey/whitelist_clients and see the enormous list of mail servers that Postgrey knows NOT to greylist, to be scared into never using Postgrey again. This includes yahoo.com, ebay.com, a bunch of airlines, and so on. The list goes back to 2005 and obviously is an incomplete list, since it only includes servers that people reported having problems with. I had to add gmail.com to it myself to avoid losing mail from my wife (domains that use large pools of mail servers will always be greylisted, it seems).

Losing mail is the reason I stopped using Postgrey. So I'm back to SpamAssassin alone and dealing with an occasional spam or two, while my spam inbox balloons.

Anti-spam field still holding

Mon, 23 Mar 2009 19:38:27 -0700

So far my silly anti-spam measures are working. Since last week I've had 1861 spam comment attempts, of which 0 were successful. 1857 of them didn't even alter the text my the captcha text field at all. Four of them inexplicably HTML-escaped the < into a <.

One feature I didn't implement from Wordpress is subscribing to comments via email. Sending an email from Java is possible but a little bit painful to implement. The Javamail API is a monster.

I do think it's useful to be able to know when someone responds to comment you left, but is spamming your inbox really the best way? I have to think there's a better way.

I did implement an RSS feed for each individual post's comments. And separate RSS feeds for all the tags on my blog, and all the categories. When RSS feeds are generated dynamically, why not? This is all of the code for the tag feeds:

(defn tag-rss [tagname]
  (if-let [tag (get-tag tagname)]
    (rss
        (str "briancarper.net Tag: " (:name tag))
        (str "http://briancarper.net/" (:url tag))
        "briancarper.net"
        (map rss-item (take 25 (all-posts-with-tag tag))))
    (error-404 )))

Plus the routing code:

(GET "/feed/tag/:name" (tag-rss (route :name)))

But I haven't uploaded the comment-feed feature because I don't know if it's overkill. Personally I am liberal with my RSS feeds, I just pop them into my Akregator and off I go. But I don't know if other people take their feeds more seriously, or what. RSS feeds can be a bit heavyweight. Maybe I should make a feed for all of my comments across all posts.

Blog is still going strong

Wed, 18 Mar 2009 22:01:11 -0700

After I implemented that silly CAPTCHA yesterday, the spam was stopped. There's also a honeypot form field (it's hidden via CSS so humans don't know it's there, and if any bot POSTs text for that field, the data is rejected automatically). It's silly and easily defeated, yet it stopped all 262 spam attempts since yesterday. It looks like all the spam is for one site, but it's coming from a huge range of IPs. So it's probably a botnet. Thanks, MS Windows!

I rewrote my whole CRUD layer so that I could use it for more than one database at once, and then rewrote my gallery code to take advantage, and now two hours later I have my origami gallery back up and running. Both sites are running from the same JVM. I wonder how many sites I can have going at once before the server melts into a puddle of Java-inflicted goo.

  PID PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
11338 16   0  512m 128m  12m S    0  0.3   0:28.33 java

Good thing I have plenty of RAM on the server. From looking at before and after shots of the memory usage, 66 MB is the JVM itself, and 40MB more is Jetty and Compojure and my code and all the dependencies. Then the last ~20 MB or so is my database slurped into RAM. So I can probably fit another few tens of thousands of posts and comments in here before I have to worry much. The real test will be letting this thing run for a couple weeks and see how hard it leaks.

Fun with HTTP headers

Tue, 17 Mar 2009 22:14:10 -0700

One fun thing about playing with Compojure is that it doesn't do much with HTTP headers for you, which is a good learning opportunity. RFC 2616 is rather helpful here.

For example I learned that if you don't set a Cache-Control or Expires header, your browser will happily re-fetch files over and over, which is a bit of performance hit. Static files that don't change often like images etc. can be set with a higher Expires value so they're cached.

Another thing to keep in mind (note to self) is that using mod_proxy to forward traffic to a local Jetty server means that the "remote IP" you get from (.getRemoteAddr request) will always be 127.0.0.1. If you want the user's real remote IP, you have to look in the X-Forwarded-For header (easily accessed as (:x-forwarded-for headers) in Compojure. Given that Identicons are generated from a hash of an IP address, this has resulted in some screwed up (wrongly identical) avatars for a bunch of people in posts for the past couple days. Oops. Not much I can do to fix that now.

In other non-news, I just the spam logging for the blog so I can see the kinds of things bots are doing to get around my feeble anti-spam measures. Sadly the spam seems to have stopped entirely, right after I set this up. How annoying.

Darn you, spammers.

Tue, 17 Mar 2009 18:35:18 -0700

I was in a rush to get this darn blog finally done, so I threw some stupid anti-spam measures on here. Namely, the comment form included 20 textareas, 19 of which were display: hidden and one of which was randomly the right one, and any text in the hidden ones would cause the comment posting to fail.

It only took a spam bot 48 hours to figure this out, I guess, because the last hour I've been hammered. So I implemented a CAPTCHA as another short-term holdover until I can code up something good. At least it immediately stopped this spam bot whose crap I've been deleting for the past hour.

Hopefully this isn't too intrusive. I think it fits the site fairly well, as you will probably agree once you see it.

Email woes

Fri, 18 Apr 2008 20:49:16 -0700

I own my own domain (or five) and one of the good things about that is having nearly infinitely many email accounts if you want them. So I tend to make up a new account for every site I register at. This leads to amusing things like getting an email from a marketing firm asking me to complete a survey for an airline "who wants to remain STRICTLY ANONYMOUS". Sent of course to UNITED@briancarper.net. Oops.

Because of laziness I set up a catchall account on my domain so every email sent to anything @briancarper.net would be sent to me. This was such a horribly bad idea, I'm unsure how I lasted for a couple of years this way. I was getting about a few hundred spam emails per day. Amazingly spamassassin + Thuderbird's junk mail filter caught almost every single one of them to the point where I hardly even noticed. Spam filters can be bad in the same way pain killers can be bad. They don't solve a problem, they only mask the pain so you can ignore the problem.

So I decided to stop using a catchall. Problem is that I already have around a hundred email addresses I've used for various message boards and companies and friends and family, and there's no way I'm going around to change them all. So I decided to just get a list of them all and set up a big list of postfix aliases for now.

So, I downloaded my whole email account in mbox format and wrote a Ruby script to crawl it and make a list of all the email accounts I've ever received mail from. Thank you Linux mailserver for storing email sanely in plaintext. Luckily for me, I haven't deleted any emails from my server since 2005; so my generated list of emails is likely to be pretty complete. It pays to be obsessive sometimes.

Even a braindead brute-force Ruby script is fast enough to do this. Took a minute or two to scan 200MB of plaintext.

#!/usr/bin/ruby
require 'find'
found = {}
Find.find(ARGV[0]) do |fn|
  next unless File.file? fn
  File.read(fn).scan(/[A-Za-z0-9_-]+@briancarper.net/) do |email|
    next if found[email]
    found[email] = true
    puts email
  end
end