briancarper.net (λ) (Tag: Email)

KMail is slow

Sun, 18 Oct 2009 13:38:48 -0700

KMail is a pretty good app, except that it's slow as glacier. If I select a few thousand spam emails in KMail (4.3.2) sitting on my IMAP server, and I try to delete them, KMail laboriously iterates through them deleting them one at a time and updating the GUI after every deletion. I'd say it averages about 4 or 5 emails deleted per second. Yes, for a few thousand emails this adds up to 10-15 minutes of waiting.

I got tired of waiting, so I decided to try out Mutt for the first time. In Mutt apparently you can delete a folder full of emails by pressing D and then specifying ~s .* as the search pattern. Mutt deleted 8,000 spam emails for me instantaneously.

This is what I get when I ignore the collective wisdom of the Linux group-mind. I've long heard that Mutt is good but I never bothered trying it, because Thunderbird and KMail and friends were "good enough".

Complacency, my old nemesis. You have beaten me again. But I am now going to give Mutt a good try. Next on the list is zsh.

Spam spam spam spam spammity spam

Sat, 17 Oct 2009 11:59:05 -0700

I woke up this morning to about 50 spam emails and some notifications from my host that my CPU usage was about 200% over the past four hours. Turns out spamd was going mental. Not sure what caused it but it seems to be working again after I restarted it.

One of the worst things about running your own mail server is spam. I don't much about how to do it properly. I have SpamAssassin running, I tweaked the settings and trained it well, and it works OK. Of 8,000 spams in the past week or two, I think only two made it through to my inbox. But I keep thinking there must be a better way.

For a while I tried greylisting. Greylisting means you pseudo-bounce every email you get, and force the mail server to resend it. Once it's resent, that server is added to a whitelist. The idea is that spam servers won't bother resending and genuine mail servers will.

I ran this way via Postgrey for a couple months. The good thing is that it works pretty much as advertised. I went from hundreds of spam emails per day, to fewer than a dozen. SpamAssassin caught all of those dozen and I never saw them. It was nice.

The problem with this, however, is twofold.

All mail from people you've never heard from before is delayed 5-10 minutes. This is very annoying in certain circumstances, e.g. registering for an account at a new message board or buying something from an online store you never used before. I'd rather like to see the receipt or user registration right away. So to get around this I had to go add them to a whitelist on the server every time, which was ridiculous.
Not all genuine mail servers bother resending after the temporary bounce, so you lose mail. You need only look in /etc/postgrey/whitelist_clients and see the enormous list of mail servers that Postgrey knows NOT to greylist, to be scared into never using Postgrey again. This includes yahoo.com, ebay.com, a bunch of airlines, and so on. The list goes back to 2005 and obviously is an incomplete list, since it only includes servers that people reported having problems with. I had to add gmail.com to it myself to avoid losing mail from my wife (domains that use large pools of mail servers will always be greylisted, it seems).

Losing mail is the reason I stopped using Postgrey. So I'm back to SpamAssassin alone and dealing with an occasional spam or two, while my spam inbox balloons.

Comcast blocks port 25

Sat, 15 Aug 2009 15:22:00 -0700

Testing a remote SMTP server is kind of hard when Comcast is secretly blocking all inbound and outbound traffic on port 25. I only figured it out when telnet wouldn't even work. I could've saved myself an hour of frustration if I'd known this. Thanks, Comcrap.

Reverse DNS

Wed, 23 Jul 2008 18:04:03 -0700

Sometime a month or two ago, sending email to my family stopped working. My emails vanished into the void and were never heard from again.

I figured my SMTP was set up incorrectly so no big deal, I used another (gmail etc.). Recently I got around to looking in my logs and I discovered:

Jul 23 17:06:49 ffclassic postfix/smtp[30408]: 73B3C4654756: host mx2.comcast.net[76.96.30.116] refused to talk to me: 421 IMTA24.emeryville.ca.mail.comcast.net comcast Reverse DNS failure : Try again later

Well that's no fun. I guess Comcast doesn't like talking to mail servers without reverse DNS set up properly. So now I fixed it (hopefully, we'll see in 24 hours).

These are the joys of running my own server. I had the option from my host of getting CPanel or its equivalent, or setting up everything by hand, so I chose by hand. The bad thing is that I have no idea what I'm doing. The good thing is there's no better way to learn than writing all of your config files by hand.

Email woes

Fri, 18 Apr 2008 20:49:16 -0700

I own my own domain (or five) and one of the good things about that is having nearly infinitely many email accounts if you want them. So I tend to make up a new account for every site I register at. This leads to amusing things like getting an email from a marketing firm asking me to complete a survey for an airline "who wants to remain STRICTLY ANONYMOUS". Sent of course to UNITED@briancarper.net. Oops.

Because of laziness I set up a catchall account on my domain so every email sent to anything @briancarper.net would be sent to me. This was such a horribly bad idea, I'm unsure how I lasted for a couple of years this way. I was getting about a few hundred spam emails per day. Amazingly spamassassin + Thuderbird's junk mail filter caught almost every single one of them to the point where I hardly even noticed. Spam filters can be bad in the same way pain killers can be bad. They don't solve a problem, they only mask the pain so you can ignore the problem.

So I decided to stop using a catchall. Problem is that I already have around a hundred email addresses I've used for various message boards and companies and friends and family, and there's no way I'm going around to change them all. So I decided to just get a list of them all and set up a big list of postfix aliases for now.

So, I downloaded my whole email account in mbox format and wrote a Ruby script to crawl it and make a list of all the email accounts I've ever received mail from. Thank you Linux mailserver for storing email sanely in plaintext. Luckily for me, I haven't deleted any emails from my server since 2005; so my generated list of emails is likely to be pretty complete. It pays to be obsessive sometimes.

Even a braindead brute-force Ruby script is fast enough to do this. Took a minute or two to scan 200MB of plaintext.

#!/usr/bin/ruby
require 'find'
found = {}
Find.find(ARGV[0]) do |fn|
  next unless File.file? fn
  File.read(fn).scan(/[A-Za-z0-9_-]+@briancarper.net/) do |email|
    next if found[email]
    found[email] = true
    puts email
  end
end

Spammery

Sat, 21 Jul 2007 15:47:29 -0700

Spamassassin on my mail server stopped working today. You don't realize just how much work a spam filter saves you until it stops working. I've probably gotten 50 spam emails since this morning.

I've been using Thunderbird in Linux for a while, and it works surprisingly well nowadays. I remember being dissatisfied with it a year or two back, but whatever I didn't like (I can't even remember) seems to be fixed. One problem with Thunderbird is that as soon as you leave home, you don't have it with you. Webmail is good in that regard. But I know there are also versions of Thunderbird that you can install on a USB flash drive. What I'm not sure is whether it's possible to get such a thing to play nicely when used on multiple operating systems. Or alternatively to sync the thing with my home desktop every time I use it. I'll have to play with it a while. I don't have a Windows machine at home to test it on, sadly.

Webmail continued

Fri, 06 Oct 2006 11:23:06 -0700

I've blabbered on quite a bit about finding a good webmail client since I ditched Gmail. I am enjoying being free of ads and knowing my email is not being stored / used / read by some corporation. Well, except potentially by the company that hosts my website. And by my ISP. And maybe by the government. But other than that I'm good.

I think I've finally settled on Horde Imp. I guess this is one of the (few) times I've decided to go with "tested and stable" rather than "new and shiny". It's often so rare to even have the option of "tested and stable" in the Open Source world, I may as well take advantage. Given that I almost missed a job offer due to an email ending up in the wrong place, I think email is a good candidate for sticking with something that works.

A perhaps valid question that comes to mind is why do I even need webmail? There are other options. In the days of my Linux youth, I used fetchmail and procmail on my local machine to auto-fetch my mail every 5 minutes, and store it all locally using courier-imap to access it. There are some advantages to that: I could auto-sort the mail into local IMAP folders. I could fetch from multiple remote email accounts. I could open my mail locally without having to download anything; it was all already download.

However there are also distinct disadvantages. Namely, that my computer is a desktop machine. I'm pretty good about making backups, but even I forget sometimes. On the other hand a remote host likely has someone paid full-time to do backups on a regular basis, and there is likely a huge incentive to actually do it consistently (i.e. if they fail, they will lose customers). Also, given that all sorting is done only once it hits my local machine, accessing mail MUST be done through that local machine. So if I want to view my mail remotely, I have to remotely access my home machine. This is a problem because, again, my local machine is a desktop computer. I don't have a backup power supply. I have a cat who likes to head-butt my power button. So there are times when I'll be stuck somewhere without access to my email. Again, uptime is something I think my website host is better at than I am.

Both of these issues can be semi-solved by leaving a copy of all mail on the remote hosts, and only fetching a copy to my local machine; then I can fall back to using the remote hosts in case my home machine is ever down. However this is a hack at best. Accessing a server full of 5,000 unread unsorted emails isn't my idea of fun. Far better to use the remote host directly all the time. Then I don't have to worry about backups (as much) and access is exactly the same from home or from work or from anywhere else.

Another option is to use a desktop-installed email client program. The only problem with that is that I'd have to install a client it everywhere I go. I am away from home far more often than I am at home, nowadays. Home is where I sleep while I'm waiting to go back to work. Given this, webmail is about the only option I see open to me. A lot of computing tasks really becoming more distributed nowadays, aren't they? I couldn't manage if home was the only place from which I could access my email. Interesting.

Migrating away from Gmail

Sun, 24 Sep 2006 19:02:28 -0700

After two years of using Gmail, I have exactly 5,531 emails. These are all sitting on some server who knows where, and getting them all off of there and onto my own server was an interesting task.

It turned out to be pretty easy thanks to Thunderbird. If you enable POP in Gmail, you can use Thunderbird to access Gmail that way. So I set that up, and set Gmail to serve up all messages including read / archived ones. Then I let Thunderbird fetch them all. This took a while (maybe an hour?).

The thing that makes this work is that Thunderbird stores mail locally as plain old mbox format files. So if you browse to ~/.thunderbird/blargaljdf.default/Mail/pop.gmail.com, you can see files with names matching the "mailboxes" in Thunderbird. There are multiple files per "mailbox". The one without an extension is the mbox file.

All I needed to do was upload these mbox files to my IMAP server and subscribe to them, and that's it. I actually used Thunderbird to split up all those Gmails into multiple mbox files first and then uploaded those. The Thunderbird GUI makes this all fairly easy. I have no use for Thunderbird normally, but it's good for this task.

It strikes me that this would also be a good way to make a backup of Gmail, if someone still plans to use it. Or any other POP email service, I suppose.

Interestingly, doing this caused IlohaMail to meltdown. It would say it knew I had 500 emails in folder X, but it would only let me see 120 of them. So now I'm using Horde again. Someday, I'll find a webmail program that works. Someday.

Webmail decisions

Tue, 19 Sep 2006 00:38:29 -0700

My host comes with Squirrelmail and Horde, and I dislike them both. Squirrelmail is too bare-bones for me. It's hideous to look at. And Horde is too over-the-top and slow for me. It takes about 17 clicks to get to the inbox once you log in. No call for that.

Really, what's the difference between one webmail app and the next, except the interface? So I'm looking for the nicest interface I can find, so long as the backend still works too.

So first I tried RoundCube, the much-acclaimed AJAX webmail app. It's pretty impressive in the "Wow, someone got a web browser to do that" sense. You can drag-and-drop things etc. But you can tell it's an AJAX hack. Things subtly don't work. If something is loading in the background, and you try another action, things often freeze or break. The folder list on the left doesn't update correctly; you have to click off a folder and click back on it again to get it to reset the message count. If you move some items to a folder, and quickly move some more to a different folder, one or the other operation will silently fail. Etc. etc. The problem (with all AJAX apps) is that there's all kinds of crazy server communication going on in the background and you never know what it's doing or if it's done or what. At least in a traditional web page you click a link or button and then you see it loading and you wait for it to finish. And then you click another.

Other things are broken in Roundcube too, like putting a < in a subject causing the subject to be blank. Probably some kind of relic of having an interface that is itself made of a bunch of changing HTML tags. I started the bugzilla for RoundCube and there's just an enormous number of problems with text parsing and encoding etc. So yeah, I passed on this. Maybe in 6 months if it's more stable I'll try again.

I considered Hastymail next. But development has apparently been stopped for a long time, and although it's not as bare as Squirrelmail, it's possibly a bit over the top in terms of "Let's make an interface that looks like it came from 1992". I like minimalistic, but Hastymail looks huge and clunky to me. I don't think it has threaded messages either, which is one thing I really love(d) about Gmail. Hastymail does have some neat extra stuff like showing emails from multiple folders at once in a multi-levelled list, but none of it looks all that useful.

I finally settled on Ilohamail. It's fairly minimalistic, but it looks nice. And it has just the slightest touch of AJAX, like threading conversations into expandable blocks in the message list. Development also seems to have stalled on it for a while, but we'll see how this goes.

And now, in any case, my emails are no longer being auto-read, stored away and turned into ads by Gmail's server farm. Sweet freedom.

Email woes

Sun, 17 Sep 2006 22:19:29 -0700

Email is such a fragile thing. It's nerve-wracking setting up an email server properly. If you set it up wrong, emails disappear into a black hole and it's sometimes not readily obvious that there's even a problem, especially if the emails don't even bounce. Same goes with setting up a procmail filter, for example.

One of my old CS professors said there are three kinds of programming bugs:

Bugs that break things in obvious and catastrophic ways, causing a complete crash of the program.
Bugs that break things without causing a full crash; the program keeps running. But you know about it.
Bugs that break things without causing a full crash; the program keeps running. And you don't know about it.

In that order, they increase from bad to worse. By far the worst is when things look like they work, but they don't. You can't even begin to fix a problem until you know a problem exists. (Programs' sweet debug-info-filled death-cries are programmers' best friends.)

Emails vanishing without being delivered or bounced is the third kind of bug, i.e. really really bad. My host's email system is very hard to work with and leads to many of this kind of bug; mysterious vanishing emails. All I want is all emails to both of my parked domain names to go to a single address at a single one of my domains. I'm pretty sure I got it right this time, but I always think that and then two weeks later someone calls me and says "Hey, how come you never responded to my email?" and it turns out the server ate it for dinner. But I will admit, it's very likely my troubles are PEBKAC-related.

If I get this set up, I'm going to stop using Gmail once and for all. I'm tired of 1) being advertised at while I read emails, 2) the Gmail servers being either down, non-responsive, or dead-set on not giving me my emails, 3) having stupid extra options to wade through like Google Talk and Google Calendars and "Invite your friends to use Gmail! We need more ad-reading monkeys!". And consarnit, I miss real folders. Labels don't do it for me.

Email providers are really in a good position. Once you start using an email address, it's REALLY hard to stop using it. So once people are hooked into using your service, you have them in your claws and you can start milking them for all they're worth. Imagine how many people, organizations, mailing lists etc. out there have your current email address and how long it would take to get them all to switch over. The only good thing is if you can forward from your old address to your new, and slowly do the transition that way.

Gmail died

Thu, 31 Aug 2006 19:07:42 -0700

So today at work, Gmail suddenly died. I couldn't log on to the Gmail webpage, and it was giving a "File Not Found" error instead of an actual Gmail error page. I tried logging on to Google in every way I could think of, and nothing worked. I tried logging on to Adsense, and the page loaded OK, but on the right instead of username/password fields I got the same File Not Found page.

I tried another computer at work, and it worked fine. And now I'm at home, and it works fine here too. I tried deleting my cache/cookies at work, and I tried it in IE and Firefox, and nothing worked, so I don't think it's a browser problem at work. Maybe something screwed with my hosts file? Maybe Google is blocking me via IP address?

The sucky thing about email addresses is once you get one and give it to tons of people, you're stuck using it forever. I really don't like Gmail at all, but it's the best I can find. I tried Horde and/or Squirrelmail at my domain, but it's just not powerful enough and I don't have full access to my server to actually configure those things like I want. Someone needs to make a simplistic, powerful, geek-based email service. (Or is there such a thing already?) I don't need fancy options. I don't need AJAX. I don't want emoticons or stationary or God only knows what else. I'd use anything which was reliable and fit those criteria.