Just a Theory

By David E. Wheeler

Posts about Email

Need Suggestions for IMAP Solution and Migration

For the last several years, I’ve run a Courier-IMAP mail server for all of the mail for this site, Kineticode, Strongrrl and other domains. We mainly used Mail.app on Mac OS X to communicate with the server, and it worked really well. Today, Julie has over 3 GB of mail data, and I have around 1.5 GB, all managed via IMAP.

Recently, I decided it was time to move the mail elsewhere. I’ve been meaning to do it for a while, primarily because the server I was using is now used for the Bricolage project, and because I never set up any spam filtering. Julie was suddenly getting 100s of spam messages in her inbox. (It really didn’t help that she was still using Panther.) So on the advice of a good friend who had been evaluating various mail services–and who for now shall go nameless and therefor blameless–I moved all of our mail to FuseMail.

At first this seamed like a pretty good solution. Our spam rates went way down, I could set up unlimited mail lists, aliases, and forwards, and there was a migration tool that automated moving all of our existing mail from the old IMAP server to the new one. There were some glitches with the migration tool, but in the end all of our mail was moved and in tact.

But that’s when I started to notice the issues. To summarize:

  • Mail put into the “Sent Items” folder by Mail.app was marked as unread. This didn’t happen on the old server, and apparently has something to so with how FuseMail names the sent folder: “Sent Items” rather than “Sent Messages.”
  • Mail.app is syncing constantly. Even once it had successfully synced the all of our email in all of our IMAP folders (which took days, it is syncing all the time, to the extent that I am sometimes waiting for up to a minute to read a mail when I double-click it, because there are all these other threads doing stuff and taking up all the resources. It can take several minutes for mail I’m sending to be sent (though that might be a delay in Mail.app copying the message to the Sent Items folder rather than the actual sending).
  • Deleting mail takes forever! This is probably the same issue as the syncing problem, but when I delete 1000s of messages from my Junk mail folder, it runs forever, and all other activities are delayed eve further. It turns out to be much more efficient to empty the Junk and Deleted Items folders using the webmail interface. And even then, Mail.app can take a while to delete locally-cached items from the folder when it syncs.
  • Suddenly, Julie is getting a lot less spam. She went from several hundred messages showing up in her Junk mailbox a few days ago to just five on Friday and two yesterday–one of which was a false positive). As she had been expecting a message from someone that she never got, this naturally made her very suspicious. Where is all the spam? Is she getting all of her mail?
  • Since FuseMail uses a mailbox named “Sent Items” instead of the traditional “Sent Messages” for all sent mail, I asked if they could move the 1.8 GB of messages from Julie’s Sent Messages to their Sent Items, since Mail.app would just choke on such a task. Though my request was escalated to the FuseMail developers, the answer came back “no.” Which I guess means that they’re not using Maildir, because in that case it would be a cinch, n’est pas?
  • Backups are not really feasible. Of course FuseMail has its own backup regimen, but if I ever want to move elsewhere or deal with some sort of catastrophic failure, I want my own backups. There is no rsync service available for this (remember: no maildir), so I have to use the IMAP interface. I’ve been trying for the past two weeks to get Offline IMAP to back up all of Julie’s and my mail, but it keeps choking. It gets a little further every time I run it; eventually it will get it all. But this only allows me to backup those accounts for which I happen to have a password. I have accounts set up for a few other users, but don’t have access to their passwords, so I can’t back them up. This does not make for very good support for corporate backup and retention policies.
  • Mail forwarded by FuseMail has its Return-Path header modified. This made RT break until I hacked it to ignore that header (which is its by-default preferred header for identifying senders.

So I’m pretty fed up. It took me a week to get all of our mail on FuseMail, and now I’m looking at moving it off again (once OfflineIMAP finishes a full sync). Grr. I’m considering finding a virtual host somewhere and setting up my own IMAP server again, but then I have the spam problem again. So then I could use a forwarding service like Pobox, or I can set up my own spam filtering (something I had hoped never to get into managing myself). My old IMAP server required very little maintenance, which was nice, but then the span filtering stuff always seemed daunting. Don’t you have to update things all the time?a

But before I go off and do something else, and unlike before I moved to FuseMail, I wanted to get an idea what other folks are doing? Do you use IMAP? Do you use it to manage a shitload (read: Gigabytes) of mail? Do you get very little spam and still get all of your valid mail? Are IMAP folder maintenance actions fast for you (in Mail.app in particular)? Are you paying a not-unreasonable amount of money for your setup? If you answered yes to all of these questions, please, for the love of all that is good in this world, tell me how you do it. I’m looking for something that I don’t have to work very hard to maintain (hence my original attempt to have some company that specializes in this stuff do it), but I’ll do what I have to to make this thing right. So how do you make it right? And if I have to run my own server, where should I host it that won’t cost me an arm and a leg?

Thanks for your help!

Looking for the comments? Try the old layout.

More about…

SVN::Notify 2.50

SVN::Notify 2.50 is currently making its way to CPAN. It has quite a number of changes since I last wrote about it here, most significantly the slick new CSS treatment introduced in 2.47, provided by Bill Lynch. I really like the look, much better than it was before. Have a look at the SVN::Notify::HTML::ColorDiff output to see what I mean. Be sure to make your browser window rally narrow to see how all of the sections automatically get a nice horizontal scrollbar when they’re wider than the window. Neat, eh? Check out the 2.40 output for contrast.

Here are all of the changes since the last version:

2.50 2005-11-10T23:27:22
  • Added --ticket-url and --ticket-regex options to be used by those who want to match ticket identifers for systems other than RT, Bugzilla, GNATS, and JIRA. Based on a patch from Andrew O’Brien.
  • Removed bogus use lib line put into Makefile.PL by a prerelease version of Module::Build.
  • Fixed HTML tests to match either “’” or “'”, since HTML::Entities can be configured differently on different systems.
2.49 2005-09-29T17:26:14
  • Now require Getopt::Long 2.34 so that the --to-regex-map option works correctly when it is used only once on the command-line.
2.48 2005-09-06T19:14:35
  • Swiched from <span class="add"> and <span class="rem"> to <ins> and <del> elements in SVN::Notify::HTML::ColorDiff in order to make the markup more semantic.
2.47 2005-09-03T18:54:43
  • Fixed options tests to work correctly with older versions of Getopt::Long. Reported by Craig McElroy.
  • Slick new CSS treatment used for the HTML and HTML::ColorDiff emails. Based on a patch from Bill Lynch.
  • Added --svnweb-url option. Based on a patch from Ricardo Signes.
2.46 2005-05-05T05:22:54
  • Added support for “Copied” files to HTML::ColorDiff so that they display properly.
2.45 2005-05-04T20:38:18
  • Added support for links to the GNATS bug tracking system. Patch from Nathan Walp.
2.44 2005-03-18T06:10:01
  • Fixed Name in POD so that SVN::Notify’s POD gets indexed by search.cpan.org. Reported by Ricardo Signes.
2.43 2004-11-24T18:49:40
  • Added --strip-cx-regex option to strip out parts of the context from the subject. Useful for removing parts of the file names you might not be interested in seeing in every commit message.
  • Added --no-first-line option to omit the first sentence or line of the log message from the subject. Useful in combination with the --subject-cx option.
2.42 2004-11-19T18:47:20
  • Changed “Files” to “Paths” in hash returned by file_label_map() since directories can be listed as well as files.
  • Fixed SVN::Notify::HTML so that directories listed among the changed paths are not links.
  • Requiring Module::Build 0.26 to make sure that the installation works properly. Reported by Robert Spier.

Enjoy!

Looking for the comments? Try the old layout.

Windows Virus Hell

So to finish up development and testing of Test.Harness.Browser in IE 6 last week, I rebooted my Linux server (the one running justatheory.com) into Windows 98, got everything working, and rebooted back into Linux. I felt that the hour or two’s worth of downtime for my site was worth it to get the new version of Test.Simple out, and although I had ordered a new Dell, didn’t want to wait for it. And it worked great; I’m very pleased with Test.Simple 0.20.

But then, in unrelated news, I released Bricolage 1.9.0, the first development release towards Bricolage 1.10, which I expect to ship next month. One of the things I’m most excited about in this release is the new PHP templating support. So on George Schlossnagle’s advice, I sent an email to webmaster@php.net. It bounced. It was late on Friday, and I’m so used to bounces being problems on the receiving end, that I simply forwarded it to George with the comment, “What the?” and went to fix dinner for company.

Then this morning I asked George, via IM, if he’d received my email. He hadn’t. I sent it again; no dice. So he asked me to paste the bounce, and as I did so, looked at it more carefully. It had this important tidbit that I’d failed to notice before:

140.211.166.39 failed after I sent the message.
Remote host said: 550-5.7.1 reject content [xbl]
550 See http://master.php.net/mail/why.php?why=SURBL

“That’s curious,” I thought, and went to read the page in question. It said I likely had a domain name in my email associated with a blacklisted IP address. Well, there were only two domain names in that email, bricolage.cc and justatheory.com, and I couldn’t see how either one of them could have been identified as a virus host. But sure enough, a quick search of the CBL database revealed that the IP address for justatheory.com—and therefore my entire home LAN— had been blacklisted. I couldn’t imagine why; at first I thought maybe it was because of past instances of blog spam appearing here, but then George pointed out that the listing had been added on August 18. So I thought back…and realized that was just when I was engaging in my JavaScript debugging exercise.

Bloody Windows!

So I took steps to correct the problem:

  1. Update my router’s firmware. I’ve been meaning to do that for a while, anyway, and was hoping to get some new firewall features. Alas, no, but maybe I’ll be able to connect to a virtual PPTP network the next time I need to.

  2. Blocked all outgoing traffic from any computer on my LAN on port 25. I send email through my ISP, but use port 587 because I found in the last year that I couldn’t send mail on port 25 on some networks I’ve visited (such as in hotels). Now I know why: so that no network users inadvertently send out viruses from their Windows boxes! I’d rather just prevent certain hosts (my Windows boxen) from sending on port 25, but the router’s NAT is not that sophisticated. So I have to block them all.

  3. Rebooted the server back into Windows 98 and installed and ran Norton AntiVirus. This took forever, but found and fixed two instances of WIN32Mimail.l@mm and removed a spyware package.

  4. Rebooted back into Linux and cleared my IP address from the blacklist databases. I don’t expect to ever use that box for Windows again, now that I have the new Dimension.

The new box comes with Windows XP SP 2 and the Symantec tools, so I don’t expect it to be a problem, especially since it can’t use port 25. But this is a PITA, and I really feel for the IT departments that have to deal with this shit day in and day out.

What I don’t understand is how I got this virus, since I haven’t used Windows 98 in this computer in a long time. How long? Here’s a clue: When I clicked the link in Norton AntiVirus to see more information on WIN32Mimail.l@mm, Windows launched my default browser: Netscape Communicator! In addition, I don’t think I’ve used this box to check email since around 2000, and I never click on attachments from unknown senders, and never .exe or .scr files at all (my mail server automatically rejects incoming mail with such attachments, and has for at least a year).

But anyway, it’s all cleaned up now, and I’ve un-blacklisted my IP, so my emails should be deliverable again. But I’m left wondering what can be done about this problem. It’s easy for me to feel safe using my Mac, Linux, and FreeBSD boxes, but, really, what keeps the Virus and worm writers from targeting them? Nothing, right? Furthermore, what’s to stop the virus and worm writers from using port 587 to send their emails? Nothing, right? Once they do start using 587—and I’m sure they will—how will anyone be able to send mail to an SMTP server on one network from another network? Because you know that once 587 becomes a problem, network admins will shut down that port, too.

So what’s to be done about this? How can one successfully send mail to a server not on your local network? How will business people be able to send email through their corporate servers from hotel networks? I can see only a few options:

  • Require them to use a mail server on the local network. They’ll have to reconfigure their mail client to use it, and then change it back when they get back to the office. What a PITA. This might work out all right if there was some sort of DNS-like service for SMTP servers, but then there would then be nothing to prevent the virus software from using it, either.
  • You can’t. You have to authenticate onto the other network using a VPN. Lots of companies rely on this approach already, but smaller companies that don’t have the IT resources to set up a VPN are SOL. And folks just using their ISPs are screwed, too.
  • Create a new email protocol that’s inherently secure. This would require a different port, some sort of negotiation and authentication process, and a way for the hosting network to know that it’s cool to use. But this probably wouldn’t work, either, because then the virus software can also connect via such a protocol to a server that’s friendly to it, right?

None of these answers is satisfactory. I guess I’ll have to set up an authenticating SMTP server and a VPN for Kineticode once port 587 starts getting blocked. Anyone else got any brilliant solutions to this problem?

Looking for the comments? Try the old layout.

SVN::Notify 2.41 Adds Plain Text Issue Tracking Links

I expect that this will be my last release of SVN::Notify for a while. I’ve already spent more time on it than I had anticipated. But anyway, this is a pretty solid release. It doesn’t change the API or anything, but I feel that the jump from 2.30 to 2.40 is justified because of the sheer number of changes. From now on, I expect that it will mostly be maintenance, like 2.41, which fixes a minor formatting bug. Grab it now from CPAN.

First, I’ve added a new, complex example of the SVN::Notify::HTML::ColorDiff output that I will keep up-to-date with all future changes. This will allow people to get a better idea of what it’s capable of than my previous contrived examples allowed.

The biggest change is that I’ve moved the Request Tracker, Bugzilla, and JIRA support from SVN::Notify::HTML to SVN::Notify. I realized, after the release of 2.30, that it might be cool to add links to the text-only email message generated by SVN::Notify, too. So I’ve done that, including for ViewCVS links. Unlike in SVN::Notify::HTML, the links won’t be inline in the message (that doesn’t work too well in plain text, IMO), but will come in their own sections after the message. So you’ll get something like this (extreme example):

Log Message:
-----------
Let's try a few links to other applications. First, we have
A Bugzilla Bug # 709. Then we have a JIRA key, TST-1608. And
finally, we have an RT link to Ticket # 4321.

Hey, we could add one to ViewCVS for a Subversion Revision
#606, too!

ViewCVS Links:
-------------
    http://viewsvn.bricolage.cc/?rev=606&view=rev

Bugzilla Links:
--------------
    http://bugzilla.mozilla.org/show_bug.cgi?id=709

RT Links:
--------
    http://rt.cpan.org/NoAuth/Bugs.html?id=4321

JIRA Links:
----------
    http://jira.atlassian.com/secure/ViewIssue.jspa?key=TST-1608

The nice thing is that, for many mail clients, these will be turned into clickable links. You’ll also notice that the text that creates the ViewCVS link is split over two lines. This is new in this release, and works for SVN::Notify::HTML, too. I made a few other tweaks to the regular expressions, as well. Here’s a complete list of changes:

  • Fixed accessor generation so that accessors created for the attributes passed to register_attributes() but a subclass are created in the subclass’ package instead of in SVN::Notify.
  • Changed parsing for JIRA keys to use any set of capital letters followed by a dash and then a number, rather than the literal string “JIRA-” followed by a number. Reported by Garrett Rooney.
  • Modified the regular expression patterns for the RT, Bugzilla, RT, and ViewCVS links to properly match on word boundaries, so that strings like “humbug 12” don’t match.
  • Modified the ViewCVS link regular expression pattern so that it matches strings like “rev 12” as well as “revision 12”.
  • Modified the RT link regular expression pattern so that it matches strings like “RT-Ticket: 23” as well as “Ticket 1234”. Suggested by Jesse Vincent.
  • Added complicated example to try to show off all of the major features. I will keep this up-to-date going forward in order to post sample output on the Web.
  • Fixed the parsing of log messages so that empty lines are no longer eliminated.
  • HTML::ColorDiff now properly handles the listing of binary files in the diff, marking them with a new class, “binary”, and using the same CSS as is used for the “propset” class.
  • In HTML::ColorDiff, Fixed CSS for the “delfile” class to properly wrap it in a border like the other files in the diff.
  • Added labels to the HTML::ColorDiff diff file sections to indicate the type of change (“Modified”, “Added”, “Deleted”, or “Property changes”).
  • Moved the rt_url, bugzilla_url, and jira_url parameters from SVN::Notify::HTML to SVN::Notify, where they are used to add URLs to the text version of log messages.

Enjoy!

Looking for the comments? Try the old layout.

SVN::Notify 2.30 Adds Issue Tracking Links

I released a new version of SVN::Notify last night, 2.30. This new version has a few things going for it.

First, and most obviously from the point of view of users of the HTML subclass, I’ve added new options for specifying Request Tracker, Bugzilla, and JIRA URLs. The --rt-url, --bugzilla-url, and --jira-url options have an effect much like the parallel feature in CVSspam: pass in a string with the spot for the ID represented by %s, such as http://rt.cpan.org/NoAuth/Bugs.html?id=%s for RT or http://bugzilla.mozilla.org/show_bug.cgi?id=%s for Bugzilla. SVN::Notify::HTML will then look for the appropriate strings (such as “Ticket # 1234” for RT or “Bug # 4321” for Bugzilla) and turn them into URLs.

This functionality has been extended to the old --viewcvs-url option, to. For the sake of consistency, it now also requires a URL of the same form (although if SVN::Notify doesn’t see %s in the string, it will append a default and emit a warning), and will be used to create links for strings like “Revision 654” in the log message.

SVN::Notify::HTML has an additional new option, --linkize, that will force any email addresses or URLs it finds in the log message to be turned into links. Again, this works like it does for CVSspam; I’m grateful to Jeffrey Friedl’s Mastering Regular Expressions, Second Edition for the excellent regular expressions for matching URLs and email addresses.

All of this was made possible by moving the processing of options from svnnotify to SVN::Notify->get_options and adding a new class method, SVN::Notify->register_attributes. This second method allows Bricolage subclasses to easily add new attributes; register_attributes() will create accessor methods and add command-line option processing for each new attribute required by a subclass. Then, when you execute svnnotify --handler HTML, SVN::Notify->get_options processes the default options, loads the SVN::Notify::Handler subclass, and then processes any options specified by the subclass. The short story is that all of this is the detail-oriented way of saying that it is easier to subclass SVN::Notify and be able to automatically load the necessary options and attributes via the same executable, svnnotify.

This change was motivated not only by my desire to add the new features to SVN::Notify::HTML, but also by Autrijus’ new modules, SVN::Notify::Snapshot and SVN::Notify::Config. Thanks Autrijus!

I’ll try to get a nice example of all this functionality up in the next few days; if anyone else creates one first, send it to me! But in the meantime, enjoy!

Looking for the comments? Try the old layout.