Moving away from the cloud Part III - Mail

Sun 14 April 2013 by Javex

Note

This is the third part of the series of how to pull your data out of the cloud. You can find the introduction together with a list of currently published articles here.

Mail

In this entry, I will go over how to migrate your mail. Mail is a really complicated issue and I will not go over all those things I did which are already covered by good guides. I combined many guides, searched google many times and invested at least two full days + some hours over the following weeks to get this working. This is the hardest part of all the tasks I performed but it is also the most rewarding and useful.

Think about it this way: Much of your private conversation goes through eMail and you share a lot of private information in it (or maybe even business information) and you certainly don't want that in someone else's hand. I would probably publish my whole calendar and contacts list - there is no confidential information in it. Mails on the other hand contain a lot of that. So while this part may be frustrating I strongly encourage you to do this - it is worth the effort.

First of all, let me describe what we will use in our setup, then I will go over what I did with them and how I did it. I use the following programs:

Postfix - Part I

Postfix is the so-called mail transfer agent (MTA). It will be your "actual" mail server. You can see it this way: Postfix accepts mail from the outside, it sends mail to the outside and you can contact it directly from your commandline and don't need any other application. However, that would be really complicated and that's why we need additional tools.

But first of all to Postfix: You need to install it (there's most likely a package in your package manager for it) and then configure it. Configuring Postfix to your needs is a really time-intense process and so you will want to take your time, read documentation, seach guides and do your best to actually understand how Postfix works.

For my goal, I wanted to have a virtual setup. That's a little more complicated but it adds a nice benefit: You gain a lot of flexibility and don't rely on Unix users to recieve and deliver mail. Also it's a lot easier to add users later on.

But here is where the first complicated step happens: I manage those users with Postfix Admin and so it is probably a good idea if you take your interest in that right now. We can get back to Postfix once we are done with it.

Postfix Admin

Postfix Admin is a small web-application, designed to help you manage your virtual users. It has its own layout for virtual users and so it is a good idea to set that one up first. Download it (see URL above) and read its documentation. It is easy to set up and so I won't cover that here.

Note

While Postfix Admin has a login mechanism itself, I do not trust it. Thus, I also protected the site with .htaccess using Nginx. That's not a requirement but I recommend it, because it doesn't make any problems and you won't access it a lot anyways.

After you have read about Postfix Admin and set it up, you probably have already been introduced to some Postfix configuration files. Use them, combine them with other guides and you can get back to Postfix

Postfix - Part II

After you have set up Postfix Admin, it is not hard to get the rest of Postfix running. I made it this way around because otherwise you will probably end up with a database schema that you have to change later on. Also, you already get some useful hints from configuring it, so when you now configure Postfix, it will be easier for you.

So again: Read some guides, use your already existing schema and configuration and set up Postfix. Hopefully your guide already had a description on how to ensure it is working, otherwise error tracking later on might be hard.

Note

Make sure to add at least one user that you can test with.

Note

Be aware that you must have port 25 open. I closed it because I use SSL on all services, but some mail servers on the outside don't. And they won't be able to send you mail (it took me some time to notice that that's the problem of missing mails).

Dovecot

Dovecot is what adds some use to your mail server. Know of IMAP? Well, without Dovecot you cannot send mail from outside your server (at least not in the regular way). So head over to their website, install & configure it and you will be able to use IMAP to recieve mails. Sending will still be done through SMTP and that's Postfix' job.

You have to get this straight: Postfix is actually responsible for sending and recieving mails. But you can delegate the delivery to another client (e.g. Dovecot). Now Dovecot provides the IMAP service and some other useful stuff so you really want that kind of bonus.

If you got it installed and running, test that it's working and then we can do the more fancy stuff. This is the easier and more fun part: Contrary to any foreign server, you now have full control over what happens to your mail on the server's side. Even before a mail reaches your inbox you can do an awful lot of stuff with it, like filtering (spam, custom filters, etc.) and even some editing (e.g. I tag some mails before they reach me so they are automatically marked Important).

Sieve

Sieve is part one of the fun: It is a server side filtering system. Read its documentation and see how super-powerful it is. For example, I sort log mails I recieve into a special folder even before it reaches my inbox. Thus, the Thunderbird client side filtering I had to do is now done by the server.

Go read the documentation on it on the Dovecot Wiki (see above). However, here is a small example I use similarly:

require ["fileinto", "imap4flags"];

if address :is "to" "[email protected]" {
    setflag "\\Seen";
    fileinto "Logs";
} elsif address :contains "to" "@domain.com" {
    redirect "[email protected]";
}

if address :is "from" "[email protected]" {
    setflag "important";
}

This script has three parts: The first condition checks for the logs recipient address (that is actually a foreign server forwarding mail to me). The script sees the log target and stores it into my Logs folder. It also marks it as seen so I don't get bugged by it in any way.

Step two is mail from the same domain, but not a log. I forward them to another domain where I read and handle them.

The third condition checks for a sender called icinga which is my server monitoring (see later post). The only relevant part here is that I deem mails from this address as important and so I set the flag important. Note that this is not a special flag at all, but in Thunderbird I have set this flag to a red color (they call them tags), so when I open it and see that mail, I know it requires my attention now.

And here is what I had to do to the Dovecot config to activate it:

protocol lda {
  ...
  mail_plugins = $mail_plugins sieve
}

plugin {
    sieve = /etc/dovecot/sieve/%u.sieve
}

The %u will be replaced by the username and so you can do per-user configuration.

Roundcube

Roundcube is fun part two. It is a web mail client much like GMail and while being extremely advanced it does not match all the abilities that the GMail web client has. Nonetheless, it is a very powerful tool and surely sufficient to read mail on the go. If the features it provides are not enough, you can always use a desktop client (e.g. Thunderbird) or use/write plugins for Roundcube.

Note

This part is optional, so if you don't need this web-based access, then ignore this section and use a desktop client instead.

Download and install roundcube on your server. You may protect this login with .htaccess as well, but I don't recommend it. You will probably access it a lot and that might annoy you (though you might want the added layer of security).

During the configuration phase you can enter the SMTP and IMAP addresses that you configured earlier on. Then you have working webmail client.

Now, if you have read the part on Contacts you might want to export your contacts from there and import them into Roundcube. I have not found a way for automatic synchronization (though I have to admit, I didn't really search).

Roundcube & Sieve

You have now seen the power of Roundcube & Sieve and you may also now about Thunderbird's client side filters. Well there's actually something better if you use Roundcube: You can use server-side filtering with Sieve but don't have to deal with configuring it on the server's side.

On the server side you will have to activate managesieve:

plugin {
    sieve = .../%u.sieve
    sieve_dir = ...
}

That's actually it: You don't need more than those lines and the sieve line is most likely already present if you configured Sieve above. Just set the directory there as well and you are fine here.

Note

Mabe a backup of your sieve scripts if you already have some.

And for Roundcube edit the main.inc.php file:

<?php
$rcmail_config['plugins'] = array('managesieve');
$rcmail_config['managesieve_port'] = 4190;
?>

Should be fairly self-explanatory. Now you can even manage your filters with a pretty GUI :-)

Conclusion

This concludes my coverage of the Mail topic: It is a setup of multiple clients, multiple new open ports and multiple configuration files. It can get pretty nasty, but in the end, you get a really strong flexibility and so I find this totally worth it.

Also you now only have a quota limited by your disk size and you can control everything yourself.

Note

I did not talk about spam filtering here. You will probably need that in the future, but there are a lot of guides out there, so this should be a non-issue.

If you have any questions, feel free to contact me.


Comments