Adventures in De-Googling (Part 2) - ProtonMail

anelson October 15, 2018 #degoggling #tech

In the last episode of Adventures in De-Goggling, I laid out the principles behind my desire to reduce the amount of intrusive Google privacy violations in my life. The next step was to try to migrate one Google account to ProtonMail and see how it went. This post is the result of that next step.

TL;DR

The quick summary of this post is:

imapsync is a great tool
ProtonMail isn’t the right solution for me.

Read on for the gory details…

Background

One of my oldest domains has been in continuous operation by me since 1998. For security purposes let’s just call this one graybeard.org. Over the years graybeard.org has used many different email solutions, most of which were self-hosted and managed by me until Google came along. For at least the last several years it’s been using Google’s hosted service for email (this is called “G Suite” now but initially it was a free tier of something called “Google Apps for Domains”, and being grandfathered into this old scheme I’ve never paid anything to Google for hosting all of my email). On this particular domain I don’t use any other G Suite features, not even contacts or calendars. So this one should be a great candidate for migration to ProtonMail.

I have a Visionary account with ProtonMail, paid anonymously with cryptocurrency. This entitles me to all of the premium ProtonMail features, including the one that is required in order to perform this migration: the ProtonMail Bridge. At the time of this writing the Linux version of the bridge was in closed beta, so I had to submit a request to be granted access, but that request was granted promptly.

I already added graybeard.org to ProtonMail’s configuration as a custom domain, although of course I didn’t actually update the DNS MX records until the migration was confirmed.

The Plan

I’m going to use imapsync to perform the migration by simply reading all of the messages from the Gmail IMAP servers, and re-creating them on the IMAP server exposed by the ProtonMail bridge. This isn’t as efficient as I would have liked, but it seems to be the best available option at the time of this writing. Though ProtonMail now have an import/export app in beta, this was not available when I first performed this migration. I can’t comment on the quality of this tool, maybe it’s rock solid but I doubt it’s more robust than imapsync.

I’ll do one big migration first, while Google are still hosting the graybeard.org mail. Once it completes successfully and my spot checks in ProtonMail leave me satisfied that the migration properly handled all metadata and attachments and such, I’ll update the MX records to switch over to ProtonMail for incoming email, then re-run the migration to pick up whatever messages came into Google in the meantime.

Fortunately there are only two users on graybeard.org in total, so coordinating this migration will be easy. If I had more than a handful of users this would need to be done more carefully, but thankfully that’s not my problem.

Initial Setup

I don’t want to write a post that duplicates the existing documentation for ProtonMail, ProtonMail Bridge, and imapsync. However I do want to make a few notes about the initial setup which might not be obvious.

ProtonMail Bridge

This approach requires you have ProtonMail Bridge installed and running on the same system that will be running imapsync. If you don’t have a paid ProtonMail plan, you’re out of luck. The ProtonMail docs cover setting up the bridge in great detail, so read all about it there. Suffice to say that I had the bridge set up and working, which I verified by using it with Thunderbird.

I also used “Switch to split addresses” mode in the Bridge, because I want each address ([email protected], [email protected], etc) to be presented via IMAP as its own account. I suggest you do this also, as there is essentially no support for switching between multiple ProtonMail user accounts the way that Google’s apps allow you to do. When the Bridge is configured as I suggest, it will generate and show in the UI a separate password for each email address, so that IMAP clients must be configured to log in to the Bridge with one login per email address, as if these were separate accounts. In ProtonMail itself they are not separate, but the Bridge presents this illusion to IMAP clients, in our case imapsync.

Google Security Config

Google’s default security settings are strict enough that it’s probably not possible to perform the migration; in any case I couldn’t figure it out. You’ll have to disable the more advanced security settings in order for this to work. As long as you have strong passwords which aren’t shared with any other sites there shouldn’t be a significant risk in doing this.

I logged into my Google account, went to My Account and then Sign-in and Security, all the way at the end of the page is an option “Allow less secure apps”. This needs to be enabled.

`imapsync`

Reading the man pages for imapsync inspires great confidence. The tool has the feel of an instrument which has been refined over many long years of in-the-trenches use, with flags for all manner of edge cases. I found perusing the imapsync Gmail FAQ to be quite useful as preparation for this migration. You’d be wise to do the same.

Understanding Folders and Labels in ProtonMail

Unfortunately ProtonMail Bridge has a strange way of exposing the structure of its folders and labels. In the root of the IMAP tree it’s not possible to create any folders. New folders go under the Folders/ folder and labels under the Labels/ folder. Messages moved into one of the Labels/ folders are not moved there but labeled with that label, while messages moved to a Folders/ subfolder are moved to that folder. It’s stupid and I can’t understand why they would take this approach.

This requires the use of complicated regex trickery to map properly. In Gmail, the folder Inbox represents the inbox, and other folders under the [Gmail] folder correspond to actual folders. All top-level folders other than [Gmail]/ and Inbox are actually labels. So our challenge is to tell imapsync how to map the labels to “folders” under Labels/ in the ProtonMail bridge, while leaving the real GMail folders alone. Fortunately imapsync is flexible enough to support this via its regextrans2 option, as you’ll see below.

For now just understand that this conceptual difference exists, particularly when you’re trying to navigate your ProtonMail email via an IMAP client like Thunderbird.

Running the migration

I opened a terminal window on my Arch Linux system to run these commands. I would have preferred to use a VPS for better persistence, but it wasn’t obvious how to run the ProtonMail Bridge headless. It should go without saying that you must not interrupt this process, by turning off your computer or letting it go to sleep.

The actual incantation to make imapsync work looks like this:

imapsync -gmail1 --user1 [email protected] \
  --host2 127.0.0.1 --user2 [email protected] --password2 BRIDGE_PASSWORD_HERE \
  --port2 1143 \
  --regextrans2 's/^((?!INBOX|\[Gmail\]).+)$/Labels\/$1/' \
  --regextrans2 's/^\[Gmail\]\/Starred$/Labels\/Starred/' \
  --regextrans2 's/^\[Gmail\]\/Important/Labels\/Important/' \
  --regextrans2 's/^\[Gmail\]\/Drafts/Labels\/Drafts/' \
  --exclude '^\[Mailbox]\/.+$'

This will prompt on STDIN for the Google account’s password. You can avoid that by passing it on the command line with --password1, but I didn’t do that because it is foolish to put credentials on the CLI. It’s ok for --password2 (the ProtonMail Bridge password) because that’s only used on this local system to connect to the ProtonMail Bridge. You can optionally omit --password2 and be prompted for both passwords each time you run the command.

Using the --gmail1 option automatically configures imapsync to use Google’s IMAP servers as the input. This saves a lot of duplication, and importantly also throttles IMAP operations to one message per second. Google apparently rate-limit their IMAP interface so slamming it too fast will get your IP banned, and that’s not fun for anyone.

Unfortunately this means the migration is slow. How slow? I have about 3GB of email, almost 60,000 individual messages, and it took three days to run. Your mileage may vary.

Assuming everything works, after a very long time the migration will finish. Remember that I initiated this migration while Google was still the mail handler for graybeard.org according to the MX records. This means that I received about three days’ worth of mail while the migration is running. Fortunately the imapsync command is idempotent; it can be run repeatedly it won’t create a duplicate copy of already-migrated messages.

So, once this migration finished and I spot-checked a few messages to ensure they migrated properly, I switched the MX records over to ProtonMail and ran the migration again. It took another three days, after which I had every last one of my emails migrated to ProtonMail.

Post-migration experience

As I write this I’ve had about a month of experience with ProtonMail as the host of record for one of my domains. As much as I really want ProtonMail to succeed, and as much as I support their philosophical stance on privacy, frankly I’ve had a pretty shitty experience overall, and won’t be migrating any more domains to ProtonMail.

I don’t want this to turn into a rant, but here’s a quick list of issues I’ve run into. If you’re considering migrating to ProtonMail, don’t let this dissuade you, but do make sure you understand each of these issues and be prepared to deal with them if they matter to you.

ProtonMail Bridge Sucks

According to their own FAQ:

On macOS, we have tested the Bridge on Apple Mail, Thunderbird, and Outlook 2011/2016. On Windows, we have tested the Bridge on Thunderbird and Outlook 2010/2013/2016. Every client implements the IMAP standard slightly differently, so we cannot make any guarantees about how the Bridge will behave on clients other than the ones listed.

I thought that was just the usual caveat from a cautious software engineer. I mean, of course they can’t guarantee it will work with other clients.

But in fact what this means is that it pretty much will not work with other clients. For example, MailSpring doesn’t work at all. On Linux, you are stuck with Thunderbird. If you like Thunderbird then I guess that’s not a problem for you, but despise it and can’t bring myself to use it.

Even if you like Thunderbird, you still should be prepared for random hangs or crashes of the Bridge, and often Thunderbird operations will time out.

I can’t speak to the quality of the bridge on Windows or macOS but on Linux it’s rubbish.

Mobile and Web don’t sync

If you archive a message in the Web interface, you’ll still see it on the Android client, and vice versa. Sometimes I see messages on one device that I already archived on another. It’s maddening.

No multi-user support

Google’s apps and web interface all work well with multiple user accounts. You can set up multiple logins, and easily switch between them in the UI. In the GMail mobile app you can see unified list of all messages across all your logins.

ProtonMail can’t do that. You literally have to log out of one account and into another. It’s useless.

No calendaring

Recall that I chose graybeard.org precisely because I don’t use the Google Calendar there. But I do rely daily on Google Calendar on another Google account, and as a result I cannot migrate that account to ProtonMail. They claim this is in the backlog but I am tired of waiting.

Sluggish, glitchy web interface

Because of all the problems with the Bridge, I use the web interface on my desktop and laptop systems. It’s…not fun. On my 2018 XPS 13 it’s quite sluggish, presumably due to all the asymmetric key crypto they’re running in Javascript. It often spins up the fan, and the keyboard shortcuts are not consistently responsive. In particular, a common workflow for me is to multi-select several messages in the inbox using keyboard shortcuts, then archive them. The “archive” shortcut it a, not e which Google has drilled into my muscle memory, but even after I remember that and press a, it often doesn’t actually archive, or archives all but one of the messages, or archives but after I’ve given up waiting and move the cursor to click the “Archive” button.

Perhaps I’ve just been spoiled by the Gmail web interface, but ProtonMail’s feels like going back in time to a much less pleasant era. It’s not exactly RoundCube-level bad, but I curse it every day.

Conclusion

ProtonMail as it exists at the end of 2018 is not robust enough to take the place of Google for my email and calendaring needs. I’ll migrate another domain over to Fastmail in the hopes that’s a better result.

Bonus conclusion: despite being written in Perl (!!!), imapsync is great. I heartily recommend it for all your IMAP migration needs.