In the last episode of Adventures in De-Goggling, I laid out the principles behind my desire to reduce the amount of intrusive Google privacy violations in my life. The next step was to try to migrate one Google account to ProtonMail and see how it went. This post is the result of that next step.
The quick summary of this post is:
imapsyncis a great tool
- ProtonMail isn’t the right solution for me.
Read on for the gory details…
One of my oldest domains has been in continuous operation by me since 1998. For security purposes let’s just call this
graybeard.org. Over the years
graybeard.org has used many different email solutions, most of which were
self-hosted and managed by me until Google came along. For at least the last several years it’s been using Google’s
hosted service for email (this is called “G Suite” now but initially it was a free tier of something called “Google Apps
for Domains”, and being grandfathered into this old scheme I’ve never paid anything to Google for hosting all of my
email). On this particular domain I don’t use any other G Suite features, not even contacts or calendars. So this one
should be a great candidate for migration to ProtonMail.
I have a Visionary account with ProtonMail, paid anonymously with cryptocurrency. This entitles me to all of the premium ProtonMail features, including the one that is required in order to perform this migration: the ProtonMail Bridge. At the time of this writing the Linux version of the bridge was in closed beta, so I had to submit a request to be granted access, but that request was granted promptly.
I already added
graybeard.org to ProtonMail’s configuration as a custom domain, although of course I didn’t actually
update the DNS
MX records until the migration was confirmed.
I’m going to use
imapsync to perform the migration by simply reading all of the messages from the Gmail IMAP servers, and
re-creating them on the IMAP server exposed by the ProtonMail bridge. This isn’t as efficient as I would have liked,
but it seems to be the best available option at the time of this writing. Though ProtonMail now have an import/export
app in beta, this was not available when I first performed this
migration. I can’t comment on the quality of this tool, maybe it’s rock solid but I doubt it’s more robust than
I’ll do one big migration first, while Google are still hosting the
graybeard.org mail. Once it completes
successfully and my spot checks in ProtonMail leave me satisfied that the migration properly handled all metadata and
attachments and such, I’ll update the MX records to switch over to ProtonMail for incoming email, then re-run the
migration to pick up whatever messages came into Google in the meantime.
Fortunately there are only two users on
graybeard.org in total, so coordinating this migration will be easy. If
I had more than a handful of users this would need to be done more carefully, but thankfully that’s not my problem.
I don’t want to write a post that duplicates the existing documentation for ProtonMail, ProtonMail Bridge, and
imapsync. However I do want to make a few notes about the initial setup which might not be obvious.
This approach requires you have ProtonMail Bridge installed and running on the same system that will be running
imapsync. If you don’t have a paid ProtonMail plan, you’re out of luck. The ProtonMail docs cover setting up the
bridge in great detail, so read all about it there. Suffice to say that I had the bridge set up and working, which
I verified by using it with Thunderbird.
I also used “Switch to split addresses” mode in the Bridge, because I want each address (
firstname.lastname@example.org, etc) to be presented via IMAP as its own account. I suggest you do this also, as there is
essentially no support for switching between multiple ProtonMail user accounts the way that Google’s apps allow you to
do. When the Bridge is configured as I suggest, it will generate and show in the UI a separate password for each email
address, so that IMAP clients must be configured to log in to the Bridge with one login per email address, as if these
were separate accounts. In ProtonMail itself they are not separate, but the Bridge presents this illusion to IMAP
clients, in our case
Google Security Config
Google’s default security settings are strict enough that it’s probably not possible to perform the migration; in any case I couldn’t figure it out. You’ll have to disable the more advanced security settings in order for this to work. As long as you have strong passwords which aren’t shared with any other sites there shouldn’t be a significant risk in doing this.
I logged into my Google account, went to My Account and then Sign-in and Security, all the way at the end of the page is an option “Allow less secure apps”. This needs to be enabled.
Reading the man pages for
imapsync inspires great confidence. The tool has the feel of an instrument which has been
refined over many long years of in-the-trenches use, with flags for all manner of edge cases. I found perusing the
imapsync Gmail FAQ to be quite useful as
preparation for this migration. You’d be wise to do the same.
Understanding Folders and Labels in ProtonMail
Unfortunately ProtonMail Bridge has a strange way of exposing the structure of its folders and labels. In the root of
the IMAP tree it’s not possible to create any folders. New folders go under the
Folders/ folder and labels under the
Labels/ folder. Messages moved into one of the
Labels/ folders are not moved there but labeled with that label,
while messages moved to a
Folders/ subfolder are moved to that folder. It’s stupid and I can’t understand why they
would take this approach.
This requires the use of complicated regex trickery to map properly. In Gmail, the folder
Inbox represents the inbox,
and other folders under the
[Gmail] folder correspond to actual folders. All top-level folders other than
Inbox are actually labels. So our challenge is to tell
imapsync how to map the labels to “folders” under
Labels/ in the ProtonMail bridge, while leaving the real GMail folders alone. Fortunately
imapsync is flexible
enough to support this via its
regextrans2 option, as you’ll see below.
For now just understand that this conceptual difference exists, particularly when you’re trying to navigate your ProtonMail email via an IMAP client like Thunderbird.
Running the migration
I opened a terminal window on my Arch Linux system to run these commands. I would have preferred to use a VPS for better persistence, but it wasn’t obvious how to run the ProtonMail Bridge headless. It should go without saying that you must not interrupt this process, by turning off your computer or letting it go to sleep.
The actual incantation to make
imapsync work looks like this:
imapsync -gmail1 --user1 email@example.com \ --host2 127.0.0.1 --user2 firstname.lastname@example.org --password2 BRIDGE_PASSWORD_HERE \ --port2 1143 \ --regextrans2 's/^((?!INBOX|\[Gmail\]).+)$/Labels\/$1/' \ --regextrans2 's/^\[Gmail\]\/Starred$/Labels\/Starred/' \ --regextrans2 's/^\[Gmail\]\/Important/Labels\/Important/' \ --regextrans2 's/^\[Gmail\]\/Drafts/Labels\/Drafts/' \ --exclude '^\[Mailbox]\/.+$'
This will prompt on STDIN for the Google account’s password. You can avoid that by passing it on the command line with
--password1, but I didn’t do that because it is foolish to put credentials on the CLI. It’s ok for
ProtonMail Bridge password) because that’s only used on this local system to connect to the ProtonMail Bridge. You can
--password2 and be prompted for both passwords each time you run the command.
--gmail1 option automatically configures
imapsync to use Google’s IMAP servers as the input. This saves
a lot of duplication, and importantly also throttles IMAP operations to one message per second. Google apparently
rate-limit their IMAP interface so slamming it too fast will get your IP banned, and that’s not fun for anyone.
Unfortunately this means the migration is slow. How slow? I have about 3GB of email, almost 60,000 individual messages, and it took three days to run. Your mileage may vary.
Assuming everything works, after a very long time the migration will finish. Remember that I initiated this migration
while Google was still the mail handler for
graybeard.org according to the MX records. This means that I received
about three days’ worth of mail while the migration is running. Fortunately the
imapsync command is idempotent; it
can be run repeatedly it won’t create a duplicate copy of already-migrated messages.
So, once this migration finished and I spot-checked a few messages to ensure they migrated properly, I switched the MX records over to ProtonMail and ran the migration again. It took another three days, after which I had every last one of my emails migrated to ProtonMail.
As I write this I’ve had about a month of experience with ProtonMail as the host of record for one of my domains. As much as I really want ProtonMail to succeed, and as much as I support their philosophical stance on privacy, frankly I’ve had a pretty shitty experience overall, and won’t be migrating any more domains to ProtonMail.
I don’t want this to turn into a rant, but here’s a quick list of issues I’ve run into. If you’re considering migrating to ProtonMail, don’t let this dissuade you, but do make sure you understand each of these issues and be prepared to deal with them if they matter to you.
ProtonMail Bridge Sucks
According to their own FAQ:
On macOS, we have tested the Bridge on Apple Mail, Thunderbird, and Outlook 2011/2016. On Windows, we have tested the Bridge on Thunderbird and Outlook 2010/2013/2016. Every client implements the IMAP standard slightly differently, so we cannot make any guarantees about how the Bridge will behave on clients other than the ones listed.
I thought that was just the usual caveat from a cautious software engineer. I mean, of course they can’t guarantee it will work with other clients.
But in fact what this means is that it pretty much will not work with other clients. For example, MailSpring doesn’t work at all. On Linux, you are stuck with Thunderbird. If you like Thunderbird then I guess that’s not a problem for you, but despise it and can’t bring myself to use it.
Even if you like Thunderbird, you still should be prepared for random hangs or crashes of the Bridge, and often Thunderbird operations will time out.
I can’t speak to the quality of the bridge on Windows or macOS but on Linux it’s rubbish.
Mobile and Web don’t sync
If you archive a message in the Web interface, you’ll still see it on the Android client, and vice versa. Sometimes I see messages on one device that I already archived on another. It’s maddening.
No multi-user support
Google’s apps and web interface all work well with multiple user accounts. You can set up multiple logins, and easily switch between them in the UI. In the GMail mobile app you can see unified list of all messages across all your logins.
ProtonMail can’t do that. You literally have to log out of one account and into another. It’s useless.
Recall that I chose
graybeard.org precisely because I don’t use the Google Calendar there. But I do rely daily on
Google Calendar on another Google account, and as a result I cannot migrate that account to ProtonMail. They claim this
is in the backlog but I am tired of waiting.
Sluggish, glitchy web interface
Because of all the problems with the Bridge, I use the web interface on my desktop and laptop systems. It’s…not fun.
It often spins up the fan, and the keyboard shortcuts are not consistently responsive. In particular, a common workflow for me is
to multi-select several messages in the inbox using keyboard shortcuts, then archive them. The “archive” shortcut it
e which Google has drilled into my muscle memory, but even after I remember that and press
a, it often
doesn’t actually archive, or archives all but one of the messages, or archives but after I’ve given up waiting and move
the cursor to click the “Archive” button.
Perhaps I’ve just been spoiled by the Gmail web interface, but ProtonMail’s feels like going back in time to a much less pleasant era. It’s not exactly RoundCube-level bad, but I curse it every day.
ProtonMail as it exists at the end of 2018 is not robust enough to take the place of Google for my email and calendaring needs. I’ll migrate another domain over to Fastmail in the hopes that’s a better result.
Bonus conclusion: despite being written in Perl (!!!),
imapsync is great. I heartily recommend it for all your IMAP