Why can't anyone get IMAP right? - Technology Evangelist

I'm extremely flustered with e-mail right now. When it comes to options for retrieving e-mail we have POP3 protocol or IMAP4 protocol. POP3 is basically one way and only works with the inbox and only the inbox. You can grab the messages from the inbox but if you have 4 computers you can't keep them in sync. Read a message on computer 1 and computer 2 has no idea if you have read the message. IMAP aims to fix this by keeping all messages not only on the server but also keeps the flags for each of these messages in sync. Computer 1 reads a message and computer 2, 3 and 4 all show that message as read. Reply to a message and all computers show that the message was replied to and have a copy of the reply so you can access it anywhere. Sounds great, right?

WRONG!

The IMAP protocol itself is pretty good although it is missing commands like 'move' but these problems are easily overcome. The biggest problem with IMAP are the clients available. Finding a good e-mail client that not only supports mail but also calendar and contacts is darned near impossible for Windows and while easier for the Mac it is still missing features. Frankly whatever client I go with should be cross-platform so I can use whatever computer I want.

Lets look at Windows first. I have a couple of options here for e-mail but as soon as I want to organize my life it becomes much more difficult. Outlook 2003/2007 has all the features I want but Microsoft has severely crippled IMAP. If I want a global inbox of my mail messages which also has my calendar and contact data I can't do it. I have to have a completely different PST file for those objects and they show in 'Personal Folders' while my IMAP data is in a completely different area. While PSTs offer more than 2GB now they only do so for POP3 and MAPI, not for IMAP. When you want to delete or move a message it won't actually process the commands on the server like you would expect, instead it merely changes the flag and copies the message. You have to purge the mailbox if you want to get rid of the messages with lines through them or hide messages marked for deletion. All in all it is possible to get a system moderately working, but in the end it is a royal pain that only a true techie would endure. Clearly Microsoft wants to sell more Exchange servers (MAPI) and does not care about IMAP at all. End result: I won't be buying MS Office any time soon.

Thunderbird is another option although the future of the project is a bit unknown. The problem with Thunderbird is that the UI is a bit unpolished, still no global inbox, and the calendar plugin Lightning which is based on Sunbird is really not ready for release. While there is contact support it is really, really weak. Want to sync your Thunderbird contacts to your iPhone? I have yet to find a good way though there are posts that try and outline methods that may help. Thunderbird is a good work in progress, assuming it will still move forward, but still nowhere near a good polished e-mail client. Sunbird and Lightning are even further off in terms of a complete project and it is just not ready for prime time yet. My calendar, contacts and e-mail should be seamlessly linked (not necessarily one application but definitely linked together) with a well polished, consumer friendly interface. Thunderbird/Lightning/Sunbird are great for the uber techies but don't help push IMAP forward. I am being a bit harsh when it comes to the UI, but I guess my biggest gripe here comes in the form of the calender and contacts which simply don't sync anywhere. I know, I know, this post is about IMAP but lets be real. The total package matters.

On the Macintosh there is Mail.app linked to Address Book linked to iCal. iCal leaves a lot to be desired but the UI is nice and it is a lot closer to a real product than Sunbird/Lightning. While I wish iCal had more power, it is a releaseable product. Mail.app is by far the best IMAP client I have seen and with Leopard it seems to work really nice with my IMAP Idle system. I can't seem to unsubscribe from my 'All Mail' in my GMail inbox which means that I end up with a bunch of dupes since I use my GMail Apps account to backup my mail. A lot of Outlook attachments won't come though correctly but all in all those are my only two huge complaints. I do have a global inbox or a separate inbox for each account depending on how I want to view my mail. This means I can look at all of the mail across all of my accounts at the same time to get a eagle eye view of my messages, or I can click on the specific inbox and see only messages sent to that specific account. Really simple concept that Microsoft, Mozilla and others do not seem to understand. I can set any mailbox as any object that I want such as junk mail, sent items, drafts and have these be local to my computer or on the server so that all clients can access them. Again, really simple concept but something that Mozilla makes more painful than it need to be and something Microsoft just refuses to implement. All in all Mail.app is the best e-mail client I have ever used, even with its quirks. There are add-ons that extend functionality and help fix a few things which are nice. This is nice if you have a Macintosh, but I also have a Windows system or two. What do I use for them?

The iPhone is a different story. What should be a painfully simple thing is much too hard to do, actually impossible. Things like subscribing to folders is simply not possible. If you have a server side rule that places all mail from your parents into an associated folder it is not possible to see those messages unless you first click on the folder on the iPhone. Then and only then will the iPhone connect to the server and attempt to actually retrieve any new messages. If you have an e-mail organization system that does anything other than grabbing items from the inbox it will fail with IMAP on the iPhone. Want to BCC someone? Nope. Want to attach more than 1 picture? Nope. The benefit of the iPhone e-mail is that I can actually read and fully reply to the messages unlike other devices like Windows Mobile or Palm where I can barely read the messages that come through unless that are plain text with no attachments or anything.

So here we have a protocol with great potential and everyone seems to be screwing it up. If Mozilla really wanted to steal marketshare from Microsoft they would throw some great UI designers and new developers at the Thunderbird and Sunbird projects to give them a swift kick in the pants. Don't just accept that what we have today is good enough, push for a system that is usable and allows me to check for mail on any device and keep them all in sync. Why limit this to e-mail? I should be able to keep my e-mail, calendar and contacts in sync all via the cloud. Exchange does this but only if you stay inside of a Microsoft certified environment. That is silly. I don't want to run Exchange. I don't want to be tied to Microsoft. I don't hate them but I also don't like their clients. Entourage for Mac is pretty nasty, Outlook for PC is nice unless I want to use it on anything other than Windows. I don't understand why this is so hard for anyone to do. Where is my IMAP client that is easy and sexy enough for consumers to use with calendar, contacts and e-mail sync?

If you're running IMAP on Windows or simply using IMAP to share your mail across multiple platforms and multiple devices with calendar and contact support, what are you using? How do you make it work? Is there a client I should be looking at that I'm missing?

EDIT: Fixed PSD to PST. No more Photoshop work while typing :)

The purge command or expunge is actually a good thing. A copy/move command has the potential of corrupting and possibly losing the message. Simply marking a flag is exactly how I would want it to work. Mail.app, Thunderbird and just about every other IMAP client under the sun do the proper thing which is hide the message from the inbox and show it in a 'deleted messages' box of sorts. Outlook completely screws this up and I guess they think it is better. Maybe no one at Microsoft actually uses IMAP, dunno... But it's a horrid way to implement. Expunge would be the same as emptying the trash... Just like one would expect.

I would be curious to know if you have more than 2GB in one IMAP folder. Each sub folder seems to get its own PST (brilliant, er, no) in Outlook. While POP3 and MAPI have been able to use unicode PSTs I have not been able to force an IMAP account to do the same in either 2003 or 2007 (both clients can use UNicode PSTs by the way). Can you confirm your PST state by using a PST analysis tool?

I'll correct the PSD to PST spelling in the article. Was working in Photoshop a bit too much today I guess :)

IMAP does not really have a MOVE command. It does have a COPY command. I have used more IMAP clients on various platforms than I can remember over the last decade. Most show the deleted items in the folder (such as the inbox) in which they resided upon deletion. That is the IMAP delete model. (RFC 2060) Outlook 2003/2007 and Outlook Express/Windows Mail follow this model.

Some clients have managed to mimic the missing MOVE command to send content to the Deleted Items (or Trash). I am sure you might want the Microsoft guys to implement IMAP in Outlook as they have in Entourage , which does as you prefer - copies messages to a deleted items folder, hides deleted items from the inbox view, and purges deleted items from the inbox silently. Or something like that.

Yes, both Outlook 2003 and 2007 can use Unicode .psts. In Outlook 2003, however, IMAP accounts default to ANSI .pst files. My comment was specific to IMAP.

Thunderbird 2.0 seems to be a more stable IMAP client than 1.5 was. Of course, it has support for IMAP special folders like Trash. This is not a requirement for IMAP, but is a common preference.

Of course. Outlook 2007 allows unicode .pst files with IMAP accounts. Currently at 4GB, down from 9GB.

That is not by design. I have never seen separate .pst files for the many subfolders I have in IMAP accounts with Outlook.

IMAP sometimes prompts religious discussions like Mac vs PC or Firefox vs Internet Explorer. The deleted item model of IMAP takes a lot of heat. Many people would prefer IMAP clients to behave like Outlook does with MAPI.

My Outlook 2003 at work seems to handle IMAP differently (better) than Outlook 2007. Did we make a giant leap backwards in Office 2007?

I'm new to trying to use Outlook 2007 with IMAP. Everyone here keeps talking about the Move command only as regards moving an item to the Trash or deleting it. But when I try to "move" messages to folders I have created in Outlook, it will move them once, out of the Inbox. If I try to move a second time, to a different folder, I get the strikethrough and the message "moves" nowhere. Is it not possible to effectively organize my IMAP messages & folders in Outlook? Or do I have to download them all first (a command that does not seem to take place immediately)?

Outlook 2007 will perform the expunge for you now as well, though I never found the option in 2003.

My ONLY complaint with Outlook 2007 is that I can't get it to automatically download files to the local PST file...and actually, there are two local pst files...one which is synched, I guess, and the other is not?

Well, add one more mail program to the list of bad ones: Windows Mail for Vista (Outlook Express replacement). It does not purge/expunge messages at all. It removes the message from the Inbox UI when you Delete it but once a message is removed, you can't purge it (regardless how you set the optional parameters). Messages marked for deletion are not shown by Windows Mail in the UI. So what you end up with is a full mail server of messages marked for deletion.

I used Zimbra at our school, which I'm used to, and so I encouraged by employer to sign up for a business account once I left the faculty with a Zimbra hosting provider, too. While I used the Zimbra web client at school, I've been trying a variety of mail clients at work, since the business has people that use Windows and Mac. I agree with the representations on Outlook and Mail.app, and although Thunderbird has worked well, I have had an easier time persuading people to use Zimbra Desktop, which you haven't yet reviewed. While it's only just out of beta, and still some bugs, it seems to address the issues you bring up, it's cross platform, and we haven't found a limit to the amount of data it can store and manage yet... I recommend using it with a Zimbra host, so you gain access to the advanced features like free/busy scheduling, and can also sync mobile devices like iPhone or native Blackberry.

Before I got my Mac, and was on Linux and Windows boxes most of my day I came to truly love Opera, it had the best IMAP implementation outside of PINE. I never understood why a web browser should have a built-in mail application but they did it perfectly and for awhile I used Opera as a dedicated mail client for IMAP and News Groups.

The smart filtering techniques they employed where fantastic, and they automatically created new folders for news groups.

Most corporate environments are supported by an Exchange Server. Most corporate employees are cogs in the gears and have no opportunity to affect the corporate choice of mail server. As such, we are doomed to use Outlook (aka Lookout) all our working days. Given Microsoft's willful disregard for all things outside Microsoft, it is pleasently surprising that Outlook supports IMAP at all. Doubly surprising that it ranks near the top of IMAP clients. I must use it for corporate communication -- how nice that it is mostly perfect for my own company and my personal accounts! And Outlook connector gives a little more, too. In fact, there is no comparison when it comes to third party support or integrated calendars, sharable calendars, contact lists, public contact lists, public and/or sharable folders, etc., etc., etc.

So, given my enthusieism for all things Linux, I am saddened that an equal product is not available. Meanwhile, I will explore Zimbra.

The way Outlook 2003 handles IMAP is indeed horrible; particularly if you are trying to use MobileMe and an iPhone.
There are two problems: First, if you delete an item in Outlook, it just gets a strikethrough; there is no way to move the item to any deleted item folder. (and if you purge, the message is gone forever)
Second, when you send from Outlook, your message gets put in the standard "Sent" folder, not in the "Sent" folder that is associated with the IMAP account. As a result, that message does not get synced with Mobileme.
The bottom line is that Outlook 2003 and MobileMe do not work together. However, Apple will claim that they do, but then they won't answer any questions about Outlook (my tech support person -- and I use that term loosely-- just suggested I buy a Mac).
So I am back using POP3 from Outlook to connect to GMail and IMAP from my iphone. I automatically bcc myself on all my iphone messages and then have a rule that move them from my Outlook inbox and into my Outlook sent Items. This combination sorta works, but I still can't delete something on my iPhone and have it deleted in Outlook.
Oh well.

Not sure when you did you research but even with Thunderbird 2, IMAP support is great! It even synch's while I'm offline so that if I respond to a bunch of email off line and go online, it updates the server correctly. Every computer connected gets the right info as to replies, forward, etc. Thunderbird 3 b4 is as good if not better.

And since you choose to bring other features into your analysis... The latest Thunderbird release (v3) puts all the other mail apps to shame wrt search (speed and flexibility).

Yeah, Thunderbird could be better as far as Calendar and Todo integration but there are good solutions for plugins if you are willing to spend 5 minutes and install one.

Mail.app dosn't support push befor leopard, this is a problem if you have an older system and can not update and there are multipe users one the same mail account simultaniusly. Also i'm not able to specify what folders should be used for trash, spam and drafts witch realy makes a mess of things if you either have multiple clients or a server that preferes it organised differently to Apple.
We switched to Thunderbird 2 witch solved the issues, but it has a horible search and is very unpolished (no multiple signitures per account), Thunderbird 3 seams to solve most of this but the local version is not done yet :(
Personaly i prefered Windows Live Mail, though i realy miss the icons that was in version 12.

Have you considered the Outlook connector from bynari? I have used several iterations that they have put out in our corporate environment over the years. Some of their earlier versions were flaky but the latest one they have is really nice. I use a Cyrus IMAP server with Postfix as the MTA. I have heard that their product works with a variety of other servers too. Their website is www.bynari.net

Comments

1. Posted by: William Lefkovics on February 22, 2008 2:06 AM:

Microsoft Outlook has never enjoyed a really good implementation of IMAP. That said, part of what you are complaining about is actually how the protocol works.

When you delete a message in an IMAP4 Rev1 account, it should set a flag for deletion. E-mail clients would display that message in the current folder with some indication, like a line through it, that it has been deleted. Then another command removes the flagged messages from the current folder (most likely the inbox). While Outlook uses the word Purge, the IMAP command is actually EXPUNGE. (see RFC 2060)

I used IMAP with Outlook 2003 daily for about 2 years. Currently using Outlook 2007 with IMAP and multiple accounts. Servers are Qmail, Imail, and Zimbra Collaboration Suite.

Also, for Outlook, the Personal STorage is a .PST file (as opposed to the Photoshop .PSD format). Outlook 2003 did force the user to maintain an ANSI .PST file, which had a maximum size of 2GB. Outlook 2007 removes this limitation. My IMAP accounts are all well over 2GB and as such use the Unicode .PST files. (See MS KB 830336).

Regards.

Required fields marked with: *
Name*:

Email Address*:

URL:
Remember personal info?

Comments*:

HTML Tags you can use in your posts:
<b>Bold</b> = Bold
<i>Italicized</i> = Italicized
<a href="http://www.othersite.com">Link to Other Site</a> = Link to Other Site

Please keep comments on-topic. Contact authors or other commenters
directly for off-topic conversations.

Notify me of future comments via e-mail

Technology Evangelist Digest - Free Newsletter

Sign up for the free Technology Evangelist Digest to receive daily updates, editorials, and practical advice on emerging technology trends in hardware, software, webware, marketing and beyond.

Technology Evangelist Digest will keep you up to date on the technology trends that will help make you more productive and efficient both in business and your personal life.

Let's face it: If you made it to this line, you must have found something valuable on this page, right? Think about how cool it would be to have something free and interesting to read every day from Technology Evangelist by signing up today.

1. Fill in your email below,
2. Then click on the confirmation email you receive.
3. That's it. Your first Technology Evangelist Digest will arrive within 24 hours.