The new Gmail API

This week Google announced their new Gmail API. Predictably, certain segments of the internet went insane.

My response was “oh, its about time”.

To understand its purpose, and why its not (yet) a replacement for IMAP, you have to understand how it fits with the rest of the integration points Gmail offers to developers.

Years ago Gmail started offering a feature called contextual gadgets. They were actually quite clever. As well as the code, you’d write a manifest that described a bunch of match rules, not unlike a filter. When a user opened an email, if it matched the rules, then the gadget would be embedded in an iframe at the bottom of the email. The gadget was served from your own web server and could do whatever you liked. This mechanism is the basis for many of the “rich content” features you see in Gmail today, for example the way YouTube links are turned into embedded videos.

One of the reasons this feature never really gained a lot of traction with developers is that you couldn’t directly manipulate the message from your gadget. You were given enough context to identify the message, but if you wanted to do any direct manipulations you had to do an OAuth handshake with the user to get access to their mail via IMAP. And then you had to do IMAP.

IMAP is an… interesting protocol. “Powerful” and “arcane” are probably the best words to describe it. Its definitely the standard for building full-featured desktop mail clients, but the client has to do a fair bit of work if it wants to be efficient and correct. For your average HTTP+REST-speaking Ruby/Python/Node/whatever web developer, its painful. Its really not well suited for a one-shot “here’s a message ID, do X to it” type of action.

To really make it easy for developers to actually manipulate messages from their gadget, a simple way of accessing the messages is required. And that’s exactly what the Gmail API is. I think its a good and necessary thing for Google to do, about four years late. So well done Google!

But of course, loads of people are alternately expressing outrage that Google is getting rid of IMAP or expressing relief that they use a different service that does use IMAP (and I’m not going to lie; there has been a small jump in FastMail signups today which may or may not be related). This despite a prominent message on the Gmail API page that reads “The Gmail API should not be used to replace IMAP for full-fledged email client access. Instead, see IMAP and SMTP”.

Now I can’t say that Gmail will never drop IMAP support. I have no idea what their plans are. But if they do drop it while still wanting to allow decent third-party clients, they’re going to have to do something better than this API. Email is quite a difficult application for a REST-style protocol. You usually want to fetch or manipulate portions of a message (get only the To, From and Subject, or get everything except the attachments, for example). For building an inbox-style message list, you typically want decent paging and support for incremental updates. You want to be able to receive partial updates to the messages you already have. It gets harder when you want to work with collections of messages likes folders, threads and conversations. You need strong versioning so you can request the latest changes to things without having to re-download everything. And then you want to be able to do this in as few round-trips as possible, because there’s just so much data to manipulate.

I’m not saying that its impossible to build a REST-style API rich enough and fast enough to implement a good standalone mail client. Indeed, the REST style has proven remarkably malleable over the years, and there’s loads of developer ingenuity to go around. But its certainly not an easy thing. Whatever Google’s plans are for Gmail’s API, I doubt its yet good enough for anything other than the simplest of clients.

At FastMail we took a different approach to the problem. Our client uses a JSON-RPC API that we call JMAP1. Its a protocol designed specifically for building full-featured mail clients, and anyone that has used our web client will surely agree that we got it right.

I’m not advocating JMAP as the “better” way of doing an email API but rather simply noting that a full email client is a big, stateful program that deals with a lot of data, and you need an API that will support that. JMAP is that. The Gmail API is probably not. But if we were to do something like Gmail is doing within our web client, we’d likely also need a simple REST API.

I’ll be really interested to see what kind of new extensions for Gmail appear over the next few months that sit on top of the Gmail API. Email is currently seeing a burst of innovation which is very exciting. This just pushes that along further, and I’m glad its happened.

  1. For developers reading this, JMAP doesn’t exist publically as more than this spec which we’re seeking comments on. We’re working on a standalone server, client and client libraries but they’re still a fair way off. If you think this is something you can use please contact me. ↩︎