What Is AMQP All About?

By Jeff Gould, CEO & Director of Research, Peerstone Research

In this three part series I discuss the emerging open source standard for business messaging, AMQP (Advanced Message Queuing Protocol). Part 1 is a non-technical introduction to the basic ideas behind the protocol. In Parts 2 and 3, I chat with two leading AMQP developers, Carl Trieloff of Red Hat (Part 2) and Alexis Richardson of RabbitMQ (Part 3).

I got my first e-mail account in 1991. That may seem like ancient history to many of you, but it was actually rather late in the Internet era. Many of my friends and business correspondents had already been using e-mail for years at that point, and they made fun of my quaint habit of communicating with them by fax. So I broke down and got a CompuServe account.

Boy, was CompuServe ever primitive. Remember those weird addresses that took the form long_string_of_digits@compuserve.com? And that clunky “fat client” Windows program they had? It couldn’t even send file attachments, though that was perhaps just as well considering that my modem back then only ran at 2400 bits per second (in those days people said “baud”). All the same, I was proud of my new cyberspace address. After all, I could now reach anyone in the world who had e-mail. Well, almost anyone.

You see, back then not every e-mail system was compatible with the Internet standard. Many large corporations used proprietary e-mail systems internally that couldn’t communicate with the outside world. At the time I ran a small market research firm in Paris specializing in surveys of IT users. My biggest customers were IBM, Hewlett Packard and the French phone company (aka France Telecom). All of them had their own e-mail systems, but none of them could receive e-mail sent from my CompuServe account. So whenever I produced a new research report I would carefully print it on my outrageously expensive Apple laser printer and then pay a motorbike messenger to carry it through traffic-choked Paris streets to the customer’s office. The only exception to this routine was when I had to deal with HP. Their offices were in Grenoble, hundreds of miles to the south of Paris, so motorbike messengers were out. But HP did have fax machines, of course, and I had just splurged on a cutting-edge contraption called a fax modem, which allowed me to fax a document with all its formatting straight from Microsoft Word, and thus economize a little on those expensive laser consumables. I was in the electronic equivalent of hog heaven, but I still couldn’t send a simple e-mail to my customers.

Of course, the inexorable march of progress soon swept away all these impediments to e-mail connectivity. A year or two after I got my fax modem, I went to work for a large American tech publisher, and by 1994 I had an e-mail address that contained my actual name instead of a string of numbers. And I soon discovered that HP and IBM had decided to take down the walls surrounding their proprietary e-mail systems. All of a sudden, my correspondents at IBM could send me mail from their Profs system (though I recall that it arrived all in caps). And HP even launched a product called OpenMail whose claim to fame was that it actually understood standard e-mail protocols. At the time this was a newsworthy innovation!

It’s easy to look back and laugh at the primitive state of e-mail in those days. Today no big IT user would even consider buying messaging software that failed to interoperate smoothly using basic Internet standards. And no self-respecting IT vendor would dare peddle such stuff. Right?

Actually, wrong. While ordinary e-mail settled on standards long ago, its high-end alter ego known as “message oriented middleware” – aka MOM – is still wandering around in the wilderness of proprietary isolation. MOM is what mission critical enterprise apps use to send messages to each other, usually in the same asynchronous style as e-mail (i.e. without waiting for a response, in contrast to more primitive RPC mechanisms). To be sure, these app-to-app messages have much greater business value than your typical person-to-person office missive, and therefore require VIP treatment. That’s why MOMs like IBM’s WebSphere MQ (formerly known as MQSeries) and Tibco’s Rendezvous pack all kinds of fancy features to guarantee that a message will actually be delivered and that it will only be delivered once. These MOMs can also scale to handle tremendous volumes of data in very short time frames, which comes in handy if you need to do something like distributing thousands of stock price updates to thousands of traders every second. MOMs have been around for the better part of two decades and have evolved into amazingly sophisticated pieces of software that are widely used by such well-heeled customers as Wall Street investment banks and big telcos. About the only thing they can’t do is exchange messages with each other natively, that is to say without relying on expensive single-point-of-failure kludges like gateways.

To be fair, making incompatible MOMs interoperate would no doubt be a non-trivial task. For example, there are fundamental semantic differences between IBM’s original queue-oriented model and Tibco’s publish and subscribe model. But over the years most of the MOM vendors have converged on each other’s feature sets without ever giving so much as a polite nod to the idea of interoperability. If you are a MOM user, you are for all intents and purposes still living in 1991.

The MOM vendors’ calculated refusal to offer standards-based interoperability goes against the grain of both history and common sense. True, they have embraced the Java world’s JMS standard (Java Message Service). In fact, IBM and Tibco took the lead in defining it. But JMS is only an API. It doesn’t specify the format of the headers and the packets that actually go down the wire, and it doesn’t promise let alone guarantee interoperability. Suppose you have two instances of the identical Java application, one at bank A and the other at bank B, each using JMS to communicate with a local MOM implementation. If bank A’s MOM is IBM MQ and bank B’s is Tibco, then they won’t be able to exchange messages unless they use a gateway (commonly called a “bridge” in the industry, even though it operates far above the traditional data link layer). And anyway, JMS is for Java only, it does nothing for apps written in PHP, or Ruby, or C#, to name only three popular examples.

You might be wondering – how can these guys get away with stonewalling on such a basic requirement as interoperability? The answer is simple. According to Gartner, IBM and Tibco between them control a whopping 93% of the MOM market, which the research firm estimates will be worth around $725 million this year. With a market share like that, IBM and Tibco can pretty much charge whatever they like (using IBM’s arcane “processor value unit” pricing scheme, WebSphere MQ will cost you tens of thousands of dollars per processor). The third ranked vendor, Progress Software’s Sonic unit, has less than a 2% share. In a young and fast growing market an innovative upstart like Sonic might be able to give the big guys a run for their money. But MOM is a mature space growing only a few percent per year. In short, IBM and Tibco share a cozy and lucrative duopoly that no conventional challenger is likely to upset. Customers have little choice but to play ball with them, even when they thumb their noses at interoperability.

Until now, that is. Some big-time MOM users have finally decided they aren’t going to take “no” for an answer anymore, and have banded together with some ambitious open source vendors to put together a new and totally open standard for business messaging called AMQP, short for “Advanced Message Queuing Protocol”.

AMQP’s founding genie is John O’Hara, a VP and Distinguished Engineer at the London branch of bulge-bracket investment bank JPMorgan. O’Hara launched the project back in 2003 and hired a small European open source design bureau named iMatix to build the first implementation. The beta version went live in 2006 and by the following year it was processing 300 million messages per day for the bank. Many Wall Street investment banks have built their own middleware over the years, but few of these efforts ever see the light of day beyond their own corporate firewalls. O’Hara’s plan for AMQP was far more ambitious. From the start he wanted the new protocol to match the functionality of the high-end proprietary MOMs. It had to handle all the major use cases, including queue-style store-and-forward messaging, Tibco-style publish and subscribe, and reliable file transfer. The protocol had to be capable of carrying any kind of message, but the focus was on more efficient binary formats rather than text, because in app-to-app messaging human readability is not a paramount concern. AMQP thus marked a deliberate departure from the emphasis on XML built into the superficially similar WS-ReliableMessaging standard. (Recall that WS-RM, part of the Oasis collection of web services standards, was designed to provide reliable transport of XML messages over inherently unreliable HTTP, which is a bit like putting an armored gun turret on a VW Beetle – though see this.)

But O’Hara also wanted something more than a full set of MOM features. To have a chance of displacing the proprietary products, AMQP would have to become a new Internet standard. That meant guaranteeing interoperability between different, competing implementations. It also meant bringing other users and commercial sponsors into the work group, as well as ensuring that the emerging standard enjoyed unencumbered intellectual property rights. (For background on AMQP, see O’Hara’s article “Toward a Commodity Enterprise Middleware” and this presentation.)

Today AMQP is edging toward early maturity. A full-fledged version 1.0 is expected sometime before the end of the year. At least three different open source implementations are commercially available: the original iMatix C implementation; Red Hat’s MRG distro incorporating Apache Qpid (AMQP servers implemented in Java and C++, AMQP clients in C++, Java, Python, Ruby and C#), and the RabbitMQ implementation on Ericsson’s high-performance Open Telecom Platform. Other open source vendors who participate in the AMQP Working Group but have not yet released their own versions include Novell, WS02 and Iona (recently acquired by JMS pioneer Progress Software).

Although AMQP was born in the privileged environment of wealthy, big name investment banks, its proponents see applications for it far beyond the financial sector. AMQP can serve as the underlying transport for services oriented architectures (SOAs) in any industry where app-to-app messages have significant business value. But it would be a mistake to think of AMQP as just an open source replacement for proprietary middleware behind the corporate firewall or in a few tightly-knit trading communities. John O’Hara’s vision goes much further than that. In that vision, AMQP will one day become the universal fabric for open business transaction networks between companies, where each company will have its own message exchange that communicates directly with other exchanges instead of going through proprietary gateways. That would make high-value business-to-business messaging as easy to do as Internet e-mail is today.

Recently I had the opportunity to chat with two of the leading AMQP developers: Red Hat’s Carl Trieloff, who in addition to running Red Hat’s MRG initiative is also a key member of the Apache Qpid project (see Part 2); and RabbitMQ’s Alexis Richardson, who is also the co-founder of U.S. cloud computing company CohesiveFT (see Part 3).