Message Queuing Architectures

An architecture for distributed systems based on the concept of reliable message queuing. Messages are queued asynchronously between applications and systems.

(Alleged?) Benefits: Examples:

IBM's MQSeries:

MicrosoftCorporation's MSMQ: or

Tibco's Active Enterprise:

One would suppose this architecture is suitable for wrapping legacy systems and gluing them together. But it's not clear to me how this type of architecture fits into a large distributed OO system model. What would be the equivalent of IDL for a system built along these lines? -- some guy

Actually, in one system I've worked on the IDL that you used WAS IDL. That project spent a great deal of effort building a CORBA Event services layer on top of a Message Queueing system so that they would have both IDL and reliability. Conceptually, it's not that much of a stretch, and it worked out well for that project.

P.S. One more thing about reliability -- Queues are persistent only in the case of so-called "store and forward" messaging systems. Many messaging systems support this capability, but in *every* delivered system I've ever seen the developers had to turn that capability off at deployment time to achieve the needed transaction rates. Thus the system as a whole becomes exactly as "reliable" as the machine the queue is running on. In the case of an OS/390 box, that's pretty darn reliable, but in the case of an NT, Solaris or AIX box, your guess is as good as mine.

-- KyleBrown

The utility of Message Queuing is orthogonal to the OO-ness of the distributed model. The basic idea is this: Invoke a method (send a message), and forget about it. This allows a more loose coupling between disparate systems, and loose coupling generally means (a) simple to implement, (b) simple to maintain, (c) scalable.

So where do they meet? For example, suppose you could insert a queue between the client and the remote object it is using. Obviously, not a general application, but suitable in some circumstances. Now the client can instantiate the remote object (via proxy), manipulate it via a well-defined interface, and then deactivate it. All of these operations can be queued, if you submit to some of the limitations, eg: your logic must not depend on the results of prior method invocations.

For an illustration of the combination of OO and message queueing, see ComPlus, specifically Queued Components (but also see DnaVsOo for a discussion of how OO is DNA (and Com+)). Also, I believe OMG has attempted to include some form of asynchronous comms into CORBA, but I must confess that I haven't followed it (has anyone?).

-- DinoChiesa, 12 April 2000

The two 'orthogonal' aspects of messaging and OO are combined nicely in the OMG's Typed Notification Channel. This provides strongly-typed asynchronous communication: ideal for distributed systems. Quality of Service parameters can be used to request message persistence if the service needs to be reliable. Bare in mind, this works primarily for operations which don't require a response. -- Brenton Camac

On Reliability and Persistence

The appeal of a system such as IBM's MQSeries is flexibility. People used it for a distributed comms mechanism, just to get data or requests, or messages, or whatever... from one place to another. In many cases, persistence of the messages was not a requirement. In other caes, though, people used the queue to buffer transaction requests; here persistence would obviously be required. It is true that persistence can be turned off at deployment time, and that your message throughput will go thru the roof if you take the disk i/o out of the pipeline. But this is not the only way message queues are deployed. There are many successful implementations of message queueing middleware; some use persistence, and some don't.

-- DinoChiesa

Kyle, or anyone:

If a message queue is used without persistent queues, then is there a difference between such a system and a "transaction processing monitor"? Is a TP monitor *more* reliable than a "non persistent message queue"?


-- PatrickLogan

DistributedComputing architectures are usually one of two types -- either you have RemoteProcedureCalls across a network, or you have message oriented systems, usually MessageOrientedMiddleware like MQSeries or Tuxedo. Message oriented systems have a lot of advantages over RPC systems, but they're also more complex to maintain.

The problem begins with Ellen Ullman's unattributed quote in CloseToTheMachine: "I never saw a person with two systems who didn't want to connect them together." It's not simple to get two systems which use different messaging protocols to talk to each other: this is the whole point of MiddleWare.

-- WillSargent

I'm going to try to add to (clarify?) the remarks by WillSargent to better answer Patrick's question. A TP System (as Will states) can use either an RPC mechanism or a Message Queue to move data in and out of itself. However, the communication mechanism is not the 'same' as the TP system, it's a technology that is 'used' by the TP system. The TP monitor is concerned with handling the transaction context and propagating that context around to all of the processing nodes and programs that deal with the data involved so that the transaction can either entirely succeed, or fail in such a way that consistency is preserved. -- KyleBrown

A TP Monitor is more than a distributed transaction coordinator (which implies a communication mechanism and a commit coordinator). The class of products called TP Monitor evolved to fulfill the requirements of large-scale distributed systems, many of which were doing transactions (and some of those transactions were distributed). Though the term is not rigorously defined, TP Monitors typically provide:

TP Monitors did not focus on providing web integration or pretty front ends. As a class, these products evolved long before the browser became ubiquitous.

In general, another shortcoming of TP Monitors was poor development tool support. Basically, it was: "here's a bag of APIs you can use to get at our TP Monitor services, grab a text editor and go to it!" Contrast this to the EJB-based app server market, where it seems all vendors recognize the importance of both the tooling and the runtime.

CORBA tried to standardize many of the common but proprietary application services available in TPM's.

Some of the CORBA offerings tried to reach the established TP Monitor space; for example Iona was marketing Orbix as an Object Transaction Monitor (OTM): you get an OO design and programming model but all the services you like from the TP Monitor model.

In any case, it doesn't quite make sense to ask, is a TP Monitor more or less reliable than message queueing with non-persistent queues. The comms is but one part of the TP Monitor product.

-- DinoChiesa, 12 April 2000

Surely there must be patterns for such an old software architecture. Is anyone collecting them? -- some guy

I think any asynchronous interaction requires a queue of some sort, even if it's a single entry buffer. So the "pipe/filter" paradigm would fit wouldn't it?. -- Richard Henderson.

Some common paradigms: MessagePipeline?, WorkCrew (parallelism), MessageRouter and PublishAndSubscribe. -- RossLonstein

More patterns...

Counterpoint. There is no such thing as a "message queueing architecture." There are systems in which all components are required to communicate via message passing, and there are systems in which all IPC is required to pass through message queues. However, these two constraints are not enough to define an architecture of significance. While these constraints say something about the connectors of the architecture, they say nothing at all about the components. Without that, you have described somewhat less than an architecture or architectural style.

As a RelationalWeenie, I would be inclined to use queries or stored procedures to put messages in the queues (tables). Then other apps can do whatever they want to with the messages, including delete them, mark them "complete", etc. Since most large orgs already have RDBMS infrastructure in place, it seems easier just to use that rather than add a dedicated message queuing system.

See also: DistributedComputing, MultiCaster, TupleSpace, MessagingPatterns, FlowBasedProgramming.

View edit of April 10, 2012 or FindPage with title or text search