Multi Caster

CategoryPattern
The ObserverPattern family is too damned useful. We see countless variants and extensions of it in the literature - MediatorPattern, EventNotifier, Reactor, EventChannel?, Broker, TryThrowCatch?, the various agent architectures, GUI callbacks, comms handlers and so on. Almost all hark back to the concepts in craggy old visicalc, and no doubt that's derived from someone's woodpulpware system - nothing new under the sun.

The problem is that we often see more than one observer-variant in the same development. This raises a number of issues:

Therefore,

We define five generic participants that will take part in ongoing processes of observation. These are

To summarize this, Agents communicate by subscribing to Addresses in a space managed by MultiCaster and distributed via Proxy, and publishing Values on Addresses in this space. A subscription associates a callback (an object and one of its member functions, whose signature accepts a Value) with an Address. Like observer/mediator/broker, when a Value is published on an Address, all the subscribing Agents receive it via their callbacks.

This is obviously similar to DougSchmidt's ReactorPattern. The difference is the explicit definition of Address, which models the problem domain rather than the process architecture. By making explicit the map between this Address space and the process architecture, the various issues identified before the Therefore are resolved:

Notes:

Examples: ObjectSpaceVoyager? (addresses modelled as adornments), UseNet, Teknekron InformationBus, LindaTupleSpaces, spreadsheets, windowing systems ...
It sounds good. I've been thinking for a while about alternate paradigms of computing. One of them is massively event-based. In DesigningObjectSystems? by CookAndDaniels? (the book outlines the Syntropy approach) they make the point that reality is really event based, so OO conceptual models are really a slight mismatch.

Hmm. Off topic, but "reality is event-based" seems to ignore quantum SpookyActionAtADistance. A much better case can be made for the idea that reality is transaction-based - see http://www.npl.washington.edu/AV/altvw16.html

Just to set the record straight: In DesigningObjectSystems? we didn't say that reality is event based, just that it might be simpler to create models of the world using events rather than point-to-point messages (as has been the trend in OO methods, following along from OOPLs) -- John Daniels

And MultiCaster isn't really any alternate computing paradigm - it's just adding two old concepts together. Address spaces aren't new, and neither are Reactors. The reason I like putting them together is that this way you can think about the Space of the problem domain a la OrganicArchitecture.

I have this intuition that another way of going would be to allow all objects to broadcast to no one in particular, and receive from anyone. Systems could be progressively grown by adding software where the newer bits get first crack at the messages. Sure enough, there are performance issues.

A lot of the performance issues then have to do with your communications/database architecture. The traditional TCPish arrangements of DAGs of domains work quite well with this approach.

This is a beehive style of computing. I'm really interested in getting past the limitations inherent in type systems. The kind of ossification that occurs when systems get to be too specialized because no one had the time or knowledge to refactor effectively.

Yah. I look at the marvels of CPAN and wonder what the hell I'm doing wasting my time with a strongly typed language like C++. CompositeTypes - what other types do you need? I can't think of anything I can do with STL (or JGL) that I can't do better with Perl's builtin types.

Doubtless, there is some theoretical reason why this is nuts, but I make a habit of thinking about the impossible. -- MichaelFeathers (Man of La Mancha).

I've got a number of systems in the field based on this pattern, so I can say it's far from impossible. It may not be optimal - I'm very interested in alternatives - but so far it's served me very well. -- PeterMerel
It reminds me of BlackBoard models (although it's different). The MultiCaster is like the blackboard that everyone can read or write to.

You can do a BlackBoard this way. Depends on whether you cache publications or not. More commonly, in my experience at least, you don't.

What are the Values used for?

They encapsulate whatever parameters make sense for a Path. For example, an Address is often associated with some particular device, and published Values there may represent its state. If the device is a sensor, a camera perhaps, then the Value may be a reference to an image it has captured. If the device is an actuator, then it might only publish there only if it strikes an exceptional condition.

The notion of explicitly defining the address is interesting. Do addresses usually mean something in real-world terms; is that a pre-condition for successful use of the pattern?

Seems natural for them to do so. They generally seem to match up with analysis-level classes, though I wouldn't call that a precondition.

Are addresses too useful - is there a temptation to encode other things into the address space, and/or is this a good thing?

Seems like a good thing. For example, you can cause otherwise uncaught exceptions to appear as publications on some distinctive path. Diagnostic traces of particular agents (qua device drivers) can be done the same way.

By "other things", I mean that incrementing a variable could have a different address from decrementing the same variable. Using the space to describe what has happened as well as to which object.

'Yes, that's doable too. Constructing the semantics of the Address space becomes the significant design process once you employ a MultiCaster. So I guess I think of this as an architectural rather than a design pattern. Actually, one early example of MultiCaster that occurs to me is usenet nntp; given the excellent scaling properties of that, I think calling the pattern architectural is fair.

It seems to me that the border between Addresses and Values could be blurred.

You're right, especially when Values assume the form of, say, associative arrays and you cache the publications. But the blurring goes away if you restrict Values to be either scalar or iterator types, or if you don't cache. --PeterMerel


I appear to have independently (re)invented this in a few projects. I agree that the border between Address and Value can be blurred - initially I didn't separate Address from Value at all. I used hierarchical objects that were simultaneously Address and Value. Processing an outgoing object starts at a common root (or a distributed emulation of one) and at each level of the hierarchy a few parameters are removed and used to route the remainder of the object toward its destination. This seemed like a good idea at the time, but recently I've explicitly separated the two because it seems more natural that way. One problem with having no distinction at all is that it makes protocol analysis more difficult, as you don't really know (without knowledge of every object in the system) where the address stops and the value begins.

-- ZygoBlaxell
PeterMerel has the temerity to suggest this pattern meets Wright's OrganicArchitecture criteria. Here's why:

"NATURE" is transaction-based - cf http://www.npl.washington.edu/AV/altvw16.html . By defining the Address space in terms of Fields, this pattern can emulate natural systems.

PartToWholeAsWholeIsToPart - fair enough, MultiCaster(s) can be composited into multicast domains.

FormAndFunctionAreOne - this pattern serves a central architectural purpose but there's no excess baggage in it; compare with great reflective brutes like CORBA and COM.

MultiCaster is creative (Alexander's generative) - it enables rather than dictates a great array of diverse forms, serving Wright's "ROMANCE"

The pattern does not document a tradition or habit but a fundamental basis for a space of development - a "TRUTH"

If the pattern is to be successful the Agent and Proxy semantics must mirror the MultiCaster semantics in terms of the Address space - so these are all OfTheThingAndNotOnIt

MultiCaster does not constrain the forms of the Address, Value and Agent classes, but enables them from below, meeting Wright's "SPIRIT".

This pattern scales well, so it has Wright's "THIRD DIMENSION"

Wright-style SPACE is really the point of the pattern. By shifting design concerns away from functional dependency and on to provision of service, the pattern creates an open development space within which designers and administrators can interact. Compare with the various layered and hierarchical architectural patterns that work to separate designers and administrators, reducing cooperation between them. The history of usenet exemplifies this - checkout http://www.danenet.wicip.org/bcp/history_of_usenet.html.
Peter, when implementing this, have you done anything to detect cycles? It seems that a topological sort can be used to make sure they don't occur, but it is also true that notifications you send which come back to you need not spur other notifications.

publish() takes a flag which says whether or not to propagate a publication outside the local process. When proxies publish they set this flag false. Or you can include the domain/host/pid as an argument to the callbacks and have the proxies make a decision on that basis. Or both. Usually the MultiCaster doesn't propagate an agent's publications back to itself either, figuring that the agent can manage its own state without assistance.

In the synchronous case, have you done this so that the order of notification is implied by the order of subscription, or is it a bad thing to start depending on that?

I'd say that's not very reliable, and gets less reliable the more things are distributed. About all you should rely on here is that a subscriber will receive events from a particular publisher in the same order that their generated by that publisher. You might be able to add other guarantees if you implement this over a RealTime OS - QNX say - but I'm not certain what that buys you.

When you speak of iterators are you talking about iterating over the address space?

Not really - they were ways of iterating over particular datasets. The application in that case was a very high-dimension, high-volume, geographically distributed data warehouse. The question was more about whether to send a concrete dataset from site to site or whether to iterate; the solutions that are presently being investigated for resolving that are mainly market-based - check out http://s2k-ftp.cs.berkeley.edu:8000/mariposa/

Regarding the use of the proxies, you said that they make things demand driven but isn't this all push?

Not to my thinking. Subscriptions get broadcast to all processes (if that's too costly, then all within a domain ...) but publications only get sent to subscribers. If no agent in process A is subscribed to some Address published by an agent in process B, then A's proxy in B isn't subscribed to that Address and its publication is never communicated to A. If one or more of A's agents have subscribed that address then the publication will go. Because multiple agents in one process may subscribe a path in another, the MultiCaster should be smart enough to ignore duplicate subscriptions.

As you can probably see, error recovery could get messy with this, especially if you permit unsubscription on the fly. But it's doable either via mementoes per proxy or per agent ... probably plenty of other ways too. How hairy it gets depends on how bumpless you want it to be.

Do agents typically hardcode addresses, leave them to configuration, dynamically create them, or it just depends on the problem?

Depends on the problem. I generally like to let configuration be dynamic because if I find there's some performance bottleneck in MultiCaster for comms between two particular agents it generally means I've badly factored the agents - so it's not the Address hash that's the problem.

Was taking a crack at coding a MultiCaster today to ferment the ideas. -- MichaelFeathers

Good luck with it. An area I'm still playing with has to do with event queues. You need at least one - in the MC. If you're doing this with preemptive threads then that's a critical section, and if not you still want to prevent a functional deadly embrace. But with preemptive threads what about having one event queue per agent? -- PeterMerel
Regarding the "countless variants of observer patterns". Maybe this is just an incarnation/application of the ConstructionPrincipleForDesignPatterns?

The basic for all the different Observer patterns is a common semantic: Inform dependent objects about changes. The implementation, however, is driven by the actual context (the requirements on this part of the system) - and this results "just" in a different combination of TemplateClass(es) and HookClass(es) as sketched in ConstructionPrincipleForDesignPatterns.

-- ThomasWeidenfeller

The problem being solved here is not the countless observer pattern variants that exist but the patchwork combination of several variants, or several instances of the same variant, in the same development. The difficulties that result are detailed at the top of this page.

The countless pattern variants, imho, are partly due to folk coming to patterns from different backgrounds, and partly the PatternOfBabel. But I'm not certain of the use of the ConstructionPrincipleForDesignPatterns so will ask further questions there. -- PeterMerel

You are right, I am digressing from the original problem. I was just trying to identify a root cause of the problem, leaving the problem solving to others.

-- ThomasWeidenfeller


To me, it looks like MultiCaster uses a variety of GOF design patterns in a particular set of configurations. I wonder if ArchitecturalPattern would be a good word for large-scale patterns which are singly instanced in systems or subsystems, and are composed from lower level patterns? Or, is this just creating a new word for no good reason? -- MichaelFeathers

ArchitecturalPattern is a PartyOfFive concept, and not a bad one.


How is this different from PublishAndSubscribe? - GabrielWachob

Just as the original ObserverPattern, it's not different from, it's underpinning for.


MultiCaster has a separate meaning in TCP/IP used to refer to software which broadcasts UDP packets using IGMP (ftp://ftp.rfc-editor.org/in-notes/rfc3376.txt) into the Multicast address space as defined by RFC2375 (ftp://ftp.rfc-editor.org/in-notes/rfc3307.txt).

-- Martin Spamer


Ive been working in this area and am struck with how much this observer-type pattern looks like forward-chaining expert-systems.

-- DamienMorton?


See also: EventNotifier, ObserverPattern, MediatorPattern, ObjectsInGroups, TupleSpaces, OrganicThreads, StaleEvents

EditText of this page (last edited March 21, 2007) or FindPage with title or text search