Q1: pros and cons of vector vs linked list?
Q1b: Given a 100-element collection, compare performance of … (iteration? Lookup?)
Q: UDP vs TCP diff?
%%A: multicast needs UDP.
Q: How would you add reliability to multicast?
Q: How would you use tibco for trade messages vs pricing messages?
Q5: In your systems, how serious was data loss in non-CM multicast?
%%A: Usually not a big problem. During peak volatile periods, messaging rates could surge 500%. Data loss would deteriorate.
Q5b: how would you address the high data loss?
%%A: test with a target message rate. Beyond the target rate, we don’t feel confident.
Q7: how is order state managed in your OMS engine?
%%A: if an order is half-processed and pending the 3nd reply from ECN, the single thread would block.
Q7b: even if multiple orders (for the same security) are waiting in the queue?
%%A: yes. To allow multiple orders to enter the “stream” would be dangerous.
Now I think the single thread should pick up and process all new orders and keep all pending orders in cache. Any incoming exchange messages would join the same task queue (or a separate task queue) – the same single thread.
3 main infrastructure teams
* exchange connectivity – order submission
* exchange connectivity – pricing feed. I think this is incoming-only, probably higher volume. Probably similar to Zhen Hai’s role.
* risk infrastructure – no VaR mathematics.
http://solacesystems.com/news/fastest-jms-broker/ solace JMS broker (Solace Message Router) support 100,000 messages per second in persistent mode and 10 million messages non-persistent. In a more detailed article, http://solacesystems.com/solutions/messaging-middleware/jms/ shows 11 million 100-byte non-persistent messages.
A major sell-side’s messaging platform chief said his most important consideration was the deviation of peak-to-average latency and outliers. A small amount of deviation and (good) predictability were key. They chose Solace.
In all cases (Solace, Tibco, Tervela), hardware-based appliances *promise* at least 10 fold boost in performance compared to software solutions. Latency within the appliance is predictably low, but the end-to-end latency is not. Because of the separate /devices/ and the network hops between them, the best-case latency is in the tens of microseconds. The next logical step is to integrate the components into a single system to avoid all the network latency and intermediate memory copies (including serializations). Solace has demonstrated sub-microsecond latencies by adding support for inter-process communications (IPC) via shared memory. Developers will be able to fold the ticker feed function, the messaging platform, and the algorithmic engine into the same “application” , and use shared memory IPC as the data transport (though I feel single-application design need no IPC).
For best results you want to keep each “application”  on the same multi-core processor, and nail individual application components (like the feed handler and algo engine) to specific cores. That way, application data can be shared between the cores in the Level 2 cache.
 Each “application” is potentially a multi-process application with multiple address spaces, and may need IPC.
Benchmark — Solace ran tests with a million 100-byte messages per second, achieving an average latency of less than 700 nanoseconds using a single Intel processor. As of 2009, OPRA topped out at about a million messages per second. OPRA hit 869,109 mps (msg/sec) in Apr 2009.
Solace vs RV appliance — Although Solace already offers its own appliance, it runs other messaging software. The Tibco version runs Rendezvous (implemented in ASIC+FPGA), providing a clear differentiator between the Tibco and Solace appliances.
Solace 3260 Message Router is the product chosen by most Wall St. customers.
http://kirkwylie.blogspot.com/2008/11/meeting-with-solace-systems-hardware.html provides good tech insights.
- Culprit: deadlock
- Culprit: all threads in the pool are blocked in wait(), lock() or …
- Culprit: bounded queue is full. Sometimes the thread that adds task to the queue is blocked while doing that.
- Culprit: in some systems, there’s a single task dispatcher thread like swing EDT. That thread can sometimes get stuck
- Suggestion: dynamically turn on verbose logging in the messaging module within the engine, so it always logs something to indicate activity. It’s like the flashing LED in your router. You can turn on such logging by JMX.
- Suggestion: snoop
- Suggestion: for tibrv, you can easily start a MS-windows tibrv listener on the same subject as the listener inside the trading engine. This can reveal activity on the subject
jms message selector is executed on the broker.
rvd executes the same duty — “Filter subject-addressed messages.”
Among the defining features of Tibrv, [ A) decentralization and B) SBA ] are 2 sides of the same coin. For me, it's a struggle to find out the real difference between SBA vs jms topics. Here's my attempt.
#1) In SBA(subject-based-addressing), there's no central server holding a subject. In tibrv, any sender/receiver can specify any subject. Nice — No admin to create subjects. I tested it with tibrvlisten/tibrvsend. By contrast, In JMS, “The physical creation of topics is an administrative task” on the central broker , not on the producer or consumer.
) If a jms broker goes down, so do topics therein.
) a jms publisher (or subscriber) must physically connect to a physical broker process. The broker has a physical network address. Our topic is tied to that physical address. A tibrv subject has no physical address.
) tibrv SBA (subject-based-addressing) uses a subject tree. No such tree among JMS topics. The tree lets a subscriber receive q(prices.stocks.*) but also q(*.your.*). See rv_concepts.
 such as the weblogic server in autoreo.
Imagine a typical request/reply messaging system. I think in JMS it’s usually based on temp queues, reply-to and correlation-Id — See other blog post. In contrast, RV has no broker. It’s decentralized into multiple peer rv daemons. No difference in this case —
Suppose a message broker holds a lot of queues. One of the queues is for a request message, from requester system to a pricing system. Another queue is for pricing system to return the new price to the requester.
Now, pricing system is slow. Requester should wait for no more than 5 minutes. If the new price comes back through the reply-queue 301 sec later, requester will ignore this stale price since it’s too risky to place an order on a stale price in a fast market. How do you implement this?
My design — Requester main thread can wait(5*60*000). Another thread in requester JVM can block forever in onMsg(), and notify main thread when something received.
(I actually implemented this in a few trading engines.)
Message overflow is a frequent headache in MOM. Basically consumption rate (possibly 0) is slower than production rate, eating up storage capacity, either in memory or on disk.
eg: Topic with NDS (non-durable-subscriber) only — almost free of overflow. Bonnie said broker will discard. Jerry said broker keeps track of alive subscribers. Every incoming message is guaranteed to be delivered (by retry) to each. Broker must persist pending messages. When a subscribe disconnects gracefully, broker no longer needs to keep any pending message for her.
eg: queue. While any number of producers can send messages to the queue, each message is guaranteed to be delivered, and consumed by one consumer. If no consumers are registered to consume the messages, the queue holds (on disk) them until a consumer registers to consume them.
eg: temp queue??
eg: chatrooms don’t persist data but what type of MOM is that? probably NDS topic
eg: tibrv non-CM? market data feed? Discarded but how soon? 60 sec by default?
Yes, in slow consume and fast producer scenario message lost will be possible — http://arun-tibco.blogspot.com/2010/09/tibco-rv-faqs.html
eg: tibrv CM? Storage is the ledger file. –persist–
eg: regular queue? In SIM upfront, unread messages –persist–
eg: mailing list (topic with durable subscribers)? –Persist–
eg: topic without durable subscriber? see post on message retention
guaranteed delivery (GD) is a “service” provided by the JMS infrastructure — the part of the system outside your custom application code. It's also known as the vendor API, which is software provided by IBM, Tibco, BEA etc. A formal name is “JMS provider”. It helps to think of this infrastructure as encrypted source code (ESC).
To provide GD (or any JMS service), this infrastructure must include client-side ESC — formally known as client-runtime.
ESC is simpler in the JDBC world. Sybase provides both a server and a client-side ESC. Your code calls the client-side ESC as a lower-layer API. eSpeed also ships a client side ESC. You link into your custom trading engine, and the client side ESC talks to the eSpeed server.
In tibrv, the daemon runs in each client host and is comparable to a client-side ESC but
$ it runs in a separate process
$ tibrv daemons are peer-to-peer, no client no server.
onMessage() is one of the most important part of the client-side ESC. But back to GD, ESC supports
* Stored and Forward
* auto_ack is sent by the client ESC
* retry at each leg (sender->broker->receiver)
synchronous – really means “blocking”
asynchronous – really means onMessage()
client – both a sender and receiver are clients of the broker, if you ask me.
server – “broker” is more specific. In a real world request/response system, A can ask B and B can ask A.
(However, in tibrv, there’s no broker. All rvd instances are peers. Decentralized.)
acknowledgment – which leg on the 2 legged flow — sender->broker->receiver?
Subjects are arranged in a subject tree by using a dot notation, and clients can either subscribe to a single subject (e.g., finance.quotes.NASDAQ.FooInc) or use wildcards (e.g., finance.quotes.NASDAQ.*).
JMS topic is basically subject-based, or more precisely “Channel-based”, as a (selector-less) subscriber gets everything in the channel.
jms queue “can be compared to TIBCO Rendezvous Distributes Queues (RVDQ)”. I feel tibrv inbox is also like a queue. See http://www.cs.cmu.edu/~priya/WFoMT2002/Pang-Maheshwari.pdf.
Q: is RV still the market leader in the “max throughput” MOM market?
Q: for trading apps, i feel the demand for max-throughput MOM is just
as high as before. In this space, is RV (and multicast in general)
facing any competition from JMS or newer technologies? I feel answer
A: I think RV is still the market leader. 29west is a strong
competitor. TIBCO now sells a “RV in hardware appliance” partly to
address that. I feel JMS is not really targeting max-throughput
Update — tibrv’s flexibility make tmp queue look unnecessary, rigid, uptight and old fashioned.
tmp queue is (tmp topic?) used for request/reply. I feel the essential elements of JMS request/reply are
1) tmp queue
2) set and get reply-to — setJMSReplyTo(yourTmpQueue)
3) correlation id — setJMSCorrelationID(someRequestID)
Server-side (not the broker, but another JMS user) uses getJMSCorrelationID() and then setJMSCorrelationID() in the response message, then replies to getJMSReplyTo()
tmp queue is like a Sybase tmp table — (I feel this is the key to understanding tmp queue.)
* created by requestor
* removed by the “system janitor”, or by requestor
* private to the session. Therefore creator threads must keep the session alive.
 temp topic too. p67 [[JMS]]
Start 3 “black” DOS windows A B C.
tibrvlisten -daemon localhost:7500 -service 9012 -network “” your.subject “your msg”
tibrvsend -daemon localhost:7500 -service 9012 -network “” your.subject “your msg”
## listener and sender both complain about missing daemon.
C) Now type nothing but “rvd” to start the daemon process. It starts in 0.1 second, reporting the ip endpoint bound + admin website.
Now restart the listener and it starts listening.
Now restart sender – no more errors. Listener window shows the message.
Also, http://localhost:7580/ shows one client only — our listener
Every trading server invariably uses some non-http network daemon. There’s always more than 1 process (JVM, C# or c++) on the server side. There’s usually some MOM daemon such as JMS, tibrv and gemfire notification daemon. Here are some Fundamental questions:
Q22: on top of tcp/udp, what specific network protocol between the server-side and GUI?
A: I have seen rmi and protobuf over tib ems.
Q22a: how about JMS between server and swing? Did we see 160 subscribers on a given topic, due to that many swing installations?
Q33: on top of tcp/udp, what specific network protocol among the server-side processes?
A: I have see tibrv, JMS, RMI, gemfire data distribution protocols …
Q44: since most trading servers must avoid DB latency, where does the trading data live? In memory?
A: i have seen gemfire, rttp,
Q45: in case of distributed cache (not replicated), how does one cache listener update another node?
Q55: how does the daemon stay alive after main() exits?
A: Look at ion, gemfire, activemq. There’s often at least 1 (1 enough) non-daemon thread that’s stuck in wait()
(Note OO is a development-time concept. Once compiled, an OO program looks no different from a C program with tons of structs and function pointers.)
Most financial systems today are concurrent OO systems. The lowly callback pattern is extremely common but actually requires a thorough understanding of both thread and object interaction. Let’s examine some. Note just about every player in the callback scenarios is a pointer, including function pointers.
In the simplest 1-thread case, an observer pointee registers interest with a subject object, by giving its own address. After subject state changes in Thr 1, the same thread loops through the list of observers and invokes the callback method on each observer. Note the observer object must not be deallocated.
For onMsg() in tibrv or onMessage() in JMS, there’s a remote “upstream” thread in the message broker’s PID/runtime, and there’s a separate downstream thread in the listener’s PID. I believe this thread starts out blocking for messages. Upon receiving a msg, the runtime somehow wakes up this blocking thread and this thread invokes onMsg() on the listener pointee. Meanwhile what’s the remote upstream thread doing? Simplest design is synchronous — upstream thread waits for the listener thread to finish. This means the upstream thread can only dispatch the message to remote listeners sequentially. I feel it’s much better to get the upstream thread to return right away after sending the one-way-message (see Doug Lea’s book) so this thread can contact the 2nd listener in the list. Rewind to registration time — Any thread could register the listener with “Hollywood”. This thread could terminate immediately but the listener pointee must live.
 in classic Reo, we actually send one-way messages to the market-facing gateway. We get the status message on a separate thread via onMessage().
In C/C++, you can also register into “Hollywood” a ptr to a free func. No object, usually no state (unless the free func uses stateful static objects). Free functions are free of destruction. Hollywood could save the ptr in any collection. When there’s an event, Hollywood calls us back, in any thread.
A tibrv Transport is like a JMS session. 2 Main actors in a CM conversation are the transports (“sessions”).
* sending CM transport
* listening CM transport
P 155 of [[rv concepts]] says
The sending CM transport is responsible to record each outbound message on
that subject, and to retain the message in its ledger until it receives
confirmation of delivery from the listener (or until the time limit of the
In return, the listening CM transport is responsible for confirming delivery of
Since the sender is a CM sender, i would assume receiver can’t be non-CM.
To send or receive messages using certified delivery features, a program must first
create a CM transport (also called a delivery-tracking transport). Each CM
transport employs an ordinary transport and adds some additional capabilities, such as a ledger.
JMS unicast topic has a centralized  broker. RV is decentralized. All the rvd processes are equal in status. This is a little bit similar to
– email servers among Ivy League universities.
– file-sharing peer networks
RV is peer-to-peer; JMS is hub/spoke. Note in JMS request/response, “server” has ambiguous meanings. Here we mean the broker.
 You can load-balance the broker but the “outside world” views them as a single broker. It’s the Hub of hub/spoke. Scalability bottleneck.
* account cache invalidation -> re-fetch from DB for that account.
However, after (not before) cache server starts up and rebuilds itself, all messages are guaranteed. When it’s down, such notifications are ignored. No guaranteed delivery please.
* management console to put an app server into and out of debug mode without restarts. J4? Cheap, fast, light. EMS is heavier.
* centrally log all emails sent out from many app servers
* market data feed intra-day
[[ rv concepts ]] has a concise comparison. A few highlights on CM:
(CM is one of the most important RV use cases in trading.)
* A producer sends a message to consumers directly (peer to peer, decentralized). The producer stores the message until each consumer has acknowledged receipt.
**Note the rv daemon is not a central server. There is usually one daemon on each host. These function a bit like cisco routers in a network. They don’t store or retry messages.
**I think p2p is a defining feature of rv (not only CM) relative to JMS.
* Disk failure on a peer host computer affects only the messages that its programs produce or consume. However, disk mirroring (often done on jms servers) for each individual peer is often impractical.
* my app creates “my private queue”, calling the no-arg constructor TibrvQueue(void)
* my app (or the system) creates a callback-method object, which must implement onMsg(TibrvListener,TibrvMsg). This is not a bean and definitely not a domain entity.
* my app creates myListner, calling its constructor with
** a callback-method object
** a subject
** other arguments
* my app calls queue.dispatch(void) or poll(void). Unlike JMS, NO message is returned, not on this thread!
* Messages arrive and onMsg(..) runs in the dispatch thread, asynchronously, with
** first arg = myListner
** 2nd arg = the actual message
Component – Description
Event Object – Represents program interest in a set of events, and the occurrence of a matching event. See Events on page 81.
Event Driver – The event driver recognizes the occurrence of events, and places them in the appropriate event queues for dispatch. Rendezvous software starts the event driver as part of its process initialization sequence (the open call). See Event Driver on page 83. No java api i believe.
Event Queue – A program (“”my app””) creates event queues to hold event objects in order until the program (“”my app””) can process them.
Event Dispatch Call – A Rendezvous function call that removes an event from an event queue or queue group, and runs the appropriate callback function to process the event. I think my app calls this in my app’s thread.
Callback Function – A program (“”my app””) defines callback functions to process events asynchronously. See Callback Functions on page 89.
Dispatcher Thread – Programs (“”my app””) usually dedicate one or more threads to the task of dispatching events. Callback functions run in these threads. class com.tibco.tibrv.TibrvDispatcher extends java.lang.Thread. Event Dispatcher Thread is a common design pattern in java, c#…
Rendezvous daemon (rvd) processes service all network communications for Rendezvous programs (“my apps”). Every Rendezvous program (in any supported language) uses a Rendezvous daemon, usually on local host, but possibly in another machine.
* When a program sends a message, rvd transmits it across the network.
* When a program listens to a subject, rvd receives messages from the network, and presents messages with matching subject names to the listening program.
(I think “program” means my app program that I write using RV api.)
Within Rendezvous apps, each network transport object represents a connection (“session??”) to an rvd process. Because rvd is an essential link in Rendezvous communications, if the transport cannot connect to an rvd process that is already running, it automatically starts a rvd daemon.
RVD is like a gateway. RVD sits between the network and “my app”.
— http://www.cs.cmu.edu/~priya/WFoMT2002/Pang-Maheshwari.pdf says —
An RV Sender app (java or c++) passes the message and destination topic (subject) to RVD. RVD then broadcasts this message using User Data Packet(UDP) to the entire network. All subscribing computers with RVDs on the network will receive this message. RVD will filter the messages which non-subscribers will not be notified of the message.
Rendezvous Java apps can connect to the network in either of two ways:
? An rvd transport connects to a Rendezvous daemon process (rvd).
? An rva transport connects to a Rendezvous agent process (rva), which in turn
connects to a Rendezvous daemon process.
For connections to the local network, a direct connection to rvd is more efficient
than an indirect connection through rva to rvd. In all situations we recommend
rvd transports (read “”connections””) in preference to rva transports—except for applets connecting to a remote home network, which must use rva transports
In RV, an Event object can represent
1) a listener (or program’s interest in events) and also
2) an “event occurrence” (but NOT a TibrvMsg instance). I think we need to get comfortable with this unusual design.
As described in the “flow” post, probably the main event object we use is the listener object. In fact, Chapter 5 of the java manual seems to suggest that
* TibrvListener is (probably the most important) subclass of TibrvEvent
* TibrvTimer is (probably the 2nd most important) subclass of TibrvEvent
* For beginners, you can think of TibrvEvent.java as an abstract base class of the 2
Looks like the consensus is
* tibco rv — fastest, high volume, high throughput, less-then-perfect reliability even with certified delivery
* JMS — slower but perfect reliability.
Before adding bells and whistles, core RV was designed for
* efficient delivery — no bandwidth waste
* high volume, high throughput
* multicast, not hub/spoke, not p2p
* imperfect reliability. CM is on top of core RV
* no distributed transaction, whereas JMS standard requires XA. I struggled with a Weblogic jms xa issue in 2006
— simple scenario —
If a new IBM quote has to reach 100 subscribers, JMS (hub/spoke unicast topic) uses tcp/ip to send it 100 times, sequentially. Each subscriber has a unique IP address and accepts his 1 message, and ignores the other 99. In contrast, Multicast sends  it once, and only creates copies when the links to the destinations split. RV uses a proprietary protocol over UDP.
 http://www.cs.cmu.edu/~priya/WFoMT2002/Pang-Maheshwari.pdf benchmark tests reveals “The SonicMQ broker requires a separate TCP connection to each of the subscriber. Hence the more subscribers there are, the longer it takes for SonicMQ to deliver the same amount of messages.“
 Same benchmark shows the sender is the RVD daemon. It simply broadcasts to the entire network. Whoever interested in the subject will get it.
P233 [[weblogic definitive guide]] suggests
* multicast jms broker has a constant workload regardless how many subscribers. This is because the broker sends it once rather than 100 times as in our example.
* multicast saves network bandwidth. Unicast topic requires 100 envelopes processed by the switches and routers.
The tibrv documentation [[RV concepts]] (https://docs.tibco.com/pub/rendezvous/8.3.1_january_2011/pdf/tib_rv_concepts.pdf) has good examples of when to use Certified Messaging. (minor edits by me)
Certified delivery is appropriate when a sending program requires individual confirmation of delivery for each message it sends. For example, a traveling sales representative enters sales orders on a laptop computer, and sends them to a central office. The representative must know for certain that the order processing system has received the order he sent.
Certified delivery is also appropriate when a receiving program cannot afford to miss any messages. For example, in an application that processes orders to buy and sell inventory items, each order is important. If any orders are omitted, then inventory records are incorrect. It’s like missing a incremental backup.
Certified delivery is appropriate when each message on a subject builds upon previous message (with that subject) — in sequence. For example, a sending program updates a receiving database, contributing part of the data fields in a record, but leaving other fields of the record unchanged. The database is correct only if all updates arrive in the order they are sent.
Certified delivery is appropriate in situations of intermittent physical connectivity—such as discontinuous network connections. For example, consider an application in which several mobile laptop computers must communicate with one another. Connectivity between mobile units is sporadic, requiring persistent storage of messages until the appropriate connections are re-established.
Everything is “over UDP”. See P 19 of [[ RV concepts ]]
TRDP broadcast or multicast protocol over UDP channel
TRDP unicast protocol over UDP channel
–Direct Communication (RV Version 7)
RPTP over UDP channel
* addressing — rv protocol is different from tcp/ip socket. No IP addresses. Rather, rv uses
** Subject-based addressing — For programs to communicate, they must agree upon a subject name at which to rendezvous (hence the name of this product). Subject-based addressing technology enables anonymous rendezvous, an important breakthrough
* rvd — is the name of the daemon. in most environments, one daemon process runs on each host computer.
* transport — one of the most important java classes to developers like us. There’s no jms counterpart, but the best you can get is the Session object + physical delivery options.
* unicast vs multicast. No other choice.
** multicast is prevalent
** unicast — sends a msg to exactly one Unix process id. point to point. inbox name
Q1: Why is certified messaging so widely used in trading systems when CM is not as reliable as JMS? I guess if a system already uses Rendezvous, then there’s motivation to avoid mixing it with JMS. If the given system needs higher reliability than the standard non-guaranteed RV delivery, then the easiest solution is CM. Is that how your users think?
A: I don’t think CM is not as reliable as JMS. However you can say CM’s acknowledgment mechanisms are not as flexible as JMS/EMS. The latter supports several acknowledgment modes. An important advantage of RV is it supports multi-cast which makes broadcasting quotes more efficient. JMS 5 supports something similar for topic subscriber but not as natural as RV.
Q2: in JMS, a receiver can 1) poll the queue or 2) wait passively for the queue to call back, using onMessage() — known as a JMS listener. These are the synchronous receiver and asynchronous receiver, respectively. I feel Rendezvous supports asynchronous only, in the form of onMsg(). The poll() or dispatch() methods of the TibrvQueue object don’t return the message, unlike the JMS polling operation.
If you can only remember one thing about JMS Ack, it’s
#1) 2 legs – there’s ACK on both legs of producer => broker => consumer. The 2 ACK can be present or absent independent of each other. See diagrams in [[JMS]]
#2) reliability – one of the most important features of JMS is reliability. Ack is a cornerstone. (tibrv is slightly less reliable.)
In JMS, session represents (roughly) a connection, or a socket. More precisely, one or more sessions are created from a single live connection. In JMS, sessions are single-threaded.
In RV, transport is the terminology. Means the same as a session, a socket, a connection to the “network”, to the daemon… Transport is a real object in memory.
In hibernate, session represents a connection to the DB. More precisely, a hibernate session HAS-A real connection object in memory
Uses a multicast IP address and UDB rather than TCP.
is not supported in JMS standard.
JMS (but no multicast?) unicast supports durable subscription.
RV is the prime example. Weblogic JMS also supports multicast.
Multicast (and therefore RV) is more efficient in terms of resources and redundant data transfer.
Q: usually not guaranteed to be once-and-only-once?
A: TIBCO offers a transactional augmentation of RV (RVTX) that guarantees a 1-and-only-once delivery; however, this solution essentially converts RV into a hub-and-spoke system. Consequently, very few RV implementations are transactional.
Q: the JMS synchronous receiver (poller) won’t work. You must use async like onMessage()?
A: i think every receiver must stand ready. A receiver can’t decide when to pull from server.
Q: out-of-sequence is possible in multicast and JMS?
A: both possible
* msg objects — well-defined entity beans
* transport objects
The 3 objects below build on top of msg and transports … These are described in one single chapter [[Events and Queues]] in the java documentation.
* listener objects
* callback objects, implementing onMsg()
* queue objects and dispatchable interface
* dispatcher object (subclass of Thread)
Once you understand the above, you can see that certified messaging builds on top of them:
* CM transport objects
* CM listener objects
* CM review callback — for reviewing “missed calls”
* CM msg objects