tibRV java app receiving msg – the flow

The flow:
* my app creates “my private queue”, calling the no-arg constructor TibrvQueue(void)
* my app (or the system) creates a callback-method object, which must implement onMsg(TibrvListener,TibrvMsg). This is not a bean and definitely not a domain entity.
* my app creates myListner, calling its constructor with
** a callback-method object
** myPrivateQueue
** a subject
** other arguments

* my app calls queue.dispatch(void) or poll(void). Unlike JMS, NO message is returned, Not on this thread!
* Messages arrive and onMsg(..) runs in the dispatch thread, asynchronously, with
** first arg = myListner
** 2nd arg = the actual message

Component – Description

Event Object – Represents program interest in a set of events, and the occurrence of a matching event. See Events on page 81.

Event Driver – The event driver recognizes the occurrence of events, and places them in the appropriate event queues for dispatch. Rendezvous software starts the event driver as part of its process initialization sequence (the open call). See Event Driver on page 83. No java api i believe.

Event Queue – A program (“”my app””) creates (private) event queues to hold event (message) objects in order until the program (“”my app””) can process them.

Event Dispatch Call – A Rendezvous function call that removes an event from an event queue or queue group, and runs the appropriate callback function to process the event. I think my app calls this in my app’s thread.

Callback Function – A program (“”my app””) defines callback functions to process events asynchronously. See Callback Functions on page 89.

Dispatcher Thread – Programs (“”my app””) usually dedicate one or more threads to the task of dispatching events. Callback functions run in these threads. class com.tibco.tibrv.TibrvDispatcher extends java.lang.Thread. Event Dispatcher Thread is a common design pattern in java, c#…

tibrv is decentralized, peer-to-peer, !! hub/spoke

JMS unicast topic has a centralized [1] broker. RV is decentralized. All the rvd processes are equal in status. This is a little bit similar to
– email servers among Ivy League universities.
– file-sharing peer networks

RV is peer-to-peer; JMS is hub/spoke. Note in JMS request/response, “server” has ambiguous meanings. Here we mean the broker.

[1] You can load-balance the broker but the “outside world” views them as a single broker. It’s the Hub of hub/spoke. Scalability bottleneck.

tibrv subject based address`^jms topics

tibrv subject based address` (SBA)^jms topics

Among the defining features of Tibrv, [ A) decentralization and B) SBA ] are 2 sides of the same coin. For me, it’s a struggle to find out the real difference between SBA vs jms topics. Here’s my attempt.

#1) In SBA(subject-based-addressing), there’s no central server holding a subject. In tibrv, any sender/receiver can specify any subject. Nice — No admin to create subjects. I tested it with tibrvlisten/tibrvsend. By contrast, In JMS, “The physical creation of topics is an administrative task” on the central broker [2], not on the producer or consumer.

) If a jms broker goes down, so do topics therein.

) a jms publisher (or subscriber) must physically connect to a physical broker process[2]. The broker has a physical network address. Our topic is tied to that physical address. A tibrv subject has no physical address.

) tibrv SBA (subject-based-addressing) uses a subject tree. No such tree among JMS topics. The tree lets a subscriber receive q(prices.stocks.*)  but also q(*.your.*). See rv_concepts.

[2] such as the weblogic server in autoreo.

message requester to wait5min for response #wait()

Imagine a typical request/reply messaging system. I think in JMS it’s usually based on temp queues, reply-to and correlation-Id — See other blog post. In contrast, RV has no broker. It’s decentralized into multiple peer rv daemons. No difference in this case —

Suppose a message broker holds a lot of queues. One of the queues is for a request message, from requester system to a pricing system. Another queue is for pricing system to return the new price to the requester.

Now, pricing system is slow. Requester should wait for no more than 5 minutes. If the new price comes back through the reply-queue 301 sec later, requester will ignore this stale price since it’s too risky to place an order on a stale price in a fast market. How do you implement this?

My design — Requester main thread can use condVar wait(5*60*000). Another thread in requester JVM can block forever in onMsg(), and notify main thread when something received.

Timer is the 2nd usage of condVar as Stoustrup described.

(I actually implemented this in a few trading engines.)

msg overflow ] JMS^tibrv #60sec

Message overflow is a frequent headache in MOM. Basically consumption rate (possibly 0) is slower than production rate, eating up storage capacity, either in memory or on disk.

  • eg: JMS Topic with NDS (non-durable-subscriber) only — almost free of overflow. Bonnie said broker will discard. Jerry said broker keeps track of alive subscribers. Every incoming message is guaranteed to be delivered (by retry) to each. Broker must persist pending messages. When a subscribe disconnects gracefully, broker no longer needs to keep any pending message for her.
  • eg: chatrooms don’t persist data but what type of MOM is that? probably NDS topic
  • eg: JMS queue. While any number of producers can send messages to the queue, each message is guaranteed to be delivered, and consumed by one consumer. If no consumers are registered to consume the messages, the queue holds (on disk) them until a consumer registers to consume them.
  • eg: JMS temp queue??
  • eg: regular queue? In SIM upfront, unread messages –persist–
  • eg: mailing list (topic with durable subscribers)? –Persist–
  • eg: JMS topic without durable subscriber? see post on message retention
  • eg: tibrv CM? Storage is the ledger file. –persist–
  • eg: tibrv non-CM? market data feed? Discarded but how soon? 60 sec by default? Yes, in slow consume and fast producer scenario message lost will be possible — http://arun-tibco.blogspot.com/2010/09/tibco-rv-faqs.html


subject-based addressing and tibrv

http://citeseerx.ist.psu.edu/viewdoc/download?doi= says

Subjects are arranged in a subject tree by using a dot notation, and clients can either subscribe to a single subject (e.g., finance.quotes.NASDAQ.FooInc) or use wildcards (e.g., finance.quotes.NASDAQ.*).

JMS topic is basically subject-based, or more precisely “Channel-based”, as a (selector-less) subscriber gets everything in the channel.

jms queue “can be compared to TIBCO Rendezvous Distributes Queues (RVDQ)”. I feel tibrv inbox is also like a queue. See http://www.cs.cmu.edu/~priya/WFoMT2002/Pang-Maheshwari.pdf.

tibrv vs EMS — answer from a Tibco insider

Q: is RV still the market leader in the “max throughput” MOM market?

Q: for trading apps, i feel the demand for max-throughput MOM is just
as high as before. In this space, is RV (and multicast in general)
facing any competition from JMS or newer technologies? I feel answer
is no.

A: I think RV is still the market leader.  29west is a strong
competitor.  TIBCO now sells a “RV in hardware appliance” partly to
address that.  I feel JMS is not really targeting max-throughput

JMS temporary queue, tibrv and sybase temp table

Update — tibrv’s flexibility make JMS tmp queue look unnecessary, rigid, uptight and old fashioned.

tmp queue is (tmp topic?[4]) used for request/reply. I feel the essential elements of JMS request/reply are
1) tmp queue
2) set and get reply-to — setJMSReplyTo(yourTmpQueue)
3) correlation id — setJMSCorrelationID(someRequestID)

Server-side (not the broker, but another JMS user) uses getJMSCorrelationID() and then setJMSCorrelationID() in the response message, then replies to getJMSReplyTo()

tmp queue is like a Sybase tmp table — (I feel this is the key to understanding tmp queue.)
* created by requestor
* removed by the “system janitor”, or by requestor
* private to the session. Therefore creator threads must keep the session alive.

[4] temp topic too. p67 [[JMS]]

generic callback in a concurrent java/c++ sys

(Note OO is a development-time concept. Once compiled, an OO program looks no different from a C program with tons of structs and function pointers.)

Most financial systems today are concurrent OO systems. The lowly callback pattern is extremely common but actually requires a thorough understanding of both thread and object interaction. Let’s examine some. Note just about every player in the callback scenarios is a pointer, including function pointers.

In the simplest 1-thread case, an observer pointee registers interest with a subject object, by giving its own address. After subject state changes in Thr 1, the same thread loops through the list of observers and invokes the callback method on each observer. Note the observer object must not be deallocated.

For onMsg() in tibrv or onMessage() in JMS, there’s a remote “upstream” thread in the message broker’s PID/runtime, and there’s a separate downstream thread in the listener’s PID. I believe this thread starts out blocking for messages. Upon receiving a msg, the runtime somehow wakes up this blocking thread and this thread invokes onMsg() on the listener pointee. Meanwhile what’s the remote upstream thread doing? Simplest design is synchronous — upstream thread waits for the listener thread to finish. This means the upstream thread can only dispatch the message to remote listeners sequentially. I feel it’s much better to get the upstream thread to return right away after sending the one-way-message[3] (see Doug Lea’s book) so this thread can contact the 2nd listener in the list. Rewind to registration time — Any thread could register the listener with “Hollywood”. This thread could terminate immediately but the listener pointee must live.

[3] in classic Reo, we actually send one-way messages to the market-facing gateway. We get the status message on a separate thread via onMessage().

In C/C++, you can also register into “Hollywood” a ptr to a free func. No object, usually no state (unless the free func uses stateful static objects). Free functions are free of destruction. Hollywood could save the (function) ptr in any collection. When there’s an event, Hollywood calls us back, in any thread.

tibrv-CM transport i.e. session #again

A tibrv Transport is like a JMS session. 2 Main actors in a CM conversation are the transports (“sessions”).
* sending CM transport
* listening CM transport
P 155 of [[rv concepts]] says

The sending CM transport is responsible to record each outbound message on
that subject, and to retain the message in its ledger until it receives
confirmation of delivery from the listener (or until the time limit of the
message elapses).

In return, the listening CM transport is responsible for confirming delivery of
each message.

Since the sender is a CM sender, i would assume receiver can’t be non-CM.

To send or receive messages using certified delivery features, a program must first
create a CM transport (also called a delivery-tracking transport). Each CM
transport employs an ordinary transport and adds some additional capabilities, such as a ledger.

tibrv-CM ^JMS

[[ rv concepts ]] has a concise comparison. A few highlights on CM:

(CM is one of the most important RV use cases in trading.)

* A producer sends a message to consumers directly (peer to peer, decentralized). The producer stores the message until each consumer has acknowledged receipt.

**Note the rv daemon is not a central server. There is usually one daemon on each host. These function a bit like cisco routers in a network. They don’t store or retry messages.

**I think p2p is a defining feature of rv (not only CM) relative to JMS.

* Disk failure on a peer host computer affects only the messages that its programs produce or consume. However, disk mirroring (often done on jms servers) for each individual peer is often impractical.

tibrv-UDP-multicast vs tcp-hub/spoke-JMS


Looks like the consensus is
* tibco rv — fastest, high volume, high throughput, less-then-perfect reliability even with certified delivery
* JMS — slower but perfect reliability.

Before adding bells and whistles, core RV was designed for
* efficient delivery — no bandwidth waste
* high volume, high throughput
* multicast, not hub/spoke, not p2p
* imperfect reliability. CM is on top of core RV
* no distributed transaction, whereas JMS standard requires XA. I struggled with a Weblogic jms xa issue in 2006

— simple scenario —

If a new IBM quote has to reach 100 subscribers, JMS (hub/spoke unicast topic) uses tcp/ip to send it 100 times, sequentially [3]. Each subscriber has a unique IP address and accepts this 1 message, and ignores the other 99. In contrast, Multicast sends [4] it once, and only duplicates/clones a message when the links to the destinations split [5]. RV uses a proprietary protocol over UDP.

[5] see my blogpost https://wordpress.com/post/bintanvictor.wordpress.com/32651

[3] http://www.cs.cmu.edu/~priya/WFoMT2002/Pang-Maheshwari.pdf benchmark tests reveals “The SonicMQ broker requires a separate TCP connection to each of the subscriber. Hence the more subscribers there are, the longer it takes for SonicMQ to deliver the same amount of messages.
[4] Same benchmark shows the sender is the RVD daemon. It simply broadcasts to the entire network. Whoever interested in the subject will get it.

— P233 [[weblogic definitive guide]] suggests
* multicast jms broker has a constant workload regardless how many subscribers. This is because the broker sends it once rather than 100 times as in our example.
* multicast saves network bandwidth. Unicast topic requires 100 envelopes processed by the switches and routers.

keywords of TibRV

* addressing — rv protocol is different from tcp/ip socket. No IP addresses. Rather, rv uses
** Subject-based addressing — For programs to communicate, they must agree upon a subject name at which to rendezvous (hence the name of this product). Subject-based addressing technology enables anonymous rendezvous, an important breakthrough

* rvd — is the name of the daemon. in most environments, one daemon process runs on each host computer.

* transport — one of the most important java classes to developers like us. There’s no jms counterpart, but the best you can get is the Session object + physical delivery options.

* unicast vs multicast. No other choice.
** multicast is prevalent
** unicast — sends a msg to exactly one Unix process id. point to point. inbox name

2 questions on tibRV to a Tibco friend #LS

Q1: Why is certified messaging so widely used in trading systems when CM is not as reliable as JMS? I guess if a system already uses Rendezvous, then there’s motivation to avoid mixing it with JMS. If the given system needs higher reliability than the standard non-guaranteed RV delivery, then the easiest solution is CM. Is that how your users think?

A: I don’t think CM is not as reliable as JMS. However you can say CM’s acknowledgment mechanisms are not as flexible as JMS/EMS. The latter supports several acknowledgment modes. An important advantage of RV is it supports multi-cast which makes broadcasting quotes more efficient. JMS 5 supports something similar for topic subscriber but not as natural as RV.

Q2: in JMS, a receiver can 1) poll the queue or 2) wait passively for the queue to call back, using onMessage() — known as a JMS listener. These are the synchronous receiver and asynchronous receiver, respectively. I feel Rendezvous supports asynchronous only, in the form of onMsg(). The poll() or dispatch() methods of the TibrvQueue object don’t return the message, unlike the JMS polling operation.

JMS acknowledgment – cheatsheet

If you can only remember one thing about JMS Ack, it’s

#1) 2 legs – there’s ACK on both legs of producer => broker => consumer. The 2 ACK can be present or absent independent of each other. See diagrams in [[JMS]]

#2) reliability – one of the most important features of JMS is reliability. Ack is a cornerstone.

tibrv is slightly less reliable. but CM-transport also uses confirmation on each individual message… see other blogpost.

tibRV transport^connection^session

In JMS, session represents (roughly) a connection, or a socket. More precisely, one or more sessions are created from a single live connection. In JMS, sessions are single-threaded.

In RV, transport is the terminology. Means the same as a session, a socket, a connection to the “network”, to the daemon… Transport is a real object in memory.

In hibernate, session represents a connection to the DB. More precisely, a hibernate session HAS-A real connection object in memory

multicast tibrv ^ JMS #1st take

multicast is not supported in JMS standard. Note JMS (but no multicast?) unicast supports durable subscription.

RV is a good example of multicast. Weblogic JMS also supports multicast.

Multicast (and therefore RV) is more efficient in terms of resources and redundant data transfer.

Q: Unlikely TCP, multicast messages are usually not guaranteed to be once-and-only-once?
A: TIBCO offers a transactional augmentation of RV (RVTX) that guarantees a 1-and-only-once delivery; however, this solution essentially converts RV into a hub-and-spoke system. Consequently, very few RV implementations are transactional.

Q: the JMS synchronous receiver (poller) won’t work. You must use async like onMessage()?
A: i think every receiver must stand ready. A receiver can’t decide when to pull from server.

Q: out-of-sequence is possible in multicast and JMS?
A: both possible

solace^tibcoApplicance #OPRA volume

http://solacesystems.com/news/fastest-jms-broker/ solace JMS broker (Solace Message Router) support 100,000 messages per second in persistent mode and 10 million messages non-persistent. In a more detailed article, http://solacesystems.com/solutions/messaging-middleware/jms/ shows 11 million 100-byte non-persistent messages.

A major sell-side’s messaging platform chief said his most important consideration was the deviation of peak-to-average latency and outliers. A small amount of deviation and (good) predictability were key. They chose Solace.

http://www.hpcwire.com/hpcwire/2009-09-14/solace_systems_sets_the_pace_in_the_race_to_zero_latency.html has good details.

In all cases (Solace, Tibco, Tervela), hardware-based appliances *promise* at least 10 fold boost in performance compared to software solutions. Latency within the appliance is predictably low, but the end-to-end latency is not. Because of the separate /devices/ and the network hops between them, the best-case latency is in the tens of microseconds. The next logical step is to integrate the components into a single system to avoid all the network latency and intermediate memory copies (including serializations). Solace has demonstrated sub-microsecond latencies by adding support for inter-process communications (IPC) via shared memory. Developers will be able to fold the ticker feed function, the messaging platform, and the algorithmic engine into the same “application” [1], and use shared memory IPC as the data transport (though I feel single-application design need no IPC).

For best results you want to keep each “application” [1] on the same multi-core processor, and nail individual application components (like the feed handler and algo engine) to specific cores. That way, application data can be shared between the cores in the Level 2 cache.

[1] Each “application” is potentially a multi-process application with multiple address spaces, and may need IPC.

Benchmark — Solace ran tests with a million 100-byte messages per second, achieving an average latency of less than 700 nanoseconds using a single Intel processor. As of 2009, OPRA topped out at about a million messages per second. OPRA hit 869,109 mps (msg/sec) in Apr 2009.

Solace vs RV appliance — Although Solace already offers its own appliance, it runs other messaging software. The Tibco version runs Rendezvous (implemented in ASIC+FPGA), providing a clear differentiator between the Tibco and Solace appliances.

Solace 3260 Message Router is the product chosen by most Wall St. customers.

http://kirkwylie.blogspot.com/2008/11/meeting-with-solace-systems-hardware.html provides good tech insights.

JMS guaranteed delivery — guaranteed by who@@

guaranteed delivery (GD) is a “service” provided by the JMS infrastructure — the part of the system outside your custom application code. It's also known as the vendor API, which is software provided by IBM, Tibco, BEA etc. A formal name is “JMS provider”. It helps to think of this infrastructure as encrypted source code (ESC).

To provide GD (or any JMS service), this infrastructure must include client-side ESC — formally known as client-runtime.

ESC is simpler in the JDBC world. Sybase provides both a server and a client-side ESC. Your code calls the client-side ESC as a lower-layer API. eSpeed also ships a client side ESC. You link into your custom trading engine, and the client side ESC talks to the eSpeed server.

In tibrv, the daemon runs in each client host and is comparable to a client-side ESC but
$ it runs in a separate process
$ tibrv daemons are peer-to-peer, no client no server.

onMessage() is one of the most important part of the client-side ESC.  But back to GD, ESC supports

* Stored and Forward
* Ack
* auto_ack is sent by the client ESC
* retry at each leg (sender->broker->receiver)