strategic value of MOM]tech evolution

What’s the long-term value of MOM technology? “Value” to my career and to the /verticals/ I’m following such as finance and internet. JMS, Tibrv (and derivatives) are the two primary MOM technologies for my study.

• Nowadays JMS (tibrv to a lesser extent) seldom features in job interviews and job specs, but the same can be said about servlet, xml, Apache, java app servers .. I think MOM is falling out of fashion but not a short-lived fad technology. MOM will remain relevant for decades. I saw this longevity deciding to invest my time.
• Will socket technology follow the trend?
• [r] Key obstacle to MOM adoption is perceived latency “penalty”. I feel this penalty is really tolerable in most cases.
• — strengths
• [r] compares favorably in terms of scalability, efficiency, reliability, platform-neutrality.
• encourages modular design and sometimes decentralized architecture. Often leads to elegant simplification in my experience.
• [r] flexible and versatile tool for the architect
• [rx] There has been extensive lab research and industrial usage to iron out a host of theoretical and practical issues. What we have today in MOM is a well-tuned, time-honored, scalable, highly configurable, versatile, industrial strength solution
• works in MSA
• [rx] plays well with other tech
• [rx] There are commercial and open-source implementations
• [r] There are enterprise as well as tiny implementations
• — specific features and capabilities
• [r] can aid business logic implementation using content filtering (doable in rvd+JMS broker) and routing
• can implement point-to-point request/response paradigm
• [r] transaction support
• can distribute workload as in 95G
• [r] can operate in-memory or backed by disk
• can run from firmware
• can use centralized hub/spoke or peer-to-peer (decentralized)
• easy to monitor in real time. Tibrv is subject-based, so you can easily run a listener on the same topic
• [x=comparable to xml]
• [r=comparable to RDBMS]

Jump IV Aug 2012

Q1: pros and cons of vector vs linked list?
Q1b: Given a 100-element collection, compare performance of … (iteration? Lookup?)

Q: UDP vs TCP diff?
%%A: multicast needs UDP.

Q: How would you add reliability to multicast?

Q: How would you use tibco for trade messages vs pricing messages?
%%A CM for trade and non-CM for price?

Q5: In your systems, how serious was data loss in non-CM multicast?
%%A: Usually not a big problem. During peak volatile periods, messaging rates could surge 500%. Data loss would deteriorate.

Q5b: how would you address the high data loss?
%%A: test with a target message rate. Beyond the target rate, we don’t feel confident.

Q7: how is order state managed in your OMS engine?
%%A: if an order is half-processed and pending the 3nd reply from ECN, the single thread would block.
AA: in ETS, Every order is persisted till EOD (to support bust/corrections by exchange or cancels by clients)

Q7b: even if multiple orders (for the same security) are waiting in the queue?
%%A: yes. To allow multiple orders to enter the “stream” would be dangerous.

Now I think the single thread should pick up and process all new orders and keep all pending orders in cache. Any incoming exchange messages would join the same task queue (or a separate task queue) – the same single thread.

3 main infrastructure teams
* exchange connectivity – order submission
* exchange connectivity – pricing feed. I think this is incoming-only, probably higher volume. Probably similar to Zhen Hai’s role.
* risk infrastructure – no VaR mathematics.

tibRV java app receiving msg – the flow

The flow:
* my app creates “my private queue”, calling the no-arg constructor TibrvQueue(void)
* my app (or the system) creates a callback-method object, which must implement onMsg(TibrvListener,TibrvMsg). This is not a bean and definitely not a domain entity.
* my app creates myListner, calling its constructor with
** a callback-method object
** myPrivateQueue
** a subject
** other arguments

* my app calls queue.dispatch(void) or poll(void). Unlike JMS, NO message is returned, Not on this thread!
* Messages arrive and onMsg(..) runs in the dispatch thread, asynchronously, with
** first arg = myListner
** 2nd arg = the actual message

————————
Component – Description

Event Object – Represents program interest in a set of events, and the occurrence of a matching event. See Events on page 81.

Event Driver – The event driver recognizes the occurrence of events, and places them in the appropriate event queues for dispatch. Rendezvous software starts the event driver as part of its process initialization sequence (the open call). See Event Driver on page 83. No java api i believe.

Event Queue – A program (“”my app””) creates (private) event queues to hold event (message) objects in order until the program (“”my app””) can process them.

Event Dispatch Call – A Rendezvous function call that removes an event from an event queue or queue group, and runs the appropriate callback function to process the event. I think my app calls this in my app’s thread.

Callback Function – A program (“”my app””) defines callback functions to process events asynchronously. See Callback Functions on page 89.

Dispatcher Thread – Programs (“”my app””) usually dedicate one or more threads to the task of dispatching events. Callback functions run in these threads. class com.tibco.tibrv.TibrvDispatcher extends java.lang.Thread. Event Dispatcher Thread is a common design pattern in java, c#…

tibrv is decentralized, peer-to-peer, !! hub/spoke

JMS unicast topic has a centralized [1] broker. RV is decentralized. All the rvd processes are equal in status. This is a little bit similar to
– email servers among Ivy League universities.
– file-sharing peer networks

RV is peer-to-peer; JMS is hub/spoke. Note in JMS request/response, “server” has ambiguous meanings. Here we mean the broker.

[1] You can load-balance the broker but the “outside world” views them as a single broker. It’s the Hub of hub/spoke. Scalability bottleneck.

## if a event-driven trading engine is!! responding

• Culprit: deadlock
• Culprit: all threads in the pool are blocked in wait(), lock() or …
• Culprit: bounded queue is full. Sometimes the thread that adds task to the queue is blocked while doing that.
• Culprit: in some systems, there’s a single task dispatcher thread like swing EDT. That thread can sometimes get stuck
• Suggestion: dynamically turn on verbose logging in the messaging module within the engine, so it always logs something to indicate activity. It’s like the flashing LED in your router. You can turn on such logging by JMX.
• Suggestion: snoop
• Suggestion: for tibrv, you can easily start a MS-windows tibrv listener on the same subject as the listener inside the trading engine. This can reveal activity on the subject

message filter – jms ^ tibrv

jms message selector is executed on the broker.

rvd executes the same duty — “Filter subject-addressed messages.”

tibrv subject based address`^jms topics

tibrv subject based address` (SBA)^jms topics

Among the defining features of Tibrv, [ A) decentralization and B) SBA ] are 2 sides of the same coin. For me, it’s a struggle to find out the real difference between SBA vs jms topics. Here’s my attempt.

#1) In SBA(subject-based-addressing), there’s no central server holding a subject. In tibrv, any sender/receiver can specify any subject. Nice — No admin to create subjects. I tested it with tibrvlisten/tibrvsend. By contrast, In JMS, “The physical creation of topics is an administrative task” on the central broker [2], not on the producer or consumer.

) If a jms broker goes down, so do topics therein.

) a jms publisher (or subscriber) must physically connect to a physical broker process[2]. The broker has a physical network address. Our topic is tied to that physical address. A tibrv subject has no physical address.

) tibrv SBA (subject-based-addressing) uses a subject tree. No such tree among JMS topics. The tree lets a subscriber receive q(prices.stocks.*)  but also q(*.your.*). See rv_concepts.

[2] such as the weblogic server in autoreo.

message requester to wait5min for response #wait()

Imagine a typical request/reply messaging system. I think in JMS it’s usually based on temp queues, reply-to and correlation-Id — See other blog post. In contrast, RV has no broker. It’s decentralized into multiple peer rv daemons. No difference in this case —

Suppose a message broker holds a lot of queues. One of the queues is for a request message, from requester system to a pricing system. Another queue is for pricing system to return the new price to the requester.

Now, pricing system is slow. Requester should wait for no more than 5 minutes. If the new price comes back through the reply-queue 301 sec later, requester will ignore this stale price since it’s too risky to place an order on a stale price in a fast market. How do you implement this?

My design — Requester main thread can use condVar wait(5*60*000). Another thread in requester JVM can block forever in onMsg(), and notify main thread when something received.

Timer is the 2nd usage of condVar as Stoustrup described.

(I actually implemented this in a few trading engines.)

msg overflow ] JMS^tibrv #60sec

Message overflow is a frequent headache in MOM. Basically consumption rate (possibly 0) is slower than production rate, eating up storage capacity, either in memory or on disk.

• eg: JMS Topic with NDS (non-durable-subscriber) only — almost free of overflow. Bonnie said broker will discard. Jerry said broker keeps track of alive subscribers. Every incoming message is guaranteed to be delivered (by retry) to each. Broker must persist pending messages. When a subscribe disconnects gracefully, broker no longer needs to keep any pending message for her.
• eg: chatrooms don’t persist data but what type of MOM is that? probably NDS topic
• eg: JMS queue. While any number of producers can send messages to the queue, each message is guaranteed to be delivered, and consumed by one consumer. If no consumers are registered to consume the messages, the queue holds (on disk) them until a consumer registers to consume them.
• eg: JMS temp queue??
• eg: regular queue? In SIM upfront, unread messages –persist–
• eg: mailing list (topic with durable subscribers)? –Persist–
• eg: JMS topic without durable subscriber? see post on message retention
• eg: tibrv CM? Storage is the ledger file. –persist–
• eg: tibrv non-CM? market data feed? Discarded but how soon? 60 sec by default? Yes, in slow consume and fast producer scenario message lost will be possible — http://arun-tibco.blogspot.com/2010/09/tibco-rv-faqs.html

 

subject-based addressing and tibrv

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.60.5735&rep=rep1&type=pdf says

Subjects are arranged in a subject tree by using a dot notation, and clients can either subscribe to a single subject (e.g., finance.quotes.NASDAQ.FooInc) or use wildcards (e.g., finance.quotes.NASDAQ.*).

JMS topic is basically subject-based, or more precisely “Channel-based”, as a (selector-less) subscriber gets everything in the channel.

jms queue “can be compared to TIBCO Rendezvous Distributes Queues (RVDQ)”. I feel tibrv inbox is also like a queue. See http://www.cs.cmu.edu/~priya/WFoMT2002/Pang-Maheshwari.pdf.

tibrv vs EMS — answer from a Tibco insider

Q: is RV still the market leader in the “max throughput” MOM market?

Q: for trading apps, i feel the demand for max-throughput MOM is just
as high as before. In this space, is RV (and multicast in general)
facing any competition from JMS or newer technologies? I feel answer
is no.

A: I think RV is still the market leader.  29west is a strong
competitor.  TIBCO now sells a “RV in hardware appliance” partly to
address that.  I feel JMS is not really targeting max-throughput
space.

JMS temporary queue, tibrv and sybase temp table

Update — tibrv’s flexibility make JMS tmp queue look unnecessary, rigid, uptight and old fashioned.

tmp queue is (tmp topic?[4]) used for request/reply. I feel the essential elements of JMS request/reply are
1) tmp queue
2) set and get reply-to — setJMSReplyTo(yourTmpQueue)
3) correlation id — setJMSCorrelationID(someRequestID)

Server-side (not the broker, but another JMS user) uses getJMSCorrelationID() and then setJMSCorrelationID() in the response message, then replies to getJMSReplyTo()

tmp queue is like a Sybase tmp table — (I feel this is the key to understanding tmp queue.)
* created by requestor
* removed by the “system janitor”, or by requestor
* private to the session. Therefore creator threads must keep the session alive.

[4] temp topic too. p67 [[JMS]]

generic callback in a concurrent java/c++ sys

(Note OO is a development-time concept. Once compiled, an OO program looks no different from a C program with tons of structs and function pointers.)

Most financial systems today are concurrent OO systems. The lowly callback pattern is extremely common but actually requires a thorough understanding of both thread and object interaction. Let’s examine some. Note just about every player in the callback scenarios is a pointer, including function pointers.

In the simplest 1-thread case, an observer pointee registers interest with a subject object, by giving its own address. After subject state changes in Thr 1, the same thread loops through the list of observers and invokes the callback method on each observer. Note the observer object must not be deallocated.

For onMsg() in tibrv or onMessage() in JMS, there’s a remote “upstream” thread in the message broker’s PID/runtime, and there’s a separate downstream thread in the listener’s PID. I believe this thread starts out blocking for messages. Upon receiving a msg, the runtime somehow wakes up this blocking thread and this thread invokes onMsg() on the listener pointee. Meanwhile what’s the remote upstream thread doing? Simplest design is synchronous — upstream thread waits for the listener thread to finish. This means the upstream thread can only dispatch the message to remote listeners sequentially. I feel it’s much better to get the upstream thread to return right away after sending the one-way-message[3] (see Doug Lea’s book) so this thread can contact the 2nd listener in the list. Rewind to registration time — Any thread could register the listener with “Hollywood”. This thread could terminate immediately but the listener pointee must live.

[3] in classic Reo, we actually send one-way messages to the market-facing gateway. We get the status message on a separate thread via onMessage().

In C/C++, you can also register into “Hollywood” a ptr to a free func. No object, usually no state (unless the free func uses stateful static objects). Free functions are free of destruction. Hollywood could save the (function) ptr in any collection. When there’s an event, Hollywood calls us back, in any thread.

tibrv-CM transport i.e. session #again

A tibrv Transport is like a JMS session. 2 Main actors in a CM conversation are the transports (“sessions”).
* sending CM transport
* listening CM transport
P 155 of [[rv concepts]] says

The sending CM transport is responsible to record each outbound message on
that subject, and to retain the message in its ledger until it receives
confirmation of delivery from the listener (or until the time limit of the
message elapses).

In return, the listening CM transport is responsible for confirming delivery of
each message.

Since the sender is a CM sender, i would assume receiver can’t be non-CM.

To send or receive messages using certified delivery features, a program must first
create a CM transport (also called a delivery-tracking transport). Each CM
transport employs an ordinary transport and adds some additional capabilities, such as a ledger.

TibRV usage]%%department

* account cache invalidation -> re-fetch from DB for that account.

However, after (not before) cache server starts up and rebuilds itself, all messages are guaranteed. When it’s down, such notifications are ignored. No guaranteed delivery please.

* management console to put an app server into and out of debug mode without restarts. J4? Cheap, fast, light. EMS is heavier.

* centrally log all emails sent out from many app servers

* market data feed intra-day

tibrv-CM ^JMS

[[ rv concepts ]] has a concise comparison. A few highlights on CM:

(CM is one of the most important RV use cases in trading.)

* A producer sends a message to consumers directly (peer to peer, decentralized). The producer stores the message until each consumer has acknowledged receipt.

**Note the rv daemon is not a central server. There is usually one daemon on each host. These function a bit like cisco routers in a network. They don’t store or retry messages.

**I think p2p is a defining feature of rv (not only CM) relative to JMS.

* Disk failure on a peer host computer affects only the messages that its programs produce or consume. However, disk mirroring (often done on jms servers) for each individual peer is often impractical.

tibRV daemon — briefly

Rendezvous daemon (rvd) processes service all network communications for Rendezvous programs (“my apps”). Every Rendezvous program (in any supported language) uses a Rendezvous daemon, usually on local host, but possibly in another machine.

* When a program sends a message, rvd transmits it across the network.
* When a program listens to a subject, rvd receives messages from the network, and presents messages with matching subject names to the listening program.

(I think “program” means my app program that I write using RV api.)

Within Rendezvous apps, each network transport object represents a connection (“session??”) to an rvd process. Because rvd is an essential link in Rendezvous communications, if the transport cannot connect to an rvd process that is already running, it automatically starts a rvd daemon.

RVD is like a gateway. RVD sits between the network and “my app”.

— http://www.cs.cmu.edu/~priya/WFoMT2002/Pang-Maheshwari.pdf says —
An RV Sender app (java or c++) passes the message and destination topic (subject) to RVD. RVD then broadcasts this message using User Data Packet(UDP) to the entire network. All subscribing computers with RVDs on the network will receive this message. RVD will filter the messages which non-subscribers will not be notified of the message.

tibrv daemon ^ agent

Rendezvous Java apps can connect to the network in either of two ways:
? An rvd transport (session?) connects to a Rendezvous daemon process (rvd).
? An rva transport (session?) connects to a Rendezvous agent process (rva), which in turn
connects to a Rendezvous daemon process.

For connections to the local network, a direct connection to rvd is more efficient
than an indirect connection through rva to rvd. In all situations we recommend
rvd transports (read “”connections””) in preference to rva transports—except for applets connecting to a remote home network, which must use rva transports

tibRV event object@@

C# “event” also has a dual-meaning…

In RV, an Event object can represent
1) a listener (or program’s interest in events) and also
2) an “event occurrence” (but NOT a TibrvMsg instance). I think we need to get comfortable with this unusual design.

As described in the “flow” blogpost, probably the main event object we use is the listener object. In fact, Chapter 5 of the java manual seems to suggest that
* TibrvListener is (probably the most important) subclass of TibrvEvent
* TibrvTimer is (probably the 2nd most important) subclass of TibrvEvent
* For beginners, you can think of TibrvEvent.java as an abstract base class of the 2

tibrv-UDP-multicast vs tcp-hub/spoke-JMS

http://narencoolgeek.blogspot.com/2006/01/tibco-rv-vs-tibco-ems.html
http://stackoverflow.com/questions/580858/tibco-rendezvous-and-msmq

Looks like the consensus is
* tibco rv — fastest, high volume, high throughput, less-then-perfect reliability even with certified delivery
* JMS — slower but perfect reliability.

Before adding bells and whistles, core RV was designed for
* efficient delivery — no bandwidth waste
* high volume, high throughput
* multicast, not hub/spoke, not p2p
* imperfect reliability. CM is on top of core RV
* no distributed transaction, whereas JMS standard requires XA. I struggled with a Weblogic jms xa issue in 2006

— simple scenario —

If a new IBM quote has to reach 100 subscribers, JMS (hub/spoke unicast topic) uses tcp/ip to send it 100 times, sequentially [3]. Each subscriber has a unique IP address and accepts this 1 message, and ignores the other 99. In contrast, Multicast sends [4] it once, and only duplicates/clones a message when the links to the destinations split [5]. RV uses a proprietary protocol over UDP.

[5] see my blogpost https://wordpress.com/post/bintanvictor.wordpress.com/32651

[3] http://www.cs.cmu.edu/~priya/WFoMT2002/Pang-Maheshwari.pdf benchmark tests reveals “The SonicMQ broker requires a separate TCP connection to each of the subscriber. Hence the more subscribers there are, the longer it takes for SonicMQ to deliver the same amount of messages.”
[4] Same benchmark shows the sender is the RVD daemon. It simply broadcasts to the entire network. Whoever interested in the subject will get it.

— P233 [[weblogic definitive guide]] suggests
* multicast jms broker has a constant workload regardless how many subscribers. This is because the broker sends it once rather than 100 times as in our example.
* multicast saves network bandwidth. Unicast topic requires 100 envelopes processed by the switches and routers.

tibrv-CM example usage scenarios #nice

The tibrv documentation [[RV concepts]] (https://docs.tibco.com/pub/rendezvous/8.3.1_january_2011/pdf/tib_rv_concepts.pdf) has good examples of when to use Certified Messaging. (minor edits by me)

Certified delivery is appropriate when a sending program requires individual confirmation of delivery for each message it sends. For example, a traveling sales representative enters sales orders on a laptop computer, and sends them to a central office. The representative must know for certain that the order processing system has received the order he sent.

Certified delivery is also appropriate when a receiving program cannot afford to miss any messages. For example, in an application that processes orders to buy and sell inventory items, each order is important. If any orders are omitted, then inventory records are incorrect. It’s like missing a incremental backup.

Certified delivery is appropriate when each message on a subject builds upon previous message (with that subject) — in sequence. For example, a sending program updates a receiving database, contributing part of the data fields in a record, but leaving other fields of the record unchanged. The database is correct only if all updates arrive in the order they are sent.

Certified delivery is appropriate in situations of intermittent physical connectivity—such as discontinuous network connections. For example, consider an application in which several mobile laptop computers must communicate with one another. Connectivity between mobile units is sporadic, requiring persistent storage of messages until the appropriate connections are re-established.

tibRV network protocols

Everything is “over UDP”. See P 19 of [[ RV concepts ]]

–Multicast
TRDP broadcast or multicast protocol over UDP channel

–Point-to-Point
TRDP unicast protocol over UDP channel

–Direct Communication (RV Version 7)
RPTP over UDP channel

keywords of TibRV

* addressing — rv protocol is different from tcp/ip socket. No IP addresses. Rather, rv uses
** Subject-based addressing — For programs to communicate, they must agree upon a subject name at which to rendezvous (hence the name of this product). Subject-based addressing technology enables anonymous rendezvous, an important breakthrough

* rvd — is the name of the daemon. in most environments, one daemon process runs on each host computer.

* transport — one of the most important java classes to developers like us. There’s no jms counterpart, but the best you can get is the Session object + physical delivery options.

* unicast vs multicast. No other choice.
** multicast is prevalent
** unicast — sends a msg to exactly one Unix process id. point to point. inbox name

2 questions on tibRV to a Tibco friend #LS

Q1: Why is certified messaging so widely used in trading systems when CM is not as reliable as JMS? I guess if a system already uses Rendezvous, then there’s motivation to avoid mixing it with JMS. If the given system needs higher reliability than the standard non-guaranteed RV delivery, then the easiest solution is CM. Is that how your users think?

A: I don’t think CM is not as reliable as JMS. However you can say CM’s acknowledgment mechanisms are not as flexible as JMS/EMS. The latter supports several acknowledgment modes. An important advantage of RV is it supports multi-cast which makes broadcasting quotes more efficient. JMS 5 supports something similar for topic subscriber but not as natural as RV.

Q2: in JMS, a receiver can 1) poll the queue or 2) wait passively for the queue to call back, using onMessage() — known as a JMS listener. These are the synchronous receiver and asynchronous receiver, respectively. I feel Rendezvous supports asynchronous only, in the form of onMsg(). The poll() or dispatch() methods of the TibrvQueue object don’t return the message, unlike the JMS polling operation.

JMS acknowledgment – cheatsheet

If you can only remember one thing about JMS Ack, it’s

#1) 2 legs – there’s ACK on both legs of producer => broker => consumer. The 2 ACK can be present or absent independent of each other. See diagrams in [[JMS]]

#2) reliability – one of the most important features of JMS is reliability. Ack is a cornerstone.

tibrv is slightly less reliable. but CM-transport also uses confirmation on each individual message… see other blogpost.

tibRV transport^connection^session

In JMS, session represents (roughly) a connection, or a socket. More precisely, one or more sessions are created from a single live connection. In JMS, sessions are single-threaded.

In RV, transport is the terminology. Means the same as a session, a socket, a connection to the “network”, to the daemon… Transport is a real object in memory.

In hibernate, session represents a connection to the DB. More precisely, a hibernate session HAS-A real connection object in memory

multicast tibrv ^ JMS #1st take

multicast is not supported in JMS standard. Note JMS (but no multicast?) unicast supports durable subscription.

RV is a good example of multicast. Weblogic JMS also supports multicast.

Multicast (and therefore RV) is more efficient in terms of resources and redundant data transfer.

Q: Unlikely TCP, multicast messages are usually not guaranteed to be once-and-only-once?
A: TIBCO offers a transactional augmentation of RV (RVTX) that guarantees a 1-and-only-once delivery; however, this solution essentially converts RV into a hub-and-spoke system. Consequently, very few RV implementations are transactional.

Q: the JMS synchronous receiver (poller) won’t work. You must use async like onMessage()?
A: i think every receiver must stand ready. A receiver can’t decide when to pull from server.

Q: out-of-sequence is possible in multicast and JMS?
A: both possible

G5 key tibRV objects for Java developers

* msg objects — well-defined entity beans
* transport objects

The 3 objects below build on top of msg and transports … These are described in one single chapter [[Events and Queues]] in the java documentation.

* listener objects
* callback objects, implementing onMsg()
* queue objects and dispatchable interface

Less important:
* dispatcher object (subclass of Thread)

—
Once you understand the above, you can see that certified messaging builds on top of them:
* CM transport objects
* CM listener objects
* CM review callback — for reviewing “missed calls”
* CM msg objects

java calling win32 tibrvsend.exe

            /**

            * –daemon and the arg must be distinct strings

            *

             * quotes aren’t necessary

            */

            String[] strArrayWithoutQuote = new String[] {

                        “C:\\tibrv\\8.3\\bin\\tibrvsend”, “-daemon”,

                        “localhost:7500”, “-service”, “9013”, “-network”,

                        “;239.193.224.50,239.193.224.51;239.193.224.50”,

                        “GENERIC.g.TO_MTS_TRADE_REQUEST”, xml };

            System.out.println(Arrays.asList(strArrayWithoutQuote));

            execAndWait(strArrayWithoutQuote);

      }

      /**

      * http://www.rgagnon.com/javadetails/java-0014.html says — If you need to

      * pass arguments, it’s safer to a String array especially if they contain

      * spaces.

      */

      static private void execAndWait(String[] command)

            try {

                  Runtime runtime = Runtime.getRuntime();

                  Process p = runtime.exec(command);

                  BufferedReader stdInput = new BufferedReader(new InputStreamReader(p

                              .getInputStream()));

                  String s = null;

                  while ((s = stdInput.readLine()) != null) {

                        System.out.println(s);

                  }

                  BufferedReader stdError = new BufferedReader(new InputStreamReader(p

                              .getErrorStream()));

                  while ((s = stdError.readLine()) != null) {

                        System.err.println(s);

                  }

                  p.waitFor(); // advised to do this after streams

            } catch (IOException e) {

                  throw new RuntimeException(e);

            } catch (InterruptedException e) {

                  throw new RuntimeException(e);

            }

      }

solace^tibcoApplicance #OPRA volume

http://solacesystems.com/news/fastest-jms-broker/ solace JMS broker (Solace Message Router) support 100,000 messages per second in persistent mode and 10 million messages non-persistent. In a more detailed article, http://solacesystems.com/solutions/messaging-middleware/jms/ shows 11 million 100-byte non-persistent messages.

A major sell-side’s messaging platform chief said his most important consideration was the deviation of peak-to-average latency and outliers. A small amount of deviation and (good) predictability were key. They chose Solace.

http://www.hpcwire.com/hpcwire/2009-09-14/solace_systems_sets_the_pace_in_the_race_to_zero_latency.html has good details.

In all cases (Solace, Tibco, Tervela), hardware-based appliances *promise* at least 10 fold boost in performance compared to software solutions. Latency within the appliance is predictably low, but the end-to-end latency is not. Because of the separate /devices/ and the network hops between them, the best-case latency is in the tens of microseconds. The next logical step is to integrate the components into a single system to avoid all the network latency and intermediate memory copies (including serializations). Solace has demonstrated sub-microsecond latencies by adding support for inter-process communications (IPC) via shared memory. Developers will be able to fold the ticker feed function, the messaging platform, and the algorithmic engine into the same “application” [1], and use shared memory IPC as the data transport (though I feel single-application design need no IPC).

For best results you want to keep each “application” [1] on the same multi-core processor, and nail individual application components (like the feed handler and algo engine) to specific cores. That way, application data can be shared between the cores in the Level 2 cache.

[1] Each “application” is potentially a multi-process application with multiple address spaces, and may need IPC.

Benchmark — Solace ran tests with a million 100-byte messages per second, achieving an average latency of less than 700 nanoseconds using a single Intel processor. As of 2009, OPRA topped out at about a million messages per second. OPRA hit 869,109 mps (msg/sec) in Apr 2009.

Solace vs RV appliance — Although Solace already offers its own appliance, it runs other messaging software. The Tibco version runs Rendezvous (implemented in ASIC+FPGA), providing a clear differentiator between the Tibco and Solace appliances.

Solace 3260 Message Router is the product chosen by most Wall St. customers.

http://kirkwylie.blogspot.com/2008/11/meeting-with-solace-systems-hardware.html provides good tech insights.

simple tibrvlisten/tibrvsend example using a winxp local rvd

Start 3 “black” DOS windows A B C.
A)
tibrvlisten -daemon localhost:7500 -service 9012 -network “” your.subject “your msg”
B)
tibrvsend -daemon localhost:7500 -service 9012 -network “” your.subject “your msg”

## listener and sender both complain about missing daemon.

C) Now type nothing but “rvd” to start the daemon process. It starts in 0.1 second, reporting the ip endpoint bound + admin website.

Now restart the listener and it starts listening.

Now restart sender – no more errors. Listener window shows the message.

Also, http://localhost:7580/ shows one client only — our listener

JMS guaranteed delivery — guaranteed by who@@

guaranteed delivery (GD) is a “service” provided by the JMS infrastructure — the part of the system outside your custom application code. It's also known as the vendor API, which is software provided by IBM, Tibco, BEA etc. A formal name is “JMS provider”. It helps to think of this infrastructure as encrypted source code (ESC).

To provide GD (or any JMS service), this infrastructure must include client-side ESC — formally known as client-runtime.

ESC is simpler in the JDBC world. Sybase provides both a server and a client-side ESC. Your code calls the client-side ESC as a lower-layer API. eSpeed also ships a client side ESC. You link into your custom trading engine, and the client side ESC talks to the eSpeed server.

In tibrv, the daemon runs in each client host and is comparable to a client-side ESC but
$ it runs in a separate process
$ tibrv daemons are peer-to-peer, no client no server.

onMessage() is one of the most important part of the client-side ESC.  But back to GD, ESC supports

* Stored and Forward
* Ack
* auto_ack is sent by the client ESC
* retry at each leg (sender->broker->receiver)

ambiguous terms to avoid in MOM discussions

synchronous – really means “blocking”

asynchronous – really means onMessage()

client – both a sender and receiver are clients of the broker, if you ask me.

server – “broker” is more specific. In a real world request/response system, A can ask B and B can ask A.

(However, in tibrv, there’s no broker. All rvd instances are peers. Decentralized.)

acknowledgment – which leg on the 2 legged flow — sender->broker->receiver?