Q1: pros and cons of vector vs linked list?
Q1b: Given a 100-element collection, compare performance of … (iteration? Lookup?)
Q: UDP vs TCP diff?
%%A: multicast needs UDP.
Q: How would you add reliability to multicast?
Q: How would you use tibco for trade messages vs pricing messages?
Q5: In your systems, how serious was data loss in non-CM multicast?
%%A: Usually not a big problem. During peak volatile periods, messaging rates could surge 500%. Data loss would deteriorate.
Q5b: how would you address the high data loss?
%%A: test with a target message rate. Beyond the target rate, we don’t feel confident.
Q7: how is order state managed in your OMS engine?
%%A: if an order is half-processed and pending the 3nd reply from ECN, the single thread would block.
Q7b: even if multiple orders (for the same security) are waiting in the queue?
%%A: yes. To allow multiple orders to enter the “stream” would be dangerous.
Now I think the single thread should pick up and process all new orders and keep all pending orders in cache. Any incoming exchange messages would join the same task queue (or a separate task queue) – the same single thread.
3 main infrastructure teams
* exchange connectivity – order submission
* exchange connectivity – pricing feed. I think this is incoming-only, probably higher volume. Probably similar to Zhen Hai’s role.
* risk infrastructure – no VaR mathematics.
http://solacesystems.com/news/fastest-jms-broker/ solace JMS broker (Solace Message Router) support 100,000 messages per second in persistent mode and 10 million messages non-persistent. In a more detailed article, http://solacesystems.com/solutions/messaging-middleware/jms/ shows 11 million 100-byte non-persistent messages.
A major sell-side’s messaging platform chief said his most important consideration was the deviation of peak-to-average latency and outliers. A small amount of deviation and (good) predictability were key. They chose Solace.
In all cases (Solace, Tibco, Tervela), hardware-based appliances *promise* at least 10 fold boost in performance compared to software solutions. Latency within the appliance is predictably low, but the end-to-end latency is not. Because of the separate /devices/ and the network hops between them, the best-case latency is in the tens of microseconds. The next logical step is to integrate the components into a single system to avoid all the network latency and intermediate memory copies (including serializations). Solace has demonstrated sub-microsecond latencies by adding support for inter-process communications (IPC) via shared memory. Developers will be able to fold the ticker feed function, the messaging platform, and the algorithmic engine into the same “application” , and use shared memory IPC as the data transport (though I feel single-application design need no IPC).
For best results you want to keep each “application”  on the same multi-core processor, and nail individual application components (like the feed handler and algo engine) to specific cores. That way, application data can be shared between the cores in the Level 2 cache.
 Each “application” is potentially a multi-process application with multiple address spaces, and may need IPC.
Benchmark — Solace ran tests with a million 100-byte messages per second, achieving an average latency of less than 700 nanoseconds using a single Intel processor. As of 2009, OPRA topped out at about a million messages per second. OPRA hit 869,109 mps (msg/sec) in Apr 2009.
Solace vs RV appliance — Although Solace already offers its own appliance, it runs other messaging software. The Tibco version runs Rendezvous (implemented in ASIC+FPGA), providing a clear differentiator between the Tibco and Solace appliances.
Solace 3260 Message Router is the product chosen by most Wall St. customers.
http://kirkwylie.blogspot.com/2008/11/meeting-with-solace-systems-hardware.html provides good tech insights.
- Culprit: deadlock
- Culprit: all threads in the pool are blocked in wait(), lock() or …
- Culprit: bounded queue is full. Sometimes the thread that adds task to the queue is blocked while doing that.
- Culprit: in some systems, there’s a single task dispatcher thread like swing EDT. That thread can sometimes get stuck
- Suggestion: dynamically turn on verbose logging in the messaging module within the engine, so it always logs something to indicate activity. It’s like the flashing LED in your router. You can turn on such logging by JMX.
- Suggestion: snoop
- Suggestion: for tibrv, you can easily start a MS-windows tibrv listener on the same subject as the listener inside the trading engine. This can reveal activity on the subject
jms message selector is executed on the broker.
rvd executes the same duty — “Filter subject-addressed messages.”
Among the defining features of Tibrv, [ A) decentralization and B) SBA ] are 2 sides of the same coin. For me, it's a struggle to find out the real difference between SBA vs jms topics. Here's my attempt.
#1) In SBA(subject-based-addressing), there's no central server holding a subject. In tibrv, any sender/receiver can specify any subject. Nice — No admin to create subjects. I tested it with tibrvlisten/tibrvsend. By contrast, In JMS, “The physical creation of topics is an administrative task” on the central broker , not on the producer or consumer.
) If a jms broker goes down, so do topics therein.
) a jms publisher (or subscriber) must physically connect to a physical broker process. The broker has a physical network address. Our topic is tied to that physical address. A tibrv subject has no physical address.
) tibrv SBA (subject-based-addressing) uses a subject tree. No such tree among JMS topics. The tree lets a subscriber receive q(prices.stocks.*) but also q(*.your.*). See rv_concepts.
 such as the weblogic server in autoreo.
Imagine a typical request/reply messaging system. I think in JMS it’s usually based on temp queues, reply-to and correlation-Id — See other blog post. In contrast, RV has no broker. It’s decentralized into multiple peer rv daemons. No difference in this case —
Suppose a message broker holds a lot of queues. One of the queues is for a request message, from requester system to a pricing system. Another queue is for pricing system to return the new price to the requester.
Now, pricing system is slow. Requester should wait for no more than 5 minutes. If the new price comes back through the reply-queue 301 sec later, requester will ignore this stale price since it’s too risky to place an order on a stale price in a fast market. How do you implement this?
My design — Requester main thread can wait(5*60*000). Another thread in requester JVM can block forever in onMsg(), and notify main thread when something received.
(I actually implemented this in a few trading engines.)
Message overflow is a frequent headache in MOM. Basically consumption rate (possibly 0) is slower than production rate, eating up storage capacity, either in memory or on disk.
eg: Topic with NDS (non-durable-subscriber) only — almost free of overflow. Bonnie said broker will discard. Jerry said broker keeps track of alive subscribers. Every incoming message is guaranteed to be delivered (by retry) to each. Broker must persist pending messages. When a subscribe disconnects gracefully, broker no longer needs to keep any pending message for her.
eg: queue. While any number of producers can send messages to the queue, each message is guaranteed to be delivered, and consumed by one consumer. If no consumers are registered to consume the messages, the queue holds (on disk) them until a consumer registers to consume them.
eg: temp queue??
eg: chatrooms don’t persist data but what type of MOM is that? probably NDS topic
eg: tibrv non-CM? market data feed? Discarded but how soon? 60 sec by default?
Yes, in slow consume and fast producer scenario message lost will be possible — http://arun-tibco.blogspot.com/2010/09/tibco-rv-faqs.html
eg: tibrv CM? Storage is the ledger file. –persist–
eg: regular queue? In SIM upfront, unread messages –persist–
eg: mailing list (topic with durable subscribers)? –Persist–
eg: topic without durable subscriber? see post on message retention
guaranteed delivery (GD) is a “service” provided by the JMS infrastructure — the part of the system outside your custom application code. It's also known as the vendor API, which is software provided by IBM, Tibco, BEA etc. A formal name is “JMS provider”. It helps to think of this infrastructure as encrypted source code (ESC).
To provide GD (or any JMS service), this infrastructure must include client-side ESC — formally known as client-runtime.
ESC is simpler in the JDBC world. Sybase provides both a server and a client-side ESC. Your code calls the client-side ESC as a lower-layer API. eSpeed also ships a client side ESC. You link into your custom trading engine, and the client side ESC talks to the eSpeed server.
In tibrv, the daemon runs in each client host and is comparable to a client-side ESC but
$ it runs in a separate process
$ tibrv daemons are peer-to-peer, no client no server.
onMessage() is one of the most important part of the client-side ESC. But back to GD, ESC supports
* Stored and Forward
* auto_ack is sent by the client ESC
* retry at each leg (sender->broker->receiver)
synchronous – really means “blocking”
asynchronous – really means onMessage()
client – both a sender and receiver are clients of the broker, if you ask me.
server – “broker” is more specific. In a real world request/response system, A can ask B and B can ask A.
(However, in tibrv, there’s no broker. All rvd instances are peers. Decentralized.)
acknowledgment – which leg on the 2 legged flow — sender->broker->receiver?