Context — real time high volume market data feed from exchanges.
- Q: what’s depth of market? Who may need it?
- Q: what’s the different usages of FIX vs binary protocols?
- Q: when is FIX not suitable? (market data dissemination …)
- Q: when is binary protocol not suitable? (order submission …)
- Q: what’s BBO i.e. best bid offer? Why is it important?
- Q: How do you update BBO and when do you send out the update?
- Q[D]: how do you support trade-bust that comes with a trade id? How about order cancellation — is it handled differently?
- Q: how do you handle an order modification?
- Q: how do you handle an order replacement?
- Q: How do you handle currency code? (My parser sends the currency code for each symbol only once, not on every trade)
- Q: do you have the refresh channel? How can it be useful?
- Q[D]: do you have a snapshot channel? How can it be useful?
- Q: when do you shutdown your feed handler? (We don’t, since our clients expect to receive refresh/snapshots from us round the clock. We only restart once a while)
- Q: if you keep running, then how is your daily open/close/high/low updated?
- Q: how do you handle partial fill of a big order?
- Q: what’s an iceberg order? How do you handle it?
- Q: what’s a hidden order? How do you handle it?
- Q: What’s Imbalance data and why do clients need it?
- Hint: they impact daily closing prices which can trigger many derivative contracts. Closing prices are closely watched somewhat like LIBOR.
- Q: when would we get imbalance data?
- Q: are there imbalance data for all stocks? (No)
- ——— symbol/reference data
- Q[D]: Do you send security description to downstream in every message? It can be quite long and take up lots of bandwidth?
- Q: what time of the day do symbol data come in?
- Q5[D]: what essential attributes are there in a typical symbol message?
- Q5b: Can you name one of the key reasons why symbol message is critical to a market data feed?
- Q6: how would dividend impact the numbers in your feed? Will that mess up the data you send out to downstream?
- Q6b[D]: if yes how do you handle it?
- Q6c: what kind of exchanges don’t have dividend data?
- Q: what is a stock split? How do you handle it?
- Q: how are corporate actions handled in your feed?
- Q: what are the most common corporate actions in your market data? (dividends, stock splits, rights issue…)
- ——— resilience, recovery, reliability, capacity
- Q: if a security is lightly traded, how do you know if you have missed some updates or there’s simply no update on this security?
- Q25: When would your exchange send a reset?
- Q25b: How do you handle resets?
- Q26[D]: How do you start your feed handler mid-day?
- Q26b: What are the concerns?
- Q[D]: how do you handle potential data loss in multicast?
- Q[D]: how do you ensure your feed handler can cope with a burst in messages in TCP vs multicast?
- Q[D]: what if your feed handler is offline for a while and missed some messages?
- Q21: Do you see gaps in sequence numbers? Are they normal?
- Q21b: How is your feed handler designed to handle them?
- Q[D]: does your exchange have a primary and secondary line? How do you combine both?
- Q[D]: between primary and secondary channel, if one channel has lost data but the other channel did receive the data, how does your system combine them?
- Q[D]: how do you use the disaster recovery channel?
- Q[D]: If exchange has too much data to send in one channel, how many more channels do they usually use, and how do they distribute the data among multiple channels?
- Q: is there a request channel where you send requests to exchange? What kind of data do you send in a request?
- Q: Is there any consequence if you send too many requests?
Q: what if some of your requests appear to be lost? How does your feed handler know and react?
- Dalian Commodity Exchange
- SGX Securities Market Direct Feed
- CanDeal (http://www.candeal.com) mostly Canadian dollar debt securities
I have evidence (not my imagination) to believe that these exchange data feeds don’t use vanilla TCP or multicast but some proprietary API (based on them presumably).
I was told other China feeds (probably Shenzhen feed) is also API-based.
ESpeed would ship a client-side library. Downstream would statically link or dynamically link it into parser. The parser then communicates to the server in some proprietary protocol.
IDC is not an exchange but we go one level deeper, requiring our downstream to provide rack space for our own blackbox “clientSiteProcessor” machine. I think this machine may or may not be part of a typical client/server set-up, but it communicates with our central server in a proprietary protocol.
There was a requirement that within 90 seconds, any execution on any online or traditional broker system need to be reported to the “official” exchange. For each listed security, there’s probably a single “official” listing exchange.
Take IBM for example.
- On NYSE, executions only take place after 9.30am, usually after an opening auction.
- On-line electronic brokers operate 24/7 so an execution could happen and get reported any time. However, NYSE data feed only publishes it after 4am by right. I don’t know how strict this timing is. If your feed shows it before 4am I guess you are free to discard it. Who knows it might be a test message.
I could imagine that u.s. exchanges are more advanced in terms of trading rules, operational complexity, matching engine, order types … because presumably u.s. exchanges have more volume and more variations in securities. It’s like a bigger, older hospital is more sophisticated since it has treated many more patients.
On the other hand, I read more than once (over the last 7 years) that in terms of pure latency the bigger exchanges in the U.S. are often slower, but only moderately, so everything still works fine. I don’t know any capacity concerns on the horizon. Some of the smaller exchanges in Asia were aggressive to beat the bigger exchanges on latency. World #1 fastest exchange is now Bombay.
https://www.opradata.com/specs/48_Line_Notification_Common_IP_Multicast_Specification.pdf shows 48 multicast groups each for a subset of the option symbols. When there were 24 groups, the symbols starting with RU to SMZZZ were too heavy too voluminous for one multicast group, more so than the other 23 groups.
Our OPRA parser instance is simple and efficient (probably stateless) so presumably capable of handling multiple OPRA multicast groups per instance. We still use one parser per MC group for simplicity and ease of management.
From: Chen, Tao
Sent: Tuesday, May 30, 2017 8:24 AM
To: Tan, Victor
Subject: RE: OPRA feed volume
Opra data is provided by SIAC(securities industry automation corporation). The data is disseminated on 53 multicast channels. TP runs 53 instances of parser and 48 instances of rebus across 7 servers to handle it.
There are various order types usable before the 9.30 opening auction (and also before a halted security comes back). We might receive these orders in the Level 2 feed. I guess the traders use those orders to test the market before trading starts. Most of these orders can be cancelled, since there’s no execution.
The Imbalance data is a key feature by the exchange auction engine, and possibly very valuable to traders. It has
- indicative match price, which at 9.30 becomes final match price
- imbalance at that match price
The auction engine looks at all the orders placed and works out an optimal match price to maximize execution volume. Traders would monitor that value published by exchange, and adjust their orders. This adjustment would be implemented in an execution algorithm.
In one major market data provider, there are 20 to 30 million “instruments”, mostly derivatives. Globally there are more than 300,000 stocks on various trading venues, as of 2017.
This site aggregates 300 to 400 “sources”. Each source could be a data feed or a distinct trading venue. For example, Nyse has at least 3 independent trading venues — nyse classic, Arca and Amex.
Required nlg — NYSE operates four exchanges
- Arca: 4 multicast streams
- American: 8 multicast streams
- NYSE classic: 4 multicast streams
- NYSE National? Acquired in 2017, to be relaunched in 2018
A Market (a particular exchange or an instrument) shows its true liquidity upon a Client-market-order. The other types of order are typically market-maker orders, which are (always?) limit orders.
#1 obvious sign of every illiquid product — wide bid/ask.
#2 obvious sign — tiny sizes of quotes (i.e. limit orders)
Now the defining features of a Liquid instrument. Look at EUR/USD…
1) Immediacy — client order is immediately filled. It’s necessary but not sufficient to have a tight bid/ask spread.
2) Resiliency — low-impact. Minimal impact on equilibrium price even in the face of a large trade. Depth of market. If there are relatively large bid/ask volumes close to the best bid/ask, then impact will be low.
2a) strong demand — large bids around mid-price
2b) firm and adequate supply — large offers around mid-price
Even the largest instrument in the world — EUR/USD can show a (temporary) drop in mid-price after a 100mio trade.
Most common Alt Trading Service is the dark pool, often operated by a sell-side bank (GS, Normura etc).
A “transparent” exchange (my own lingo) provides the important task of _price_discovery_. A dark pool doesn’t. It receives the price from the exchanges and executes trades at the mid-quote.
Market order can’t specify a price. You can think of a market buy order as a marketable limit order with price = infinity. Therefore, when a market order hits a limit order, they execute at the limit price. When 2 limit orders cross, they execute at the “earlier” limit price.
Therefore, on the exchange, I believe all trades execute either on the best bid price or best ask. I guess all the mid-quote executions happen on the ATS’s.
Dark pool is required to report trades to the regulator, but often with a few sec longer delay than an exchange.
Dark pool may define special order types beside the standard types like limit orders or market orders.
Forex is quote driven, not order driven. Forex has no exchange. The dominant market is the interbank market. Only limit orders  are used. However, within a private market operated by a single dealer, a “market order” type can be defined. I feel the rules are defined by the operator, rather than some exchange regulator.
 A Forex limit order is kind of fake – unlike the exchange’s guarantee, when you hit a fake limit order that dealer may withdraw it! I believe this is frowned upon by the market operator (often a club of FX banks), so dealers are pressured to avoid this practice. But I guess a dealer may need this “protection” in a fast market.