This is a probably the biggest sell-side equity order-management-system (OMS) out there, written in c++11. Daily order volume is probably highest among all investment banks, presumably 6 to 7 figures based on my speculation, though a lot of them get canceled, rejected or unfilled. I can’t reveal too many internal details due to compliance.
In contrast, GS used to get about a million individual trades a day, probably not counting the high-frequency small trades.
- I have not seen a message queue so far but they could be hidden somewhere. Earlier I heard people telling me Tibco (and similar messaging middlewares) were popular in fixed income and other trading but now I doubt it. Queues add latency.
- We do use some pub-sub MOM but not for order messages therefore not part of order flow.
- I haven’t noticed any locking or condition variable so far. I think single-threaded mode is faster than synchronized multi-threading. Multiple instances of the same software runs in parallel across machines. I think this is in many ways better than one big monolithic process hosting many threads. We have 4 threads per instance in some cases.
- socket programming is not needed in any module. I believe the applications communicate via FIX, SOAP etc, on top of well-encapsulated TCP library modules.
- RDBMS is loaded into cache at Start-of-Day and seldom accessed intra-day. I confirmed it with an ex-DBA colleague
- no garbage collection like in java and dotnet
- heavy use of CRTP. I don’t remember seeing many virtual functions.
- The most important message is the order object, represented by a FIX message. The order object gets enriched and modified by multiple functions in a chain. Then it is sent out via FIX session to the next machine. As in any OMS, the order object is stateful. I still don’t know where the order objects are saved. I would think they are persisted somewhere so a crash won’t wipe out pending orders.
- (Elsewhere, I have seen very lean and mean buy-side OMS systems that don’t persist any order! After crash, it would query the exchange for order states.)
- The 2nd most important message is probably the response object, represented by a FIX msg. If there are 100,000 order objects then there are roughly 300,000 response objects. Each order generates multiple responses such as Rejection, PendingNew, New, PartialFill, PendingCancel, Cancelled… Response objects probably don’t need to be persisted in my view.
- The 3rd most common message is the report message object, again in FIX format. Each order object probably generate at least one report, even if rejected. Report objects sound simple but they carry essential responsibilities , not only regulatory reporting and client confirmations, but also trade booking, trade capture… If we miss an execution report the essential books and records (inventory, positions..) would be messed up. However, these reports are not so latency sensitive.