too many DB-writes: sharding insufficient #Indeed

Context: each company receives many many reviews. In a similar scenario, we can say a stock receives many investor comments

Interviewer: OK you said horizontal sharding by company id can address highly concurrent data store updates. Now what if one company, say, Amazon, by itself gets 20% of the updates so sharding can’t help this one company.

me: I guess the update requests would block and possibly time out. We can use a task queue.

This is similar to WordPress import/export requests.

  • Each task takes a few seconds, so we don’t want user to wait.
  • If high server load, the wait could be longer.
  • More important — this task need not be immediate. It can be scheduled.
  • We should probably optimize for server efficiency rather than latency or throughput

So similarly, all the review submissions on Amazon can be queued and scheduled.

In FIX protocol, an order sender can receive 150=A meaning PendingNew. I call it “queued for exchange”.  The OMS Server sends the acknowledgement in two steps.

  1. Optional. Execution Report message on order placement in OMS Server’s Order Book. ExecType (150) = A (Pending New)
  1. Execution Report message on receiving acknowledgement from exchange. ExecType (150) = 0 (New). I guess this indicates placement in exchange order book

Note this optimization is completely different from HFT latency engineering.

Advertisements

FIX^tibrv: order flow

Background — In a typical sell-side an buy/sell order goes through multiple nodes of a flow chain before it goes out to the trading venue. The order message is payload in a messaging platform.

FIX and Tibrv (and derivatives) are dominant and default choices as messaging platforms. Both are highly adaptable to complex combinations of requirements.

FIX tibRV
TCP UDP transport
unicast multicast pub/sub fan-out
designed for low-latency eq trading popular for bond trading latency
order msg gets modified along the flow chain ? editable payload

overcoming exchange-FIX-session throughput limit

Some exchanges (CME?) limits each client to 30 orders per second. If we have a burst of order to send , I can see two common solutions A) upstream queuing B) multiple sessions

  1. upstream queuing is a must in many contexts. I think this is similar to TCP flow control.
    • queuing in MOM? Possible but not the only choice
  2. an exchange can allow 100+ FIX sessions for one big client like a big ibank.
    • Note a big exchange operator like nsdq can have dozens of individual exchanges.

Q: is there any (sender self-discipline) flow control in intranet FIX?
A: not needed.

ExecType[150] to complement 35=8

Tag 150 shows the execution report type, so it only (and always, according to Rahul) accompanies 35=8 not other 35=* messages.

With 35=8 alone, we won’t know what type report this is.

The first execution report we get on a given order should show 150=0 i.e. new, not a fill. I believe only the real exchange (not the intermediate routing nodes) would send this message.

I think sometimes we get 35=8;150=A i.e. pending-new, presumably before 150=0. Why? I don’t think we need to bother now.

FIX^MOM: exchange connectivity

The upshot — some subsystem use FIX not MOM; some subsystem use MOM not FIX. A site could send FIX over MOM (such as RV) but not common.

It’s important to realize FIX is not a type of MOM system. Even though FIX uses messages, I believe they are typically sent over persistent TCP sockets, without middle-ware or buffering/queuing. RTS is actually a good example.

Q: does FIX have a huge message buffer? I think it’s small, like TCP, but TCP has congestion control, so sender will wait for receiver.

Say we (sell-side or buy-side) have a direct connectivity to an exchange. Between exchange and our exterior gateway, it’s all sockets and possibly java NIO — no MOM. Data format could be FIX, or it could be the exchange’s proprietary format — here’s why.

The exchange often has an internal client-server protocol — the real protocol and a data flow “choke point” . All client-server request/response relies solely on this protocol. The exchange often builds a FIX translation over this protocol (for obvious reasons….) If we use their FIX protocol, our messages are internally translated from FIX to the exchange proprietary format. Therefore it’s obviously faster to directly pass messages in the exchange proprietary format. The exchange offers this access to selected hedge funds.

As an espeed developer told me, they ship a C-level dll (less often *.so [1]) library (not c++, not static library), to be used by the hedge fund’s exterior gateway. The hedge fund application uses the C functions therein to communicate with the exchange. This communication protocol is possibly proprietary. HotspotFX has a similar client-side library in the form of a jar. There’s no MOM here. The hedge fund gateway engine makes synchronous function calls into the C library.

[1] most trader workstations are on win32.

Note the dll or jar is not running in its own process, but rather loaded into the client process just like any dll or jar.

Between this gateway machine and the trading desk interior hosts, the typical communication is dominated by MOM such as RV, 29West or Solace. In some places, the gateway translates/normalizes messages into an in-house standard format.

y FIX needs seqNo over TCP seqNo

My friend Alan Shi said … Suppose your FIX process crashed or lost power, reloaded (from disk) the last sequence received and reconnected (resetting tcp seq#). It would then receive a live seq # higher than expected. This could mean some executions reports were missed. If exchange notices a sequence gap, then it could mean some order cancellation was missed.  Both scenarios are serious and requires request for resend. CME documentation states:

… a given system, upon detecting a higher than expected message sequence number from its counterparty, requests a range of ordered messages resent from the counterparty.

Major difference from TCP sequence number — FIX specifies no Ack though some exchange do. See Ack in FIX^TCP

Q; how could FIX miss messages given TCP reliability? I guess tcp session disconnect is the main reason.

https://kb.b2bits.com/display/B2BITS/Sequence+number+handling has details:

  • two streams of seq numbers, each controlled by exch vs trader
  • how to react to unexpected high/low values received. Note “my” outgoing seq is controlled by me hence never “unexpected”
  • Sequence number reset policy — After a logout, sequence numbers is supposed to reset to 1, but if connection is terminated ‘non-gracefully’ sequence numbers will continue when the session is restored. In fact a lot of service providers (eg: Trax) never reset sequence numbers during the day. There are also some, who reset sequence numbers once per week, regardless of logout.

 

binary data ] FIX

It’s possible to “embed” arbitrary binary data in a FIX tag-value pair. However, the parser has to know where it ends. Therefore, the “value” consist of a length and a binary payload.

Q: Any example?

Q: can we send a image?

Q: can we send a document?

Q: Encryption? Need to read more

retrans: FIX^TCP^xtap

The FIX part is very relevant to real world OMS.. Devil is in the details.

IP layer offers no retrans. UDP doesn’t support retrans.

 

 

TCP FIX xtap
seq# continuous no yes.. see seq]FIX yes
..reset automatic loopback managed by application seldom #exchange decision
..dup possible possible normal under bestOfBoth
..per session per connection per clientId per day
..resumption? possible if wire gets reconnected quickly yes upon re-login unconditional. no choice
Ack positive Ack needed only needed for order submission etc not needed
gap detection sophisticated every gap should be handled immediately since sequence is critical. Out-of-sequence is unacceptable. gap mgr with timer
retrans sophisticated receiver(ECN) will issue resend request; original sender to react intelligently gap mgr with timer
Note original sender should be careful resending new orders.
See https://drivewealth-fix-api.readme.io/docs/resend-request-message

 

session^msg^tag level concepts

Remember every session-level operation is implemented using messages.

  • Gap management? Session level
  • Seq reset? Session level, but seq reset is a msg
  • Retrans? per-Msg
  • Checksum? per-Msg
  • MsgType tag? there is one per-Msg
  • Header? per-Msg
  • order/trade life-cycle? session level
  • OMS state management? None of them …. more like application level than Session level

(proactive)PossResend^PossDupFlag(reactive)

A common interview question. http://fixmeisters.blogspot.com/2006/04/possdupflag-or-possresend-this.html has details.

msg with PossDupFlag msg with PossResend
handled by FIX engine application to check many details. Engine can’t help.
seq # old seq # new seq #
triggered by retrans request not automatically
generated by FIX engine not automatically
reactive action by engine proactive decision by application

FIX.5 + FIXT.1 breaking changes

I think many systems are not yet using FIX.5 …

  1. FIXT.1 protocol [1] is a kinda subset of the FIX.4 protocol. It specifies (the traditional) FIX session maintenance over naked TCP
  2. FIX.5 protocol is a kinda subset of the FIX.4 protocol. It specifies only the application messages and not the session messages.

See https://www.fixtrading.org/standards/unsupported/fix-5-0/. Therefore,

  • FIX4 over naked TCP = FIX.5 + FIXT
  • FIX4 over non-TCP = FIX.5 over non-TCP

FIX.5 can use a transport messaging queue or web service, instead of FIXT

In FIX.5, Header now has 5 mandatory fields:

  • existing — BeginString(8), BodyLength(9), MsgType(35)
  • new — SenderCompId(49), TargetCompId(56)

Some applications also require MsgSeqNo(34) and SendingTime(52), but these are unrelated to FIX.5

Note BeginString actually look like “8=FIXT1.1

[1] FIX Session layer will utilize a new version moniker of “FIXTx.y”

FIX over non-TCP

based on https://en.wikipedia.org/wiki/Financial_Information_eXchange…

Background — The traditional FIX transport is naked TCP. I think naked TCP is most efficient.

–TCP-based:

Some adaptations of FIX can use messaging queue or web services. I think these are built on top of TCP.

MOM would add latency.

–UDP:

FIX is traditionally used for trading. In contrast, (high volume) market data usually go over multicast for efficency. However, there are now  adaptations (like FAST FIX) to send market data over multicast.

checksum[10] value is ascii

https://www.onixs.biz/fix-dictionary/4.2/tagNum_10.html asserts the value has 3 characters, always unencrypted.

In contrast, some non-FIX protocols use an int16 binary integer, which is more efficient.

However, very few protocols would use 10 bits to represent a 3-digit natural number. The only usage I can imagine is a bit-array like 24 bits used to combine 5 different fields, so bit 0 to 9 could be clientId…

SOH can be embedded in binary field value

Usually, a FIX tag/value “field” is terminated by the one-byte SOH character, but if the value is binary, then SOH can be embedded in it at the beginning or the END (resulting in two SOH in a row?)…

https://stackoverflow.com/questions/4907848/whats-the-most-efficient-way-to-parse-fix-protocol-messages-in-net confirms SOH can be embedded.

https://www.ibm.com/support/knowledgecenter/en/SSVSD8_8.3.0/com.ibm.websphere.dtx.packfix.doc/concepts/c_pack_fix_message_structur.htm says in a binary field, there must be a length

FIX heartBtInt^TCP keep-alive^XDP heartbeat^xtap inactivity^MSFilter hearbeat

When either end of a FIX connection has not sent any data for [HeartBtInt] seconds, it should transmit a Heartbeat message. This is first layer of defense.

When either end of the connection has not received any data for (HeartBtInt + “some reasonable transmission lag”) seconds, it will send a TestRequest probe message to verify the connection. This is 2nd layer of defense. If there is still no message received after (HeartBtInt + “some reasonable transmission lag”) seconds then the connection should be considered lost.

Heartbeats issued as a response to TestRequest must contain the TestReqID transmitted in the TestRequest. This is useful to verify that the Heartbeat is the result of the TestRequest and not as the result of a HeartBtInt timeout. If a response doesn’t have the expected TestReqId, I think we shouldn’t ignore it. If everyone were to ignore it, then TestReqId would be a worthless feature added to FIX protocol.

FIX clients should reset the heartbeat interval timer after every transmitted message (not just heartbeats).

TCP keep-alive is an optional, classic feature.

NYSE XDP protocol uses heartbeat messages in both multicast and TCP request server. The TCP heartbeat requires client response, similar to FIX probe message.

Xtap also has an “inactivity timeout”. Every incoming exchange message (including heartbeats) is recognized as “activity”. 60 sec of Inactivity triggers an alert in the log, the log watchdog…

MS Filter framework supports a server-initiated heartbeat, to inform clients that “I’m alive”.  An optional feature — heartbeat msg can carry an expiration value a.k.a dynamic frequency value like “next heartbeat will arrive in X seconds”.

%%FIX xp

Experience: special investment upfront-commission. Real time trading app. EMS. Supports
* cancel
* rebook
* redemption
* sellout

Experience: Eq volatile derivatives. Supports C=amend/correction, X=cancel, N=new

FIX over RV for market data dissemination and cache invalidation

Basic FIX client in MS coding interview.

Ack in FIX #TCP

TCP requires Ack for every segment? See cumulative acknowledgement in [[computerNetworking]]

FIX protocol standard doesn’t require Ack for every message.

However, many exchanges spec would require an Ack message for {order submission OR execution report}. At TrexQuant design interview, I dealt with this issue —

  • –trader sends an order and expecting an Ack
  • exchange sends an Ack and expecting another Ack … endless cycle
  • –exchange sends an exec report and expecting an Ack
  • trader sends an Ack waiting for another Ack … endless cycle
  • –what if at any point trader crashes? Exchange would assume the exec report is not acknowledged?

My design minimizes duty of the exchange —

  • –trader sends an order and expecting an Ack
  • exchange sends an Ack and assumes it’s delivered.
  • If trader misses the Ack she would decide either to resend the order (with PossResend flag) or query the exchange
  • –exchange sends an exec report and expecting an Ack. Proactive resend (with PossResend) when appropriate.
  • trader sends an Ack and assumes it’s delivered.
  • If exchanges doesn’t get any Ack it would force a heartbeat with TestRequest. Then exchange assumes trader is offline.

TimeInForce [59] FIX

0 = Day (or session)
1 = Good Till Cancel (GTC)
2 = At the Opening (OPG)
3 = Immediate or Cancel (IOC)
4 = Fill or Kill (FOK)
5 = Good Till Crossing (GTX)
6 = Good Till Date
7 = At the Close

Many of these values are interesting. Most confusing is IOC ^ FOK. The IOC order can (but does not have to) be filled in full. If it cannot be filled immediately or fully, the pending part gets canceled (a partial execution is generated) by the market. It differs from a fill or kill order in that in fill or kill orders, partial executions are not accepted.

FIX support for multi-leg order # option strategy

  • There are multi-leg FIX orders to trade two stocks without options
    • I believe there’s no valid multi-leg order to trade the same stock using market orders. The multiple legs simply collapse to one. How about two-leg limit orders? I would say Please split into two disconnected limit orders.
  • There are multi-leg FIX orders to trade two futures contracts
  • There are multi-leg FIX orders to trade 3 products like a butterfly.
  • There are multi-leg FIX orders with a special ratio like an option “ratio spread”

But today I show a more common multi-leg strategy order — buy 5 buy-write with IBM. At least four option exchanges support multi-leg FIX orders — Nasdaq-ISE / CBOE / ARCA / Nasdq-PHLX

35=AB;
11=A1;
38=5; // quantity = 5 units
54=1;
555=2; // how many legs in this order. This tag precedes a repeatingGroup!
654=Leg1; //leg id for one leg
600=IBM; // same value repeated later!
608=OC; //OC=call option
610=201007; //option expiry
612=85; //strike price
624=2; //side = Sell
654=Leg2;
600=IBM; …

Note that in this case, the entire strategy is defined in FIX, without reference to some listed symbol or productID.

essential FIX messages

Focus on just 2 app messages – new order single + a partial fill. Logon/Logout/heartbeat… LG2 — important session messages.

Start with just the 10 most common fields in each message. Later, you need to know 20 but no more.

For the most important fields – tag 35, exectype, ordstatus, focus on the most important enum values.

This is the only way to get a grip on this slippery fish

25-13:34:27:613440 |250—-> 16 BC_ITGC_14_CMAS_104 1080861988 |

8=FIX.4.2^A9=0162^A35=D^A34=00250^A52=20120725-17:34:27.610^A49=BARCAP014A^A56=ITGC^A115=BARCAP1^A11=1040031^A22=5^A38=100^A40=1^A44=0.01^A47=A^A48=RIM.TO^A54=1^A55=RIM^A59=0^A100=ITGC^A376=-863982^A10=124^A

|

— A NewOrderSingle followed by 2 exec reports —

25-13:34:27:802535 |<—-252 13 BC_ITGC_14_CMAS_104 1079105241 |

8=FIX.4.2^A9=220^A35=8^A128=BARCAP1^A34=252^A49=ITGC^A56=BARCAP014A^A52=20120725-17:34:27^A37=869-1^A17=10^A150=0^A20=0^A39=0^A55=RIM^A54=1^A38=100^A32=0^A31=0.000000^A151=100^A14=0^A6=0.000000^A11=1040031^A40=1^A22=5^A48=RIM.TO^A60=20120725-17:34:27^A59=0^A47=A^A10=002^A

|

25-13:34:27:990564 |<—-253 13 BC_ITGC_14_CMAS_104 1079105241 |

8=FIX.4.2^A9=230^A35=8^A128=BARCAP1^A34=253^A49=ITGC^A56=BARCAP014A^A52=20120725-17:34:27^A37=869-1^A17=20^A150=2^A20=0^A39=2^A55=RIM^A54=1^A38=100^A32=100^A31=7.010000^A151=0^A14=100^A6=7.010000^A11=1040031^A40=1^A22=5^A48=RIM.TO^A60=20120725-17:34:27^A59=0^A47=A^A30=MSCN^A10=076^A

|

client conn IV#FIX/java

Q: OrdStatus vs ExecType in FIX?
http://fixwiki.fixprotocol.org/fixwiki/ExecutionReport
http://fixprotocol.org/FIXimate3.0/en/FIX.5.0SP2/tag39.html — good to memorize all of these

Each order can generate multiple execution reports. Each report has an exectype describing the report. Once an order status changes form A to B, OrdStatus field changes from A to B, but in the interim the exectype in the interim execution reports can take on various values.

Enum values of exectype tag and ordstatus tag are very similar. For example, a partial fill msg will have exectype=F (fill) and ordstatus=1 (partial fillED).

exectype is the main carrier of the cancellation message and the mod message.

In a confusing special case, The exectype value can be the letter “i”, representing OrderStatus, spelt “OrderStatus”.

Q: what’s a test request in FIX
AA: force opposite party to send a heartbeat.

Q: what’s send Message() vs post Message in COM?
http://wiki.answers.com/Q/What_is_the_difference_between_PostMessage_and_SendMessage

Q: intern()?

Q: are string objects created in eden or ….?

design considerations – my first homemade FIX client

In a typical sell-side / market maker, the FIX gateway is a FIX client for a liquidity venue. The liquidity venue (typically an exchange, but also ECNs, dark pools) runs a massively multi-threaded, FIX server farm, possibly with custom-made FPGA hardware. I believe the technical requirements for an exchange FIX server is different from a sell-side FIX client.

Therefore, I assume my audience are far more interested in the client code than the server code.

My mock server implementation is a mock-up, with no optimization, no resilience/robust design, no fault-tolerance, no select()-based sockets, no high-availability, no checksum validation.

— design priorities —
Since this is a from-scratch implementation of a large spec, I focused on basic functionality. Debugging and instrumentation is important in the early stage. A lot of extra code was created to help developers verify the correct operation of all those nitty gritty details.

I don’t aim to build a complete engine, but the basic functionality I build, i try to build it on solid ground, so hopefully it doesn’t become throw-away code.

At the heart of the OO design was the abstraction expressed by AbstractClient, AbstractState and ClientSession. These form the backbone of the core object graph. The fields and constructors were chosen with care. Cohesion and low-coupling were my ideals but sometimes hard to achieve.

These core classes help meet most of the fundamental requirements related to object state of a client connection, sequence number, state transition, output stream sharing, connection liveness monitoring… I wouldn’t say these are well-design classes, but they are a reasonable first attempt at modelling the problem using java classes.

Reusable components were created whenever possible, such as the message parser, message formatter.

FIX engine are highly configurable. All config parameters are extracted into a config object, which is obtained from a config factory. One config for Test, one for Production or UAT etc.

— features —
– parsing a raw FIX msg into an array, which is faster to access than a hashmap. By our design, we avoid hash lookup completely.
– Destroyer.java is a home-made solution to the potential forever-blocking of readLine(). On some platforms, socket read is not interruptible and the only way to unfreeze a socket read is to close the socket. We do that in this implementation but only after sending a final logout message.
– Various template methods to guarantee close() and guard against resource leak
– AbstractClient has most of the generic logic, while SingleOrderClient implements very small amount of custom logic.
– FIX msg formatter is generic enough to accept a map of tags and values — any tags and values
– sequence num validation
– state machines on both client and (simplified) server. State design pattern.
– server side — very few features, but ConnectionManager is reasonable encapsulation of new socket creation.

—- client-side features to be implemented —-
– heart beat — more robust. Should never fail.
– timeout waiting for ack
– re-logon if no heartbeat for 60 seconds
– checksum validation
– storing exchange order-ack into database
– Right now, given time constraint, i didn’t use nonblocking IO. All FIX system should use non-blocking IO.
———- Disclaimer ————-
I spent about 4 hours on this project, as suggested by Jonathan. I had to do some basic research on FIX since I have never programmed FIX (except some ECN connectivity modules that probably used FIX under the hood).
————— How to run the server/client —————–
Run ServerMain.java, then ClientMain.java.
To see the heart beat monitoring/intervention (Destroyer) in action, put 500 in getSilenceThreshold().

G10 fields in NOS ^ exec report

—- top 5 fields common to NOS (NewOrderSingle) and exec report —
#1) 22 securityIDSource. Notable enum values
1 = CUSIP
5 = RIC code
*** 48 SecurityID, Requires tag 22 ==NOS/exec
*** 55 optional human-readable symbol ==NOS/exec

#2) 54 side. Notable enum values
1 = Buy
2 = Sell

#3) 40 ordType. Notable enum values
1 = Market order
2 = Limit order

#4) 59 TimeInForce. Notable enum values
0 = Day (or session)
1 = Good Till Cancel (GTC)
3 = Immediate Or Cancel (IOC)
4 = Fill Or Kill (FOK)

—- other fields common to NOS and exec type
*11 ClOrdID assigned by buy-side
*49 senderCompID
*56 receiving firm ID
*38 qty
*44 price

—- exec report top 5 fields
* 150 execType
* 39 OrdStatus. Notable enum values
0 = New
1 = Partially filled
2 = Filled

* 37 OrderID assigned by sell-side. Not in NOS. Less useful than the ClOrdID

socket, tcp/udp, FIX

I was told both UDP and TCP are used in many ibanks. I feel most java shops generally prefer the simpler TCP.
Specifically, I learnt from many veterans that some ibanks use multicast (UDP) for internal messaging.

FIX is not much used. It is sometimes used as purely message formatting protocol (without any session management) on top of multicast UDP. (In contrast, full-featured FIX communication probably prefer TCP for connection/session control.)

Similar – Object serialization like protobuf.

##essential features@FIX session #vague

Top 5 session-message types– Heartbeat(0), Test Request(1) ResendRequest(2), Logout(5) / Logon(A).[1] This short list highlight the essential functionality of the concept known as “FIX session”. I feel the most important features are sequence number and heart beat.

FIX session concept is different from a database server session. I feel FIX session is more like a TCP connection with keep-alive — http://fixprotocol.org/documents/742/FIX_Session_Layer_rev1.ppt says A FIX Session can be defined as “a bi-directional stream of ordered messages between two parties within a continuous sequence number series”… very much like TCP.

Each FIX Engines is expected to keep track of both sequence numbers — The Incoming Sequence Number expected on inbound messages received from the counterparty

http://javarevisited.blogspot.com/2011/02/fix-protocol-session-or-admin-messages.html is a blog post by a fellow developer in finance IT.

http://aj-sometechnicalitiesoflife.blogspot.com/2010/04/fix-protocol-interview-questions.html is another blogger

[1] In parentheses are MsgType values, which are ALWAYS single-characters.

parse FIX msg to key/value pairS #repeating group

http://bigblog.tanbin.com/2012/03/design-considerations-my-first-homemade.html is the context.

    /**
     * Parsing to array is faster than parsing to a HashMap
     */
    public static ArrayList quickParse(String fromServer) {
        ArrayList ret = new ArrayList(Collections.nCopies(
            MAX_FIX_TAG, (String) null));
        for (String pair : fromServer.split(SOH)) {
            String[] tmp = pair.split("=");
            if (tmp.length != 2)  continue;
            int tag = Integer.parseInt(tmp[0]);
            ret.set(tag, tmp[1]); //treat ArrayList as array -- efficient
        }
        return ret;
    }

Repeating group is not supported in this simplistic scheme, but could be accommodated. I can think one simple and one robust design:

Design 1: array<list<string>>, so a regular non-repeating tag will hit a singular list.

Design 2: we ought to know which tags are repeating. The message specification must tell us which tag (NoRoutingIds in the example) signals the arrival of a repeating group. Therefore, at compile time (not runtime) system knows what tags are special i.e. potentially repeating. We will build a separate data structure for them, such as a list<groupMember>. So each FIX message will have one regular array + one list

state maintenance in FIX gateway

Even the fastest, most dumb FIX gateways need to maintain order state, because exchange can send (via FIX) partial fills and acks, and customer can send (via other MOM) cancels and amends, in full-duplex.

When an order is completed, it should be removed from memory to free up. Before that, the engine needs to maintain its state. State maintenance is a major cost and burden on a low-latency system.

A sell-side FIX engine is a daemon process waiting for Response from liquidity venue, and mkt/limit orders from buy-side clients.

[[Complete guide to capital markets]] has some brief coverage on OMS and state maintenance.

FIX session msg vs session tags

The “content” of FIX data is surrounded by session related data, including session-Messages and session-Tags. As a beginner, I was constantly confused over the difference between session-Messages vs session-Tags

A) There are only a few (I guess 5 to 10) session message types, such as heart beat, logon/logout, request for resend…
** All the types — session msg subtypes or app msg subtypes — show up under Tag 35.

B) In ANY (session/app) Message, there are (too many) session-related Tags such as (34)seqNum, (10)checksum, (8)beginString, (9)bodyLength, (52) timeStamp, encryption. I’d say even the all-important (35)MsgType tag qualifies as session tag.

In fact, application data are “surrounded” by session data. App data are the stars of the show, surrounded by a huge “supporting crew” —
– A session msg’s fields are 100% session-related;
– An app msg is not free of session information. It is partly a session msg — due to those session tags.
– If you snoop the conversation, most messages are session messages — heartbeats
– every business data exchange starts with and ends with session messages — logon and logout.

FIX head^tail

Every Single msg (session or app) has exactly 1 head and 1 tail.

Up to FIX.4.4, the header contained three fields: 8 (BeginString), 9 (BodyLength), and 35 (MsgType) tags.

Every Single msg (session or app) has a tail, containing a single tag (10) ie the checksum.

==> Given a long string, how do you carve cout a full message (session or app)? Look for 8= and 10=

I feel head/tail =~= tcp envelope, and Body =~= payload

sequence num ] FIX

Sequence num can be considered part of the header for many app messages. It’s a control device. I consider it as a session-related tag on a non-session message.

Officially, sequece num is NOT part of the header. Up to FIX.4.4, the header contained three fields: 8 (BeginString), 9 (BodyLength), and 35 (MsgType) tags.

There are several tags about sequence num:
(34) MsgSeqNum Integer message sequence number.
(36) NewSeqNo New sequence number
(43) PossDupFlag Indicates possible retransmission of message with this sequence number
(45) RefSeqNum Reference message sequence number

https://drivewealth-fix-api.readme.io/docs/resend-request-message describes a resend request message i.e. a 35=2 message, containing —

(7) BeginSeqNo and (16) EndSeqNo specifies the requested messages

FIX session^app messages

http://btobits.com/fixopaedia/fixdict11/index.html

I feel even an app message has session-related tags.

Top-down learning of the FIX tags:

* learn basic fields of head and tail — see another post
* learn basic message types (msgType or Tag 35) within session messages aka session messages
** remember — Each session starts with a _session_ message LOGON and ends with session message LOGOUT
*** LOGON is a full message with a head containing a field msgType=A meaning LOGON

* learn basic message types (msgType or Tag 35) within _app_ messages
** learn basic Execution Report (35=8), the most common app message
** learn basic trading scenarios consisting of FIX dialogs

FIX common app message types

This post is about “Business / Application messages”, not “session/admin messages

http://en.wikipedia.org/wiki/Order_(exchange) — different order types

common tag=value pairs

35=D — msgType = NewOrderRequest
35=8 — msgType = executionReport
35=G — msgType = requestForCancel/Replace
12=0.05 — Commission (Tag = 12, Type: Amt) Note if CommType (13) is percentage, Commission of 5% should be represented as .05.

Used in:

*
* Quote Status Report (AI)
* Quote Response (AJ)
* Quote (S)

FIX session is like TCP + sqlplus

Each session starts with an session-message LOGON and ends with session-message LOGOUT. Session-messages are also known as “admin messages”. Main purpose is session management and flow control as in TCP

To maintain the session, I think FIX uses not only the session messages — each app message also has session-related tags like (10)checksum, (9)bodyLength and (34)sequence numbers — used to maintain a session between client and server.

retrans-request — is a message to ask for a resend of a missing message

heartbeat messages — between 2 parties

FIX keywords and intro

tag=value, where tag is always a number, and value is often char(1) — Both need the dictionary for interpretation.

Whenever there’s an enum of possible values, FIX would encode them into very short codes (often char(1) ). As a result, encoding/decoding using dictionary is prevalent.

http://www.ksvali.com/fix-protocol-faqs/#2.1

http://www.ksvali.com/2009/02/fix-protocol-videos-on-youtube-finally/