sending unsupported FIX tag on an algo order

EPA (End Point Adapter) is a generic name for a buy-side FIX gateway, responsible for crafting FIX orders and transmitting to a FIX liquidity venue. A venue could be a ECN or a pass-through broker or a private liquidity venue in an ibank…

Some broker declare that “Orders sent that include Algorithmic parameters that are not supported for that strategy will be rejected.” This warning relates directly to our earlier discussion “validation based on algo name”. Our conclusion is – EPA to pass through any supported or unsupported params to broker, and let broker decide to reject or ignore.

The alternative is EPA internal reject. Based on my experience, Inviting a broker reject is no worse than internal reject. Broker reject is usually preferred. When EPA rejects, there’s always a lingering doubt “Are we sure this unsupported tag would be rejected by broker? If not 100% certain, then why don’t we try placing the order? Maybe broker would accept the order even though they say they would reject.”

In other words, EPA internal reject is premature pessimism.

— validation based on algo name

For example, if tag 8081 is unsupported for the algo in my order, then I believe it’s usually harmless to pass it through to broker. Unsupported means “irrelevant” or “inapplicable” therefore no effect whatsoever. Conversely, If this #8081 had any side effect on this particular algo, then the side-effect would have been documented for this algo, and become a relevant tag for this algo.

So I think it’s harmless to pass through to broker. Whether broker ignores this unsupported tag or rejects, it’s anyway no worse than EPA implementing the same ignoring/rejecting.

absolute^relative(toBmark) algo perf: paradox

For a big Buy order,

IS benchmark could be $79
Vwap benchmark could be $76

A vwap algo may deliver $77, under-performing its benchmark.
An IS algo may deliver $78, beating its benchmark, but worse than the vwap algo performance.

I assume most clients want absolute performance, but they are the ones to specify the benchmark.

VWAP=a bmark^exectionAlgo

In the context of a broker algos (i.e. an execution algo offered by a broker), vwap is

• A benchmark for a bulk order
• An execution algo aimed at the benchmark. The optimization goal is to minimize slippage against this benchmark. See other blogposts about slippage.

The vwap benchmark is simple, but the vwap algo implementation is non-trivial, often a trade secret.

exec algo: with-volume

— WITH VOLUME
Trade in proportion to actual market volume, at a specified trade rate.

The participation rate is fixed.

— Relative Step — with a rate following a step-up algo.

This algo dynamically adjusts aggressiveness(participation rate) based on the
relative performance of the stock versus an ETF. The strategy participates at a target percentage of overall
market volume, adjusting aggressiveness when the stock is
significantly underperforming (buy orders) or outperforming (sell orders) the reference security since today’s open.

An order example: “Buy 90,000 shares 6758.T with a limit price of ¥2500.
Work order with a 10% participation rate, scaling up to 30%
whenever the stock is underperforming the Nikkei 225 ETF (1321.OS)
by 75 basis points or more since the open.”

If we notice the reference ETF has a 2.8% return since open and our 6758.T has a 2.05% return, then the engine would assume 6758.T is significantly underperforming its peers (in its sector). The engine would then step up the participation to 30%, buying more aggressively, perhaps using bigger and faster slices.

What if the ETF has dropped 0.1% and 6758.T has dropped 0.85%? This would be unexpected since our order is a large order boosting the stock. Still, the other investors might be dumping this stock. The engine would still perceive the stock as underperforming its peers, and step up the buying speed.

POV: execution algo

• minimizing market impact
• Percent of Volume
• participation ratio

https://empirica.io/strategies-catalog/pov has a the formula to determine the slice sizes.

symbology #RIC/NYSE/RTS

I think Symbology is a strict naming convention, esp. in derivatives.

RIC is one symbology. NYSE symbology is well known. RTS also has its own symbology .

typical options HFT shop annual profit

Optiver $450M profit, 700 employees. IMC is another options HFT with 500 employees. These specialist shops do in-house theoretical pricing, whereas the fastest HFT shops usually do not. Apparently, there’s still some nice profits for smaller hft shops despite the arms race!

execution algo #!! alpha-algo

I think the “exec algo” type is lesser known.

• VWAP is best known example
• bulk orders
• used by many big sell-sides (as well as buy-sides) to fill client orders
• the machine acts as a robot rather than a strategist
• goal is not to generate alpha, but efficient execution of a given bulk order
• Execution algorithms predict future market movement, mostly based on analyzing historical patterns, by machine learning
• Market Movement prediction is a data intensive job. Sophisticated AI systems are set up to solve this problem.
• 10’s of TB of tick data needed
• volume profile

 

mean reversion, deviation detector, pattern recognition

Nothing, including real time mkt data analyzers, can predict the future. They can point out unusual deviations which often precede reversions to norm. In such a case timing is unpredictable though.

Case in point — When I saw historical highs in copper price, I thought it would drop (reversion) within hours, or at most couple of days, but it just kept rising and destroyed my position. (A necessary condition for my undoing is margin. No margin, no collapse.)

I guess China Aviation Oil might have something like this?

Such a reversion is one type of pattern. Some patterns have a strong rational, logical basis. Consider historical vol’s mean reversion pattern. Consider the widening spread on-the-run vs off-the-run.

Mean reversion is one type of deviation detector.

execution risk in a VWAP execution algo

Background: VWAP strategy is known to have minimal market impact but bad “execution risk”.

Suppose you are given a large (500,000 shares) Sell order. Suppose your goal is minimal market impact i.e. avoid pushing up the price. What execution strategy? I don't know about other strategies, but VWAP strategies generally participate according to market volume, so given a decent implementation the market impact is often … reduced.

I think the idea of the Exec risk is the _uncertainty_ of the final block price. If an implementation offers a very tight control and results in well-controlled final block price, then exec risk is small. http://www.cis.upenn.edu/~mkearns/finread/impshort.pdf explains with an example —

suppose MSFT trades 30 million shares on an average day. If a trader has three million MSFT shares to trade, a VWAP algorithm may be appropriate. However, if the trader

gets 30,000 shares of MSFT to trade, then the savings of market impact (by spreading the trade over the whole day) is not significant compared against the opportunity cost the trader could save by trading the stock within the next few minutes. Quick execution means the uncertainty (or std) in “final block price” is much reduced. With a small order you would achieve something close to the arrival price.

vwap execution chasing the wrong signal – my guess

A vwap algo starts with a “model profile”, which tells us each hour (or minute) of the trading day typically experiences how many percent of the total daily volume.

Then the algo tries to execute according to the model profile, executing 10% in the first hour. The actual market profile may show a spike in the second hour. Suppose 2nd hour usually gets below half of first hour according to the model profile, but we see it's going to more than double the first hour, because the past 5 minutes show a very high spike in volume.

Question is, should we increase our trade rate? I guess there's reason to do so. When the volume spikes, we should trade bigger chunks so as to let the spike “mask and absorb” our market impact. If we don't capture this spike, then 2nd hour might end up being 80% of daily volume, but we only put in 4% our quantity, so our remaining quantity would cause market impact.

However, it's also possible to chase the wrong signal. The spike might cause a large rise or (more likely in realistic panic-prone markets) drop in price, which could reverse soon. Suppose we are selling a big quantity and the spike causes a big drop. Our active participation would further deepen the drop. We might do better to patiently wait for a reversal.

##HFT: some defining features relative to similar systems

I learned from a seminar. I didn’t do any research. Just some personal observations.

• HFT is different from a broker execution algo in terms of holding period. HFT never holds a position overnight.
• HFT is alpha-driven. In contrast, a strictly sell-side trading engine is driven by customer flow.
  • however, HFT doesn’t use market orders as much as limit orders, so it may appear to be market-making.
• an HFT engine makes many, many trades in a day, but so does an execution algo.
• HFT usually makes no directional bet. In contrast, fundamental strategies have a view. I know many sell-side traders have a view and may hold inventory overnight.

I think the distinction can become unclear between HFT and execution algos.

slippage: several related meanings

–definition 1

slippage = difference between 2 prices — the earlier quoted by an OTC dealer (FX, bond, swap…) vs the requote when you take that quote. In many quote-driven markets, all public quotes are always indicative.

–definition 2

slippage = difference between 2 prices — the EOD benchmark vwap vs your block order’s /implementation/realized/ vwap, assuming order filled within a day. If order is to be executed from 2pm to 4pm, then the benchmark vwap should be calculated accordingly.

As an execution benchmark, VWAP has remained the most common benchmark. TWAP is an alternative benchmark. There are two other related and simpler benchmarks — arrival Price and decision Price

• Decision price = the live price number (as a snapshot) that triggered decision to trade. Trading signal price. see http://en.wikipedia.org/wiki/Implementation_shortfall
• Arrival price = the live price number (as a snapshot) when order received by execution engine.

Nomura algo spec defines arrival price as the snapshot mid price between bid/ask at arrival time. So this arrival price is the benchmark, and the average execution price of your order should end up very close.

=== 3 execution algorithms to minimize 3 slippages

1. Implementation Shortfall — Minimize slippage relative to arrival price. Note decisionPrice is Not used by the implementation guys,
2. VWAP – Minimize slippage relative to the Volume-Weighted Average Price (VWAP) in the market
3. TWAP — Minimize slippage relative to the Time-Weighted Average Price (TWAP) in the market

 

 

VWAP: market impact = slippage @@

Suppose we have a 5000 IBM buy-order. VWAP is the benchmark chosen. Our goal is to predicted market impact. If the predicted MI is too large, then we should either be very careful, or decline the block order.

The predicted market impact is also the predicted slippage.

Let's put in some fake but concrete numbers. Suppose we estimate that submitting this block buy would raise today's eod_vwap by 100 bps. That means whatever today's eod_vwap happens to be, our execution_vwap will be higher by 1%.

If eod_vwap is $70 then our market impact is about $0.70

If eod_vwap is $75 then our market impact is about $0.75

I think we should inform our client that we can execute the block order at the benchmark eod_vwap + some spread. I would quote a spread like $1 given spot is $69 and I estimate today's eod_vwap to be $69.50.

Suppose client accepts and we execute the order and execution_vwap = $69.9. What's the realized market impact? We can only imagine “what if I had not done this big order”. However, we can observe the benchmark eod_vwap = $69.10. We performed better than quoted. Slippage = execution_vwap – eod_vwap, so our slippage is smaller than our quote!

[[all about HFT]]

Author is an option specialists (currently teaching derivatives at a university). Many mentions of HFT on options.

 

chapters (80p) on technology. Author believes the 3 legs are {strategy, math, tech}

chapter (50p) on strategy

**first part seems to be uninteresting, math-light but might be important in practice

**chapter (12p) on arbitrage strategies

1 page on native API vs FIX.

a few pages on cpu offloading, including running Monte Carlo on GPGPU

compares c++ vs c#java in a HFT context

compares buy vs build in a HFT context

[[Quatitative Trading]] by E.P.Chan

I still feel it's too complicated …

 

chapter (5p) on Matlab. This author singles out Matlab and Excel, not R or c++

chapter (40p) on back testing

**Chapter on historical database

chapter (2P) on HFT

chapter (7p) on factor models

[13]some modules of algo trading system #YH

A few modules I have heard of

* MM — mkt datafeed reader
* OMS — order book …
* order submission gateways and smart order routers.
* PP — pretrade pricer, not needed in stock or FX
* BB — trade booking engine, when an execution comes back successful. BB is the main module affecting the position master DB
* PNL — real time pnl engine
* RR — real time risk?

I guess BB, PP or PNL might be absorbed into the OMS module. In some places, OMS might be the name for any real time component of the entire system. There’s a narrower definition of OMS though, so I will just refer to that definition as the EMS (execution management system). For a micro hedge fund,

PP EMS RR may not exist, and the rest (MM BB PNL) can be manual. In contrast algo shops must automate all steps.

–Where does MOM fit in? It’s an enabling technology, usable in many of the functional modules above.

–How does dynamic data fabric fit in? Also known as data grid or in-memory DB. There are 2 types
* generic technology, often integrating SQL, MOM technologies
* functional module with complex business logic, such as CEP engines, often tightly integrated with MM OMS PP

These two descriptions are not mutually exclusive. Many data grid systems include features of both. However, for simplicity, in this write-up I treat data grid as the generic technology just like a DB or MOM.

–How do reference data fit in? I guess it’s just another separate service to interface with database, which in turn provide reference data to other modules when needed?

Ref data tend to be fairly static — update frequency would be once a few hours at most, right? It is essentially a readonly component to the algo engine modules. Readonly means those modules don’t update ref data. It’s one-way dependency. In contrast, MM is also readonly, but more dynamic. MM is the driver in an event-driven algo engine.

I feel ref data read frequency can be high, but update frequency is low. Actually, i feel the occassional update can be a performance issue. Those dependent modules can’t cache ref data if ref data can change mid-day. There are techniques to address this issue. A fast engine must minimize DB and network access, so if ref data is provided on the network, then every read would be costly.

Some moules are simpler, like PNL and BB so no big deal. Pricing lib is used for PP and perhaps RR, which are quantatitively complex. MM EMS are technically challenging.

The “algo” tends to be in the MM PP EMS and RR modules

A High frequency shop also needs to assess market impact. Not sure where it fits in, perhaps the EMS

Where does STP fit in? I feel STP is largely non-realtime.

TWAP – basics

TWAP has multiple meanings

1) twap is name of an Execution benchmark
2) twap is name of a trading algo that aims to beat twap benchmark

One site says — The algo breaks up a large order and releases dynamically determined slices of the order to the market using evenly divided time slots between a start and end time. The aim is to execute the order close to the average price between the start and end times thereby minimizing market impact.

Another site says — Executes an order evenly over a user defined time period. Unlike VWAP, scheduling is done on an even basis rather than trying to anticipate the volume curve of the period as VWAP does. The most common use of TWAP is for distributing big orders throughout the trading day. For example let’s say you want to buy 100,000 shares of Morgan Stanley. Putting one such a big order would vastly impact the market and the price most likely would start to raise. To prevent that, investor can define time period in TWAP Strategy over which they want to buy shares. It will slice evenly big order into smaller ones and execute them over defined period.

Higher risks of market impact when market volume is ultra thin but our strategy continues to send the same order size at the same pace as before — our orders would likely become large orders causing deep impact.

Can work well as an alternative to VWAP over short periods, particularly into the close. Basic illustration of vwap vs twap benchmark — Suppose there are only two trades during 10-11am:

– 100 shares for $10 at 10am
– 10000 shares for $11 at 10:30am

* The TWAP is $10.50 because the last traded price was at $10 for 30 minutes and at $11 for 30minutes
* The VWAP is $10.99 — a more realistic reflection of the price one would likely get

https://empirica.io/blog/twap-strategy/ describes intra-day TWAP formula

 

vwap exec-algo is non-trivial

See other posts on ExecutionRisk.

Some friends say vwap is the simplest trading algorithm, compared to most algorithms. I am newbie. I feel vwap is not a no-brainer that you can master in an hour.

2 datetimes — formula can apply between any 2 datetimes, but by default is [StartOfDay – now]

VWAP has 3 meanings —
1) vwap is name of a live soft mkt datum displayed as a live creeping curve similar to a “moving average price of last 3 trades”. VWAP is averaged by volume, and averaged over all trades since SOD (on that cusip). Do we include just sells? Well, each buy trade matches exactly one sell.

2) vwap is name of a common Execution Benchmark when a sell-side receives a block order from a buy-side
** guaranteed VWAP execution.
** Say you had a sizeable buy order executed below average price. Well done, but how do you define “below average price”? Default definition is vwap. Arrive Price is another definition.

Suppose a buy-side has a large (500,000 shares) block sell – tough job. A sell side (usually a i-bank) typically alleviates the buy-side’s problem immediately by buying the shares wholesale from the buy-side, and then selling them off later. The contract price might be the next 30 days’ vwap Minus 1 cent, which is 1 cent WEAKER than vwap. With a block-Sell, it’s hard to guarantee a price Above vwap; it’s more achievable to guarantee a price Below vwap.

3) vwap is name of a standard execution algo for Trading machines
** impact – minimal market impact. This is an advantage of vwap strategies. Many passive traders prefer vwap strategies. (But large order may not get fully filled)
** The other side of the coin — execution risk is high with VWAP

** price, quantity, timing, LO/MO, when to cancel — the key output of this or any algo. Here are Some of the trading tips
**** empirically, to beat vwap benchmark, you need good forecast of intraday volume pattern
**** rule of thumb — trade most heavily when markets are most liquid.
**** Note there’s usually a desire to finish working the block order within a time-frame, rather than indefinitely. If indefinitely or order is small, you can beat vwap benchmark by using limit orders
**** Note using limit orders earns us the b/a spread while using mkt orders costs us the spread but is faster.
**** if you want to aggressively participate in current volume and avoid missing the wave, then use MO rather than LO.
**** block traders sometimes divide the given timeframe into, say, 10 bins and allocate more/less than 10% of the block to each bin according to volume forecast, and trade according to the plan. Just like project planning and execution.
**** Note minimizing mkt impact is one of the top 3 criteria. If you sell first part of a block-Sell with massive impact, price would stay high for the rest of the day and you can’t complete the order.

BNP (NY) FX algo trading IV 2012

Tick data needed for … back testing, analysis? Quants use R, matlab etc to analyze data. IT don’t use those.

Q: how can you help my desk make more money?

Q: from a low-level interviewer — describe to me a challenging technical design (and expect me to drill in).

Q: We know the drawback of auto_ptr. Now write an auto_ptr template class spec (without function bodies) with private op=()

Q: describe some threading challenge in your low-latency system.

Q: how would you use pigeons for IPC (possibly across machines)?
Q: what’s a unix domain socket?

Q: is Throwable checked or unchecked exception?

Q: how many incoming/outgoing orders a day?
Q: peak incoming order rate

(I feel they noticed my pretrade pricer as it’s similar to their market making engine.)

Q3: what pretrade pricing rules did you have? Describe how they work in real life
Q3b: what’s the peak price update frequency?

Q: in your pretrade pricing engine, how many messages do you send out each day?

Q: how does the pretrade pricing system interact with other systems?
Q: are the conduits more retail or institutional?
Q: how does the asset valuation work?

%%Q: what do quants here do differently from quants in der pricing?
A: tune parameters; analyze data and come up with strategies; Sounds simple but a lot of work.
%%A: constantly improve the strategies and parameters.

%%Q: 20,000 euro loss would be realized loss?
A: yes

%%Q: what if multiple strategies trade the same currency pair without knowing each other?
A: multiple live strategies can trade the same pair. They should be designed to minimize correlation and know each other.

non-event-driven trade origination (Charles River)

(Background — all order origination must be driven by something, sometimes by the clock!)

Apart from event driven algo trading, a lesser-known type of semi-algo trading system is what I call trading signal generation. In some trading software (Charles River?), you can configure portfolio balancing “rules” to automatically generate candidate trades — trades needed to balance a portfolio. These candidate orders start a “workflow” and show up on the order blotter. Trader can take her time to decide when to hit the market.

These trading signals are not event driven, not CEP type. They could be driven by a periodic revaluation. I think it’s also possible to start a book-revaluation on-demand, perhaps when the asset manager decides to do so.

The asset manager could be a private wealth advisor or a fund manager. Portfolio balancing is important to asset allocation ratios.

Unlikely high frequency shops, these candidate trades are not automatically sent to the exchange. Therefore, the impact and risk are less severe. Without human intervention, the risk is much higher. I guess that’s why the high frequency shops usually use small orders, making many small gains and many small losses.

There’s also a 3rd (middle-ground) type of algo-trading, where a human trader (not an operations person) monitors the automated trading machine, either by market order, limit order, RFQ, RFQ-auto-response, or ECN quotes (always executes off the ECN). Under each category, human intervention level (and risk) are rather different. The human trader watches the orders sent, requests received, quotes sent/received and can jump in and take over. This 3rd type is also event-driven.

prop trading^mkt-making – intertwined

These terms’ meaning become progressively vague as they spread into everyday English, but for now, let’s look at the strict definitions.

Prop Trading basically means “orders initiated regardless of client requests”, which is different from sell-side Mkt-maker’s trading.

However, a market maker often needs to initiate orders as a hedge, or to exit a position.

Additionally, a market maker always has discretion as to which client order to ignore.

A lot of prop trading takes place in sell-side firms, along with the market-making system, so these special prop traders can often gain insight from the MM order flows. Look at front-runners.

##autoREO quote pricing rules, again

For each position in a trader’s book, she can pick one of the pricing rules. Once a rule is attached to a position, the rule has a life of its own in this “algo-trading” engine. Whenever one of the observables change, the rule would fire a cache trigger to update the quotes, which exists off that position. Everything lives in the cache. CEP-like.

#1 pin rule – “pin my position” to the primary YieldCurve of the desk. We basically add a spread on the pin yield to get the bid-ask yield.

#2 eval rule – use last traded price on the live market for that security. We basically add a spread on the eval yield to get the bid-ask yield.

#3 DCA rule – use dollar-cost-average to get a “cost price”. Convert to yield, and add a spread on it to get the bid-ask yield.

#4 Treasury rule – we follow the live Treasury quotes. We periodically (10 sec) poll our live cache to reprice our quotes. We basically add a spread on the T yield to get the bid-ask yield.

** Less popular – Libor futures quotes, T-futures quotes.

#7 Arb rule — There are 2 “model” accounts, aka prop trading accounts. Any security in those accounts is priced by the model. Outputs both bid and ask. We just take them as is, without adding any spread.

#8 Lastly, MTM rule – there’s real time mark-to-market done on most positions – supposed to be fair valuations. This rule basically says take the “mark” yield and apply a spread to get the bid-ask yield. Now the confusing part – many of the marking rules are the _same_ ones listed above. However, there was a separate family of marking rules and a separate family of quote pricing rules.

I now think this can give rise to a cascade or ripple effect. Position A has a mark rule triggered by an eval, so position A mark-to-market valuation is updated. Then Position A or B has a quote pricing rule referencing A’s mark, so in a 2nd wave the bid/ask gets updated.

small old-guard quant fund – some facts and figures

This is one of the earliest quant funds – stat arb sort of thing. “Definitely a quant fund, rather than a macro fund, but not high frequency.”

Quant group 12 guys. Write most of (what i call) the business modules. (In contrast, mkt data consumption and OMS – 2 infrastructure pillars — are owned by the System group — what i call the pure tech, math-free modules) These business modules generate orders, achieve certain target positions, manage risks… As [[blackbox]] pointed out, quant traders think in terms of entering/exiting positions.

System group – 2 system admins and 4 c++ developers. I guess one of them is a network guy.

Mostly equity cash, but also commodities and (eq?) options. Trading is fully automated, forecast-based.

One model may have a trading horizon of hours
One model may have a trading horizon of days
These 2 models may overlap if they cover the same name. There is a feature of the trading system to manage the overlap.

(No model with trading horizon of seconds – not really high frequency.)

GUI is x-win. Whole company is linux/solaris, without windows.

Sybase – used lightly, now phased out. Persistence is done without database, probably in flat files.

Primary app language is c++ — threading, IPC, TCP, multicast for mkt data feed handler. No RV.

Various scripts – often in perl.

Truly algorithmic trading on autopilot

Bulk of the software is written by “infrastructure engineers” using STL, sockets, system calls(yes)  and C tricks — all low-level stuff to optimize speed, and a small chunk is written by the algorithm researchers.

At SOD, operations start the trading engine and it runs on autopilot. The human control is provided not by a human trader but by the operations team. In a sense, the ops team are the traders.

Treasury algo trading (low-latency), basics

In this European megabank, “treasury algo-trading” basically means RFQ auto-quoting.

Bloomberg and tradeweb allow an institutional client to send RFQ to a selection of (her “favourite”) Treasury dealers (also Agency bond dealers and IRS dealers). RFQ can be both bid and ask.

Off-the-run securities have much lower liquidity than on-the-run, but still higher liquidity than muni or corp bonds. Basically, who wants to buy the off-the-run when they easily get the on-the-run?

Typical latency [ from receiving request at gateway, to sending out response from gateway ] is typically 60~100ms (some say up to 200ms). 500ms (half a second) would be too high. I mean millis, not micros.

The RFQ itself takes 20-30ms to price.

who sells/buys quant algorithms and strategy

Q: what exactly is advertized, sold and implemented when we see a big bank (like GS) selling a quantitative strategy/algorithm/model?
%%A: trade signals/algo/intelligence, execution service, and trade “sponsorship”

On the buy-side, traders namely “account owners” are always users of a quant algo. I believe Retail traders or institutional traders receive trade signals from the algo (owned by the sell side) and can either ignore it or follow it.

Q2: Physically where is the trade signal generated?
%%A: either in the sell-side system or in the buy-side system These signals are never guaranteed to be profitable. Just imagine you are using your hard-earned savings to trade. Trade signals might contain size, order type (mkt order or limit order or OCO…), and price. Actually, There are resellers for this “product” ie trade signals. A small broker can be the buy-side of an algo vendor, and the sell side to its own retail clients. The algo is the IP of the algo vendor. The same vendor may also provide market data, or DMA, smart order routing …

Q2b: can a client take the signal and trade through another broker?
%%A: i believe she won’t be able to profit from it. The vendor will make sure you route your orders to them. Assuming signal is generated in the sell-side system, then you probably pay for the signals.

In a complex multilateral set-up, always ask
Q: which entity makes trading decisions?
%%A: probably not the algo vendor

In contrast, the block-trade deal is simpler. Trading decisions are made by the sell-side (say GS), on behalf of the buy-side (say Metlife). GS might be the counter-party to Metlife or just a middleman. If GS routes the order to CBOT, then GS sponsors the trade. See http://bigblog.tanbin.com/2009/09/adp-broker-means.html

tick data feed – mkt-data distributor

• You have a ticker feed, a queue that the feed can push onto, data workers that consume the queue, and subscribing listeners to the data workers.

• Explain a non-blocking, scalable algorithm for how the data workers can transfer the data to the listeners.
• Suppose some listeners consume messages slower than others. Suppose some listeners are only interested in certain tickers.
• Suppose the ticker feed pushes data on the queue in bursts, how might you ensure the data workers don’t needlessly poll an empty queue?

That’s original question. I feel this is fairly realistic market data system. I covered a a similar requirement in http://bigblog.tanbin.com/2011/08/max-throughput-live-quote-distribution.html

I feel a data worker deals with 2 “external systems” — big queue + listeners. The queue is a messaging FIFO thing (not necessarily JMS queue) to be polled. When it’s empty, data worker threads should wait in wait(). A socket thread can probably receive new data and notify data workers. When all the workers have processed a given message, it needs to be removed — See the same blog above.

I feel non-blocking kind of comes by-default, but maybe I don’t understand the question.

Message filter should probably be implemented in the data worker. See the same blog above.

To ensure slow listeners don’t block fast listeners, I feel we need multiple threads per data worker. In contrast, the Simple design is that data worker thread is also the thread running the onMessage() methods of all the listeners.  Multi-threaded data worker is probably a common design.

trading decisions require human !! just machine intelligence

XR,

(another blog post)

You questioned if so much of trading decision-making is reliably handled by computers, then what special skills would a trader need? If not much, then you and me can also be traders. Well, you have more trading experience than I have (0.000), but here's my 2 cents worth.

There are important pricing decisions beyond pre-trade (PnL check, funding, borrowing…) but I assume your question is about pre-trade esp. pricing decisions. In that space, I feel below are some skills/tasks a machine won't, can't or shouldn't do —

* configuration – trade decision making software offers a bunch of tunable parameters on a per-name basis, name being a cusip, a stock symbol etc. Generally machines don't derive these values as these are inputs to the machines.

** pricing rules – A special type of configuration is a pricing rule. A trader often has to pick one from a pool of rules. Each rule follows some market data to compute prices, which drive pre-trade decisions. A machine won't pick the rule — it applies rules you pick.

* hedging – after (sometimes before) a non-trivial trade, there's usually a need to adjust hedging. I feel often a machine needs guidance from a human. I feel it's dangerous to let machines make hedging decision. Even in a liquid market, machines may pay too high. Hedging is a more complex task than pricing a bid/offer, perhaps too complex for a machine.

* timing – a trader often steps in to change pricing rules, configuration numbers, or stop/start a machine, when she sees some market event

* conflicting signals – most algorithms process tons of market data to pick up signals according to some strategy — strategy chosen by humans, but in a storm, market data can present conflicting signals. One machine will see signals to sell; another in the same desk may see signals to buy. There are basic rules to prevent computers from make self-contradictory trades, but a human may have a better chance making out what's happening in the market. After that, we can put together a coherent view and plan.

** exception is high-frequency trading. I feel HFT algos can tolerate more ambiguity and conflicting signals. They aim to make quick in-and-out trades, so they may not need any view at all.

* RFQ – some RFQ are answered by machines, but other RFQ are large or unique, (or the trader knows the requester well) so traders may step in to review these RFQs.

* news – machines can't understand news and adjust trading.

* trends – machines can spot statistical trends, but how about industry trends and economic trends. I don't have good examples of such trends. Although such insights don't affect everyday trading, they set long term goals.

* information overload — In all financial markets, there is lots of information. The amount of information (sometimes misleading or distracting) makes trading not easier but trickier. Evolution has given human brains special capacity to handle information overload. We discard 99% of the information to focus on critical factors; we see through smoke and mirrors; we question; we debate… That allows us to figure out what's happening. Not for a machine.

* specialization — Quite often, the more complex, the more human insight is required. Many mainstream trading desks are organized by product, so that each trader can specialize and focus on a limited body of information and knowledge so as to gain insight. Similarly, a stock/bond exchange often appoints 1 or 2 traders as specialist market-makers for a particular “name”, so they should know everything about that security.

* mistakes – machines make many stupid trades in terms of timing, pricing, quantity etc. That's my short answer and ultimate answer to your question.

limit^stop order in FX/eq#my take

See also post on stop orders…

2 common order types to let you buy at a specified price — A) limit order B) stop loss order

My CFA book made it clear that a Stop-Loss-Buy@$5.55 will become a market order once the offer side “touches” that point (compare binary options!). (Note You use a SLB after you enter a short position at a LOWER price like $4. It fires when market moves against you — up in this case.)

– if best offer remains at that price, then your SLB will (partially) fill right there.
– if that 5.55 offer is wiped out and market deteriorates further to 5.58, and your order gets to head of the market-order-queue, then you execute at that price. You suffer a bit. This is the “fall-through” scenario. (For our SLB, this is “break-through”. Anything-through is bad!)
– if market quickly recovers (perhaps offer side gets aggressive) and a LOWER offer is in effect when your order is at head of the queue, then you execute at a BETTER price. This is a touch-and-go scenario.
=== In conclusion, a SL order may execute a bit better/worse than your order price. You can’t control that (but you can quickly send cancels). See http://ibkb.interactivebrokers.com/node/255

In contrast, a regular Limit-Buy order let’s you control the worst case scenario. If you are filled you are always filled at $5.55 or better.

Note the abbreviated terminology. Stop-Buy is a Buy order, for the purpose of stop-loss.

Some (Forex, eq etc) people claim that a SL executes at a worse price than specified. I believe that’s assuming touch-and-go is rare in practice, and fall-through is far more likely.

real time risk monitor ^ real time quote repricer (vol trading)

–Most used app in EQD — real time risk monitor
A) #1 workhorse — tick driven position updater with real time greek/unrealizedPNL. Real time unrealized pnl is still important, but EQD trader’s #1 concern is the potential gain/loss in MV of her open derivative positions — real time exposure. Most of the trading decisions are based on this exposure assessment. Real time unrealized pnl doesn’t address questions like VAR or “what if”.

B) Just like Autoreo, EQD real time risk engine manages entire books of open positions, real time mark-to-market/pnl and risk assessment. However, no automatic adjustment of offer/bid on inventories — #1 function of Autoreo

C) As another comparison, BofA Quartz is first a risk engine, and 2nd a pricing engine. In many asset classes, the risk engine doubles as a pre-trade pricing engine.

–Most important business factor in EQD — volatility
In the interest rates business, every player is constantly trying to forecast (and hedge) interest rate fluctuations over various timeframes ranging from 24 hours to 30 years — the entire yield curve. That forecast drives the entire market and the rates business of every bank. In eq der biz, #1 factor is not IR, not the stock market per se, but volatilities (of stocks and ETF and indices). Volatility as a  concept measures how “wild” is the swing (over a timeframe like 30 days) of any fluctuating price. Over the next 30 days, there’s just one such vol, but using bid/ask at different strike prices, you derive different implied-vol values — welcome to the Smile. A smile is one curve. Using different curves at different expiration timeframes, you get a 3D surface.

Every bank needs to forecast vol using such a surface. Each bank uses a proprietary algorithm, because vol forecast is never straightforward. Yet we must forecast, because vol is the real driver.

Typically, an eq der trader has a lot (hundreds) of open positions to manage throughout the day. (A structured eq der trader has even more long-standing positions.) The main market data of interest is tick data i.e. live bid/ask quotes. Tick drives implied vol surfaces, which drive valuation of non-listed options. Not sure about variance swaps and other eq der instruments, but volatility is the #1 driver.

Q: beside vol, what other business factors are important?
A: open position MV is also sensitive to projected dividends, but they change less often.

In a nutshell, eq der traders need to monitor sensitivity/VAR/pnl-impact in their open positions. These positions change value in response to market data “swings”.

(ADP) algo trading system @ sell-side

Refer to my “ADP” posts for background

I feel the most prominent algo engines are buy-side, or prop trading systems. Example — Citi muni prop traders operate a so-called arbitrage systems that publish bid/offer prices.

If you see a sell-side system advertising itself as algo-trading, then perhaps it caters to buy-side algo engines. Example — Barc DMA. Probably an execution-algo engine, rather than an order-origination strategy/alpha.

Sell side is often a market maker, though the term is ambiguous. As market maker, you may not have the freedom to issue a lot of market orders. Example — in fixed income, a dealer often holds positions, so they maintain firm offers and bids on the market.

I feel High-Frequency trading is not the bread-and-butter business of a sellside.

##high frequency trading system expertise

Background — what asset classes are we talking about? Equities, FX, index futures, listed options, US/German government bonds, ED futures, …See also http://bigblog.tanbin.com/2011/06/algorithmic-vs-high-frequency-vs-low.html.

Here are some basic features/techniques of HF (high-frequency) systems. In other words, HF trading systems compete on these —

1) latency – see other blog posts such as http://bigblog.tanbin.com/2011/08/low-latency-key-expertise.html

2) algo trading — a mature body of knowledge and technique starting from vwap/twap, Arrival-Price, Implmentation-Shortfall …
** trade origination

3) market data — basis of algo and latency techniques. HF systems have higher requirement for market data

?) real time risk measurement/control — i feel risk system is different if you do thousands of trades a minute. I know some HF shops run trading engine on autopilot, so risk system is not a barometer for a trader to look at

—- Above are the 3 aspects I know —-

4) architecture? Do HF system demand a special architecture? I think A) latency and B) market data volume dictates architecture

5) pricing model? It’s Part of every trading algorithm, but are they any different from the regular run-of-the-mill pricing engine in every automated trading system? Well, there are models and there are models. I feel algo trading models are slightly more [1] sophisticated than pricing models in other pretrade pricing engines.

[1] sometimes less. In practice, some HF engines compete on latency and seeks small profit on each trade, so pricing is probably simplified.

I feel HF systems sometimes use simpler pricing models than non-HF. However, it’s possible that for a given instrument a large percentage of HF trades come from a single trading engine, and that engine happens to use a simple or complex, fast or slow pricer. I won’t argue with such statistics. I’m only giving a personal perspective.

The more sophisticated pricing models usually exist in time-insensitive pre-trade Aanalysis or post-trade phases (no such luxury in the millisecond real time battlefield). They often analyze a larger body of market data, unconstrained by latency of data analysis. They often involve time-consuming simulations. They often involve optimizations. They may output a price decorated with a bunch of metrics about the quality of the price. Such output is often intended for a human consumption not machine consumption.

Now, if another buy-side runs a traditional trading system to compete against a HF, latency-gap alone could spell defeat – consider front-runners

## if a event-driven trading engine is!! responding

• Culprit: deadlock
• Culprit: all threads in the pool are blocked in wait(), lock() or …
• Culprit: bounded queue is full. Sometimes the thread that adds task to the queue is blocked while doing that.
• Culprit: in some systems, there’s a single task dispatcher thread like swing EDT. That thread can sometimes get stuck
• Suggestion: dynamically turn on verbose logging in the messaging module within the engine, so it always logs something to indicate activity. It’s like the flashing LED in your router. You can turn on such logging by JMX.
• Suggestion: snoop
• Suggestion: for tibrv, you can easily start a MS-windows tibrv listener on the same subject as the listener inside the trading engine. This can reveal activity on the subject

Treasury algo quoting engine #CS

1) In market-making, a quoter engine is driven by a specific model (among many models), and trades using a model account. It reacts to market data [1], and based on inventory will publish/withdraw bids and offers – collectively known as quotes. In theory, a quoter could maintain quotes at different price levels, but more typically, a quoter maintains multiple small[2] quotes at the same price level, but why? [3]

2) Beside market-making, another model is RFQ or bid-wanted / offer-wanted. See post on [[2 ways to sell a security]]. In treasury market, a RFQ is often sent to a small number of dealers only.

3) In terms of alpha strategy, there are 2 main “views” of price movement – mean-reverting vs trending. These are “views” on the market.

Same system can trade T futures, but it’s more complex due to the basket.

[3] A: those different quotes will join the order queue at different queue-positions. They will execute at different times. You can withdraw them individually.
[1] interdealer – BrokerTec and eSpeed; dealer-customer markets like tradeweb; orders from private clients not on the open market,
[2] low quantity, low risk — fine grained

Since each treasury position ties up a large sum of money, automatic financing is important. No margin as far as I see.
– After you buy a bond, automatically the bond is lent out on the repo market, so trader can use that money to pay the counter party.
– After you short sell a bond to get $20M, automatically you borrow the bond on the repo market, using the $20M proceeds.

algo trading – 2 types

Algorithmic Trading vs. Trade Origination

Here’s the key question you need to ask:

• Is a human making the trading decisions and simply having a computer help with the execution, or is the machine handling the execution and making the trading decisions?

The first category – where the computer only assists with the execution – is called algorithmic trading.

The second category – where the computer actually makes decisions – can be called trade origination, although that is not a canonical name

pricing and quantity rules in an algo trading engine

Our trading algorithms mostly control the price and quantity sent to trading venues. Quantity rules are simpler – verify max, min, increment, and also adjust price if quantity is unusual.

[10]equity HFT IV #UBS

Seems to be mostly simple (sometimes obscure) QQ questions.

Q: how does a DB index tree work?
Q: how do you record every keystroke and mouse event in swing? Codebase is large and is in production.
Q: what’s DB statistics? What’s histogram? What are they used for?
Q: When are db stats updated?
AA: not automatically, unless you use sybase Job Scheduler

Q: how do you programmatically change tx isolation level?
Q: what’s a mem barrier?
Q: how do you create an object in Perl?
Q: what’s biased locking? java6 biasedLocking^lockfree^ST-mode
Q: what does JIT mean? When does the optimized compilation happen?
AA: the code can be compiled when it is about to be executed (hence the name “just-in-time”).

Q: for a concurrent hash map, if you are iterating while another thread updates the collection…?
%%A: if you both hit the same segment, then CMExeption

Q: can you get a snapshot of the CHM?
%%A: I doubt it

Q: local vars are on heap or stack?
Q: what kind of queues do you use in your thread pool?
%A: default queue, probably a blocking queue

Q: what’s string intern?
AA: https://stackoverflow.com/questions/10578984/what-is-string-interning

Q: are they in the perm gen?
AA: yes before java7

Q: what happens when GC fails to free up enough memory for a request and JVM attempts to grab additional memory?
%A: the requesting thread blocks
Q: what about other threads?

Q: do you know any thread-safe array-based collection?
%A: vector, synchronize your own ArrayList, and concurrent queues.

Q: how do you protect a hashmap in concurrent app?
%A: i don’t know how to lock on a bucket (no such pointer exposed), so i lock the whole thing.

Q: can a final variable (constant) be modified?
%A: yes reflection.

Q: when would you favor a tree map over a hash map?
%A: range query

Q: serialize a hash map of array of ints?
%A: no problem.
Q: how about circular reference
%A: gemfire may have problem with it but standard java serialization can handle it

Q: what jdk data structure or algorithm have you found insufficient and you had to re-implement yourself?
%A: the queue or stack was based on arrays and expose the intermediate nodes.
Q: to block the get() method, would you use encapsulation or inheritance?
%A: inheritance is dubious.

Q: what’s syn_sent in netstat output?
%A: part of the handshake

Q: say your have 1,000,000 orders in a map, keyed by orderId, but now you want to query by stock symbol?
%A: in both tree and hash maps, you must scan every entry. But it’s possible to build an index on symbol.

algorithmic vs high-frequency vs low-latency trading

There is a lot of innovation, complexity and key principles in these domains. Below is a novice’s observations and hypotheses…In this blog there are other posts on the same topic.

The terms “algo trading”, “high frequency” and “low latency” are related, from a developer’s perspective. Low-latency is infrastructure optimization and engineering, like juice extraction from an orange (the hardware), which requires deep technical but modest financial or analytical knowledge. Low-latency infrastructure, once constructed, is used by many trading desks across a bank, not only by HF traders or algo traders.

HF and AT both originated on the buy side. Strictly speaking sell side (including investment banks) should seldom/never engage in these risky endeavors, but I know many do, under disguise. LL is needed on both buy-side and sell-side.

Algo trading can use many different mathematical algorithms to decide when to buy or sell (or cancel an uncompleted order) how many contracts at what price, using what type of order, in conjunction with what hedging orders. If a so-called “strategy” competes on frequency, then latency is often a key differentiator. The cold war was both /ideological/ and an arms race. Low latency infrastructure is like the nuclear weapon in the trading “arms race”, whereas the math algorithm is (very loosely) the ideology.

Some trading algorithms aren’t latency sensitive, and won’t be high-frequency. They compete only on the math. I believe most machine trading systems in reality are quite fast. If it’s very low frequency, then a human trader can make the decisions herself, aided by a machine with an algorithm

The math doesn’t need to be sophisticated or use stochastic PDE, but should match market reality and be faithfully aligned with the strategic intentions.

ION-mkt — nested event handler

MkvPlatformListener — Platform events such as connection status.
MkvPublishListener — Data Events like publish and unpublish.
MkvRecordListener — Data Events like Record updates
MkvChainListener — Data Events like Chain updates.

class CustomChainListener implements MkvChainListener {
     public void onSupply(MkvChain chain, String record, int pos,
           MkvChainAction action) {
           System.out.println("Updated " + chain.getName() + " record: " + record
                 + " pos: " + pos + "action: " + action);
     }
}
class CustomPublishListener implements MkvPublishListener {
     public void onPublish(MkvObject object, boolean start, boolean dwl) {
           if (object.getMkvObjectType() == MkvObjectType.CHAIN
                 && object.getName().equals("MY_CHAIN") && start) {
                 MkvChain mychain = (MkvChain) object;
                 System.out.println("Published " + mychain.getName());
                 try {
           ////// the new() creates a listener just like swing ActionListeners
                       mychain.subscribe(new CustomChainListener());
                 } catch (MkvConnectionException e) {}
           }
     }
     public void onPublishIdle(String component, boolean start) {
     }
     public void onSubscribe(MkvObject obj) {
     }
}
..
Mkv.getInstance().getPublishManager().addPublishListener(new CustomPublishListener());

common quote pricing rules + live feeds for munis

Rule: (same as FX day trader, and applicable to market making only) based on inventory changes
Rule: dollar cost averaging
Rule: based on BMA index, which is typically 70% of Libor, assuming a 30% tax exemption
Rule: on the muni desk, the swap positions and quotes are priced using SIFMA swap index and ED futures
Rule: on the muni desk, the T (and futures) positions and quotes are priced using T/T-futures price feed
Rule: based on ETF prices. If one of our quoted bonds is part of one (or several) ETF, we price our bid/ask using live ETF prices
Rule: based on Evaluation prices, received twice daily (JJ Kenny and InteractiveData)
Rule (one of the most important): based on MSRB last-traded quantity/price reported to MSRB
Rule: based on “pins” controlled by senior traders of the desk
Rule: based on The One Muni Curve of the trading desk
Rule: stock (hedging) positions (not quotes) are priced using stock price feeds

Other feeds – ION, SBA (SmallBizAdmin) feed

“Based on” generally
Always apply a bid/ask spread matrix (call dates/quantity as x/y dimensions)
Always apply a commission matrix
Apply odd lot discount matrix
Always convert from clean to dirty price
Always check for occurrence of bid/ask inversion

##what mkt-data feed your autoReo pricing engine@@

background — every algo pricing/trading engine needs market data feeds.

For my algo engine,
– Libor, ED futures
– Treasury futures
– Treasury ticks
– Bloomberg
– credit rating updates
– material event feed
– ION mkt view
– JJ Kenny
– MSRB, NASD TRACE

There are a few essential INTERNAL feeds to my algo engines —
– positions,
– marks by other desks,  and other algo engines.