low-latency jobs; learning python#CSY

I believe my current trading system is latency-sensitive (as a large-scale [1] order management engine), but on this job I don’t do anything about latency — this is just another equity trading system job. I feel you don’t need to aim for low latency. Just aim for any c++ job that’s not off-putting to you.

[1] I consider it large scale because it has probably 10-20 developers working on it, for at least 5 years.

Low-latency jobs are harder to get, and they usually prefer developers in their 30’s or younger.

I don’t exactly avoid such jobs, but I don’t hold up any serious expectation of an offer by those teams. The main barrier of entry is technical capabilities, either coding test or obscure c++/linux questions. Even if I’m lucky to hit familiar, well-prepared tech questions, they may still find reasons to take a pass, as I experienced in my technical wins at SIG, Bloomberg etc.

Java and python are never show-stoppers for these jobs, so
* if you only aim for low-latency jobs, then don’t worry about python or java
* if you widen your search scope, then I suggest you pick up python, not java

Taking a step back to get a perspective, I think job seekers are like smartphone makers — we need to adjust our “product” for the changing customer taste. The “product” in this case is our testable tech skills. Domain knowledge is losing importance; Python is now in-demand; coding tests are growing harder; Linux/compiler system knowledge has always been important to low-latency interviews .. so we need to decide how to adjust our “product” to attract our customers.

—- Earlier email —-

How is your python practice?

This is a personal journey. Therefore some people don’t like to discuss in details how they are learning something new. So feel free to disclose any amount of information you feel comfortable with.

I advocated solving coding problems in python. I now realize both you and me don’t want to solve every problem. Out of 10 problems, we might solve 1 or 2 in real code. So in the past 4 weeks, perhaps you didn’t look at 10 problems so the number of problems you could solve in python might be very low. Therefore, my suggestion may not work for you.

In that case, I wonder what python coding experiments you prefer.

I once said python is easier to learn, but like learning any new language, it still demands a huge commitment, sustained focus, and personal sacrifice. Therefore, it helps greatly if there’s a python project on a day job. Without it, we need self-discipline, determination, and clear targets to sustain the focus and energy. None of these is really “easy”.

Talking about clear targets, one example is “solve one coding problem of medium complexity each week, in python”.


HFT developers seldom need to optimize latency

I bet 90% of HFT developers (non-architects) Never need to optimize latency or throughput, based on my experience in

  • RTS
  • mvea

Why? The latency sensitive codebase is low-level and stable, so no change required. Regular developers only need to use existing frameworks and avoid latency penalties.

If the frameworks are strict, then there are few chances to hit latency penalty.

Best way to acquire professional experience in latency engineering — put your ideas to work in a real project and tune it until it works. Look at my wait/notify code in 95G. I think Kam did something similar in the TREP project in RTS.

If you don’t have such an opportunity, then you must read up in your spare time and ask some good questions in a good contexx.

SOR/OMS/.. skills !! upstream

See %%algo trading dev xp !! HFT

SmartOrderRouting, OrderMgmt, FIX connectivity .. are core skills in algo trading dev, but are they really upstream, advanced skills? I doubt it.

  • in terms of interviews, I feel the domain nlg (jargon or architecture) required is a thin layer. The technical screening seems to focus on generic tech skills esp. coding test.
  • In terms of technical know-how (zbs), such a developer will NOT become stronger because of this experience. If she does acquire portable tech knowledge (possibly with an industry standard API rather than local wrappers), then she is probably experienced in that technical domain, but not other domains.

algo trading developers .. salary premium@@

Hi Friends,

For years I believed algo trading (including HFT) roles pay some 10-30% higher than regular financial IT roles. Now I doubt it.

For the sake of argument, let’s say the market rate for a senior developer is 160k base + some bonus + some stocks.

  • Factor: elitism and selectivity — buy-side algo trading jobs are notoriously hard to get. I tried about 10 places (Hudson River, Mako, SquarePoint, Gelber, DRW, Jump, TrexQuant, Susquehanna, WorldQuant, Millennium, 3-arrows…) Even if salary is higher, the chance of getting it is extremely low, at least for me.
  • Factor: role expectation and workload — I don’t have experience working at buy-side algo trading shops, but i do have experience in other demanding teams. I didn’t do so well, so I know the risk of sky-high expectation is very real. So even if salary is higher, how long can we keep it?
  • Factor: where is the intelligence — virtually all the algo-trading engineer roles I have seen emphasize low-latency rather than high-frequency. High-frequency is not really a job description for an engineer — developers optimize the low-latency system, to enable high frequency trading. The developers’ skill and intelligence required is substantial, but it’s not the same intelligence portrayed in mass media — which is trading algorithm or “alpha”. That intelligence is required only for the quants, the strategists, the portfolio managers, the traders… They are paid higher than most engineers. I guess some would say the engineers play a supporting role.
  • Factor: architect level — the high salaries (say 30% above market rate) are often at the architect (or lead developer) level. At that level, a regular financial IT job also pays above market rate. If we compare salaries at architect level, then algo trading doesn’t pay such a big premium. Perhaps 10%? Actually I’m not an architect so this level is not really relevant. I’m more comfortable as a senior developer.

For a regular senior developer, I feel algo trading roles on sell-side pays no higher than average financial IT. I know it from multiple interviews (BNP, HSBC, Citi …)

For a regular senior developer on a buy-side algo-trading system, I guess it can pay 10-20% above market rate, up to 200k base (?), but I also know of many other financial IT jobs in the U.S. paying 180k to 200k base.

Update — SIG can pays 160k for mkt-data or SOR roles but too selective.

My tentative conclusion:
* algo-trading on sell-side doesn’t pay a premium
* algo-trading on buy-side is a high-end domain, but there are other high-end domains paying at a comparable level.

%%algo trading dev xp !! HFT

Let’s not try to look like a HFT pro, citing your low-speed algo trading xp…. You could look desperate and clueless.

My list experiences don’t guarantee success, because devil is in the details (Just look at all the details in xtap retransmission…) However, I should feel more confident than an outsider.

  • [95G/Citi] OMS — In Citi, I worked on automatic quote cancellation and republish. Most executions involve partial fill and subsequent quote (limit order) reduction, similar to OMS. Citi and Baml were probably 2 biggest muni houses in the world. Baml system also handles corporate bonds.
  • [Citi] real time even-driven (tick-driven, curve driving…) quote repricing
  • [Citi] generating and updating limit orders in response to market data changes. Need to brush up on the basic principles since I was asked to elaborate, but a developer probably doesn’t need thorough understanding.
  • [95G] OMS — low-speed, low volume (bond) order management using very few FIX messages with trading venues to manage order state. I actually wrote my own wait/notify framework. This is probably the most valuable algo-trading project I have worked on.
  • [OCBC] simple automated option quote pricer in response to client RFQ
  • [95G] FIX messaging — up to 6-leg per order. Another ECN uses NewOrderSingle and Cancellation messages.
  • [RTS] FIX feeds — with heartbeat, logon etc
  • [NYSE] proprietary protocol is arguably harder than FIX
  • [NYSE] high volume, fault-tolerant raw market data draining at 370,000 messages/sec per thread
  • [NYSE] order book replication — based on incremental messages. I also implemented smaller order books from scratch, twice in coding interviews. This is directly relevant to algo trading
  • [NYSE] connection recovery, under very high messaging rate. Devil in the details.
  • SOR — no direct experience, but Citi AutoReo system has non-trivial logic for various conduits.
  • [Barclays] converting raw market data into soft market data such as curves and surfaces, essential to algo trading in bonds, FX-forwards, equity options.
  • [Stirt] real time ticking risk, useful to some traders if reliable
  • home-made FIX server/client
  • [Citi/Stirt] real time trade blotter — i added some new features
  • [Citi] very limited experience supporting an ETF algo trading system
  • [UChicago] school project — pair trading

However I feel these experiences are seen by hiring managers as insufficient. What are the gaps you see between my experience and the essential skills?

? limited OMS experience
? latency optimization
? network optimization

OMS skill !! standardized !! portable

I now feel the glorified OMS know-how is vague and poorly-standardized.

  • My B2bTradingEngine at 95G does a lot of OMS.
  • Smart order router module often does some OMS.
  • FIX connectivity module often does some OMS.
  • VWAP and other execution algos are usually considered a category of OMS.
  • … Many of the above are very different skills and have almost nothing in common.

I am afraid that even after 3Y or 5Y in some OMS system, when I apply to another OMS job they may realize I’m not a veteran.

Q: In comparison, which domain is more standardized in terms of skillset?
A: raw mkt data. Note 2nd-hand market data processing is much less valuable
A: basic bond math. Note more advanced bond math is rarely needed
A: FIX connectivity

  • How about VaR?
  • quote distribution systems in Citi and OC? poorly standardized

## execution algo + other domains / skills

OMS — is probably a core module in execution system. Equity and FX systems tend to need OMS but many trading desks presumably need no OMS, very simple OMS or off-the-shelf OMS. A buy-side trading desk may simply use the OMS of the broker. The same could be said of “execution systems” like VWAP algos.

Therefore, I feel the importance of OMS/EMS is over-stated.

SOR — is more niche, less “generic” a skill, but as a module is more visible to the business.

FIX connectivity — is a more generic tech skill and shows resilience to churn.

mkt data — is more “separate” from the other skills.

market-depth^elite domains esp. algoTrading

I used to dismiss “commodity” skills like market data, risk system, J2EE… I used to prefer high-end specializations like algo-trading, quant-dev, derivative pricers.

As I get older, it makes sense to prefer market depth rather than “elite”(high-end niche) domains. A job market with depth (eg market-data) offers a large number of positions. The typical salary of top 10% vs the median are not very different — small gaps. In contrast, the elite domains feature bigger gaps. As I grow older, I may need to reconsider the specialist vs generalist-manager choices.

Reminders about this preference (See also the spreadsheet):

  1. stagnation in my orgradient
  2. may or may not use my specialist skills in math, concurrency, algorithms, or SQL …
  3. robust demand
  4. low churn — a critical criteria whenever I mention “market depth”. I don’t like the market depth of javascript and web java.
  5. salary probabilities(distro): mgr^NBA#marketDepth etc

–case study: Algo trading domain

The skillset overlap between HFT vs other algo systems (sell-side, OTC, RFQ, automated pricing/execution..) is questionable. So is “accumulation” across the boundary.  There seems to be a formidable “dragon gate” — 鲤鱼跳龙门.

Within c++ based HFT, accumulation is conceivable. Job pool is so small that I worry about market depth. My friend Shanyou agreed that most of the technical requirement is latency. C/C++ latency techniques are different from java.

However, HFT developers seldom need to optimize latency

Outside HFT, the level of sophistication and latency-sensitivity varies. Given the vague definition, there are many (mostly java) jobs related to algo trading i.e. better market depth. Demand is more robust. Less elitist.

## threading: a better specialization than algo,quant…

  • In summary — appears daunting and opaque but actually simple in practice
  • theoretical — my strength. Few coding tests and usually easy for me
  • fairly low-level — my strength. Not as low level as debugger, or template hacks …
  • GTD — no GTD challenges, since most projects use only simple threading designs. The code tracing, trouble-shooting expectation on me is relatively low.
  • opaque — for my peers. Even the basic condVar…
  • churn — low churn at the low level, but high churn at high level
  • ever-green — favorite topic of interviewers, esp. java
  • thin->thick->thin — I have achieved this for a long time.

Many of my halos are in this domain — ##halo across domains #specific=better

algo trading engine from scratch: ez route hopefully

Update: TrexQuant had an extremely simple trading system, mostly an FIX connectivity engine…

Start by identifying some high-quality, flexible, working code base that’s as close to our requirement as possible. Then slowly add X) business features + Y) optimizations (on throughput, latency etc.) I feel [Y] is harder than [X], thought [X] gives higher business value. Latency tuning is seldom high-value, but data volume could be a show-stopper.

Both X/Y enhancements could benefit from the trusty old SQL or an in-memory data store[1]. We can also introduce MOM. These are mature tools to help X+Y. [3]

As I told many peers, my priorities as architect are 1) instrumentation 2) transparent languages 3) product maturity

GTD Show-stopper: data rate overflow. Already addressed
GTD Show-stopper: frequent[4] crashes. Unlikely to happen if you start with a mature working code base. Roll back to last-working version and retest incrementally. Sometimes the crash is intermittent and hard to reproduce 😦 Good luck with those.

To blast through the stone walls, you need power tools like instrumentation, debuggers … I feel these are more important to GTD than optimization skills.

To optimize, you can also introduce memory manager such as the ring buffer and custom allocator in TP, or the custom malloc() in Facebook. If performance doesn’t improve, just roll back as in Rebus.

For backend, there are many high or low cost products, so they are effectively out of scope, including things like EOD PnL, position management, risk management, reporting. Ironically, many products in these domains advertise themselves as “trading platforms”. In contrast, what I consider in-scope would be algo executor, OMS[2], market data engine [2], real time PnL.

— The “easy route” above is probably an over-simplification, but architects must be cautiously optimistic to survive the inevitable onslaught of adversities and setbacks —

It’s possible that such a design gradually becomes outdated like GMDS or the Perl codebase in PWM-commissions, but that happens to many architects, often for no fault of their own. The better architects may start with a more future-proof design, but more likely, the stronger architects are better at adjusting both the legacy design + new requirements

Ultimately, you are benchmarked against your peers in terms of how fast you figure things out and GTD….

Socket tuning? Might be required to cope with data rate. Latency is seldom a hard requirement.

Threading? single-threaded model is probably best. Multiple processes rather than multiple threads.

Shared memory? Even though shared memory is the fastest way to move data between processes, the high-performance and high-throughput ticket plant uses TCP/Multicast instead.

MOM? for high-speed market data gateway, many banks use MOM because it’s simpler and flexible.

Inter-process data encoding? RTS uses a single simplified FIX-like, monolithic format “CTF”. There are thousands of token types defined in a “master” data dictionary — semi-static data.

GUI for trader? I would think HTML+javascript is most popular and quick. For a barebones trading engine, the GUI is fairly simple.

Scheduled tasks? Are less common in high speed trading engines and seldom latency-sensitive. I would rely on database or java/c++ async timers. For the batch tasks, I would use scripts/cron.

Testing? I would use scripts as much as possible.

[1] eg: GMDS architect chose memory-mapped-file which was the wrong choice. [2] both require an exchange interface
[3] data store is a must; MOM is optional;
[4]If it crashes once a day we could still cope. Most trading engines can shut down when market closed.


Update: I told bbg (Karin?) and Trex interviewers that domain isn’t a big concern to me. Even a back office IT role can turn out to be better.

  1. quantDev
  2. algoTrading
  3. bigData

… are the 3 big directions for trySomethingNew. I’m cautious about each.

quantDev (not pure quant) — low demand; poor market depth; unable to find another job; least jobs and only in banks. CVA is not really quant dev, based on what I gathered. See also %% poor accu ] quantDev

algoTrading — perhaps I should try java and medium frequency?

bigData — no consolidation; questionable accumulation and value-creation

quant^HFT^WestCoast, again

After I felt fairly established on Wall St (around 2011), I looked for greener pastures:

  1. quant dev
  2. HFT
  3. high-end positions in the west coast. Not sure what positions exactly — mobile? machine learning? cloud?
  4. regular Wall St c++  job

In fact my java jobs on Wall St is not that inferior, and has the advantage of reachability. In contrast, those greener pastures still look rather distant. They look closer when I’m in a positive mood. But let’s put on the black hat and be critical and skeptical:

Quant dev is low-churn but demand is kinda shrinking.

HFT is rather niche skillset, possibly less relevant to west coast than java and regular c++.

1) 2) 3) are mostly FTE, not contracts. Regular c++ is contract-friendly and more reachable.

I used to feel the quant domain is hardest. Now I feel it’s shrinking. Now I feel I’m in much better shape. I made the decision to focus on quant early, assuming I could self-study and break into HFT later.

Data science? Math is much easier than quant finance, and I have some training in it.

C++? I now have more hands-on experience

[15]essential GTD/zbs to build algo trading system


(See also http://bigblog.tanbin.com/ post on HFT)

Just to share some observations and reflections. More than one Asia (and to a lesser extent US) recruiters have reached out to me as a potential lead-developer for a HFT engine, to be created from scratch. I believe there are not many old hands in Singapore. Even in the US, this is a relatively small circle. Not a commodity skill.

A small trading shop would have very different needs than a big bank, so their HFT engine will use off-the-shelf tools for all but the most critical, customized modules. (I had a brief blog post on it.) What are the 10 essential know-how i.e. Essential functionalities you must know how to create (prove)?

  • see also my 2018 post on execution system^OMS^FIX…
  • executing strategy for order origination, i.e. machine trading
    • market data processor, sockets
    • order book replicated from market data? perhaps at the center of the engine
    • OMS (Kenny) ? in some cases we leverage the exchange query API or 3rd-party solutions
  • [W] FIX or other order submission protocol
  • in-sample testing. I used Matlab for this. Can be much simpler than the production system.
  • [3] real time pnl, updated by market data
  • [3] barebones blotter, or perhaps command line interface?
  • [3] store historical data for analysis. Perhaps SQL or KDB.
  • [3N] reference data
  • [3] post-trade STP

Low-level tech knowledge
• [N] threading
• [NW] Boost
• debugger
• [N] memory leak detection
• [W] socket programming

[3 = 3rd party products could be good enough in Every component, but this one is more likely]
[N = Not needed in every shop, but often required by interviewer]
[W = my weakness, relatively speaking]

important IV/GTD skills for algo trading developer

Traditional (“real”) algo trading is still confined to buy side. (Sell side algo trading includes things like RFQ auto pricer, limit order auto pricer etc. Dr. Chou points out execution cost reduction algos…) Such shops look for the “strategist” role which is often (usually?) a different role from the implementation engineer’s role. Allow me to focus on the latter.

In my opinion, the really important skills are all about optimization, latency, performance Engineering

#1 [IV] algorithm/data structure optimization
#2 instrumentation (incl. debugging) — profilers, memory leak detector
# [IV] threading — for multiprocessor
# network/socket programming, including NIO
# fast messaging – like RV or solace
# memory management (c++), GC tweaking (java)
# OS tuning including memory, net and CPU
# cache products – like coherence
# exchange conn beyond FIX – understand its idiosyncrasies

Other skills are important to the team but less important to the engineer
– market data gateway
– back testing
– architectures
– OO designs
– automated pricing. No need to understand BS or bond math, or hedging techniques
– basic trading algorithms
– scripting for system automation
– statistics software