churn !! bad ] mktData #socket,FIX,.. unexpected!

I feel the technology churn is remarkably low.

New low-level latency techniques are coming up frequently, but these topics are actually “shallow” and low complexity to the app developer.

  • epoll replacing select()? yes churn, but much less tragic than the stories with swing, perl, structs
  • most of the interview topics are unchanging
  • concurrency? not always needed. If needed, then often fairly simple.

c++ salaries: high-end^regular

I spoke with 5 to 10 fellow c++ developers in financial domain across a few countries. All of them seem to suggest c++ skill has higher market value than any other language like java, c# or python. One common reason they all gave is the high frequency trading jobs. My own observations also show that most HFT tech jobs use c++ and very few jobs are java or c#.

However, how many of us can ever pass the HFT interviews?

I have tried a few high-frequency or medium-frequency algo-trading jobs using c++. I didn’t pass any. Among my friends, only one person passed. He passed more than one such interviews and he’s clearly very strong.

So I feel the reality is, most of us won’t be able to get those jobs, or keep the jobs.

Q: how many percent of the c++ jobs in finance are HFT ?

I would guess 5 to 15%. So the vast majority of c++ salaries are ordinary. I think many of us can get those jobs:)

What’s your opinion?

Wall St tech projects/timelines #XR

I have heard several people complaining about the unhealthy work culture in NY investment bank IT teams.

  • · Timeline is artificially tight.
  • · A typical senior manager is motivated by trophy systems to win him promotions + bonuses, rather than really useful systems making a real impact to business
  • · Lots of systems are developed but not used or used for superficial purposes only. (This is common in many systems. Only 1% of the features are actually put to productive use.)
  • · Low quality code; low quality output data; poor quality control
  • · Constant and wide-spread pressure to retire old systems and rewrite, when the old system is still usable and sometimes rather stable.
  • · Senseless requirements, questionable value, thrown-away systems

I heard that West coast shops, Chicago buy-side and other companies don’t have exactly the same problems. I feel my current company is also different from NY banks — Older technology; Slight higher quality standard; More stable codebase; Users are mostly external paying customers…

I guess the Wall St culture is “good” for some, while other developers hate it and exit. I’m planning to stick to finance but not banks.

tiny team of elite developers

Upshot — value-creation per head and salary level would rival the high-flying manager roles.

Imagine a highly successful trading shop. Even though the trading profit (tens of millions) is comparable to a bank trading desk with hundreds of IT head count, this trading shop’s core dev team *probably* have a few (up to a few dozen) core developers + some support teams [1] such as QA team, data team, operations team. In contrast, the big bank probably have hundreds of “core” developers.

[1] Sometimes the core teams decide to take on such “peripheral” tasks as market data processing/storage, back testing, network/server set-up if these are deemed central to their value-add.

In the extreme case, I guess a trading shop with tens of millions of profit can make do with a handful of developers. They want just a few top geeks. The resultant efficiency is staggering. I can only imagine what personal qualities they want:

* code reading — my weakness
* tools – * manuals — reading tons of tech info (official or community) very quickly, when a “new” system invariably behave strangely
* local system knowledge
* trouble-shooting — and systematic problem-solving. I feel this largely depends on system knowledge.
* design — it right, and able to adjust it as requirements change * architecture?
* tuning?
* algorithms?

(soft skills:)
* clearly communicate design trade-offs in a difficult discussion * drive to get things done under pressure without cutting corners * teamwork — teamwork to back down when needed to implement a team decision

rookie coders vs old hands, my recent xp]fin IT

– Anyone (even a non-pianist) could teach piano to a kid, but perhaps not correctly;
– Anyone could teach swimming, but Qiu Yi has insight;
– I would guess any medical undergrad could treat patients, guided by a textbook – Management consultants — These guys go by case studies. Not experts in my industry. Why would I listen to them?
– Application design, low or high level, is an art. An inexperienced guy could go by textbooks, but actually, fine judgement needed at every turn.

In Stirt and Macquarie, how fast you get things done is not as important as getting things done right. Managers here are very particular about the details. I think they want things done right and stick to that long term. I can only guess they have seen hasty code that didn’t last, that required massive rework. Example
* the 2 earlier Error Memos coders
* Varun
* Some of the code in CAOS (account opening)

This is one reason they want older guys with deep experience and strategic insight.

## which skill/field CAN be self-taught

Domain knowledge is easier to “fake” than c#/c++. It’s hard to fake the mileage…

Q: which fields can be truly self-taught to in-depth? For example, mobile apps, desktop apps.

Goal #1: pass job interviews
Goal #2: reach the professional level of Theoretical expertise
Goal #3: reach the professional level of Practical expertise

* Quant? too dry. Too many doubts to be clarified only by professor. However the basic math part is standard and well-defined. Goal 3 is invalid.
* high-frequency, algo? Many books but none on real tricks. Compare swing books! Very few jobs.
* network/linux optimization? need machine to try. Extreme optimization techniques are top secrets.
* FIX? Many practical issues hard to imagine. No focus.

* swing, wpf? — tricks can be self-taught through experiment. Practical books are available. But practical problems are unknown.
* c# and core libraries? Real obstacle is the IDE. Practical books abound.
risk-mgmt? impossible to experiment.
coherence? too many subtopics. no focus. Hard to experiment.
tibco? no book at all. Hard to experiment
threading? Can hit Goal 2, not 3

mediocre Wall St developers earning Wall-St-level salary


A paradox to share with you. For years I often wonder

Q1: why does wall street (a shorthand for “world financial centers”) pay something like double the salary compared to main street, for a senior programmer? (Disclaimer — Wall St is not the only sector; premier tech firms also pay well above main street.) On many teams including my current team, I see people doing main street programming work, using main street technical skills without any financial domain knowledge. Why pay us so much higher than main street?

I feel this is a non-trivial paradox. A deeper understanding helps us manage our career, manage our competitiveness, manage our job security, manage our workload, manage our value-add.

In this letter I use “ibank” as a shorthand for all the employers in the investment-banking and security trading business, sometimes including buy-side. Yes I’m deliberately vague and imprecise.

Factor 1: ibanks have higher margin than main street, even after 2008, and crucially a small percentage of employees are responsible for the profit. Many ibank jobs are, to be honest, not critical to the money-making machinery, but many developer jobs are. To put things into perspective, traders, fund managers, deal-makers, advisors/salesmen … are in the profit center and even more instrumental to the money-making machinery. The closer to the money, the higher is one’s value to the ibank. Highest paid wall street (non-manager) tech roles are probably the high-frequency low-latency experts because they are closest to the money. Their code has a most direct impact on the competitiveness of our trading engine in the arms race. Similarly, our pricing system calculates the prices sent directly to the market and have a direct impact on our competitiveness.

Therefore, many senior programmers are considered instrumental/important for the high margin of an ibank.

Factor 2: these tasks are often fairly complex and easy to mess up, and therefore calls for expertise, backed by track record. (To be fair, these are not the most complex.) For comparison again look at the most important roles in ibank profit centers. There’s usually a lot of decision-making, a huge flood of information to digest. An ibank can hire fresh graduates to create trading applications, but they are likely to fail to meet some of the requirements. In and outside Wall St, software dev is knowledge-intensive, methodical, intricate, even fragile [1], unlike many other jobs in finance.

[1] sensitive to human mistakes. That’s why we say 99.9% of software have bugs.

Therefore ibanks generally recognize in-house app dev as high-value (sometimes instrumental/enabling) and complex. I have heard many times “we can’t introduce this new instrument to the market without the IT system”.

Factor 3: Just a small number of experienced coders are needed. In main street, we see armies of developers for a given project. (Lee Chong Wah may disagree.) On Wall St, team size is cut to a third or a quarter. In so many ibanks I have worked, I always see a lot of senior developers and few junior developers. I feel OCBC (and other traditional banks) see developers as the same “armies”.

Therefore ibanks have high profit, and want to use part of the profit to attract a small number of elite developers. This is a “perfect storm” for higher salary. Now, in reality, a team of 3 to 7 senior developers would have just 1 key developer, so the other guys are not that critical. Perhaps one of them is doing nothing special at all. Low-value, low-importance, mundane code, but still paid well above main street. (I might be one of them now.)

Factor 4: During recruitment, ibanks typically have an adequate budget for a senior developer. If market rate is 140k, and their budget is 90k then they eventually hit the reality that strong candidates (with track record) won’t be interested, so they end up scraping the bottom of the barrel. Mostly tough and unsuccessful.

So managers go get a sufficient budget, screen, select and offer to this fairly strong candidate, but for various reasons he doesn’t get to play that key role. He may even end up with some truly unimportant modules, somewhat over qualified and overpaid. Managers can’t just drop him, but can leave his contract (if contractor) to expire without renewing. If not a contractor, then bonus is in question. If the guy is actually good but the mundane job must be done in the team, then maybe unfair to mistreat him….

That’s my answer to the opening question Q1. But there’s another question, because ibank managers are not stupid….

Q2: For the non-critical roles, why don’t they hire strong main street developers and pay them main street salary, to do whatever main street work there is in the project? No track record needed.

I feel many enlightened and bold ibank managers actually do that, but there’s a risk of uncertainty in following that route. Instead, most managers much prefer candidates with proven track record in a similar project. Also, recruiters are reluctant to submit profiles without relevant experience. Therefore, main street developers are largely excluded from the talent pool.

High-speed vs high-complexity financial products

I now see 2 broad categories of financial products–

A) Analytically complex instruments — CDS, IRS, structured, IR-linked, index-linked(?), MBS, swaptions, embedded options… Usually derivatives.
** You often need sophisticated valuation engines for pre-trade quoting and/or risk
** If such a product (say bond with option) is liquid then you may get tight bid/ask and need no sophisticated valuation engine. However, I feel most of these instruments have large spread.

B) high speed, high volume markets – stocks, major indices, FX spot, ED futures, Treasury,
** These are by far the most heavily traded and prominent products, often with low (profit) margins

Many popular or dominant instruments do not fall into either of these — vanilla bonds, vanilla options, FX options, FX futures, FX fwd, VIX

Any product falling into both categories? I don’t think so.

Products in (A) often give practitioners a sense of job security due to the specialist knowledge required. However, I usually stay clear of exotics. I tend to feel INsecure away from the mainstream.

Some System developers in (B) may have no real product-specific insight. They probably do not analyze data — historical data; or summarize (to make sense of) large volumns of data; or fundamental analysis; or statistical analysis