dev jobs ] Citi SG+NY #Yifei

My friend Yifei spent 6+ years in ICG (i.e. the investment banking arm) of Citi Singapore.

  • Over 6Y no layoff. Stability is Yifei’s #1 remark
  • Some old timers stay for 10+ years and have no portable skill. This is common in many ibanks.
  • Commute? Mostly in Changi Biz Park, not in Asia Square
  • Low bonus, mostly below 1M
  • VP within 6 years is unheard-of for a fresh grad

I feel Citi is rather profitable and not extremely inefficient, just less efficient than other ibanks.

Overall, I have a warm feeling towards Citi and I wish it would survive and thrive. It offers good work-life balance, much better than GS, ML, LB etc

conclusions: mvea xp

Is there an oth risk? Comparable to MSFM, my perception of the whole experience shapes my outlook and future decision.

  • Not much positive feedback beside ‘providing new, different viewpoints’, but Josh doesn’t give positive feedback anyway
  • should be able to come back to MS unless very stringent requirement
  • Josh might remember Victor as more suitable for greenfield projects.
  • I think Josh likes me as a person and understands my priorities. I did give him 4W notice and he appreciated.
  • I didn’t get the so-called “big picture” that Josh probably valued. Therefore I was unable to “support the floor” when team is out. The last time I achieved that was in Macq.
  • work ethic — A few times I worked hard and made personal sacrifices. Josh noticed.
  • In the final month, I saw myself as fairly efficient to wrap up my final projects including the “serialization” listed below

Q: I was brought in as a seasoned c++ old hand. Did I live up to that image? Note I never promised to be an expert
A: I think my language knowledge (zbs, beyond QQ) was sound
A: my tool chain GTD knowledge was as limited as other old hands.

Q: was the mvea c++ codebase too big for me?
A: No, given my projects are always localized. and there were a few old hands to help me out.

I had a few proud deliveries where I had some impetus to capture the momentum (camp out). Listed below. I think colleagues were impressed to some extent even though other people probably achieved more. Well, I don’t need to compare with those and feel belittled.

This analysis revealed that Josh is not easily impressed.  Perhaps he has high standard as he never praised anyone openly.

  • * I identified two stateless calc engines in pspc. Seeing the elegant simplicity in the design, I quickly zoomed in, stepped over and documented the internal logic and replicated it in spreadsheet.
  • * my pspc avg price sheet successfully replicated a prod “issue”, shedding light into a hitherto murky part of the codebase
  • * I quickly figure out serialization as root cause of the outage, probably within a day.
  • * I had two brave attempts to introduce my QOT innovation
  • * My 5.1 Brazil pspc project was the biggest config project to date. I single-handedly overcame many compilation (gm-install) and startup errors. In particular, I didn’t leave the project half-cooked, even though I had the right to do so.
  • I make small contributions to python test set-up

buy-side interviewer’s criteria #DQH

Some buy-side tech interviewers have two bars:

  • The low bar – Can this candidate do the job? Some 80% of rejected candidates can do the job, at least in theory.
  • The high bar – can this candidate add value to the team, surpassing current team average standard in at least one tech domain? I feel this high bar is mostly QQ — theoretical and low-level, to my delight.

Out of 100 candidates, if 5 clear the high bar but all become unavailable, then the position may stay vacant for months and years, even though 80 of them clear the low bar.

The hiring team would allocate some man-hours every month to recruitment. Some of these highly paid engineers would take the interview as a relatively relaxed session, and she may learn something from the interaction with the candidates.

Above is the buy-side hiring culture. On the opposite end of the spectrum is WallSt contractor hiring. High bar is very close to the low bar, more like “out of 30 resumes, and 10 onsite candidates, we will make 2 offers.”

G9 asset classes,by dev-job mkt depth

Beware many big domains don’t need lots of developers.

  1. Bonds including sovereign
  2. [V] Eq (including ETF) cash and swap
  3. [V] FX cash and fwd
  4. Eq/FX options
  5. [Q] IRS
  6. IR (including bond) futures
  7. [Q] CDS
  8. [Q] MBS
  9. [d=will see higher demand for developers??]
  10. [V=volume and velocity drive the demand for developers]
  11. [Q=low volume but j4 automation is quantitative in terms of automated risk and pricing.] I believe these quantitative asset classes play to my “theoretical” strength and not too niche, but these domains aren’t growing.

y some backoffice tech jobs=more stressful than FO

Paradox. Here are some answers from colleagues

  • Reason: a front office system can be very stable and mature
  • Reason: a front office system’s business logic can be fairly simple. An outsider would think business functions A/B/C/D should be part of this FO system, but in reality, they are are offloaded to other (FO or MO) systems.
  • Reason: a FO team manager can have good control on the requirements given by business and external teams.
  • Reason: even though front office users can be demanding and want immediate answers, good localSys knowledge can help you find those immediate answers. Back office users can also be demanding.


Most c++guys have no insight in STL

When I recall my STL interviews on Wall St, it’s clear that majority of c++ app developers use STL almost like a black box, every year for 5-10 years. Only 1% has the level of insight as Henry Wu.

Reason? such insight is never needed on the job. This factor also explains my advantage in QQ interviews.

  • Analogy — Most of us live in a house without civil engineering insight.

STL uses many TMP and memory techniques. Some may say STL is simple but I doubt they have even scratched surface on any of the advanced topics.

  • Analogy — some may say VWAP execution algo is simple.

Many interview questions drill in on one or two essential STL functionality (which everyone would have used), just below the surface. These questions filter out majority[1] of candidates. Therein I see an opportunity — You can pick one or two topics you like, and grow an edge and a halo, just as Henry did. Intellectual curiosity vs intellectual laziness. 不求甚解,不加咀嚼,囫囵吞枣 — I see it in many self-taught developers. That’s exactly what these Wall St interview questions are designed to identify and screen out.

How about west coast and other high-end tech interviews? I’m not sure.

[1] sometimes 90%, sometimes 60%.


attitude of a student #Josh

The background was localSys, focus@GTD i.e. productivity ….

In my final catch-up, I singled out distraction as the first factor. It was something easy to identify. I was distracted by interview preparation, nutritional research, housing market research, …. but that distraction was not a key factor.

I said another side factor was raw memory capacity — lots of facts to remember and other people seem to show better retention.

I also said I didn’t get a project where I could exceed expectation and gain momentum.

Boss suggested that I lost motivation. Then he suggested — “Take the attitude of a student. Be willing to learn and to take on anything.”

By “student” he didn’t mean a fresh grad or an intern, a desire for new knowledge.

ROTI@learning ED/IRS/FX-swap/repo #math-lite

Note this effort is after my basic bond math study, though I often count this effort as part of the broader “bond math” study.

Basic bond-math knowledge has robust demand on Wall St. Without hard evidence I feel ROTI is decent in basic bond math study. Q1: How is the ROTI in this study?

I feel many of the jargon terms in this space are common and essential knowledge:)

  • swap rate; comparative advantage;
  • OIS; Libor;
  • basis risk;
  • collateral;
  • curve building

However, this self-study rarely helped me:

  • MSFM course
  • Stirt job interview

Q1b: How is the market depth and robust demand of this skill?
A: not used much in the trading buy-side, but some asset management and most sell-side do need this know-how.

Note this topic is generally math-lite and much simpler than option math, so I was able to self-study:) See fixation@ROTI…dampens job satisfaction+joy@learning

Q2: how is the knowledge retention rate?
A2: decent. Thin->thick yes but not yet thick->thin

Risk analytics dev, hopefully !! another broken dream#AshS

Update: many developers seem to suggest that any risk analytics experience would not help me move into “front office”, though I’m not so hung up about front office.

Many people also point out budget is typically lower in back office.

Hi Ashish,

Thanks for your input. I feel bad about repeated failed attempts to break into a few quantitative domains (listed below) so I need to consider some negative what-ifs. What if the risk system is similarly “underwhelming” like:

* volatility surface fitter in Barclays
* curve building and real time “ticking” risk sensitivity in Baml
* quant library integration at Macquarie

* equity and FX volatility product pricing in OCBC
* (I guess your Barclays mortgage system could be another quant domain?)

Armed with my formal education and analytical competence, I have tried valiantly to find a foothold in a quant dev domain. I was unable to dig in my heels.

I will not go into details on these “broken dreams”. Suffice to say that I had enough of these … that recently I decided to walk away from a higher-paying ($120/hr) contract because it looks like another system with some complicated financial math.

I don’t want to give up completely on my (large) investment in quant study. This is the source of one of the biggest pains in my career. As I said, I am also cautious about throwing good money after bad money…

Thanks for pointing out the “infrastructure” part — I can imagine that like your friends, most tech guys in risk systems are only responsible for the infrastructure. In some banks, there’s an army of developers supporting the risk systems, so the infrastructure is presumably quite complex and large scale. Presumably, if a developer can understand a sizable infrastructure or the complex “plumbing” in a risk system, he would be considered competent, valuable, and strong even though he doesn’t understand the math.

Across domains, math is not the only domain knowledge — I believe 1) system architecture and 2) jargon are important domain knowledge too, esp. in risk systems.

Among the millions of java/c++ developers in risk systems, I guess 2% to 10% need some quant knowledge? In such a case, the market depth would be rather poor, because risk quant developer roles would be few and far between, just like the pricing quant developer market.

Q: your #1 domain xp.. pricing^mktData@@

See also marketable_domain_xp spreadsheet… Don’t spend too much time on this question… illusion. Though I see myself a veteran in some domain but hiring managers may see a haphazard collection of marginally related experiences. I may say AA is my #1 domain, but I may still choose BB domain.

E-trading (AutoTrading) and OMS are too vague as domains. Means different things to different people.

In terms of absolute length, pricing would be my #1 domain

  1. Citi pre-trade bond pricing
  2. Barcap
  3. Stirt risk bucketing
  4. OC Quest
  5. MSFM + self-study

In terms of depth, I’m very comfortable with the (limited) math therein, which is an entry barrier to many.

  • 😦 However, this domain has relatively few jobs and poor accumulation
  • 😦 Mostly in ibanks
  • 🙂 has contract roles
  • 🙂 salary premium? occasionally
  • 🙂 entry barrier? reasonable

specialty jobs outpay commodity financial IT jobs@@ #Mithun

In 2007 I worked on the most common type of financial IT job — 1) web java with 2) a big database and stored-procedures  3) small amount of javascript (possibly with ajax). 4) In my case, there was also some scheduled batch job.
It was in Private-Wealth-Management but no financial domain knowledge required.
(I have since moved on to “fancier” systems, not only pricing or market data.) I started to think the most common dev job like the PWM job doesn’t pay as well, but what’s the reality that you see throughout your career?
Q: does the most common type of financial IT job pay lower because it only needs commodity skills?
[Mithun] Not always. It depends on how critical the demands are and how much $ impact it has.
Trading as such they have money to spend, sometimes demanding & definitely impacts $ 
there is a short-term trend —  depends on demand & supply. 
I have lately seen lot of money thrown at angular even thought it might not be challenging as concurrency, but then u need many angular developers for a few server side developers. But then this skill fast retires.
Perhaps it pays just as well, or 5% lower than the specialty dev jobs in trading systems?
Commodity java dev will get around 145k while concurrency guy will get 175k…. 
Things have changed in that “most common type of financial IT job”, which I can only define in vague terms.
* once there was lots of spring/hibernate usage
* once there were lots of xml-related server-side libraries
* I guess there are more RESTful interface?
* more javascript code
* fewer stored procedures
* more noSQL systems, less SQL
* I guess there’s now more Hadoop??
yes Hadoop , spark, but I see a lot of demand for python, and they are coming with rich libraries.

churn !! bad ] mktData #socket,FIX,.. unexpected!

I feel the technology churn is remarkably low.

New low-level latency techniques are coming up frequently, but these topics are actually “shallow” and low complexity to the app developer.

  • epoll replacing select()? yes churn, but much less tragic than the stories with swing, perl, structs
  • most of the interview topics are unchanging
  • concurrency? not always needed. If needed, then often fairly simple.

dominant server-side language@WallSt: evolution

Don’t spend too much time.. Based on my limited observations,

  • As of 2007, the top dog was java.
  • The dominance is even stronger in 2018.
  • Q: how about in 10 years?
  • A: I feel java will remain #1

Look at the innovation leaders — West coast. For their (web) server side, they seem to have shifted slightly towards python, javascript, RoR

Q: Why do I consider buy-side, sell-side and other financial tech shops as a whole and why don’t I include google finance?
A: … because there’s mobility between sub-domains within, and entry barrier from outside.

Buy-side tend to use more c++; Banks usually favor java; Exchanges tend to use … both c++ and java. The latency advantage of c++ isn’t that significant to a major exchange like Nsdq.


latency QQ ]WallSt IV #java,c++..

Latency knowledge

  • is never needed on the job but … high Market Value
  • is not GTD at all but … is part of zbs
  • is not needed in py jobs
  • is needed in many c++ interview topics but .. in java is concentrated in JIT and GC
  • is an elite skill but … many candidates try
  • some depth is needed for IV and other discussions but … relatively low-complexity .. low-complexity topics #eg:GC/socket

c++ salaries: high-end^regular

I spoke with 5 to 10 fellow c++ developers in financial domain across a few countries. All of them seem to suggest c++ skill has higher market value than any other language like java, c# or python. One common reason they all gave is the high frequency trading jobs. My own observations also show that most HFT tech jobs use c++ and very few jobs are java or c#.

However, how many of us can ever pass the HFT interviews?

I have tried a few high-frequency or medium-frequency algo-trading jobs using c++. I didn’t pass any. Among my friends, only one person passed. He passed more than one such interviews and he’s clearly very strong.

So I feel the reality is, most of us won’t be able to get those jobs, or keep the jobs.

Q: how many percent of the c++ jobs in finance are HFT ?

I would guess 5 to 15%. So the vast majority of c++ salaries are ordinary. I think many of us can get those jobs:)

What’s your opinion?

WallSt tech projects/timelines #XR

I have heard several people complaining about the unhealthy work culture in NY investment bank IT teams.

  • · Timeline is artificially tight.
  • · A typical senior manager is motivated by trophy systems to win him promotions + bonuses, rather than really useful systems making a real impact to business
  • · Lots of systems are developed but not used or used for superficial purposes only. (This is common in many systems. Only 1% of the features are actually put to productive use.)
  • · Low quality code; low quality output data; poor quality control
  • · Constant and wide-spread pressure to retire old systems and rewrite, when the old system is still usable and sometimes rather stable.
  • · Senseless requirements, questionable value, thrown-away systems

I heard that West coast shops, Chicago buy-side and other companies don’t have exactly the same problems. I feel my current company is also different from NY banks — Older technology; Slight higher quality standard; More stable codebase; Users are mostly external paying customers…

I guess the Wall St culture is “good” for some, while other developers hate it and exit. I’m planning to stick to finance but not banks.

##resilient WS tech: FIX,sh-script…

Background: the constant need to economize on my learning hours. Have to attempt to pick the “survivors”.

  • FIX as a tech skill is an example of unexpected resilience. (However, this is in a specialized domain, so competition is arguably lower.) FIX isn’t dominant. Exchange native API is faster. Many HFT shops don’t use FIX, but still FIX has good ROTI.
  • SQL is not longer dominant but still standard
  • Tibco isn’t dominant. Many teams use competing products. Still it’s resilient
  • XML, in the face of light-weight serialization protocols – json, protobuf
  • Bourne shell, in the face of python, perl…
  • STL, invented in the 80’s (??) with many limitations, challenged repeatedly
  • tibrv, challenged by 29west, solace,

I have a bias towards older technologies. They have stood the test of time.

tiny team of elite developers

Upshot — value-creation per head and salary level would rival the high-flying manager roles.

Imagine a highly successful trading shop. Even though the trading profit (tens of millions) is comparable to a bank trading desk with hundreds of IT head count, this trading shop’s core dev team *probably* have a few (up to a few dozen) core developers + some support teams [1] such as QA team, data team, operations team. In contrast, the big bank probably have hundreds of “core” developers.

[1] Sometimes the core teams decide to take on such “peripheral” tasks as market data processing/storage, back testing, network/server set-up if these are deemed central to their value-add.

In the extreme case, I guess a trading shop with tens of millions of profit can make do with a handful of developers. They want just a few top geeks. The resultant efficiency is staggering. I can only imagine what personal qualities they want:

* code reading — my weakness
* tools – * manuals — reading tons of tech info (official or community) very quickly, when a “new” system invariably behave strangely
* local system knowledge
* trouble-shooting — and systematic problem-solving. I feel this largely depends on system knowledge.
* design — it right, and able to adjust it as requirements change * architecture?
* tuning?
* algorithms?

(soft skills:)
* clearly communicate design trade-offs in a difficult discussion * drive to get things done under pressure without cutting corners * teamwork — teamwork to back down when needed to implement a team decision

rookie coders vs old hands, my recent xp]fin IT

– Anyone (even a non-pianist) could teach piano to a kid, but perhaps not correctly;
– Anyone could teach swimming, but Qiu Yi has insight;
– I would guess any medical undergrad could treat patients, guided by a textbook – Management consultants — These guys go by case studies. Not experts in my industry. Why would I listen to them?
– Application design, low or high level, is an art. An inexperienced guy could go by textbooks, but actually, fine judgement needed at every turn.

In Stirt and Macquarie, how fast you get things done is not as important as getting things done right. Managers here are very particular about the details. I think they want things done right and stick to that long term. I can only guess they have seen hasty code that didn’t last, that required massive rework. Example
* the 2 earlier Error Memos coders
* Varun
* Some of the code in CAOS (account opening)

This is one reason they want older guys with deep experience and strategic insight.

## which skill/field CAN be self-taught

Domain knowledge is easier to “fake” than c#/c++. It’s hard to fake the mileage…

Q: which fields can be truly self-taught to in-depth? For example, mobile apps, desktop apps.

Goal #1: pass job interviews
Goal #2: reach the professional level of Theoretical expertise
Goal #3: reach the professional level of Practical expertise

* Quant? too dry. Too many doubts to be clarified only by professor. However the basic math part is standard and well-defined. Goal 3 is invalid.
* high-frequency, algo? Many books but none on real tricks. Compare swing books! Very few jobs.
* network/linux optimization? need machine to try. Extreme optimization techniques are top secrets.
* FIX? Many practical issues hard to imagine. No focus.

* swing, wpf? — tricks can be self-taught through experiment. Practical books are available. But practical problems are unknown.
* c# and core libraries? Real obstacle is the IDE. Practical books abound.
risk-mgmt? impossible to experiment.
coherence? too many subtopics. no focus. Hard to experiment.
tibco? no book at all. Hard to experiment
threading? Can hit Goal 2, not 3

mediocre WallSt developers earning WallSt-level salary


A paradox to share with you. For years I often wonder

Q1: why does wall street (a shorthand for “world financial centers”) pay something like double the salary compared to main street, for a senior programmer? (Disclaimer — Wall St is not the only sector; premier tech firms also pay well above main street.) On many teams including my current team, I see people doing main street programming work, using main street technical skills without any financial domain knowledge. Why pay us so much higher than main street?

I feel this is a non-trivial paradox. A deeper understanding helps us manage our career, manage our competitiveness, manage our job security, manage our workload, manage our value-add.

In this letter I use “ibank” as a shorthand for all the employers in the investment-banking and security trading business, sometimes including buy-side. Yes I’m deliberately vague and imprecise.

Factor 1: ibanks have higher margin than main street, even after 2008, and crucially a small percentage of employees are responsible for the profit. Many ibank jobs are, to be honest, not critical to the money-making machinery, but many developer jobs are. To put things into perspective, traders, fund managers, deal-makers, advisors/salesmen … are in the profit center and even more instrumental to the money-making machinery. The closer to the money, the higher is one’s value to the ibank. Highest paid wall street (non-manager) tech roles are probably the high-frequency low-latency experts because they are closest to the money. Their code has a most direct impact on the competitiveness of our trading engine in the arms race. Similarly, our pricing system calculates the prices sent directly to the market and have a direct impact on our competitiveness.

Therefore, many senior programmers are considered instrumental/important for the high margin of an ibank.

Factor 2: these tasks are often fairly complex and easy to mess up, and therefore calls for expertise, backed by track record. (To be fair, these are not the most complex.) For comparison again look at the most important roles in ibank profit centers. There’s usually a lot of decision-making, a huge flood of information to digest. An ibank can hire fresh graduates to create trading applications, but they are likely to fail to meet some of the requirements. In and outside Wall St, software dev is knowledge-intensive, methodical, intricate, even fragile [1], unlike many other jobs in finance.

[1] sensitive to human mistakes. That’s why we say 99.9% of software have bugs.

Therefore ibanks generally recognize in-house app dev as high-value (sometimes instrumental/enabling) and complex. I have heard many times “we can’t introduce this new instrument to the market without the IT system”.

Factor 3: Just a small number of experienced coders are needed. In main street, we see armies of developers for a given project. (Lee Chong Wah may disagree.) On Wall St, team size is cut to a third or a quarter. In so many ibanks I have worked, I always see a lot of senior developers and few junior developers. I feel OCBC (and other traditional banks) see developers as the same “armies”.

Therefore ibanks have high profit, and want to use part of the profit to attract a small number of elite developers. This is a “perfect storm” for higher salary. Now, in reality, a team of 3 to 7 senior developers would have just 1 key developer, so the other guys are not that critical. Perhaps one of them is doing nothing special at all. Low-value, low-importance, mundane code, but still paid well above main street. (I might be one of them now.)

Factor 4: During recruitment, ibanks typically have an adequate budget for a senior developer. If market rate is 140k, and their budget is 90k then they eventually hit the reality that strong candidates (with track record) won’t be interested, so they end up scraping the bottom of the barrel. Mostly tough and unsuccessful.

So managers go get a sufficient budget, screen, select and offer to this fairly strong candidate, but for various reasons he doesn’t get to play that key role. He may even end up with some truly unimportant modules, somewhat over qualified and overpaid. Managers can’t just drop him, but can leave his contract (if contractor) to expire without renewing. If not a contractor, then bonus is in question. If the guy is actually good but the mundane job must be done in the team, then maybe unfair to mistreat him….

That’s my answer to the opening question Q1. But there’s another question, because ibank managers are not stupid….

Q2: For the non-critical roles, why don’t they hire strong main street developers and pay them main street salary, to do whatever main street work there is in the project? No track record needed.

I feel many enlightened and bold ibank managers actually do that, but there’s a risk of uncertainty in following that route. Instead, most managers much prefer candidates with proven track record in a similar project. Also, recruiters are reluctant to submit profiles without relevant experience. Therefore, main street developers are largely excluded from the talent pool.

High-speed vs high-complexity financial products

I now see 2 broad categories of financial products–

A) Analytically complex instruments — CDS, IRS, structured, IR-linked, index-linked(?), MBS, swaptions, embedded options… Usually derivatives.
** You often need sophisticated valuation engines for pre-trade quoting and/or risk
** If such a product (say bond with option) is liquid then you may get tight bid/ask and need no sophisticated valuation engine. However, I feel most of these instruments have large spread.

B) high speed, high volume markets – stocks, major indices, FX spot, ED futures, Treasury,
** These are by far the most heavily traded and prominent products, often with low (profit) margins

Many popular or dominant instruments do not fall into either of these — vanilla bonds, vanilla options, FX options, FX futures, FX fwd, VIX

Any product falling into both categories? I don’t think so.

Products in (A) often give practitioners a sense of job security due to the specialist knowledge required. However, I usually stay clear of exotics. I tend to feel INsecure away from the mainstream.

Some System developers in (B) may have no real product-specific insight. They probably do not analyze data — historical data; or summarize (to make sense of) large volumns of data; or fundamental analysis; or statistical analysis

packaged software less popular in front office #my take

For commercial configurable packages such as Murex, Summit, Sunguard, Calypso…, banks (big or small) are more likely to deploy to middle/back office where differentiation is unimportant. In contrast, Front office (eg. pre-trade pricing) is competitive among banks, and involves traders’ personal views, models, strategies and entire proprietary product creations. If you use software package, such a unique competitive edge is sometimes achievable by configuration, but only up to a limit. Beyond that limit, you can write plug-in modules for the vendor product (such as pricing modules) but up to a limit. Beyond that limit, you can request features but vendors are slow and/or expensive. Therefore, bigger investment banks choose Build over Buy. You can engage an external consultancy, or hire your own developers.

Middle/Back office includes booking, position master, PnL, risk, STP, clearance/settlement, GL, cash management …

Another Example — Charles River is an OMS/EMS software vendor, popular with small buy-sides. Selling points include rich feature set out-of-the-box, extension points in the form of plug-in modules client programmers can create. The last resort (Avoid!) for a desperate client is a formal Feature Request to the vendor. May take a long time. This is the dark side of the Buy (vs build) route. Vendors like to defend themselves by playing up the rich and extensible features, superior software quality, and quick FR turnaround.

ETL vendors are interested in middle/back office, and front office too. I feel CEP, Distributed cache vendors … are competing for the same customers.

For a large trading desk’s front office, the Build route is preferred. An Asia regional bank’s trading/eCommerce IT head told me.

Another practical way to customize is to add stored procs into the vendor-designed database schema. Read-only procs are powerful and safe, but functionally limited. Next level is DML stored proc. This requires intimate knowledge of table relationships. I feel with practice some API users can master it.

how important is automated pricing engine@@ ask veterans

I proposed/hypothesized with a few pricing engine veteran friends that the big make-or-break trades and successful traders don’t rely so much on pricing engines, because they have an idea of the price they want to sell or buy. However, my friends pointed out a few cases when a pricing engine can be important.

– Risk management on a single structured/exotic position can involve a lot of simulations.
– If u have too many live positions, pricing them for pre-trade OR risk can be too time consuming and error prone.
– If you have many small RFQs, trader may want to focus elsewhere and let the system answer them.

How about those make-or-break big trades? Volume of such trades is by definition small. Decision maker would take the pricing engine output as reference only. They would never let a robot make such big decisions.

how much math is needed in … #a few finance IT fields

I always maintain that math is key domain knowledge. In the few areas I know, when a bit of math is required, a lot of regular developers would need to brush up before they can “play in the band” (as in a heavy metal band).

– Bond trading platforms? Close to zero math. Definitely less than bond pricing. Developers do need to remember basics like when interest rate rises, bond prices drop, and zero-coupon bonds trade at huge discounts (due to price/yield math relation). Also high-grade bonds tend to trade at lower yield — minimal math.

– Bond pricing? Slightly more than zero math. Price/yield conversion, yield-to-worst, modified duration, dv01 rollup, time-value-of-money are widely used, but developers usually don't need any mathematical insight into them to do the job. Developers just need basic understanding. Hiring managers may think differently.

– Forex trading? Arithmetic only, I believe. Add/subtract/multiply/divide only. Bond math requires high school math. In contrast, Lots of Forex trading IT jobs require primary school math.

– Option pricing? A bit of statistics and calculus. Like in bond pricing, developers don't need any mathematical insight. However, they need a basic grasp of historical vol calculation, implied-vol inversion, delta and other greeks. Even more fundamental concepts include PCP and the option payoff diagrams of about 10 basic strategies, but I don't think developers need these.

– VaR on a portfolio including bonds and options? Requires more than bond math and option math. More statistics. But I suspect a dedicated developer in a bond pricing system may need more in-depth bond math than a VaR developer, because a stringent math requirement on software developers is impractical, overkill and giving the wrong focus to the wrong role. Instead, the real math is usually for the quant analysts. Risk systems use more math than pricing systems, and employ more quants.

profit size(!! market size)determines budget for a trading system

Q: what kind of market potential determines how much (salary and other) budget a bank has for a trading syste?

I don’t know what factors are truly important but I know market size is not. Many people say Forex securities have gigantic market sizes in _notional_. If you hear anyone describing the (growing) importance of IRS, they never fail to mention total outstanding contract value.

If you ask me, global market __profit__ size (in dollar and cents) determines how much budget a bank allocates to given trading desk. (A bank typically aims to capture x% global or regional market share.)

Example — FX options usually get quite a decent budget with much smaller aggregate contract outstanding than FX spot (B) [1], because FX option has fatter margin. FX spot margin is very thin due to competition.

Example, Equity volatility trades have lower aggregate dollar amount than cash equity (B) but fatter margin.

As yet another derivative example, I believe CDS was very lucrative and high-margin, compared to government bond (B) trading.

All of these trading systems get sizable budgets despite small market sizes compared to the bigger markets of (B).

[1] I’m not sure how to measure outstanding in FX spot, but it’s probably fair to compare aggregate transaction amount in spot vs FX options. FX spot markets have large number of large trades.

developer caliber requirement in FI vs eq drv

If I compare developer requirement in FI vs eq derivatives

* latency, volume — easier in FI, but the fastest IR markets are faster than listed options.
* market data — easier in FI
* pricing — developers need slightly less pricing knowledge in the FI space. In EQD, volatility math is an everyday reality
* domain knowledge — FI space has more complex products but I don't see deeper knowledge in FI developers.
* trade execution, position management, pnl — about the same level of skill requirement

Overall, I feel FI has lower requirement but pays high. Perhaps the desk is more profitable.

controllers vs IT — whose responsibility for garbage-out@@

Controllers are paid to validate and attest the numbers [1]. They (not IT) have the knowledge and the entitlement to confidential information to affirm that the numbers make sense. When (not “if”) mistakes happen, then controllers must answer.

90% (in some places 100%) of the developers are expected to have limited knowledge (so that they can focus on well-defined requirements) but build automated systems very quickly, often with lots of obvioius “garbage out” i.e. unreliable, inaccurate output.

In most (not just “many”) cases, Developers simply don’t know enough to recognize “garbge out”. Through experience supporting a local (not a GENERIC) system, some senior IT guys may gain that judgement, insight and knowledge.

There are mistakes controllers are unable to detect — if developers release something obscure without informing controllers. Out of 500 changes, which ones do we need to “inform”? Initially be very cautious. Over time our judgement grows.

[1] what numbers? Let’s not get into the specifics as we could get carried away.

smaller company to poach a big-bank developer

A smaller company often has no choice but to pay a premium to poach a developer from a big bank. Without a premium, i don’t see a good reason to move. In reality, a smaller company often don’t want to pay a premium, so no deal.

In some real cases, the smaller company actually wants a discount.

Before believing recruiters saying “i have placed 20 candidates from non-big-banks into big banks”, consider these formidable undercurrents —

brank and credibility – if your future is in big banks, then remember they generally trust your brank in other big banks. They trust that (and IBM brank) more than they trust brank in smaller companies.

fear (the other side of the above coin) — A big bank developer enjoys job security since next interviewer feels “safer” hiring him and “fearful” hiring an unproven candidate from a smaller firm or a big tech like Google.

Familiarity – large banks are similar in set-up. Even a big buy-side like Pimco or a hedge fund is probably seen as different from a sell-side ibank. Forget techs like Microsoft!

radiation – ibank trec radiates everywhere, so does big techs like Oracle. Blackrock/hedgeFund trec radiates less. ECN or software vendor trec won’t radiate. Remember only a tiny number of core engine developers in ECN have the experience relevant to an ibank’s trading engine core team. This applies to all the big names like murex, sunguard, calypso, … See my blog on 3rd type of domain knowledge

What if this inflated WallSt IT salary bubble bursts@@

Update: I feel it’s unlikely. The industry is becoming more tech, more data, more automated. Complexity increases. Software has to deal with more complexity, not less.

Tip: remember survival = job market beauty contest, not on-the-job productivity.
** Stay in the leading pack – marathon.
** know your weaknesses on the job_market and on the job

Tip: stay relevant to the business. Business needs critical apps rolled out fast so they can trade. They are willing to sponsor such projects.So what are the critical, non-trivial apps?
– trade booking, sub-ledger, daily unrealized pnl
– pricing – usually non-trivial
– market data – sometimes trivial as traders can simply get a low-tech data feed
– OMS – only in exchange trading

Tip: know the high margin trading desks in each city. These profit centers will continue to pay high. SG (FX …) HK (eq), Ldn (FX, IR …)

Tip: remember pure tech firms might catch up Wall St IT pay.

—(Rest of the items may not address the issue in the subject.)—

Tip? broaden trec into GUI. Many of my server-side systems have a GUI counterpart.
Tip? broaden trec from java/c++ to dotnet, math, market data.
Tip?? move up to architect, team lead
Tip?? broaden trec into BA and prj mgmt
Tip?? deepen — unix tuning, DB tuning, memory tuning, math

"experienced" developer in trading system..meaning@@

Experience (of an old timer) in “our” system means
– #1) I can rely on this person to help me, if he’s willing
– #2) local sys knowledge
problem solver track record;
– extremely fast turnaround; – knows exactly where to tweak to make it work
– knows what (not) to test
– knows what users (don’t) want;
– knows the limitations of a hell lot of paper designs that don’t work in this context
– can get most assignments done

— To the hiring side —
Experience (of a candidate in the candidate pool) in a similar investment-bank means
– #1) proven track record, so my boss won’t fire me if I hire this candidate unwisely or unsuccessfully
– knows how this kind of place breaths and sucks

Experience (of a developer) in a similar system means
– adequate knowledge of standard tech tools in this space. Best eg — threading
– hopefully 50% familiarity with our tech tools used here, if 100% is the level of my average team members.
– possible knowledge of the limitations of some tools
– possible knowledge of unique capabilities of some tools
– knows the trading conventions and how this kind of things work
– knows the jargons

##[11]upstreams(sg gov)for next 10Y #wallSt tech

(See also blog post on 2010 GS annual report.) Singapore government is smart to always look out for upstream domains. Similarly, in financial IT, there are some upstream *domains* too.

Upstream: real time risk — upstream in derivative risk … but poor Market Depth
Upstream: grid — Memory virtualization, grid computing
Upstream: Eq(and FX) HFT and algo trading is upstream to other asset classes
Upstream: Option-related (including mtg) analytics tend to be more advanced, and probably upstream. Any derivative with optinality tends to be dominated by vol in terms of pricing complexity.
Upstream: connectivity — collocation, streaming,
Upstream: latency — sub-m, multi-core, non-blocking…
Upstream: voluminous market data processing — storage, distribution…FX and eq option has the highest volume, and often dominates entire infrastructure. Many asset classes are far behind
Upstream: SecDB — firmwide singleton mkt risk engine. Upstream in m-risk technology. Most shops have data silos.

However, upstream technologies often become fads.
– C++? not really upstream. A strong c++ guy isn’t necessarily strong in java or C#.
– web service
– solace-like messaging hardware
– CEP? How much real value does it add?
– rule engines
– MortgageBackedSecurity pricing? Not really upstream, though most complex analytically.
– ETL?
– hibernate?
– python?

which banks ask differentiating questions

(another blog post)

If an interview question is too tough, then strong or fake candidates both fail. If question is too common, then both pass. Good differentiating questions let the strong candidates shine through. Differentiating candidates depends on

* non-standard (but not obscure) questions, or
* drill-in on standard questions, or
* an interviewer capable of seeing the subtle differences among answers to standard questions.

If you have “none of the above”, then you can't differentiate candidates.

Most interview questions are open-ended. I (not yet a strong candidate) can give super long answers to sleep-vs-wait, interface-vs-abstract-class, arrayList-vs-linkedList, “how to prevent deadlock”…. However, many interview questions are so common that candidates can find perfect standard answers online. A strong interviewer must drill in on such a stock question to test the depth of understanding.

GS, google, lab49, MS, barc cap have good interviewers.

front office developers for fx options trading desk

Question: Guess how many developers are needed in the fx option trading desk at one of world's biggest investment banks? Note this desk handles all fx option trading for the entire bank.
Answer: about 20 world wide including consultants.

The other trading desk in the FX space is the FX cash desk. Not sure how many developers.

If there are 20 similar trading desks in the bank, then total number of _desk_specific_ developer head count is going to be around 500, at most 1000. I think this is a very small percentage (5%?) of total IT head count.

In 2007 GS had about 6000 IT headcount. This bank is probably double. I'd say at least 10,000 IT head count. I would estimate less than 5% of them are directly funded by trading desks.

Now I know why front office trading is a rare opportunity.

Risk developers are classified into 2 types
– some are dedicated to a desk
– some are firm-wide

financial domain XP ^ domain Knowledge

Domain _experience_ is what hiring managers look for. If there’s a high-paying job in some rarified field (say analytics, algo trading, low-latency..) then domain experience on the resume is the crucial factor. Many candidates can have strong tech background, and strong domain knowledge, but only those with “relevant experience” will get short-listed, or those with good recruiter connections.

The domain experience shows in terms of 1) jargon 2) architecture, but actually is very local and not portable at all.

Generic domain knowledge is very “thin”. You can pick it up by reading or over a few months on the job. Mostly jargon terms, and very little math. Hiring managers ask questions to verify such knowledge but it’s not hard to pass.

y Singapore IBanks pay on-par with WallSt #tech

The higher the position, the more on-par.

* these international (not local) investment banks (not commercial banks) have budget.

* Most trading systems have a tiny elite team of super developers, whereas other industries employ armies of developers. An ibank would rather get 1 rock star than 10 mediocre.
* Yes an ibank could hire experienced java developers at $5k but they aren’t battle tested in trading systems.
* recruiters would only send in experienced candidates.

I don’t know how, but gov can probably influence private sector hiring. If this is the case, then Singapore gov is probably encouraging the “on-par” situation. Singapore public sector has the tradition of paying top salary for top talent.

Look at the GIC guys. Even a young graduate can earn S$120k. It’s obvious to me that they aren’t really different from their peers in “ordinary” sectors, but the vacancy simply comes with that price tag. Singapore employers (private or public) willingly pay the premium.

S’pore is in constant competition (and battle) with Hongkong, while rising up to London and New York. They compete for big banks, for hedge funds, for ECNs, for IPOs, and ultimately for talents. Singapore gov is more serious (takes more actions) than many governments to attract high-caliber talents in upstream sectors such as high-finance. The higher your value-add, the more they want you. Sadly, they can only measure your value-add by your income.

##front office technical know-how

–Most imp to hiring and productivity–

swing, wpf — more complex than all other tools
pricing math
pnl rollup, MTD, unrealized.. is imp to traders
fire fighting
c++ …rv
mastery over db query 

general low latency techniques
gemfire — imp to both risk and real time pre-trade. Let’s xx2set up
trade booking, cancel, MOD
java OOM — simple techniques
position mgmt is important to traders
trade blotter basics

y mainframe is still important in finance and airline booking

Most profitable business in IBM is mainframe.


Q: why not yet replaced by newer technologies like java and unix?

A: actually mainframe can also run java and unix


Reason: extremely reliable. Decision-makers dare not touch them. It’s hard to recreate something so reliable. May take many years to demonstrate the reliability.

Reason: mainframe is improving too, as competing technologies improve.

Reason: Terminals (like ATM and airline booking terminals) are locked into IBM mainframe. They can’t work with newer servers. You must replace terminals along with mainframe.

Reason: mainframe processing power is still formidable.

Reason: IBM killed its mainframe competitors chiefly Hitachi.  I don’t know if this prolongs or shortens mainframe shelf-life.

##some java skills no longer]demand

(another blog post)

Any powerful technology requires learning. Whatever learning effort I put into EJB, Corba, struts, JSP, JNI, J2ME, Swing, AWT, rule engines .. are becoming less and less valuable to employers. The march of java is merciless. How merciless? Even the creator of java – Sun microsystem – has to go open source on both java and Solaris just to survive, and sell itself to Oracle.

I am now very careful not to invest myself too heavily into anything including popular must-know stuff like Spring, hibernate, distributed cache, EJB3, AOP, CEP, JSF, GWT, protobuf, web services, design patterns, RMI …

I think most of the above are jxee, not coreJava.

Instead, I seek stability in older core java technologies like threading, collections, serialization, messaging, reflection, …


(C++ is even older and more stable, but this post is all about java.)