fear@large codebase #web/script coders

One Conclusion — my c++ /mileage/ made me a slightly more confident, and slightly more competent programmer, having “been there; done that”, but see the big Question 1 below.

— Historical view

For half my career I avoided enterprise technologies like java/c++/c#/SQL/storedProc/Corba/sockets… and favored light-weight technologies like web apps and scripting languages. I suspect that many young programmers also feel the same way — no need to struggle with the older, harder technologies.

Until GS, I was scared of the technical jargon, complexities, low-level API’s debuggers/linkers/IDE, compiler errors and opaque failures in java/SQL … (even more scared of C and Windows). Scared of the larger, more verbose codebases in these languages (cf the small php/perl/javascript programs)… so scared that I had no appetite to study these languages.

— many guys are unused to large codebases

Look around your office. Many developers have at most a single (rarely two) project involving a large codebase. Large like 50k to 100k lines of code excluding comments.

I feel the devops/RTB/DBA or BA/PM roles within dev teams don’t require the individual to take on those large codebases. Since it’s no fun, time-consuming and possibly impenetrable, few of them would take it on. In other words, most people who try would give up sooner or later. Searching in a large codebase is perhaps their first challenge. Even figuring out a variable’s actual type can be a challenge in a compiled language.

Compiling can be a challenge esp. with C/c++, given the more complex tool chain, as Stroustrup told me.

Tracing code flow is a common complexity across languages but worse in compiled languages.

In my experience, perl,php,py,javascript codebases are usually small like pets. When they grow to big creatures they are daunting and formidable just like compiled language projects. Some personal experiences —
* Qz? Not a python codebase at all
* pwm comm perl codebase? I would STILL say codebase would be bigger if using a compiled language

Many young male/female coders are not committed to large scale dev as a long-term career, so they probably don’t like this kinda tough, boring task.

— on a new level

  • Analogy — if you have not run marathons you would be afraid of it.
  • Analogy — if you have not coached a child on big exams you would be afraid of it.

I feel web (or batch) app developers often lack the “hardcore” experience described above. They operate at a higher level, cleaner and simpler. Note Java is cleaner than c++. In fact I feel weaker as java programmer compared to a c++ programmer.

Q1: I have successfully mastered a few sizable codebases in C++, java, c#. So how many more successful experiences do I need to feel competent?
A: ….?

Virtually every codebase feels too big at some time during the first 1-2 years, often when I am in a low mood, despite the fact that in my experience, I was competent with many of these large codebases.
I think Ashish, Andrew Yap etc were able to operate well with limited understanding.
I now see the whole experience as a grueling marathon. Tough for every runner, but I tend to start the race assuming I’m the weakest — impostor syndrome.
Everyone has to rely on log and primitive code browsing tools. Any special tools are usually marginal value. With java, live debugger is the most promising tool but still limited pain-relief. Virtually all of my fellow developers face exactly the same challenges so we all have to guess. I mean Yang, Piroz, Sundip, Shubin, … virtually all of them, even the original authors of the codebase. Even after spending 10Y with a codebase, we could face opaque issues. However, these peers are more confident against ambiguity.
Advertisements

multi-core as j4 multi-threading #STM

Q: why choose multi-threaded design instead of single-threaded processes?

Most publications mention multi-core hardware as an answer. Questionable.

With multi-threading, you can run 30 threads in one process. Or you can run 30 single-threaded processes as in RTS parser and Rebus — industrial strength proven solution. Both designs make use of all CPU cores.

Between these two designs, heap memory efficiency can be different, as the 30 threads are able to share 99GB of objects in the same address space, but the 30 processes would need shared memory.

I feel the lesser known middle-ground design is … 30 threads running in single-threaded-mode, By definition, these 30 threads can only share immutable data only,. Anything mutable is thread-local.

In any multi-threaded design, those 30 threads can share the text segment i.e. memory occupied by code. Text segment tend to be smaller than the heap footprint.

In a boss-worker design, the worker threads may need to share very few mutable object, so these worker threads are not strictly STM. They are quasi-STM

speed^soft qualities: CIV scoring

If you are very strong, you can achieve both

  • complete the required number of problems in time with no serious bugs
  • demonstrate requirement clarification, edge cases, dry-run, thorough error handling, detailed complexity analysis, coding style.

In reality, some speed coding tests at FB, Indeed etc really favor speed over the other things.

When we take such a test, there is always internal tension between two motivations — speedy completion vs demonstrating other qualities.

Thinking out loud is a common recommendation but it can create unnecessary conversation and slow you down. However, some of these conversations may lead you to the right path.

##xp: enough localSys⇒GTD #but respect≠guaranteed

Did localSys effort save me? See my rather surgical tabulation analysis of past PIP + survivals

  1. GS — No, not a lot of respect, but I was able to hold my end of the log. With insufficient localSys I would get even less respect.
  2. Quest — No. The PIP was due to many factors. I feel my Quest GTD was adequate. With insufficient localSys, I would get even less respect.
  3. 🙂 RTS — yes
  4. 95G + volFitter — limited localSys .. nice green field projects for me.
  5. Macq — No. However, With insufficient localSys, I would have been kicked out within Y1

— Macq? I now feel my localSys at Macq was deeper than at Quest or RTS

  • I put in lots of effort and was able to diagnose most build errors on MSVS and Linux.
  • I needed lots of localSys knowledge to modernize codebase for MSVS-2015 and c++14.
  • I needed localSys + innovative on-the-fly code gen + other skills to add the logging decorator into pymodels. This hack alone is worth a bonus. This hack is more effective more valuable than all my hacks in OC.

However, I feel the expectation was too high so in the end I got PIP and felt like damaged good.

I need to challenge that negative impression of the entire episode.

technology xyz is dead

I often hear people say “technology XX is dead” .. exaggerated attention-grabbing gimmick, common on social media user posts.

I like the “dead”, old-fashioned, time-honored, stable technologies that are robust (not only resilient) against churn.

The alternative technologies are use-less, worth-less, hope-less, less proven, low-value, more likely to die young or even dead-on-arrival

##spare time: what peers do to “get ahead”

Exec Summary — for all of us, strategic orgro, ROTI etc is … holy grail … elusive. My wife/in-laws often say “You spend so much family time on your studies but are you earning more?” Translating personal endeavor to income is holy grail .. frustrating..

I believe many professionals don’t have the abilities to convert spare time to tangible personal growth.

Tangibility is defined by each individual, and requires a high degree of self-knowledge.

  • investment analysis (HuKun, XR)? I doubt any of them has any ROTI
  • deeper in java for higher pay or promotion? Higher pay is basically hopeless for many of us who are already at the higher end. GTD depends more on localSys. Promotion has no correlation with deeper java knowledge.
  • coding drill
  • tsn like mobile, data science (XR), java (Sunil)
  • personal investment
  • formal education in spare time like CFA, MBA
  • Stephen Keith was able to write academic papers in his spare time .. very rare

In this realistic analysis, my c++/c#/quant/swing attempts now look commendable.

 

c++IV=much harder than GTD #Mithun

c++ IV is much harder than c++ job GTD, as I told Mithun.

  • GTD is no different from java jobs, even though the build process can be slightly hairy. Java build can also get messy.
  • In contrast, C++ IV is a totally different game.

You need a rating of 1/10 to do a decent job, but need 7/10 to pass ibank interviews. This gap is wider in c++ than in java as java interview bar is much lower.

Most technical challenges on the job are localSys, so you can just look at existing code and 照猫画虎, 如法炮制, as AndrewYap does. Venkat of RTS said we should but we still do.

Corollary — after 3Y full time c++ job, you may still fail to pass those interviews. Actually I programed C for 2Y but couldn’t pass any C interview whatsoever.

3Hr@knapsack problem..overspent@@

I spent 2-4 hours implementing/refining my own knapsack DP solution — https://github.com/tiger40490/repo1/blob/py1/py/algo_combo_perm/knapsack.py

Many “peers” would consider it overspent. This kind of “conventional” view tend to destroy the precious satisfaction, the wellspring of positive energy.

Some peers complete it in a hurry and move on.

They may not compare it against the comboSum problem … x-ref is crucial to pattern recognition

They may not learn a lot by doing it quickly. I wouldn’t learn much of anything if I do that .. 囫囵吞枣

sg19: risks@tsn

This time my TSN attitude is not driven by fear of stagnation, but a free-spirited, fun-seeking desire to explore.

My past TSN failures (and some successes) are the best guide. Given the multiple TSN failures, my expectation is rather low. However, the pain of stigma , confidence loss, impact on family can still be very real. That’s why I wrote in my blog that the safest job is back office java+SQL+scripting with low-calibre colleagues.

[19]c++guys becom`very unlucky cf java

On 22 Apr 2019 I told Greg that c++ developers like me, Deepak, CSY.. are just so unlucky — most of the WallSt c++ jobs are too demanding in terms of latency engineering, either on buy-side or sell-side.

Greg agreed that java interviews are much easier to pass. Greg said if you have reasonable java skills, then you can get a job in a week.

I told Greg that the only way Deepak or CSY could get an offer is through one of the few easy-entry c++jobs, but there are relatively few such jobs i.e. without a high entry barrier.

— widespread view that c++ developers are perceived as strongest due to c++ being a hard language

My conclusion

  1. yes in terms of QQ and zbs
  2. no in terms of GTD, as GTD challenge is mostly due to localSys. Even a python codebase can be hard.

longer u stay,more localSys nlg⇒safer@@ #5%GS

As I said in this blog, FTE-dev is often worse off than contractors.

I think if you stay for many years without moving up while some of your colleagues move up, you may or may not get a stigma.  Some of the newer members may become your manager:) But this is not the main focus here.

The longer you stay, the more knowledgeable about local system. Hopefully more relaxed and less stress? Partly true but very dangerous slippery slope. You hope that you can make do with 80% of the earlier /intensity/, but I think it won’t happen in most ibanks [1].

In my observation, most VP-level old timers operate under /unrelenting/ pressure (“threat”) to maintain high productivity. They are expected to be more proficient and more productive than earlier, not allowed to slow down and take it easy … No retirement home here.

Otherwise, they would fail the peer benchmark

Another Part of the threat comes from hungrier, younger colleagues able to reach (then surpass) half your speed within a year or two, driven by /formidable/ brain-power (energy, intelligence, ..)

[1] There are exceptions but I only know a few so I don’t want to spend too much analyzing. Exception — if you were the original architect and you keep an eye on the evolution of your brainchild, and remain knowledgeable about it, but this scenario requires some brain-power.

That’s the harsh competitive reality even if you don’t seek increasing responsibilities. A small percentage of the people are ambitious ladder climbers. (Am clearly not, because at my current level I already feel the heavy workload.)

Many people I talk to want an “easy life”, not increasing responsibilities. However, If you don’t take up increasing responsibilities, you may become too expensive. You may get a token bonus. I think you may even get a humiliating bonus.

Overall, in “peacetime” long service without moving up can feel embarrassing and uncomfortable at times, for some individuals. (It’s more noticeable if most of the peers at your level are much younger, as in Macq and OC.) Some corporate cultures may tolerate but stigmatize that

Employer claim they prefer employees staying longer rather than shorter. That’s blatant marketing. In reality, some employers wish some old timers to leave on their own, to make way for younger, cheaper fresh blood. GS annual 5% cull in peacetime is widely-reported in WSJ, Independent... A few common motivations:

  1. Old timers are sometimes perceived as obstacles to change, to be removed.
  2. some employers believe younger, newer workers are cheaper and more motivated on average
  3. Whenever a new manager comes in he would bring in his old friends, otherwise he is weak.

Down turn? All hell breaks loose. Rather than protecting you, your long service track record may make you vulnerable. You may be seen as an opportunity to “replenish fresh blood”. In contrast, the less-productive but newer colleagues may show potential, and the hiring manager don’t want to look bad — hiring then firing new guys. In other words, it’s sometimes safer for the manager to sacrifice an old timer than a recent new hire. This is different from my Stirt experience.

My personal biased conclusions —

  • no such thing as long service recognition. No such thing as two-way commitment.
  • If you can be replaced cheaper, you are at risk. The more you earn, the more risky
  • If you are earning above the market rate then you need enough value-add, regardless how long you have served.

## strategic TSN among%%abandoned

Q: name the most strategic trySomethingNew domains that I have since abandoned, given up, quit. How about the new plan to take on coding drills as a lifelong hobby?

Don’t spend too much time, because the answers are nothing new even though this is a decent question.

  1. — ranked by surprise
  2. algo trading? actually very few roles spread across a number of firms
  3. c#
  4. drv pricing quant
  5. real time risk as in GS, Qz and Athena
  6. RDBMS tuning
  7. MOM, async, message-driven design knowhow
  8. distributed cache like Coherence and Gemfire
  9. Solaris sys admin and DBA
  10. perl, SQL, Unix/Linux power-user knowledge? No longer a top 10 focus

—-

  • python? Not yet abandoned
  • web dev for dotcom? I did this for years then abandoned it. Many of the tech skills are still relevant like sessions, java thread safety, long-running jobs

importance@formal edu4tech career

I tend to dismiss the value of formal education (including degrees) and I tend to overvalue on-the-job quick-n-dirty learning. Compared to Asia, U.S. tech culture is less fixated on formal education.

  • Eg: Some inexperienced developer colleagues seem to have a good grasp of option math
  • Eg: my theoretical knowledge of comp science is completely self-taught, including concurrency, SQL, OO,..
  • eg: A more extreme example of such a domain is DynamicProgramming + Greedy algorithms. An intelligent programmer (regardless of age) can become highly competent without formal training.

Some theoretical domains are really based on field practice such as OO design.

Even if you have a solid education, either formally, or on the job over a few focused years, we all face the same challenge of continuing education —

  1. my web dev experience (self-education) is now outdated, according my interviews at Indeed , ByteDance ..
    • But luckily there are many interviews I can attend and learn from.
  2. data science, machine learning … is not so easy to self-learn
    • luckily there are many online learning resources.

 

long-term value: QQ imt ECT speed

The alpha geeks — authors, experts, open source contributors … are they fast enough to win those coding contests?

The speed-coding contest winners … are they powerful, influential, innovative, creative, insightful? Their IQ? Not necessarily high, but they are nobody if not superfast.

The QQ knowledge is, by definition, not needed on projects, usually obscure, deep, theoretical or advanced technical knowledge. As such, QQ knowledge has some value, arguably more than the ECT speed.

Some say a self-respecting programmer need some of this QQ expertise.

##tech skill superficial exposure: 5 eg

see also exposure: semi-automatic(shallow)Accu #$valuable contexx and low-complexity topics #JGC..

  • –(defining?) features (actually limitations) of superficial exposure
  • accu — (except bash, sql) I would say very few of the items below offer any accu comparable to my core-c++ and core-java accu
  • entry barrier — (except SQL) is not created since you didn’t gain some real edge
  • commodity skill —
  • traction — spinning the wheel
  1. JGC — i think most guys have only textbook knowledge, no GTD knowledge.
  2. lockfree — most guys have only textbook knowledge
  3. [e] spring, tibrv — xp: I used it many times and read many books but no breakthrough. In-depth knowledge is never quizzed
  4. bash scripting — xp: i read books for years but only gained some traction by writing many non-trivial bash scripts
  5. SQL — xp: 5 years typical experience is less than 1Y@PWM
  6. –other examples (beware oth)
  7. [e] EJB, JPA (hibernate), JMS, Hadoop(?)
  8. memory profilers; leak detectors
  9. design patterns — really just for show.
  10. Ajax
  11. [e] Gemfire
  12. java latency tuning — is an art not a science.
    • Poor portability. JVM tuning is mostly about .. GC but at application level, real problem is usually I/O like data store + serialization. Sometimes data structure + threading also matter receives disproportionate limelight.
    • conclusion — superficial, textbook knowledge in this skillset probably won’t help with latency tuning.
  13. [e=ecosystem like add-on packages outside the core language.] Ecosystem skills are hard to enable deep learning and traction as employers don’t need deep insight. GTD is all they need.

lower workload ⇏ quality free time #family

Lower workload CAN mean

… more time for kids + workout, but can also mean

… more time wasted .. burn/rot

The free time saved due to lower workload is often spent on reflective blogging… controversial

A great example of quality free time is the Bayonne -> MS commute in 2018/2019. For a few weeks I was able to do coding drill on commute, despite the segmented commute. For a few months I was doing /productive/ git-blogging

##coreJava QQ topics: stand-out from crowd

Q: what core java QQ topics (not necessarily skills) to make you stand out from the crowd?

  1. (Best example) low-level core threading features — higher market value than ecosystem features like concurrency utilities, which are so numerous that most candidates are not expected to know in-depth
  2. jGC
  3. latency tuning — including JIT
  4. collections internals

— second tier QQ topics

  • reflection, AOP, bytecode engineering? never popular
  • advanced generics? out of fashion
  • class loader internals? out of fashion
  • JNI? never popular
  • jvm internals? never popular and never needed on any project

q[visible progress]=Unreasonable expectations

See also my sample list in the sms twister blog visible progress # very rare therefore to be celebrated

Contrast with the views in [[reconciliation]]. Beware empty [g=] glorifications that don’t mean much when I look back 20Y later.

Every week, I actually make more progress than my fellow daddies with kids and commute etc. However, Once a while, in retrospect I would fall apart and cast serious doubt on (and belittle) my progress and point out the unfortunately invisible long-term effect.

I think many people implicitly follow a harsh and simplistic criteria like earning capacity, kids’ grades or absolute return, to dismiss and discredit all the “progresses”. This can become irrational, counterproductive, and /demotivating/ — engine loss of power. Such criteria are unfair to the self. If you are a teacher or coach, would you be so harsh on your student?

It can be a punishment, like a flogging whip.

Putting on a critical thinker’s hat, I feel that for most guys in my situation, it’s no mean achievements to maintain current condition and make small progress, with invisible long-term effect. Anything more is asking too much, and requires luck, talent, determination, contexx etc.

  • –ranked by …? I want to highlight the unsung heroes…
  • cholesterol, dental, belly and weight? maintaining is no mean achievement
  • loving relationship with wife? maintained, even strengthened
  • knowledge (and first hand experience) with diet, fitness, aging? building up slowly
  • more blood donation, done for my kids.
  • semi-retirement planning? improving through 5 discussions/year
  • more familiar with Bayonne residential market
  • relationship with in-laws? improved, as visible long term progress. More important — relationship with my own parents maintained
  • boy’s renzi and Chinese reading? improved slightly. Not really long term visible progress but at least he maintained
  • physical flexibility? maintained .. yes! Improvement? yes a bit of visible progress, with huge effortstamina? maintained … no mean achievement
  • [g] financial domain knowledge? I expanded to FX; market data; low-latency equity; FIX exchange trading…. Visible progress but shallow.
  • algo and coding test performance? I tend to belittle the improvement
  • bonding with kids? constantly building, deepening… Not by default, but by effort.
  • c++/c# conquered as a visible long term progress. Rather hard and long mileage, which probably means high entry barrier for the new entrants.
    • Probably more important — java skill level maintained.
  • credit score
  • financial assets (mostly holding well against inflation)? yes visible progress but I tend to belittle it. Building this portfolio actually required persistent effort, years of analysis, experiments, ..

our coding drills r different: fundamental reason #le2XR

Fundamentally, one chooses how to practice based on past interview experience of his own, not hearsays.

Question for you — how many times in the last 18 months did you get a coding interview that required 30+ minutes per question?

A: about 20 out of 25 positions in my recent experience, not including a BGC-partners onsite when they gave me 4 different questions but each 20 minutes only.

Most of my coding rounds are hackerrank/codility or over weekend.

  • Now I think for you the answer is 10% (but did you include those hacker rank tests?)
  • Now I think for you, mostly you don’t need to compile ! That’s why you call them “algorithm questions” rather than coding questions.

Even if I persuade you on the importance of edit-compile-test speed, or the value of python, sooner or later you would doubt “Really? How come I seldom need to write real code so fast in an interview?”. You would eventually stop practicing with real-code and refocus on pure algorithms, by reading key ideas behind common questions.

If you spend hours of focused energy writing real code as practice, and feel worn out, your own, personal experience would eventually kick in and remind you that it’s not worth it.

Conversely, if I were to follow your method to memorize key ideas only, my personal experience would soon shout out — “Take a step back and look again, you are already pretty fast coming up with data structure and algorithm ideas, but your REAL weakness is implementing them — too slow !”

I listed all recent interview episodes in tabular format — https://bintanvictor.wordpress.com/2018/04/28/some-worthwhile-coding-iv-experiences/.

  • 100% of big tech companies require at least one coding question lasting more than 30 minutes.
  • 100% of big buy-side shops require at least one coding question lasting more than 30 minutes.
  • More than 50% of investment bank jobs also require it.
  • For the other c++ financial companies (the so-called “third party” players) like Bloomberg, exchanges, brokers, market data providers, about 80% of the jobs require it.

local jargon, localSys architecture #WFH

Background:  3 types: Portable dnlg ] finance IT listed 3 portable types of dnlg (i.e. domain knowledge), but now I feel the local (i.e. non-portable) dnlg is more relevant to

  • your remote work
  • GTD productivity,
  • your survival,
  • your overall value-add to the team,
  • assessment by manager

Interviewers can only ask portable dnlg, Local dnlg include things like

  1. local jargon
  2. local system architecture, local “data flow”

irrational envy for all-round high flyers

When I first identify an acquaintance as an all-round high flyer, his (her) “note-worthy” achievements were invariablly rather few, thanks to my automatic filter on his other “success” stories … becasue those kinds of “successes” are, at a deep and personal level, unimportant to me. But then those things insidiously sneak past my defence into my inferiority complex and /infest/. Extremely irrational and 不值得.

I would rather feel inferior to someone (I know well) with many unrelated yet worthy achievements [3]. I doubt there’s any in my circle.

Incidentally, when a public speaker is introduced on stage, the audience often hear a list of “successes” which are mostly unimportant to me.

(Even though none of them is a friend I know well enough) Over the years there were a small number of acquaintances [1] I have singled out. Once I singe one out, I tend to lose my critical thinking and see many unimportant/insignificant/secondary “achievements” as enviable. Critical thinking is badly, badly needed at such a juncture!

Incidentally, one of the most effective ways to feel not-inferior is a lucrative job offer, even if I don’t take it.

The initial “enviable achievements” are usually one of these 5
1) income, almost always managerial [2]
2) investment, mostly property
3) beautiful wife
* fitness, flexibility and body shape
* short commute

The other factors are usually (as they should) in my “don’t-care/unimportant-to-me” list, but they sneak into my inferiority complex.

* (multiple) degreed from prestigous universities? Actually most of them are inferior to me!
* academic kids
* competitions and awards to himself or kids
* branded employers? many of them have fewer than mine
* running his own side business? I did while many of them didn’t
* wife professional career
* work-life balance… “easy job”? Questionable. Most high-paying jobs require effort
* writing and music skills? I achieved more than most of them!
* publications
* cars? is a liability not an asset!
* green card
* vacations to many places? Huge cost, no real gain for me
* magerial success at an erly age
* golf skills? i couldn’t care less when I’m not losing my critical thinking.
* networking skill, smooth personality? I’m not this type

[2] as soon as I hear the MD title of some ex-classmate, I lose my critical thinking defence.

Better consider [[compassion and self hate]] and Buddhist teaching

[1] Beside ML, Here are some questionable names. Many of them I barely know the name and job title, so my inferiority is fundamentally similar to my infatuation over the Indonesian girl Sandy, whom I spoke to fewer than 3 times.
* Lu Nuo — I only know he has a writing hobby …
* Cai Hongyu
* Tao YY — I don’t really know how he is doing
* Yang Yang
* Xie Xiaoli

[3] briefly on myself — math, piano, c#, swing, properties, blog, helping friends with job hunting

overvalued analytics applications #CSDoctor

Is GPS navigator necessary? Are side signal lights necessary? Some things are more necessary than others.

Trading system, risk systems, market data systems are mainstream. In contrast, I have found out that pricing analytics system is not really mainstream. Many buy-side and most smaller sell-side firms don’t use any pricing analytics beyond rudimentary derivations from raw market data.

There are also a number of sophisticated derivative and fixed-income analytics vendors including Bloomberg, Murex, Numerix.. These vendors focus on analytics so their customers don’t need deep expertise. OCBC’s quant team’s main job was validating the analytics offered by Murex.

Pricing analytics tools are “advisory” and not mandatory. The creators of those tools (like CSDoctor) tend to over-value their creations as if they are going to make the traders faster, safer, more profitable. In reality, traders can always choose not to use them.

As a contrast, take market data as example – 80% of trading shops need to build or buy market data systems as they can’t operate without it.

Math+intelligence ] trading ≠> high value-add

  • — Examples of math applied outside traditional proven quant domains like VaR
  • Trade analytics, execution analytics systems — analyzing past executions, and uses statistic tools to derive some empirical or parametric distribution.
  • Sell-side pre-trade analytics to evaluate a proposed trade….
  • Real-time risk analytics

Q: How much math in these systems? Not that much.

  • Fundamentally, trading domain is math-lite…
  • Risk management is slightly more mathematical due to large data set, relaxed latency requirement, many scenarios
  • The fancier and more advanced math, the more dubious

Q: Value-add? Questionable. That’s one reason why most financial institutions don’t spend billions building such systems. They do spend billions on traditional automation systems.

Q: Who would want to pay to use these systems? Rather Few.

Q: Python? Possibly.

case study: CSDoctor’s — value-add@analytics #CSDoctor

Are equities simpler than FICC@@

I agree that FICC products are more complex, even if we exclude derivatives

  • FI product valuations are sensitive to multiple factors such as yield curve, credit spread
  • FI products all have an expiry date
  • We often calculate a theoretical price since market price is often unavailable or illiquid.
  • I will omit other reasons, because I want to talk more (but not too much) about …

I see some complexities (mostly) specific to equities. Disclaimer — I have only a short few years of experience in this space. Some of the complexities here may not be complex in many systems but may be artificially, unnecessarily complex in one specific system. Your mileage may vary.

  • Many regulatory requirements, not all straightforward
  • Restrictions – Bloomberg publishes many types of restrictions for each stock
  • Short sale — Many rules and processes around short sale
  • Benchmarks, Execution algorithms and alphas. HFT is mostly on equities (+ some FX pairs)
  • Market impact – is a non-trivial topic for quants
  • Closing auctions and opening auctions
  • Market microstructure
  • Order books – are valuable, not easy to replicate, and change by the second
  • Many orders in a published order book get cancelled quickly. I think some highly liquid government bonds may have similar features
  • Many small rules about commission and exchange fees
  • Aggregate exposure — to a single stock… aggregation across accounts is a challenge mostly in equities since there are so many trades. You often lose track of your aggregate exposure.
  • Exchange connectivity
  • Order routing
  • Order management

quant≠sure good for aging dev

I now feel the quant domain knowledge doesn’t change so fast, but ..

1) Quant domain is an elitist, exclusive sector with low market depth (highly specialized).

I think many intelligent/powerful developers can succeed as a quant dev, even without formal quant training, if motivated or interested enough. Abilities + effort (due to keen interest) is all I needed when I succeeded at Barcap

Not a semi-retirement domain. When you lose the mental power (Re A.Brooks) you better get out from this hot domain.

2) The high salary + analytical complexity + limelight attracts bright young people. Bright young people tend to be innovative, even when there’s no such necessity

3) see my blogpost ruthless march of technology

fixation@ROTI@tech-xx : too result-oriented; disengaged

fixation@ROTI/payoff/success/result/accu … dampens job satisfaction+joy@learning.

This affects my “engagement”. Granted, we should not Ignore these ROTI factors, or those “smells” … instead we should evaluate our direction and take stock, but let’s not overdo it.

  • +ve Eg: Barcap option math
  • +ve Eg: Barcap swing learning
  • +ve Eg: RTS socket programming
  • -ve Eg: git
  • -ve Eg: curve building
  • -ve Eg: WCF

Consider a tour guide aiming for the tip at the end.
Consider Grandpa in his research career.
Consider a singer like 王杰 or the last few years of 邓丽君。
Consider Einstein’s violin

Q: has that increased your income or benchmark score? # more time in office, shorter commute, MSFM, c# ….

  1. This question can be posed to grandpa.
  2. This question can be posed to any education institute including the “top schools 名校”. Ironically the same questioners seem to be /fixated/ on these top schools for their kids. So for these people, this question is self-contradictory.
  3. This question can be posed to my friends engaged in quantitative investment analysis.

This question is harmful, misleading, derogatory, discriminatory, browbeating, pessimistic/fatalistic, myopic, … This question tosses aside many important things to our lives, our joys, and satisfaction —

  • career safety net
  • exploration of personal talents and personal interests
  • “in-demand” satisfaction
  • market depth
  • mobility between firms
  • freedom — I don’t want to feel “trapped”
  • observation (even conviction) on various topics, based on in-depth personal research

## Are those leetcoders really stronger@@

You wrote “I think many leetcoders are fresh graduates, or young people, who have more time and energy.  I know I cannot compete them …”

Q: Are those leetcoders really stronger in coding tests? Generally yes, but it depends on the type of coding question.

  • For a weekend take-home assignments …. I am not sure they would produce better quality code than us. (I’m not very good either). Code smells matter to my past interviewers.
    • They may not run as many tests as we do.
    • I tend to spend 200%+ more time on the assignment
  • For white-board ….. or remote dumb editor pair programming (west coast favorite), they may not be able to explain their thought process as well as I do.
    • My 2007 Google white-board interview used pseudo code, so the leetcoders’ advantage would be much lower.
  • For completely original questions created by a hiring manager then sent via hacker rank platform, leetcoders may not be fast enough to solve them, since leetcode problems can’t include every newly invented problems.
    • I hit two problems related to shortest-path algorithm, which can be similar to leetcode problems. I also hit very unusual problems.
  • For multi-threading coding questions, leetcode doesn’t help.
    • hit me at least 8 times — UBS, Barcap, Gelber, Bbg ..

exposure:=semi-automatic(shallow)Accu #$valuable contexx

Opening eg — In RTS team, granted I didn’t get deep[2] socket experience or latency /engineering/ experience, but over the years semi-automatically I would get some valuable exposures, by raising good questions about .. sockets; reliable order-book replication; error recovery; throughput engineering…

eg — in mvea team, I can get some valuable exposures to FIX; large scale and reliable equity OMS; low-latency (caching); order routing; automatic hedging; partial fills; limit orders; order cancels/amends; busts… Even if I don’t get deep experience on this job, my resume could claim genuine experience! Strategic positioning … (shallow) Accumulation

eg — in citi-muni, I got exposure to mkt-making; event-driven limit order repricing; PnL roll-up; mark-to-market; swing; JMS; Gemfire…

Key points about the “pattern”:

  • thanks to the strategic contexx, you get to accumulate (semi)automatically
  • robust commercial value in the skill
  • shallow [2] accumulation — I call it “exposure”, enough to impress some interviewers.
  • [1] small amount of effort — much lower than GTD, getting a job/certificate, losing weight
  • consistent effort ..

However, as the years go by, many developers stop digging with questions and others ignore the opportunities to dig into the difficult codebase because … they don’t have to:(. The automatic learning is a valuable option if you put in some legwork [1]. In contrast, some jobs don’t offer much automatic learning —

  • OC team: not so mainstream. I could still learn some WCF; reliable windows servers;
  • Qz team: poor portability. I could still learn some curve building; ticking risk;

[2] In contrast, here are examples of “deep” experience (hopefully serving as a protective entry barrier ) —

  1. from-scratch (95G) wait/notify solution
  2. from-scratch (95G) sybase stored proc to manage inventory in the face of competing orders
  3. home-prj order book replication in 2 coding interviews — Jump + iRage
  4. home-prj FIX client/server https://github.com/tiger40490/repo1/tree/jProj/java/com/tanbinFIX
  5. home-prj swing GUI to auto-update a table viewer

 

##[18]orgro lens:which past accu proved long-term # !!quant

(There’s a recoll on this accumulation lens concept…. )

This post is Not focused on IV or GTD. More like zbs.

Holy grail is orgro, thin->thick->thin…, but most of my endeavors fell short. I have no choice but keep shifting focus. A focus on apache+mysql+php+javascript would have left me with rather few options.

  • —-hall of famers
  • 1) [T] data structure theory + implementation in java, STL, c# for IV — unneeded in projects
  • 2) [CRT] core java knowledge including java OO has seen rather low churn,
    • comparable to c++
    • much better than j2EE and c#
  • 3) [T] threading? Yes insight and essential techniques. Only for interviews. C# is adding to the churn.
  • 4) [j] java/c++/c# instrumentation using various tools. Essential for real projects and indirectly helps interviews
  • [C] core C++ knowledge
  • [C] GTD knowledge in perl/python/sh scripting
  • [j] google-style algo quiz — Only for high-end tech interviews. Unneeded in any project
  • [R] SQL? yes but not a tier one skill like c++ or c#
  • coding IV — improved a lot at RTS
  • ————————also-ran :
  • devops
  • [C] personal productivity scripts
  • [T] probability IV
  • [C] regex – needed for many coding interviews and real projects
  • [C] low level C skills@RTS {static; array; cStr; reinterpret_cast;  enum; typedef; namespace; memcpy}
  • [!T] bond math? Not really my chosen direction, so no serious investment
  • [!T] option math?
  • SQL tuning? not much demand in the trading interviews, but better in other interviews
  • [R] Unix — power-user GTD skills.. instrumentation, automation? frequently used but only occasionally quizzed
  • [R] Excel + VBA? Not my chosen direction
  • [jR !D] JGC +jvm tuning

–strengths
C= churn rate is comfortable
D = has depth, can accumulate
R= robust demand
T= thin->thick->thin achieved
j|J = relevant|important to job hunting

self-help industry: your life is stuck in a rut

The self-help industry’s messages (SMS) resonates with me.

  • burn or rot
  • I often feel a lack of direction
  • i often feel my spare time is not productive
  • I often feel left behind on the slow track, but most of us are, anywhere I look, including the managers.
  • I often feel I’m not living life to the full
  • I often feel I’m not growing, learning anything new, but it’s the norm
  • When I feel my life is “not that bad”, the self-help industry would question me “Really?”
    • marketable skill — i feel lucky that I moved into finance tech, but a non-finance job like telecom would be fine too.
    • marketable skill — I feel it’s good that I moved out perl into java with MktDepth…
    • marketable skill — I feel lucky to discovery personal strengths in lowLevel java/c++/threading/unix…
    • I feel good about the father’s job I’m doing
    • I feel good about my investments
    • I feel good about my Singapore home and my commute
    • I feel 80% good about my healthy lifestyle
    • (just a brief subset relevant to this topic)

## clean design may look over-complicated

What is simple/clean really depends on (your knowledge of) best practices.

  • #1 eg: shared_ptr as member variable. Looks complicated but simplifies many things.
  • eg: nested container without pointer — looks scary[1] to java/python developers, but I think in c++ this is a proven design and simplifies many things.
  • eg: most of the 23 GOF design patterns introduce additional classes to address coupling and cohesion. These are classic and clean designs but add complexity. MVC pattern ditto.
  • try-with-resource looks complicated to me
  • RAII looks complicated to me

[1] inserting a entire sub-container  into the umbrella container requires copying the sub-container.

SOR/OMS/.. skill !! upstream

See %%algo trading dev xp !! HFT

SmartOrderRouting, OrderMgmt, FIX connectivity .. are core skills in algo trading dev, but are they really upstream, advanced skills? I doubt it.

  • in terms of interviews, I feel the domain nlg (jargon or architecture) required is a thin layer. The technical screening seems to focus on generic tech skills esp. coding test.
  • In terms of technical know-how (zbs), such a developer will NOT become stronger because of this experience. If she does acquire portable tech knowledge (possibly with an industry standard API rather than local wrappers), then she is probably experienced in that technical domain, but not other domains.

algo trading developers .. salary premium@@

Hi Friends,

For years I believed algo trading (including HFT) roles pay some 10-30% higher than regular financial IT roles. Now I doubt it.

For the sake of argument, let’s say the market rate for a senior developer is 160k base + some bonus + some stocks.

  • Factor: elitism and selectivity — buy-side algo trading jobs are notoriously hard to get. I tried about 10 places (Hudson River, Mako, SquarePoint, Gelber, DRW, Jump, TrexQuant, Susquehanna, WorldQuant, Millennium, 3-arrows…) Even if salary is higher, the chance of getting it is extremely low, at least for me.
  • Factor: role expectation and workload — I don’t have experience working at buy-side algo trading shops, but i do have experience in other demanding teams. I didn’t do so well, so I know the risk of sky-high expectation is very real. So even if salary is higher, how long can we keep it?
  • Factor: where is the intelligence — virtually all the algo-trading engineer roles I have seen emphasize low-latency rather than high-frequency. High-frequency is not really a job description for an engineer — developers optimize the low-latency system, to enable high frequency trading. The developers’ skill and intelligence required is substantial, but it’s not the same intelligence portrayed in mass media — which is trading algorithm or “alpha”. That intelligence is required only for the quants, the strategists, the portfolio managers, the traders… They are paid higher than most engineers. I guess some would say the engineers play a supporting role.
  • Factor: architect level — the high salaries (say 30% above market rate) are often at the architect (or lead developer) level. At that level, a regular financial IT job also pays above market rate. If we compare salaries at architect level, then algo trading doesn’t pay such a big premium. Perhaps 10%? Actually I’m not an architect so this level is not really relevant. I’m more comfortable as a senior developer.

For a regular senior developer, I feel algo trading roles on sell-side pays no higher than average financial IT. I know it from multiple interviews (BNP, HSBC, Citi …)

For a regular senior developer on a buy-side algo-trading system, I guess it can pay 10-20% above market rate, up to 200k base (?), but I also know of many other financial IT jobs in the U.S. paying 180k to 200k base.

Update — SIG can pays 160k for mkt-data or SOR roles but too selective.

My tentative conclusion:
* algo-trading on sell-side doesn’t pay a premium
* algo-trading on buy-side is a high-end domain, but there are other high-end domains paying at a comparable level.

algo-IV^QQ expertise

Background — became expert via external QQ benchmark` !!localSys xx

Q2: how are algo quizzes different from the C++ language expertise tested in job inerviews?
%%A2: I feel this algo expertise is loosely defined — mostly defined by a few hundred “fashionable and common algo interview questions.”

Ironically, the algorithm researchers and inventors may not ace these west-coast algo interviews 😉 Also, the non-trivial algorithms in textbooks are invariably too complicated for a 45-min coding interview.

There’s no deep insight or innovation in this game. This type of “expertise” doesn’t solve tough real-world problems. They can only solve simple real-world problems.

Top performers in this game are possibly students, exam-taking machines. They won’t feel proud of themselves as compared to c++ language expert or algorithm inventor.

Q2b: how are speed coding different?
%%A: in addition to A2, speed coding requires a mastery of a tiny subset of the language features and mastery of many coding idioms based thereon.

tech specialization=illusion #cod`IV

My dad is a real specialist. There are new comers who publish books to challenge the established theories, but it always takes a new comer many years of accumulation to make a name.

In contrast, Tech specialization suffers from the combined hazards of 1) churn (+volatility), 2) low entry-barrier,

3) screening interviews are the real watershed just like 高考. The screening used in job interviews are often coding tests. Your specialization is marginally relevant at best. So even if you specialize in, say, cryptocurrency, FunctionalProg, Containers, SOA, .. the hardest interview challenge you face is probably a generic algorithm coding test! Your specialization and accumulation could help you do a better job, but not getting the job. I feel this is like an actress auditioning. You don’t need such strong experience acting; you just need to wow them at audition.

4) market depth — (similar to my dad’s situation) after one job in this specialization, can you easily find another?

growing popularity@scripting languages #threat2c++

See also sg19 %% attitude on java, relative2c++ and ..

By 2019, I have noticed growing popularity of high-level languages like GO, node.js and python … all dynamic, scripting languages, in contrast to compiled languages. However, java is still fairly relevant. I won’t say too much about the possible reasons — ease of learning, ease of finding and training new hires…

i feel this trend is a continuation + evolution of a longer trend — from c/c++ to java/c#.

One direct impact of this trend — demand for c++ skill is shrinking[3], again. The remaining c++ jobs tend to pay lower than these “easier” languages or require deep expertise (like HFT shops) .. very bad deals for the c++ job seekers.

[3] It’s important to remember that demand for c# and perl developers is also shrinking.

come-n-go /Flash-in-the-pan — In my younger days (early 2000’s), there were also a growing choice of scripting languages but only one became general purpose — Perl. I feel GO, node.js may lose popularity over the coming years. These languages are popular partly because they are new. I described elsewhere that among the scripting languages, only 1 or 2 will dominate. Perl was dethroned by python.

See y re-enter c++if jobs fewer — I still believe c++ is harder and “builds character“, unless you spend 90% of your c++ career using wrappers that other people wrote and don’t bother to look under the hood.

OMS skill – !! standardized !! portable

I now feel the glorified OMS know-how is vague and poorly-standardized.

  • My B2bTradingEngine at 95G does a lot of OMS.
  • Smart order router module often does some OMS.
  • FIX connectivity module often does some OMS.
  • VWAP and other execution algos are usually considered a category of OMS.
  • … Many of the above are very different skills and have almost nothing in common.

I am afraid that even after 3Y or 5Y in some OMS system, when I apply to another OMS job they may realize I’m not a veteran.

Q: In comparison, which domain is more standardized in terms of skillset?
A: raw mkt data. Note 2nd-hand market data processing is much less valuable
A: basic bond math. Note more advanced bond math is rarely needed
A: FIX connectivity

  • How about VaR?
  • quote distribution systems in Citi and OC? poorly standardized

most(new/old)specializations turn out non-strategic

Look at my past vindicative specializations vs 

The Singapore government make technology bets. Venture capitalist make bets. Many organizations also make technology bets.. Every technology bet can lose or win.

In this post, I’m not advocating trySomethingNew. I am advocating specialization, which often requires accumulation and sticking to something not so new, like FIX, threading, SQL, sockets, bond math, …

 

learning nothing Strategic@a job@@ Normal !

Suppose after I stay on a job for 2 years, (beside lots of local system knowledge) I now only have some “familiar, un-fresh” tech topics to learn, like 2 of the following

  • Java — serialization, Eclipse, ..
  • Linux commands
  • Some domain jargons
  • Perl, or python
  • SQL
  • git
  • A bit of math # always turns me on!

… but none of the following

  1. algos
  2. low latency
  3. quant
  4. .. other hot domains but none guarantees higher income

Q: Would I lose interest and feel bored? Note 80% of my peers are in this situation. They are coping fine!
A: I think I will, but need to see the reality. Looking at my past “strategic” learning, I think many are similar to my TriTech direction — my naive preference for opamps design in my 1997 third year IA at TriTech — Nothing strategic after all.

Hoping for a job with something engaging and challenging is realistic and reasonable. Hoping for “strategic” is naive. For many years, I was driven by this TriTech motivation which inevitably made me feel I’m in the wrong job. (Only Barclays job felt “strategic” for 6 months.) Now looking back, ## past vindicative specializations shows a small number of vindicative examples. I feel that’s 30%, so most of my trySomethingNew or other specializations didn’t prove strategic.

However, Quartz is different. Learning something familiar but generic like java is still better than learning Quartz. Quartz is a killer.

[17] widely in-use, no longer quizzed #spring,SOAP.

I see a pattern — a new technology is getting adopted and quizzed in-depth at interviews. After 5 years, it is still a favorite, perhaps dominant solution, but 1) the know-how has become common knowledge and candidates are assumed to know it and 2) usage is now standardized and simplified, so the “bar” is lower, and candidates without the knowledge can easily pick it up.

No more in-depth questions needed. Therefore, time previously invested here is wasted, since only superficial knowledge is required now.

  1. Eg: spring/hibernate
  2. Eg: java servlets and JSP — From 1999 to 2008 these topics were heavily quizzed. Still widely in use but often invisible.
  3. Eg: Apache web server — In 2000 I was asked a lot on Apache details. Apache is still very popular. See https://w3techs.com/technologies/overview/web_server/all
  4. Eg: php — still widely used, but I feel not asked a lot. See https://w3techs.com/technologies/history_overview/programming_language/ms/y
  5. Eg: xml parsing — I used to get in-depth questions about DOM or SAX etc. Now I believe xml is still widely used
  6. Eg: web services, SOA, SOAP — Still very much in use
  7. Eg: HTTP protocol details like GET/POST, status codes
  8. Eg: Maven and Ant

Most of my examples are in the high-churn domains like Internet, mobile. I believe the same will happen to interview questions on big data, javascript, Android, iOS , blockchain, ..

The opposite list — some essential technologies underlying multiple technology waves were never heavily quizzed, but, after the waves subsided, remain rather relevant to many niche interviews.

  • TCP/UDP
  • SQL query — joins, subquery, case, ..
  • SQL and DB tuning
  • Unix automation — It can take years to become reasonably competent with all of bash, piping, subshells, trap, shell functions, string operators, file manipulation, and top 30 Unix commands
  • Unix system administration
  • Pthreads, a POSIX standard C library
  • http client programming
  • regular expression
  • Inter-Process-Communication
  • Java servlet session management
  • Java serialization
  • Java reflection

— You write —
There are still many projects using Spring. My current project is also using Spring, but it’s modified by internal team to create an internal framework. When people discuss in meeting, they say “Spring” to refer to this framework. But there are many pitfalls when I use it. To name a few:

  1. a) restful service is easy to implement in spring, ust add related annotations, but it doesn’t work, and after I spent a few days of research, I gave up and choose to use a internally created annotation.
  2. b) some configurations doesn’t work, parameters couldn’t be passed in. I still don’t know what’s the reason. The internal framework code is not accessible for other teams developers, so I don’t think it worth to spent more time to try to figure out.

For this project using Spring, the interview only mentioned this project is using Spring, but didn’t ask any questions about Spring.

For last year, I went through 5 interviews, 2 mentioned the projects are using Spring, and only one client asked some Spring questions.

I recall 5 years ago, 8/10 will ask spring and hibernate questions. Now, still a few clients asked Spring questions, but none asked Hibernate questions.

 

financial IT profession: !! so bad #long pre-retirement#Daniel

See also a profession u Enjoy with good {income+barrier}

(A letter I didn’t send out) Hi Daniel,

I think the programmer profession (financial software dev in particular) is not that bad:

  • Career Longevity — (I will delay or completely skip retirement) Reasonable in the U.S. Some professions (like doctors, accountants, teachers) are even better. but most professions (90% of the U.S. working population) can’t expect to have such a salary at age 65, including executives, lawyers, quants… Why do we always envy those guys?
    • Age discrimination — better than most professions including managers
  • Entry barrier — formidable, comparable to doctors or lawyers. See ##行行出状元,但有几个行业不容易出头
  • market depth — see salary probabilities(distro): mgr^NBA#market depth 
  • Job security — actually reasonable, despite the threat of globalization and threat from younger guys. I think the high entry barrier in financial IT is to our advantage
    • I was told some managers have job security concerns too
    • I was told U.S. university professors also face elimination (淘汰).
  • Work Stress — Significant variation across firms and across teams, but generally more stress than other professions.
  • Workload — much higher than other professions. Programming is a knowledge-intensive job.
  • Job market demand — very high for IT skills. Therefore people from other professions are drawn here.
    • Market depth is excellent. You can find plenty of jobs from $50k (easy to cope) to $300k
  • Income — well above average among all professions. Just look at national statistics and world-wide statistics.
  • not a sunset industry, not under threat like music, movie, journalism.
  • knowledge intensive, challenging complexity, engaging, but without excessive mental stress.

no IV muscle growth]sg..really@@

Common reason: no IV to give feedback on any “muscle growth”

Common reason: Most if not ALL of the growth areas have since become non-strategic, due to the limited job market

  • c# —  Actually my c# had tremendous and fast growth, no slower than my 2010-2012 peak period, but there was no IV to verify it
  • py — was growing fast in Mac, but no full time python jobs
  • quant — I went through a hell of growth in quant-dev, but gave up
  • c++ tool knowledge — was growing in Mac but not a QQ topic at all.
  • c++ optimization for HFT — I read quite a bit but can’t pass the interviews 😦 so I gave up

HFT guys seldom need2optimize latency;always quizzed@latency

I bet 90% of HFT developers (non-architects) Never need to optimize latency or throughput, based on my experience in

  • RTS
  • mvea

Why? The latency sensitive codebase is low-level and stable, so no change required. Regular developers only need to use existing frameworks and avoid latency penalties.

If the frameworks are strict, then there are few chances to hit latency penalty.

Best way to acquire professional experience in latency engineering — put your ideas to work in a real project and tune it until it works. Look at my wait/notify code in 95G. I think Kam did something similar in the TREP project in RTS.

If you don’t have such an opportunity, then you must read up in your spare time and ask some good questions in a good contexx.

accumulated domain xp fetching salary premium@@

Update: Sanjay in Stirt felt his domain xp should be protective but no. GTD was key in that team.

DaShan: Prior Domain experience may not translate into higher code quality or productivity on the next job, due to restrictions and important differences between systems.

salary depends on many factors like

– budget and company size
– relationship with the site owner
– urgency
– domain experience

When I first met DaShan, I felt he has some invisible upper hand as a developer with that domain experience. I felt that way because of the ignorance or “lack of domain knowledge”. Now I know that other people also see me with that Halo.

How about the high math of MSFM?