## portable skills~core c++j

I have found core java and core c++ outstanding domains for my personality. I feel there are not too many similar domains with

  1. feature: highly portable, standardized knowledge
  2. feature: requires more than shallow knowledge you can pick up in a crash course
  3. feature: low churn high stability

… so we can hope to accumulate. Here are some comparable domains:

  • pure algo 🙂 yes standardized, non-trivial questions. I can accumulate. Better than brain teasers
  • socket 🙂 Yes standardized questions
  • SQL 🙂 good despite shrinking demand
  • python 😦 Deep questions are rare and not standardized
  • quant 😦 questions are standard, but too few jobs
  • bond math 🙂 not so deep but good
  • algo trading 😦 i don’t think they ask any standard question
Advertisements

%% absorbency isn’t highest #don’t scold kids

Obviously 99% of us have limited technical talent like learning capacity, memory, problem solving speed .. but today I will focus on another factor “absorbency 耐得住寂寞” – the capacity to endure repetitive practice, sustained focus, and build the /mileage/ required for mastery. No one has unlimited absorbency. Sooner or later we all reach saturation point, or “breaking point” (Liu Shuo).

Opening example — I often tell my friends “Study is not hard work for me.”

example — When I was trying rock climbing, my finger and forearm would get too tired for me to continue. On the other hand, as a teenager, I could do 600 sit-up in 10 minutes non-stop.

Absorbency is one of the top 5 most important technical talents, esp. in the long run. Some fellow parents said the vast majority of primary school pupils are not very different in IQ. I would add that differences in absorbency is big. I often feel my son is weaker on absorbency when practicing piano, Chinese handwriting and math. I think he is improving. His swimming practice has always been good.

Some individuals are not the brightest/fastest but can tolerate high amounts of practice and can become formidable like 郭靖

Let’s shift focus to my own technical absorbency. I want to say I’m not the strongest so I should not blame my son.

  1. Coding drill – I can do more than most peers, but then some grads can solve more than 5 problems a day, and pass all Leetcode tests.
  2. Quant study – I did more than half my homework on my own. Some classmates were able to practice and study more.
  3. Getting a grasp on a big codebase – absorbency required, beside learning speed + memory.
  • eg flight — how much technical learning you get on a 15H long flight is a measure of your technical absorbency.
  • eg coding drill — Rahul, me and Abhinav each find joy in coding drill but many of my friends seem to focus on reading the ideas rather than practicing and solving problems independently. Focusing on the end result rather than the journey is no fun. The joy increases your absorbency
  • Positive feedback — is a key factor affecting absorbency. Coding practice and software programming offers immediate feedback.
  • self-doubt — I frequently (once a month?) question my endeavor (and the elusive strategic value) .. detrimental to absorbency. It’s already hard to absorb the practice even without these self-doubts.
    * yoga
    * risk analytics
    * socket

learn new tech for IV(!!GTD): learn-on-the-job is far from enough

Example — you programmed java for 6+ months, but you scored below 50% on those (basic) java knowledge question I asked you in skype chat. You only know what to study when you attend interviews. Without interviews, you won’t encounter those topics in your projects.

Example — I used SQL for at least 3 years before I joined Goldman Sachs. Until then I used no outer join no self-join no HAVING clause, no CASE, no correlated sub-query, no index tweaking. These topics were lightly used in Goldman but needed in interviews. So without interviews, I wouldn’t not know to pay attention to these topics.

Example — I programming tcp sockets many times. The socket interview questions I got from 2010 to 2016 were fairly basic. When I came to ICE I looked a bit deeper into our socket codebase but didn’t learn anything in particular. Then my interviews started showing me the direction. Among other things, interviewers look for in-depth understanding of

· Blocking/non-blocking

· Fast/slow receivers

· Buffer overflow

· Reliability

· Ack

How the hell can we figure out these are the high-value topics in TCP without interviews? I would say No Way even if I spend 2 years on this job.

[18] IV (!! GTD)body-building logbook by skill

I’d like to keep the table columns simple.

muscle delta when duration intensity
2D coding Q 3->5 2018 2D 3
perm/combo 5->8 2017 4D 3
string/array 6->7 2
backtracking 1->6 2018 2D 3
tail recursion 0->5 2017 x hours 1
big O analysis 7->8 2018 1
sorting algo nlg 7->8 2018 x hours 1
advanced recursion 3->5 2018 x Days 3
whiteboard best practice 6->7 2018 2 days 3
shared mem 2->4 2018 2 H 2 coding experiment
c++ for coding Q 3->8 2017 10-20 D 2 bbg
socket 3->6 17-18 x days 2 thanks to IV, not project
rvr/mv/forward 2->5 16-18 x days 2
noexcept 0->5 2018 1 H 1 Trex IV
other c++11 2->4 1
py basics 4 coding test 3->5 2
FIX?
STL?

spend 4Hr/D self-study@@ #XR

Hi XR,

I called you to highlight to you the realistic amount of hours we can spend on self-study.

Real example 1 —- I monitored it for months in my Master’s program — I would spend about 30 hours a week, including lectures, tutorial classes and homework(about 15-30 Hr/week, often taking up one full weekend-day). If I self-study without the program, I think it will drop to 3 hours a week. Why so low? Read on.

Eg 2 —- Some graduate students prepare for coding interviews by going through some question bank, just as you planned. Could be close to a hundred questions, during a 3M period — just my imagination, not based on any observation. They probably have 4 hours/weekday + 16 hours/weekend-day of free time. How many hours do they actually spend ? I would guess 10 – 20 hours a week. Why so low? Read on

Real example 3 —- In Singapore, I had kids, bills, grocery, household chores, home maintenance, family outing … so I had far less spare time than now, but still I probably had 1H/weekday + 8H/WED (weekend-day) for myself. In reality, I averaged 3 hours a week on self-study, but you know what a motivated person I am. Why so low? Read on

Real example 4 —- in Feb 2018 I had some very promising, valuable interview opportunities (FB). Even on the very last weekend before the interview, my time utilization rate was very very low. I lock myself up in office with no one else to distract me. I sleep only 7 hours and I eat in front my computer. In fact, I tell my friends “I spent entire weekend in office without going out of the building. You would think I could spend 15 hours a day on the preparation. Well, more like 6. Why so low?

It’s not all about motivation. I had high motivation in Singapore. I had the highest motivation over the recent weekend. But the numbers are still much lower than “theoretical limits”. There’s huge difference between wall-clock time vs real time spent:

Example 5 —- My son would spend an hour pretending to study 20 Chinese characters but only learn 3 characters and soon forget all of them. So the real time he spent was … a few minutes?

It’s about intensity. I call it laser energy intensity.

Real example 6 —- in a coding interview, my mind was working at top capacity. Over 15 minutes I solved some very tough problem. The rest of the interview (I was there for 3 hours) was far less intense, though it’s still intense, as intense as any technical interview.

Example 4 continued —- I time myself for white-board coding. Over 20 minutes my mind was working under immense pressure. Over the next 2 hours I sit down at a computer and make the actual code work. It’s still a lot of work, but my mind was at 50% peak capacity.

Use numbers to gauge the amount of workload and your time utilization rate. Say each problem (in homework or a coding exercise) takes average 30 minutes in exam / interview. Beside eating, using toilet, you have 15 hours to spend, so you would want to do 30 problems. How many can you actually complete? Perhaps 6, up to 10. Why so low? Our mind can’t operate at peak capacity for 15 hours!

In Example 4, after I spend 20 intense minutes, I need to wind down. After I spend 2 hours on lower-intensity work, I also need to wind down.

In Example 1, at the lecture/tutorial classes, the intensity was about 30% on average. That’s why I can do 2-3 hours. At a higher intensity the students’ mind would /disengage/ after an hour. All experienced teachers would slow down and give breaks when introducing tough technical topics.

In Example 1, When I spent 10 hours to complete my homework, my intensity was about 50% and I took many breaks. Actually the concentration time spent was more like 6-7 hours.

How does this mental “workload” compare to paid project work in a regular job, including work-from-home?

  • 😦 Project work is often repetitive or mundane. I can’t choose another topic.
  • 🙂 Paid work often has much bigger impact (otherwise why paid) and often a sense of achievement
  • 😦 Paid work stress is harmful, since the impact is high.
  • 🙂 Project work often has a discipline built-in, so we don’t wander off. This is similar to my Master’s program
  • 🙂 Project work requires us to keep going when the going gets tough. Also in my Master’s program

personal learn`]difficult workplace: %%tips #XR

I understand your boss could be pushing hard and you have colleagues around who may notice what you do…. Well, Not much wiggle room for self-study. A few suggestions:

  • try to put some technical ideas into the code. I did manage to put in some threading, some anonymous inner classes, some memcpy(), some local byte array buffer into my project, so I get to practice using them. (I understand the constraints of a tight time line…)
  • it takes a few minutes to write down new findings discovered at work. I put them in my blog. I also try to spend a little more time later on to research on the same topic, until I feel confident enough to talk about it in an interview.
  • I try to identify some colleagues who are willing to discuss technical issues in my project. I try to discuss only when boss is not paying attention. Boss is likely to feel we are taking too much time on some unimportant issue.

If the learning topic is not related to work, then I feel it’s similar to checking personal investment account at work. (In the ICE office now, some employees get a cubicle with 4 walls so they get more freedom than me.) Do your colleagues check their investment accounts at lunch time? I believe they always get a bit of personal time. In GS, on average very roughly 1 out of 9 working hours is spent on personal matters, and the other companies have higher than that. We all need personal time to call insurance, immigration, repair, … The managers might need even more personal time. I would guess at least 60 minutes a day is yours. Question is how to avoid drawing attention. I don’t care that much about drawing attention, so I often print technical articles to read, or read on-line, or blog on-line.

It’s more discrete to write email to record your technical learning. I often send those emails to my blog (email-to-publish) or to my personal email address.


Personal time (be it 60 minutes or 3 hours at some banks) is never enough. We just have to try harder to squeeze a bit more out of the 9 hours. If you are serious about learning in your personal time, then I see two much bigger obstacles
1) family responsibility and distractions
2) insufficient motivation and persistent effort (三天打鱼两天晒网)

In my Singapore years (4.5 years), I felt overwhelmed not by work but family duties, so my weekends/evenings were almost never put to good use for personal learning. I can’t just blame my family members though. I do get quiet time 10.30 pm to 12.30 and many hours on weekends. Somehow, I didn’t put in persistent effort so I didn’t experience significant growth in my technical capabilities.

A very capable colleague was able to do his math research at home and make progress. His wife is probably a full time home maker and takes care of their 3 kids. He is wealthy so he may have a maid and a separate study room at home. However, I feel a more important factor is his persistent effort and focus. A rolling stone gathers no moss. By the way, this same guy runs about 5 miles at least 4 times a week. Determined and efficient. Good time management and correct priorities.

If (a big IF) we are sufficient motivated, we will find time or make time, either the 60 minutes at work, or on trains, or at home, or in Starbucks. In reality, very few individuals have that level of motivation, so I believe some external factors can help, such as (my favorite) —

* jot down some idea in a draft email and do a bit of research whenever I get time to build on the idea, until it’s complete and fairly substantial. Idea could be something I overhead, an idea I’m truly interested in. The learning is mostly in the research but also in the subsequent reviews. If I don’t review the email, I will forget most of it. When I do review it, I not only refresh my memory, but I often discover connections with other things I studied, or find new ideas to learn — 温故而知新. Learning is associative, like growing a spider web.

sense-of-urgency, for kids+professionals #IBM #Alok

If there is no time frame specified, there is no sense of urgency and nothing to measure against. If you set a time frame you will be able to use this as a reason to go back to your boss at regular intervals and update them on the progress, ask for help and set appropriate expectations throughout the year.”

So said an IBM site. “Sense of urgency” is a simple but powerful propaganda slogan introduced by the 1993 incoming CEO of IBM. In my experience, workers with a consistent sense of urgency will be noticed by the boss. A boss without a sense of urgency could be a nice boss but not effective.

I like this slogan, because my son needs it, badly. It should be instilled at a younger age.

If a frog sits in a slow boiler, she won’t have the sense of urgency to jump out.

By default, folks fall into inaction — weight management, regular exercise, or self-improvement. Setting some kind of measurable goal is a simple device. My dad goes a step further — until now, he has daily plans and he gets things done.

Take for example my house search. If I don’t have any sense of urgency, I would simply postpone indefinitely any research or discussion including school-districts. Reality is, sooner or later my son needs a school. The more I postpone my research, the less time I have to do a proper research. So I do feel a sense of urgency, but hopefully I’m not hitting too many colleagues in the office.

Let’s remind each other the sacrifices we made — the cut, living away from family, additional expenses including rent.

[[Focus]] for technical xx

See also my email on “the zone”, the “creative engine”.

As the book author, Dan’s idea of focus is stricter. It’s related to a student’s focus. I told my son he needs better concentration. As a student I was very good with concentration.

Compared to my “traction” days, I now spend too much time emailing, blogging on non-tech, calling home, asking about housing, shopping, food preparation … But that’s not the key. During the work or study hours, I am not focusing as I did in primary school.

Traction days are associated with growth, concentration, positive feedback loop, tech blogging….

I think Yang was the first manager who pointed out I need better focus. Perhaps he meant I tried to study all the systems in PWM Comm all at once. I made real progress (with ErrorMemos, SQL, Java …) after I sent my wife/son back to Singapore.

I made real progress with quant stuff during the Barcap days. However, I also had to keep job hunting remotely.

Avichal, who sat beside me, said I had too many distractions…

I guess the initial code complexity requires focus… Without sufficient focus, the complexity is just too much for my capacity. Better shut out everything else and focus on delivering something non-trivial. Remember PWM, Quartz, …

Grandpa had a lot to say about focus…

##study subjects I enjoyed as student(+professional)

(You don’t have to read this lengthy blog post.)

All my examples below required a long period of persistent effort. My son is unwilling to put in even 30% of my effort, so he wouldn’t find any enjoyment.

After the initial few hours of curiosity, learning any subject was always challenging, boring, or repetitive. The mental workload always wears us down, so we always needed conscious effort to stick to it I.e. stay focused and apply ourselves. The joy comes as a light at end of the tunnel. No exception.

In contrast, watching TV, playing electronic games, eating, sight-seeing, shopping … don’t require any effort. When we get bored with it, we are free to break away — there’s no measurable progress to worry so give-up has absolutely no consequence. We choose to stick to it (say, a game) only because we enjoy it. Therefore no effort required.

If a teenager takes on professional gaming as a paid job, he will invariably find it monotonous, repetitive, uninteresting, dull, tiring, before he can earn any money. Those gamers who do earn money are able to earn money precisely because they put up with the boring stuff — persistent effort.

— Now the study subjects I enjoyed, ranked by amount of effort:

I enjoyed blogging on technical subjects. My QnA interviews directly benefit from my blogging but this is not the biggest reason for the enjoyment. In fact, most of my technical blog content is not required for interviews. I can see my blog improving in quality (and quantity) as I revise and expand each article progressively. Blogging let’s me achieve deeper understanding on many tough topics like sockets, reflection, c++template, move-semantics, quant, complex SQL, … I call it "真本事". I then notice the superficial knowledge in fellow programmers, who studied only 20% of what I read.

I enjoyed memorizing English words, in my teenage years. I developed my own method. I was able to memorize 30 – 100 words a day for a few months (I refresh each word 20 to 40 times spaced out over 6 months). The fast progress was "breathtaking". Self-confidence grew. I actually learned some words my English-educated Singapore classmates didn’t know!

I enjoyed Chinese composition in my secondary Year 2-3, when I found my literary "voice", my style of writing. For about 2 years before that break-through, I was always in the bottom 10% in my class in terms of composition and every assignment was a pain like giving birth to a short, boring baby. The pain, self-despise, self-pity was one of the deepest in my childhood. I felt inadequate almost like crippled. The breakthrough was a liberation. I achieved it by writing daily essays for a few months. My output slowly improved from very dry, simple "流水账" to have some meaning, some color, some personality, some nice expressions. It became creative expression.

(Paradoxically, as a student I was very strong in math/physics, but I never had the same enjoyment as I had learning English and Chinese composition. I think my joy was in overcoming highly visible personal limitations — sense of breakthrough and liberation.)

I enjoyed piano playing after years of repetitive, boring, tough practice. In my life I never again put up with this much practice — 5000 repeated practices x 2 minutes/repetition in some cases.

I know more c++than c#(spent X years full time)

Action: spend enough (30?) hours revisiting my 1000+ cpp (including socket, stream..) blog posts and explore just a little beyond them.

(I blogged about exactly the same observation before …) Here’s a paradox — i spent 2 years in a full time c# job, arguably longer than my full time c++ experience. However, sometimes I feel I understand dotnet less than C++.

Reason? Many aspects of the languages are never used in-depth on work projects, so length of experience != depth of experience. We need spare time self-exploration to learn those aspects like:

– template meta programming
– custom operator new/delete
– memory tuning
– memory leak detection
– threading
– async communications
– pure virtual
– ++i vs i++

(… not a complete list by any measure.) Interviewers are notorious for testing these obscure or advanced, theoretical topics.

create positive-stressful env n keep fit #UChicago;c++job

Hi XR,

We talked briefly about this human nature — no one has that “perfect” will-power, perseverance and persistent motivation, therefore as a result the environment we put ourselves in can help induce a significant personal effort.

Here’s the situation we discussed — you are in a perm job, you won’t feel motivated to keep up interviewing skills. I guess our workload + family responsibilities will “crowd out” any personal study plan to keep learning and stay fit for interviews. (This looks like one example of the boiling frog — a frog won’t jump out of hot water if on a slow cooker.)

I see many examples of this human nature:

  • I mention my blog and my github in my resume, so I sometimes worry that prospective interviewers may spot mistakes. That fear keeps me motivated to improve my “published” work. When
    I review my work, I often learn something, and I often feel excited that I’m slowly building up a positive “public” profile.
  • Without taking my current c++ job, how many (spare time) hours would I be able to put into c++ self-study? I think average 2hr/week is achievable but not easy. Nowadays I actually put in about 5 to 10 hours outside 9-6 work hours, on my c++ projects.
  • In Singapore, my wife can put in some effort to improve English but she doesn’t have time or energy. She finds it difficult to put in the effort. In the U.S. she WOULD indeed put in more effort learning English. On the other hand, many students in China could build a formidable English vocab simply by reading — they have a persistent motivation.
  • 30 minutes moderate exercise 5 times a week — is a basic guideline, but I know very few adults actually doing that consistently. I bet when one’s health condition requires that much exercise, there will be more motivation and more effort.
  • Once I paid course fees, I routinely put in 30 hours a week on my financial math studies. In theory I could self-study the same, but how many hours would I put in on average over a year? Perhaps 30 minutes/week!

(Most of my examples are about self-improvement and self-motivation. )

it takes effort to remain in financial IT#Ashish

NY vs NJ — cost is basically the same my friend, in terms of rent, transportation, food … Most of my friends probably live in New York city suburbs. (I know no one who choose to live in NYC, paying 4% city tax.) My cost estimate is always based on my experience living in NYC suburbs.

The “relief” factor you identified is psychological, subtle but fundamental. It resonates in my bones. It’s a movie playing in my head every day.

In the present-day reality, at my age I probably can still find jobs at this salary in Singapore or Hongkong, with growing difficulties. How about in 10 years? Huge uncertainty. If you were me, you have to take a long, hard look into yourself and benchmark yourself against the competing (younger) job seekers. You would find negative evidence regarding your competitiveness on the Asia financial IT job market. That’s the get-the-job aspect. The 2nd aspect is keep-the-job and even tougher for me. Therefore in the U.S. I make myself a free-wheeling contractor!

As I age, I take the growing job market competition as a fact of life. I accept I’m past my prime. I keep working on my fitness.

On both [get|keep ]-the-job, U.S. offers huge psychological relief. Why? Ultimately, it boils down to long term income (and family cash flow). Since I believe U.S. employers can give me a well-paying job more easily, for longer periods, I feel financially more secure. I will hold meaningful jobs till 65 (not teaching in polytechnic, or selling insurance etc). I can plan for the family more confidently. I could plan for a new home.

There are many more pros and cons to consider. Will stick to the bare essentials.

Singapore offers security in terms of the social “safety net”, thanks to government. Even if my Singapore salary drops by half, we still enjoy subsidized healthcare, decent education, frequent reunions with grandparents, among many other benefits (I listed 20 in my blog..).

Besides, I always remind my wife and grandparents that I have overseas properties, some paying reliable rental yield. Under some assumptions, passive income can amount to $5k/mon so the “worry” my dad noticed is effectively addressed. Another huge relief.

As a sort of summary, in my mind these unrelated concerns are interlinked –

· My long-term strength/weakness on job markets

o coding practice, lifelong learning

o green card – a major weakness in me

· family cash flow

o passive incomes

o housing

· Singapore as base camp

Do you notice I didn’t include “job security” in the above list? I basically take job-Insecurity as a foregone conclusion. Most of my friends are the opposite – taking job security as an assumption. This difference defines me as a professional, shapes my outlook, and drives me to keep working on “fitness”.

If I had no kids, I would have already achieved financial freedom and completely free of cash flow concerns. As a couple, our combined burn rate is S$3k-4k/mon but our (inflation-proof) passive income has/will reach that level. Also medical and housing needs are taken care of in Singapore. In reality, kids add S$1k-2k to our monthly burn rate. We are still on our way to financial freedom. So why the “worry”?!

From: ASHISH SINGH

Sent: Thursday, 16 March 2017 8:07 AM
To: Victor Tan
Cc: ‘Bin TAN (Victor)’
Subject: Re: it takes effort to remain in financial IT

That jewish guy and avichal are exceptions . Most of developers struggle and get success by hard work, by spending time with code for atleast 1-2years in any new company. I have not met other fast learners like them. Those are rare to find.

Regarding the worry that your dad always notices , i think this is quite obvious “more money more stress” but yeah this is the right time for you to move to USA where there are plenty of tech jobs . This move will surely give you a shy of relief.

Regarding the cost if living, why would you want to stay in NY , it is indeed costly. How about NJ?

QQ^ZZ mutual exclusion: c++/java..

Background — The QQ/ZZ framework was first introduced in this post on c++ learning topics

Focus on tough not ordinary topics in c++ or java.

QQ: By and large, tough topics in job interviews are never needed in projects.
ZZ: By and large, tough topics in work projects (zbs) are never tested in job interviews.

Mutually exclusive =>

  • effort (laser energy) I spent on textbook knowledge during job hunt basically hurts my work,
  • effort I spent on tough zbs topics for a project very seldom help with job interviews.

Therefore, some optimizing candidate would make the minimum effort to get by on projects, and channel the energy to QQ knowledge or algo challenges.

X years needed2become effective java GTD guy

Update — It would be good to have some measurable yardsticks for GTD competence. Right now the g_GTD_zbs tag includes at least 50 blogs.

XR, (letter sent in July 2010)

Here the goal is not getting the job but keeping the job. You said there’s a difference between a knowledgeable java student and an experienced java developer.

I said I only did java for 3 years.. well maybe 4. I think much of that time I was doing repetitive work, not learning. About half the time (1-2 years) I was learning, by experimenting, reading, discussing, debugging… (I actually learned a lot of java from our long phone conversations.)

I feel if a bright young guy is given a good training project, then in 6 – 24 months he might be able to reach my level of java proficiency.

I also said a lot of young coders could become faster than me with a specific java task after a few months of java learning. Well … an apprentice carpenter could become faster than his master at sawing along a line, but can’t replace the master completely. I feel the hundreds of decisions made each week by an experienced java developer are often based on more experience.

C# is probably same thing – 6 to 24 months to become effective. A very experienced c# friend told me “3 months”. I spent about 6 serious months and another 12 repetitive months on c#…

Venkat, one of the fastest-learning developers I have worked with, said (in Lau Pa Sa) “To get really competent with a new language (like java, c#, python) honestly you need at least one year.” Venkat had the strongest C++ skill I know. I witnessed how he picked up c#. He later struggled a bit with java.

The 64 million dollar question is, what are the really effective learning strategies. I don’t have good answers. Guo Qiao was an advocate of programmer online learning. Some people even suggest taking part in open source projects, to learn from practicing master programmers… I’d love to hear your input.

Thanks.

master a skill^depend@colleagues – you decide

Nikhil surprised me when he told me he frequently asks for help… Until then, I always thought I need to go through the problem solving by myself, unaided, or I will never learn.


Context — trading engine team workload is often 3 times the normal, comfortable level. Headcount is often below 33% normal headcount. (This is a result of time-to-market delivery experience — The fewer developers, the lower communication overhead, the more nimble.)

Challenge – Suppose you are suddenly asked to finish something quick and dirty in half the time previously allocated.

Higher pressure usually means you need to depend more heavily on other developers in your own team. Many things you should not *want* to understand. You need to decide what analysis/research/investigation to “outsource”, but still retain control.

A component could be your colleague’ responsibility, but familiarity improves your *velocity* and reduces your reliance on the author. With mastery, you often discover tweaks and shortcuts, saving time when there’s no time at all. You can become more knowledgeable than the author about its usage!

Look at GS colleagues Nikhil, Anil, Nicholas… They ask each other dozens of questions for everyday BAU, but each mastered how to read and experiment with JIL, sybase stored proc, CVS, unix commands, …They don’t master every subject, and they don’t lean on others in every domain.

Eg: how to programatically insert reference data into an empty data store, rather than through a complicated (buggy) GUI. The GUI route is un-automated if you must repeatedly rebuild from scratch.
Eg: key tables. Out of 20 to 50 frequently used tables, about 2 to 5 tables should be familiar to everyone, like Comm, trade, offer. Before you go through these steep learning curves, you aren’t initiated into the club.
Eg: how to locate and parse the logs
Eg: how to test-send and test-receive messages
Eg: how to restart some of the important servers
Eg: how to identify common issues in each essential server
Eg: parse a few cornerstone config files
Eg: parse autosys JIL to see dependencies and flows

But as a Greenfield dev consultant, you aren’t expected to get too deep into prod support or BAU.

A short-term greenfield developer consultant is not a “Subject Matter Expert”.

which programming languages changed toooo much

No elaboration please. Pointers will do.

perl 6?
python 3?  but the excessive changes are not taken up!
php5 was better received.
C# ?
MFC -> winforms -> WPF ? Different technologies. More like delphi/powerBuilder/coldFusion declining than a language evolving.
java 8? not much change. Lambda is the biggest
c++11? not much change. Rval ref is the biggest

## once-valuable skills – personal xp

struts
perl – lost to python
tomcat, weblogic, jboss
apache, mysql
dns, unix network config
autosys – not used after GS
[$] sybase, oracle – after ML edge project I didn’t need it again.
[$] VBA – used once only. But Excel is going strong!

[$ = still has value]

–random list (not “top 10”) of longevity skills
eclipse, MSVS
Excel and add-in
STL, boost
javascript, though I don’t use it in my recent jobs
python
Linux shell
vi
compiler skills
make, msbuild
bash and DOS batch scripting, even though powershell, vbscript and python/perl are much richer.

2 focuses when learning a new tech skill: IV^GTD

When we learn a new tech skill, we need very, very specific goals and guidelines to help us choose what NOT to dig into too early.

Focus #1: IV topics, a well-defined focus. Basically textbook knowledge. Go deep enough but not too deep on a particular subtopic. Examples — op-new, pure virtual, const ref param of copier, recursive locks, OOM calling new,

Focus #2: GTD i.e. dev productivity — as measured by managers, bench-marked against other developers. Incidentally, perceived productivity sometimes depends so heavily on local system knowledge that a new hire always lags behind no matter what. I feel productivity and GTD has low dependency on #1, as most of the textbook knowledge is irrelevant.

Focus #2b: tools — see post on 2 categories of tools.

Focus #2c? tips on StackOverflow. No such books. No way to

Focus #?: best practices — or Acceptable practices and well-trodden paths

Focus #?: learn all the syntax details to read any source code on books, interviews and online

Focus #: high-level design decisions, for technical leadership roles. Myriads of decisions are made every hour during development, some at high some at low levels. Apparently, a lot of these decisions involve theoretical understanding.

local jargon,local domain knowledge = local sys knowledge

I agree that finance IT skills tend to “converge” across big banks, so we poor developers need not learn so many damn shit new stuff like git, python, continuous build…

On the domain knowledge side, the “local system knowledge” has no convergence as such, so each time I take up a new system, I must spend the time on the stored procs, table relationships, batch jobs, feed files, etc. There are also a lot of non-technical local knowledge. I hear you when your manager complained “what do you know about the system?”

Local system knowledge can be a long learning curve. Those who write the code in the first place has a huge advantage —

“It takes me 3 months to write this damn shit, then we add new features and fix bugs, but it’s still familiar to me. A new guy would take a year to get familiar code-wise, provided he has chance to explore all the key components. If he’s only exposed to half the system, then he would never know  the other half.”

(Therefore, I like to rewrite stuff.)

In http://bigblog.tanbin.com/2011/04/3-types-of-domain-knowledge-in-finance.html I mentioned 3 types of domain knowledge — Jargon, Math, and Architecture.The Math part is stable. Local system knowledge includes a lot of local jargon. Sometimes generic jargon can help us pick up the local jargon.

constant study (over last5Y)keeps brain young#like muscle

I believe our mind (esp. our memory) is like a muscle. If we don't

seriously use it, it tends to age, weaken and lose capacity.

Over the last 5 years, i spent 2009-2011 in the US on my own —

tremendous learning and improvement, perhaps the most active learning

period in my professional life. Then after I came back to SG, i

studied financial math program, c#/dotnet, and some c++.

I feel the serious study keeps my mind active. However, over the last

18 months, I notice various signs of my learning capacity reducing,

but it's not all due to aging —

* biggest factor is lack of concentration, due to kids and family commitment

* not enough time to periodically re-visit each topic,

After all, I feel it's vital and paramount to keep our mind in

constant learning mode. 活到老学到老.

what c++ topics are valuable in the long run on Wall St

When you first pick up a language (like c++) from 11 textbooks, you form a view of what topics (know-how/insights) are important, and worth spending time on.

However, once in the field you will get a different view. Over the years, I realized
++ remote debugger is critical

++ lots of C techniques are widely used on Wall St.

++ MSVS is a dominant tool, so is GNU toolkit.

++ multiple inheritance is still relevant

— most whiteboard coding will use C constructs + vector/string, not fancy c++ features

— STL containers are widely adopted, but not everything in STL is equally important.
— portability isn’t a issue.

— template techniques are powerful but seldom used on Wall St
— boost shared_ptr is widely used, but not other boost libraries

— strings — many teams have their in-house development, so the std string is less imp

— I feel OO techniques are not as popular/easy as in java/c#

So over 50 years (if I were to work that long), i’m going to discover all the important topics. Now question is how to quickly discover them, so as to focus on the most valuable.

Answer — I feel a beginner is better off changing project frequently. The more diverse projects you take on, the faster you discover those important “topics”.

The rolling stone gathers no moss? Well, i feel this is not the case for the beginner dynamic traveling consultant:)

system knowledge about a Live financial system – 3 sources (GS

When you work in financial IT, real insights into a live financial [1] IT system comes from

A) hands-on experience, often local system knowledge
B) user interaction — so you know the 80% of the codebase that aren’t important.
B2) production problem solving, with user-impact or business impact. As explained elsewhere, these steers our effort to the key sections of codebase. 20/80 rule — 20% of the codebase holds 80% of the system functionality, business logic and production issues.

Without B, you as a developer can be familiar with many sub-systems but without confidence to support users on your own, implement numerous changes, assess impacts, estimate effort, identify root causes… In many organizations developers aren’t assigned such tasks, but GS was very different and very effective.

C) good domain knowledge — This is required only in quantitative systems. Otherwise, I feel A+B will give us (system developers) more than enough domain knowledge. When C is relevant, then A+B will make you productive without “insight”.

[1] including real-time trading systems, reporting systems, financial analysis systems, STP etc.

remain relevant in a given technology(fundamentals

Q: At age 55, will you still qualify as a swing developer, or SQL developer, or python developer, or Unix admin?

1) An important factor is the stability of the language and the changing demand on specific skillset in that Language (contrast java vs SQL), but let’s focus on another factor — fundamentals.

2) In every language, there’s a body of fundamental knowledge (“knowledge-pearls”) that are Empowering and Instrumental. They facilitate your learning of “superstructure”. Many hiring managers believe strong fundamentals are the most important skill with a given Language.

I feel low-level knowledge will help you remain /relevant/ over 10 years. Some of the most tricky and frequently quizzed sub-topics are low-level — threading, memory-mgmt, collections, subclass memory layout, casting, RTTI, vptr, c++ big 3, …

3) I feel an appreciation of relative strengths/weaknesses of alternative technologies will help you remain relevant. Fundamental knowledge would help you appreciate them.

4) essential libraries of threading, collections, I/O, networking, … are not strictly part of the fundamentals, but part of mandatory skills anyway.

learning common code smells (time-savers)

XR,

Every experienced and self-respecting developer cares about maintainability, because everyone of us has first hand experience tracing, changing, and “visualizing” convoluted logic in other people’s code. Once you know the pain, you intuitively try to reduce it for your successors — altruistic instinct.

At the management level, every experienced /application-manager/ recognizes the importance of maintainability since they all have witnessed the dire consequence (earthquake, explosion, eruption…[1]) of making changes to production codebase without fully understanding the codebase. More frequently, management knows how much developer Time (most Wall St teams are tiny — nimble and quick turnaround) it takes to find out why our system behaves this way. Bottom line — unreadable code directly impacts an application team’s value-add.

Technical debt is built up when coders take shortcuts, hide dirt under the carpet,  and “borrow time”.

If we ask every developer in each of our past teams to write down their dreaded code smells, some common themes will emerge — like rampant code duplication,  VeryLongVeryUgly methods, VLVU method signatures or 10-level nested blocks. Best practice means no universally-hated code smells.

Beyond those universally-hated, code smells can be widely hated, hated by some, frowned-on or tolerated … — different levels of smelliness.

I am trying to understand the common intolerance of the common developers (not the purists). You seem to suggest there’s no point trying to. If someone says there’s code smell in my code, i’d like to know if it’s universally hated, or hated-by-some.

In many case, our code is seldom debugged/modified/reviewed by others. In those cases, it doesn’t matter that much.

Happy new year …

[1] In reality, they often get the “patient” to sigh on the not-responsible-for-death before operating on the patient. They also have a battery of UAT test cases to make sure all the (common) use cases are covered. So the code change is a best-effort and may break under unusual circumstances.

3 types of domain knowledge in finance IT

Domain knowledge is specialized, hard-to-find, and an entry barrier. I see 3 broad categories of domain knowledge — Lingo (jargon), Math and Architecture

— 1) Lingo (Jargon) — Technical or Non-technical professionals in a given trading system must share a few hundred terms, phrases, verbs and adjectives. Each jargon term often has a superficial “face” meaning and connects with other jargon terms. The full meaning in context often requires a wikipedia page. Some Random examples that came to my mind — fixings; basis risk; option knock-in; tick DB; Basel; lockfree; move-semantic; rvalue ref.

Roughly half the jargon terms involve math. An IT guy can either memorize the derived mathematical conclusions or examine the underlying math. Random examples of such jargon include — volatility surface; yield; price sensitivities; yield’s impact on FX options; when to use stress testing vs VaR. I feel many people in IT don’t really have a firm grip on these. We go into a restaurant of financial jargon and look for fast-food. I call it fast-food culture, or quick-answer culture. Anything we don’t easily understand, we like to say we don’t need to understand. If we don’t make a conscious effort, we will always stay a financial laymen. As a result, a “conscious” programmer completely new to finance for 6 months can understand concepts deeper than a 5-year veteran.

Rather than “Jargon” I prefer “Lingo” — more general and less technical.

— 2) the Math part is a body of knowledge created by quants and PhD’s, over the past 50 years or so, perhaps starting with bond math. Needs high school math and later some calculus. For many IT folks who left school 5+ years ago, even the high school math pieces are not a piece of cake. If I don’t invest hours of spare time, i won’t fully understand half the theories in bond math which is all high-school.

The math in finance looks simple but needs to be rigorous. I feel pricing calculations in fixed income and in options often use a precision of 5 to 10. A shallow understanding could overlook details.

Financial math is a sizable body of theory, but most trading IT developers need only the basics. If I could say that I needed 6 months to grow from zero-finance-knowledge to be competent enough to help build trading engines, then obviously my roles didn’t need a lot of financial math.

— 3) system architecture + infrastructure + best practices — components like pricing, risk analytics, high speed market data, pnl explains, stress test, visualization, exchange gateways, SOR, DMA, mark-to-market, trade capture …
It’s possible to talk about an architecture of a complete suite of components but I prefer to talk about architectures of individual components. I prefer this because architectures are vastly different in terms of volume and real-time nature.

Banks want to hire experienced developers to help them re-architect, so as to stay competitive in the “arms race”.

Architecture domain knowledge changes faster than math or jargon knowledge. Just like jargon and math, architectural knowledge is specialized and not a commodity skill, so relatively few people in finance IT has it, creating a has-vs-hasnot divide among candidates. It’s difficult to gain that insight and knowledge because developers don’t need to know all the design decisions once made to fix up the live system they are now supporting. Team is set up such that you are given a lot to do just to get-things-done so you have no spare bandwidth to learn other components, but non-trivial insight into neighboring components are necessary for an architectural understanding.

I have seen a few impeccable blue-prints that fail in practice and need major changes.  Anyone can come up with great-sounding architectures but most of them are /sub-optimal/. Reason can be “hard to debug”, “inflexible”, “learn curve”… Therefore real world, proven architectural domain knowledge is rare and valuable.

Most architectures rely on solid, time-honored infrastructure software like Message-oriented-middle-wares, RDBMS, distributed cache and grids, cross-platform RPC/ORB, thread libraries, xml. Vendors always say “we are perfect” but good knowledge about their weaknesses and limitations are hallmarks of real architects.

Some say fixed income has more domain knowledge but that’s true only in the math area. Equities (and FX?) HFT probably have a larger body of domain knowledge in the architecture space.

Market data systems have substantial jargons + some architecture.

Risk systems probably (based on hearsay) has all three, but I feel only a small percentage of the software developers need math.

gaining traction ^ learning curve gradient (diminishing ROTI)

Context — learning a new software language, API, entire server (OS, DB..) or toolkit, with non-trivial concepts embedded therein.

The more common pattern is the learning curve. Initial gradient is often higher as pick up speed. After 6 months (or 1, or 12..) it flattens and tapers off and you experience diminishing returns.

A 2nd pattern is “gaining traction”. For the first 4 weeks (or 1 or 20..), you spend a lot of time reading and experimenting but without growing confidence. You fail to connect the dots. I experienced and overcame this wheel-spinning process in

– swing
– C++
– java
– python
– EJB
– JMS, RV
– low latency
– instrument pricing
– options
– IRS

Solution?

which (vendor-)implementations to invest in

I may have talent in comp science theories. But I place my highest emphasis on familiarity with implementations — tools, utilities, products, packages, jars, building blocks — download-able stuff.

With theories, a talented student can progressively go deeper, higher, sharper, and in less time as her foundation strengthens and broadens. I feel less so with tools. Familiarity with implementations takes a hell of time.

For example, when something doesn’t work with something else, it often took me a few minutes to a few days. Therefore I have a strong aversion for new technologies and a strong affinity to mature, time-honored tools.

For example, countless designs work great on paper or during interviews. The #1 thing about any tool is the limitations you are likely to hit. Even a minor drawback can derail your entire project.

Some tools I’m investing into (either my old turf or new targets, the +++) and some (–) I’m divesting
++ c/java debuggers
++ sybase, oracle, mssql
+ rv, MQ (JMS is a spec, not an implementation)
+ unix sockets
+ STL
+ eclipse
+ pthreads implemented in linux and solaris
+ python + solaris/linux latency tuning
+ pl/sql?
– spring / hibernate
– bash customization
– http
– mysql, php
– vi — needs 2 years. See [[Productive Programmer ]]
– any local system knowledge

knowledge@vendor products is far from enough4GTD

Consider typical non-trivial financial app based on 1 or a couple primary vendor products + some database + some JMS server + some app server + some web server. The #1 “primary vendor software” could be a distributed cache, spring, ETL, or something like ION or something else.

Q: How much complexity is in the vendor software, and how much is in the site-specific code including localized customizations of the vendor software?
A: 10% vs 90%

How about the ML Edge b2b trade engine? 70/30% at this stage (March).

harmless productivity tools ^ frameworks like ..Spring

[[The Productive Programmer]] by Neal Ford advocates learning a large number of _simple_ tools for productivity. Tools could be a complex but we use them in a non-intrusive, simple manner. Many other tools promise to help productivity but are traps – spring, hibernate, jaxb, xslt. Once you employ them they tend to become your master.

I have no such issues with perl/shell scripting, testing tools like junit, unix utilities, text editors, eclipse, maven/ant (?) A critical feature is, the option to use the tool in part of my project, and use alternative tools for the remainder. The tool should not be possessive and exclusive.

Hibernate is more gregarious than Spring, as you can mix Hibernate and straight JDBC in different modules of your system. Spring, being a framework, tends to engulf your entire project. You are then constrained by that framework developers’ decisions.

avoid convenience high-level wrappers; program against low-level APIs

(another blog post.)

Many examples of the same observation — convenient high-level wrappers insulate/shield a developer from the (all-important) underlying API. Consequently, he can’t demonstrate his experience and mastery in that particular “real” technology — he only knows how to use the convenience wrappers.

example — servlet — in one of my GS trading desks, all web apps were written atop an in-house framework “Panels”, which was invented after servlet but before JSP, so it offers many features similar or in addition to JSP. After years of programming in Panels, you would remember nothing at all about the underlying servlet API, because that is successfully encapsulated. Only the Panels API authors work against the underlying servlet API. Excellent example of encapsulation, standardization, layering and division of labor.

Note the paradox — guys who wrote those wrappers are considered (by colleagues and job interviewers) as stronger knowledge experts than the users of the wrappers. So, everyone agrees knowledge of the underlying technology is more valuable.

example — Gemfire — one of my trading engines was heavy on Gemfire. The first things we created was a bunch of wrapper classes to standardize how to access gemfire. Everyone must use the wrappers. If you are NOT the developer of those wrappers, then you won’t know much about gemfire API even after 12 months of development. Interviewers will see that you have close to zero gemfire experience — unfair!

example — sockets — i feel almost everywhere sockets are used, they are used via wrappers. Boost, ACE, java, c#, python, perl all offer wrappers over sockets. However, socket API is critical and a favorite interview topic. We had better go beyond those wrappers and experiment with underlying sockets. I recently had to describe my network programming experience. I talked about using perl telnet module, which is a wrapper over wrapper over sockets. I feel interviewer was looking for tcp/udp programming experience. But for that system requirement, perl telnet module was the right choice. That right choice is poor choice for knowledge-gain.

example — thread-pool — One of my ML colleagues told me (proudly) that he once wrote his own thread pool. I asked why not use the JDK and he said his is simple and flexible — infinitely customizable. That experience definitely gave him confidence to talk about thread pools in job interviews.

If you ask me when a custom thread pool is needed, I can only hypothesize that the JDK pools are general-purpose, not optimized for the fastest systems. Fastest systems are often FPGA or hardware based but too costly. As a middle ground, I’d say you can create your customized threading tools including a thread pool.

example — Boost::thread — In C++ threading interviews, I often say I used a popular threading toolkit — Boost::thread. However, interviewers repeatedly asked about how boost::thread authors implement certain features. If you use boost in your project, you won’t bother to find out how those tools are internally implemented. That’s the whole point of using boost. Boost is a cross-platform toolkit (more than a wrapper) and relies on things like win32 threads or pthreads. Many interviewers asked about pthreads. If you use a high-level threading toolkit, you avoid the low-level pthreads details..

example — Hibernate — If you rely on hibernate and HQL to generate native SQL queries and manage transaction and identities, then you lose touch with the intricate multitable join, index-selection, transaction and identity column issues. Won’t do you good for interviews.

example — RV — one of my tibco RV projects used a simple and nice wrapper package, so all our java classes don’t need to deal with tibrv objects (like transport, subject, event, rvd….). However, interviewers will definitely ask about those tibrv objects and how they interact — non-trivial.

example — JMS — in GS, the JMS infrastructure (Tibco EMS) was wrapped under an industrial-strength GS firmwide “service”. No applications are allowed to access EMS directly. As a result, I feel many (experienced) developers don’t know JMS api details. For example, they don’t use onMessage() much and never browse a queue without removing messages from it. We were so busy just to get things done, put out the fires and go home before midnight, so who has time to read what he never needs to know (unless he plans to quit this company)?

example — spring jms — I struggled against this tool in a project. I feel it is a thin wrapper over JMS but still obscures a lot of important JMS features. I remember stepping through the code to track down the call that started the underlying JMS server and client sessions, and i found it hidden deeply and not well-documented. If you use spring jms you would still need some JMS api but not in its entirety.

example — xml parsing — After many projects, I still prefer SAX and DOM parsers rather than the high-level wrappers. Interviewers love DOM/SAX.

example — wait/notify — i used wait/notify many times, but very few developers do the same because most would choose a high-level concurrency toolkit. (As far as I know, all threading toolkits must call wait/notify internally.) As a result, they don’t know wait/notify very well but luckly, wait/notify are not complicated as the other APIs mentioned earlier.

example — FIX — one of my FIX projects uses a rather elaborate wrapper package to hide the complexity of FIX protocol. Interviewers will ask about FIX, but I don’t know any from that project because all FIX details were hidden from me.

example — raw arrays — many c++ authors say you should use vector and String (powerful, standard wrappers) to replace the low-level raw arrays. However, in the real world, many legacy apps still use raw arrays. If you avoid raw arrays, you have no skill dealing with them in interviews or projects. Raw arrays integrate tightly with pointers and can be extremely extremely tricky. Half of C/C++ complexities step from pointers.

example — DB tuning — when Oracle 10 came out, my DBA friend told me the toughest DBA job is now easier because oracle 10 features excellent self-tuning. I still doubt that because query tuning is always a developer’s job and the server software has limited insight. Even instance tuning can be very complicated so DBA hands-on knowledge is probably irreplaceable. If you naively believe in Oracle marketing and rely on self-tuning, you will soon find yourself falling behind other DBAs who look hard into tuning issues.

##growing in java: where to focus when learning c++/c#

I feel stronger than a rookie java developer because we are comfortable with both GTD and IV:
1) IDE
** (remote) debugging. Also consider my Perl sss()
2) threading — idioms, principals
** thread creation, synchronized, wait…
** threadlocal
** thread pool
** immutable
3) reflection + proxy
) syntax of overriding, constructors, final, singletons, generics
) design patterns — large idioms
) unit test

other signs of strength:
) generics — creating generic classes
) serialization
) garbage collections
) RMI
) inner classes
) collections and data structures — choosing

What about spring, hibernate …? I feel these fads come and go.

##java trading interview topics

* annotations? I don’t know why so often asked
* multi-threading
* concurrent data structures
* thread pool
* deadlock prevention
* deadlock investigation
* wait-notify, synchronized
* dependency injection. I don’t know why so often asked
* garbage collection
* singletons
* serialization, externalization(??), probably needed for stream messaging
* generics
—–
* immutable
* deep copy — needed for serialization and cloning
* producer/consumer

##hard CORE java/c++ skills across domains

— c++
void ptr
double pointer
ptr to ref; ref to ptr
ptr to method
new/delete global overload to intercept new/delete operations
allocators
debugger
scoped typedef, namespace
vptr/vtbl
static initialization sequence

—java
Note: MOM, spring, hibernate, data grid … aren’t core java and out of scope.

reflections
threads
generic

— Below are needed only in rare situations, IMHO
memory management — leak detectors, garbage collection, weak (soft) references, memory profilers,

jdk and cglib proxy
JMX
custom class loaders
byte code engineering
JVMTI — (eg jprobe) powerful tools to analyze production systems.
real time java

learning curve in AutoReo

(Written in May 2010)

I thought about some friend’s suggestion to “deepen your java”. Now I feel the more I complain about being slower than others and having a lot of difficulties learning the code base, the better!

Learning curve in the Autoreo team is steep, but project risk is not too high.

By contrast, in other projects I don’t feel challenged because I was so familiar with the technologies and tools. I won’t improve. One day I step into a big trading system, I would not be prepared for the complexities. Project risk would be too high – I won’t be given a lot of time to learn.

I don’t fancy learning java in a school or getting a Master’s. I worked for 12 years and studied by myself for a long time. I know what I need to learn – stuff used in real (imperfect) systems. So my current project is probably the best classroom for my personality. I’m grateful.