depth: %%tech strength@@

Some top geeks I know are fast at reading code + logs. Few rely on documentation. I’m OK not the fastest.

Some top geeks in “sister” teams of my team are experts with instrumentation tools and techniques. I guess other top geeks seldom need a lot of instrumentation. I feel they lack the experience but make up for it in other skills.

Some top geeks keep gaining depth if they focus on one complex system. I might have potential here.

##10basic constructs4c++cod`IV

See EPI300

  1. std::string (more useful than cStr)
  2. vector (more useful than array) sorted data structure (i.e. stl map), unordered_map
  3. Node class used in a linked graph
  4. dtor, copier, op=
  5. ref only as function I/O
  6. iterator – basic usage
  7. double pointer
  8. stack, queue
  9. pointer arithmetic
  10. shared_ptr
  11. local static

no exception
stl algo? Only Citadel array-shrink
no pointer to function
no template
no (very, very seldom) threading in coding Q
adv: matrix
adv: circular buffer


describing “latest” project

Avoid naive honesty, which often backfires. Such honesty doesn’t serve any purpose. We are professionals, expected to communicate truthfully but also stay relevant. The major challenge or a key component I created may be irrelevant to the interviewer or impossible to describe over phone.

Q: which component did you *design* ?

%%A: wait/notify framework to associate order requests sent out with response received. See the SCB concurrency design question

Q: major contribution?

%%A: Preferences framework
%%A: wait/notify framework

Q: major challenges?

%%A: I will describe the cancelled trade blotter
%%A: wait/notify framework

Q: what does the system do, in layman’s term

skill: deepen^diversify^stack up

Since 2010, I have carefully evaluated and executed 3 broad strategies:

  1. deepen – for zbs + IV
  2. diversify or branch-out. Breaking into new markets
  3. stack-up – cautiously
  • eg: Deepened my java/SQL/c++/py knowledge for IV and GTD. See post on QQ vs ZZ.
  • eg: diversified to c++, c#, quant, swing…
  • eg: diversify? west coast.
  • eg: diversify? data science
  • eg: diversify? research + teach
  • eg: stack-up to learn spring, hibernate, noSQL, GWT.

Stack-up — These skills are unlikely to unlock new markets. Lower leverage.

GTD stress/survival on the job? None of these help directly, but based on my observation GTD skill seldom advance my career as a contractor. It could create a bit of spare time, but it’s a challenge to make use of the spare time.

c++GTD^IV muscle building: X years xp: U can cut by half

What specific topics to improve for c++ (not pure algo) coding IV? (X years experience doesn’t guarantee anything)

  • · Write code to print alternative item reversely
  • · Write ref-counted string
  • · Write auto-ptr

What specific topics to improve for c++ QnA IV? (X years experience doesn’t guarantee anything)

So in any of these areas, x years spent using a language could leave you still total beginner!

tough n high-leverage c++topics#IV QQ

I used to feel I have so much learning(absorption) capacity, but now I feel in my finite career I can’t really master and remember all the tough c++ topics.

Practical solution — Classify each difficult c++topic into one of

  1. QQ: high impact on QnA interview, probably the only type of high-leverage tough topic. Largely textbook knowledge. As such I’m basically confident I can learn all the basics on my own (perhaps slower than Venkat), provided I know the topics.
    1. including “coding questions” designed really for knowledge test, rather than algorithm thinking
    2. eg: HFT, Venkat…
  2. high impact on algo coding IV? No such topic. These coding IV are not about knowledge in tough topics!
  3. ZZ: high impact on GTD zbs — inevitably Low leverage during job hunt
  4. 00: no high impact on anything

Q: Is there a tough topic in both QQ and ZZ? I don’t know any.

I think 00 will be the biggest category:

  • [00] template wizardry;
  • [00] template specialization
  • [00] MI;
  • [00] operator overloading;
  • [00] pthread
  • ————-
  • [QQ]
  • [QQ] move semantics
  • [QQ] boost common lib
  • [QQ] optimization tricks. Remember MIAX and SCB IV by Dmitry
  • [QQ] singleton implementation — not really tough
  • [QQ] pimpl — not really tough
  • [QQ] op-new/malloc (interacting with ctor)
  • [QQ] memory layout
  • [QQ] specific threading constructs
  • [QQ] smart ptr details
  • [QQ] ptr as a field
  • [QQ] implement auto_ptr or ref-counting string
  • [QQ] UDP — not really tough
  • [QQ] select()
  • [QQ] multicast
  • [QQ]
  • [ZZ] IDE set-up
  • [ZZ] compiler/linker/makefile details
  • [ZZ] debuggers
  • [ZZ] crash analysis, like segmentation fault


## c++topics seldom quizzed

(master -> pearl)
some low-level details I thought would be popular but seldom asked:
* L-value
* iterator types and implementations
* static variables outside classes
* implicit acts of magic by compiler
* array and cStr – syntax, memory, … the gory details beyond the basics
* template specialization
* ref/pointer typedef inside  templates
* non-dummy-type args in template
* MI
* enum
* exception spec
* C integration
* pimp
* fwd declaration
* namespace
* linker
* extern
* double pointers
* hiding rule
* swap – all the important usages and no-fail
* overloading and  method resolution
* casting and conversion
*** OOC and  conversion ctor
— “mid-level”
* boost Any vs Variant
* i/o stream
* regex
* file access (including random)
* serialization
* memory leak detection
* details of boost thread
* boost smart pointer beyond the shared_ptr
* std::string details

##c++weakness revealed by QnA IV#+projects

See also

Relative to java/c#, c++ questions are more low-level (often using non-OO constructs). This is generally my strength.

I used to feel confident I can see patterns in tech interview questions and figure out what specific types of knowledge is required at a certain seniority. I was fairly successful with java/SQL/c#, but less so with c++. Based on the past questions i’m trying to make out the patterns.

=======weaknesses revealed by interviews. Probably need more GTD experience. Subset of these are both tough and high-leverage  — mostly QQ category.

  • couldn’t describe my vol fitter project as a c++ project
  • couldn’t describe my B2bTrading engine as c++ project
  • any threading code I wrote for coding  tests I couldn’t implement in boost::thread
  • STL useful tricks (like swing) — Shanghai, synechron, Jap, WQ
  • specific threading constructs/techniques — sapient, jump, SCB
  • boost common lib — jump, Jap NumericTech, synechron,
  • firm grasp of language features (like rvalue ref) — 3Arrow, bbg
  • how to handle heap (smart)pointer as a field
  • essential class-based techniques/constructs (eg ref counting) — Miami, espeed, nQuant,
  • c++11 features – DRW, 3arrow
  • initializing common data structures like arrays, vectors, classes — needed in a lot of coding challenges
  • implementation details of singleton, factory, pimpl, ref-counted string
  • [lg] mem management/optimization (like heap-only classes) — barc-mtg, nQuant
  • [lg] socket — nquant
  • (No one tested me on MI, advanced template, etc)
  • ====These are harder to self-study — too specialized and no pointers
  • [lg] template beyond merely using existing libraries
  • linux (instrumentation) tools like truss, LD_library_path — jump, Miami, nQuant
  • [lg] essential/deeper know-how (signals, OOM, stack size) not expected of a junior developer …– jump, Japanese, nQuant, 3Arrow, barc-mtg
  • [lg] extreme low-level optimization (CPU cache, custom-RTTI, byte alignment, placement new) ..– Miami, nquant, jump
  • =========much-needed zbs not quizzed by IV
  • headers …
  • static object – one-definition rule
  • pre-processor tricks
  • makefiles, continuous build
  • linker troubleshooting, nm commands etc
  • pthreads
  • debuggers,
  • memory leak detectors,
  • debugging — harder than java
  • random crashes

Continue reading “##c++weakness revealed by QnA IV#+projects”

zbs+nlg insight accumu due to X years@a job: !by default

Many interviewers assume the length of using something on a full time job indicates accumulation… Other interviewers test the depth of your insight.
You know you have insight, mileage, accumulation if/when ..
  • ..when  you know most of the important and relevant [1] terms in the lingo. Interviewers often go deeper about a new “construct”[2], and question its motivation/history and limitations.
    • ….if you know the __connections__ between the concepts. Without a mental map, there would be way too many unconnected lingo terms leading to info overload
    • ….if you know what new developments are only marginally important to your field, so you can focus on the really important
  •   ..if you can /separate the wheat from the chaff/滥竽充数/. I think often the real veterans would exchange a secret handshake and acknowledge each other very quickly. Eg: Dad (Tan Jiajian) knows his peers in the domain.
    • …. If you can’t quickly separate the wheat from the chaff, then probably you aren’t experienced enough.
  • ..when  you know you have grown much stronger than a junior person new to the field. How soon this happens is poorly correlated to # years
  • ..when your mental “book” has grown thin -> thick -> thin. Eg: math..
  • ..when you know you can master a “local” system quickly, thanks to the accumulation /under your belt/. Eg: Avichal Gupta…
As you can see, many of the indications/evidence of insight are related to evaluative discussions (including interviews). I’m not being narrow-minded and focus exclusively on job interviews. No. In any team,  very quickly person A would size up person B (and vice versa), and often have a rough number in his head about “how much B knows our shared domain”.
[1] it’s subjective but I feel 90% of what consider irrelevant is clearly irrelevant.
[2] this includes new choice, technique, best practice
German said one needs to be in a project for 6Y, but I disagree. I feel a wide range of non-trivial challenges is more important, and I feel we are unlikely to get that by staying in one project.
eg Perl — I used Perl for 3 to 5 years..
eg SQL — I used SQL for many years before GS, but my accumulation was solely in GS!
eg Threading — not much used. Mostly accumulation by reading…
eg option math –
eg C++ — see post on c++IV: knowledge more important than coding experience

sticky%% imprints/reputation – java ^ c++

I feel my c++ interviews aren’t so successful (, but many traditional efforts may generate diminishing returns. (eg: Home coding project can be very time-consuming). Instead, perhaps we should look for special “imprints” – knowledge pearls that help us impress and stand out during competitive interviews. In a way, we must beat the interviewer on some front since on many questions we would be beaten.

Venkat can often stand out within a short conversation, due to his imprints.

My java interview experience – I managed to impress interviewers and stood out although my coding/design may not stand out. Am I a stronger java guy than my friends? I guess only if you quiz me on those special topics
– lock free
– wait/notify intricacies
– condition variable as replacement of wait/notify
– custom thread pool
– volatile keyword intricacies
– skip list (sorted set)
– concurrent hash map
– special singletons

Targets (memory/threading helps efficiency) —
* subvert the virtual mechanism P928[[primer]]
* dtrace
* placement new – real use cases
* replacement of RTTI
* data alignment in market data wire format
* memory page locking
* alloca()
* custom new() and free list in [[eff c++]]
* lock free – real code not concept
* template partial specialization but I don’t know the use cases?
* various custom-made smart pointers with special features — [[safe c++]]
* memory leak detectors – home made (eff c++?)
* comma operator

I feel productive with coding when …

… when I Google and find solutions or “machine-verifiable” insights
… when I spend time hacking code at home
… whenever I hook up a debugger
… when I managed to gain “control” (or wrap my mind) over an opaque system

! This is in stark contrast to textbook theories.

I feel the thrill when I get the code to work. In terms of my long-term career and long term financial stability.

I feel this interest, satisfaction, positive feedback and drive is arguably more important than my aptitude. Some people are probably more brainy than me but don’t feel this thrill.

Open source systems are more rewarding than Microsoft technology. I would say scripting tools are esp. rewarding but sometimes too “transparent” to be challenging.

feeling competent(GTD+)with linux,java,perl..!cpp/py

See also post on “beat us” to understand why I feel less competent with cpp.

Nowadays I feel more competent developing in linux than windows.

If you look hard you realize this is relative to team competence. If my team member were familiar with linux but struggling in windows, then I would feel very differently!

Same thing could happen between java and Quartz — in a java team, my Quartz know-how may be quite visible and hard to match.

Perhaps the defining examples is Perl. In so many teams no one knows more about perl than me. When perl was important to the team, I felt very competent and strong, and almost effortless. In GS, there were several perl old-hands, but I still felt on-par. Key reason – not many people specialize in perl, and also none of the OO stuff was important in those projects.

I gradually grew my competence in java. My theoretical knowledge grew fast, but competence is not really about IV questions. More about GTD know-how where instrumentation (including tool-chain) /knowledge/ is probably #1.

Big lesson — get a job to learn portable skills. You don’t want to do years of python programming but only in some proprietary framework.

elements of a good tech contexx

* paid work. The income will sustain your effort. Unpaid work won't last beyond a few weeks (a few hours/week) of aggregate “investment”

* professional project, not personal/home/school projects

* paid users, the more the better

* important to the users. The higher the financial value the better

* mainstream technology, in-demand technology

* encounter real problems — should be common (but not easy) problems

* Colleagues (teachers would be too ideal) to help out and give pointers

##which episodes(over 10Y)increased%%zbs(+GTD)

Patterns — usually the “mastery”/zbs grows over a surprisingly long time. I tend to practice over and over again to slowly get the insight. Vast majority of developers lack this contexx.

Note: these portable zbs are less “valuable” than interview skills??
Note: GTD depends more on local system knowledge than generic zbs.

* {zed} unix concepts and power tools [1]
* {GS} complex SQL query and tuning. It probably takes only a few months to reach this level, but the vast majority of developers don’t have this contexx
* {GS} remote debugging
* perl, but unfortunately, unpopular now
* java collections

[1] see post on [[2 types of utility software]]

– Getting comfortable with eclipse, but any above-average insight? no
– ErrorMemos? I put in a lot of effort but is the design techniques portable? no.
– DNS? I became very familiar with it
– oracle? i spent years but am way below average
– apache administration? spent years but …
– php? spent years but …
– javascript and mysql? i became very familiar with it, but any rare insight? no
– excel VBA? i became proficient but still below the average programmer out there.
– CVI (c for virtual instrumentation)? i spent years but …

Now, next targets of growth?

mkt data tech skills: not portable not shared

Q1: If you compare 5 typical market data gateway dev [1] jobs, can you identify a few key tech skills shared by at least half the jobs, but not a widely used "generic" skill like math, hash table, polymorphism etc?

Q2: if there is at least one, how important is it to a given job? One of the important required skills, or a make-or-break survival skill?

My view — I feel there is not a shared core skill set. I venture to say there’s not a single answer to Q1.

In contrast, look at quant developers. They all need skills in c++/excel, BlackScholes, bond math, swaps, …

In contrast, also look at dedicated database developers. They all need non-trivial SQL, schema design. Many need stored procs. Tuning is needed if large tables

Now look at market data gateway for OPRA. Two firms’ job requirements will share some common tech skills like throughput (TPS) optimization, fast storage.

If latency and TPS requirements aren’t stringent, then I feel the portable skill set is an empty set.

[1] There are also many positions whose primary duty is market data but not raw market data, not large volume, not latency sensitive. The skill set is even more different. Some don’t need development skill on market data — they only configure some components.

gdb q(next) over if/else and function calls

I used an optimized binary. Based on limited testing, un-optimized doesn’t suffer from these complexities.

Conventional wisdom: q(next) differs from q(step) and should not go into a function

Rule (simple case): When you are on a line of if-statement in a source code, q(next) would evaluate this condition. If the condition doesn’t involve any function call, then debugger would evaluate it and move to the “presumed next line”, hopefully another simple statement.

Rule 1: suppose your “presumed next line” involves a function call, debugger would often show the first line in the function as the actual “pending”. This may look like step-into!

Eg: In the example below. Previous pending is showing L432 (See Rule 2b to interpret it). The presumed line is L434, but L434 involves a function call, so debugger actually shows L69 as the “pending” i.e. the first line in the function

Rule 2 (more tricky): suppose presumed line is an if-statement involving a function call. Debugger would show first line in the function as the pending.

Eg: In the example below, Previous pending was showing L424. Presumed line is L432, but we hit Rule 2, so actual pending is L176, i.e. first line in the function.

Rule 2b: when debugger shows such an if-statement as the “pending”, then probably the function call completed and debugger is going to evaluate the if-condition.

424 if (isSendingLevel1){

425 //……

426 //……….

427 //……..

428 //……….

429 } // end of if

430 } // end of an outer block


432 if (record->generateTopOfBook()

433 && depthDb->isTopOfTheBook(depthDataRecord)) {

434 record->addTopOfBookMarker(outMsg);

435 }

This message may contain confidential information and is intended for specific recipients unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient of this message, please delete it and notify the sender. This message may not represent the opinion of Intercontinental Exchange, Inc. (ICE), its subsidiaries or affiliates, and does not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the recipient of this message is expected to provide safeguards from viruses and pursue alternate means of communication where privacy or a binding message is desired.

#1challenge if u rely@gdb to figure things out: optimizer

Background: explains why “figure things out quickly” is such a make-or-break factor.

In my recent experience, I feel compiler optimization is the #1 challenge. It can mess up GDB step-through. For a big project using automated build, it is often tricky to disable every optimization flag like “-O2”.

More fundamentally, it’s often impossible to tell if the compiled binary in front of you was compiled as optimized or not. Rarely the binary shows it.

Still, compared to other challenges in figuring things out, this one is tractable.

gdb skill level@Wall St

I notice that, absolutely None of my c++  veteran colleagues (I asked only 3) [2] is a gdb expert as there are concurrency experts, algo experts [1], …

Most of my c++ colleagues don’t prefer (reluctance?) console debugger. Many are more familiar with GUI debuggers such as eclipse and MSVS. All agree that prints are often a sufficient debugging tool.

[1] Actually, these other domains are more theoretical and produces “experts”.

[2] maybe I didn’t meet enough true c++ alpha geeks. I bet many of them may have very good gdb skills.

I would /go out on a limb/ to say that gdb is a powerful tool and can save lots of time. It’s similar to adding a meaningful toString() or operator<< to your custom class.

Crucially, it could help you figure things out faster than your team peers. I first saw this potential when learning remote JVM debugging in GS.

— My view on prints —
In perl and python, I use prints exclusively and never needed interactive debuggers. However, in java/c++/c# I heavily relied on debuggers. Why the stark contrast? No good answer.

Q: when are prints not effective?
A: when the edit-compile-test cycle is too long, not automated but too frequent (like 40 times in 2 hours) and when there is real delivery pressure. Note the test part could involve many steps and many files and other systems.
A: when you can’t edit the file at all. I have not seen it.

A less discussed fact — prints are simple and reliable. GUI or console debuggers are often poorly understood. Look at step-through. Optimization, threads, and exceptions often have unexpected impacts. Or look at program state inspection. Many variables are hard to “open up” in console debuggers. You can print var.func1()


gdb stop@simple assignments #compiler optimize

Toggle between -O2 and -O0, which is the default non-optimized compilation.

In my definition, A “simple assignment” is one without using functions. It can get value from another variable or a literal. Simple assignments are optimized away under -O2, so gdb cannot stop on these lines. This applies to break point or step-through.

In particular, if you breakpoint on a simple assignment then “info breakpoint” will show a growing hit count on this breakpoint, but under -O2 gdb would never stop there. -O0 works as expected.

As another illustration, if an if-block contains nothing but simple assignment, then gdb has nowhere to stop inside it and will only stop after the if-block. You won’t know whether you entered it. -O0 works as expected.

## vi cheatsheet

  • ~~~~ command mode #roughly ranked
  • A — to append at end of current line
  • C — wipe out from cursor to end of line and puts you into insert mode
  • cw — wipe out from cursor to end of word and puts you into insert mode
    • c2w?
  • capital O — open new line above cursor
  • small O — open new line below cursor
  • ~~~~ insert mode #roughly ranked