## UPSTREAM tech domains: defined by eg

It pays to specialize in a domain that helps you break into similar domains. (Unsuccessful with my quant adventure so far.)

  • eg: [L] socket —- C++ is the alpha wolf, leader of the wolf pack
  • eg: [L] latency —- C++ is the leader of the pack in low-level optimizations .. In-line, cache-efficiency, footprint..
  • eg: collections —- C++ is the leader of the pack. There’s a lot of low level details you gain in STL that help you understand the java/c#/python collections
  • eg: threading —- java is the leader of the pack. In c++ threading is too hard, so once I master some important details in java threading, it helps me build zbs in c/c#, but only to some extent
  • eg: [L]pbref^val —- C++ is the leader. C# has limited complexity and java has no complxity
  • eg: regex/string —- Perl is the leader
  • eg: alpha —- equities trading is the leader
  • [L] heap/stack —- C++ is the leader. Java and c# are cleaner, simpler and hides the complexities.
  • defensive coding —- (and crash management) C++ is the leader, because c++ hits more errors
  • lambda —- C# is the leader. Scripting languages follow a different pattern.
  • list/stream —- Linq is the leader
  • Black-Scholes —- feels like a leader or upstream, but not /convincing/
  • execution algo —- sell side equities desk seems to be the leader
  • data analytics —- python feels like the leader
  • [L = lowLevel]



%%c++keep crash` I keep grow`as hacker #zbs#Ashish

My current c++ project has high data volume, some business logic, some socket programming challenges, … and frequent crashes.

The truly enriching part are the crashes. Three months ago I was afraid of c++, largely because I was afraid of any crash.

Going back to 2015, I was also afraid of c++ build errors in VisualStudio and Makefiles, esp. those related to linkers and One-Definition-Rule, but I overcame most of that fear in 2015-2016. In contrast, crashes are harder to fix because 70% of the crashes come with no usable clue. If there’s a core file I may not be able to locate it. If I locate it, it may not have symbols. If it has symbols the crash site is usually in some classes unrelated to any classes that I wrote. I have since learned many lessons how to handle these crashes:
* I have a mental list like "10 common crash patterns"
* I have learned to focus on the 20% of codebase that are most convoluted, most important, most tricky and contribute most to debugging difficulties. I then invest my time strategically to rewrite (parts of) that 20% and dramatically simplify them. I managed to get familiar and confident with that 20%.

* I have learned to pick the most useful things to log, so they show a *pattern*. The crashes usually deviate from the patterns and are now easier to spot.

* I have developed my binary data dumper to show me the raw market data received, which often "cause" crashes.

* I have learned to use more assertions and a hell lot of other validations to confirm my program is not in some unexpected *state*.
* I figured out memset(), memcpy(), raw arrays are the most crash-prone constructs so I try to avoid them or at least build assertions around them.
* I also figured signed integers can become negative and don’t make sense in my case so I now use unsigned int exclusively.
* I also gained quite a bit of debugger (gdb) hands-on experience

Most of these lessons I picked up in debugging program crashes, so these crashes are the most enriching experience. I believe other c++ programs (including my previous jobs) don’t crash so often. I used to (and still do) curse the fragile framework I’m using, but now I also recognize these crashes are accelerating my growth as a c++ developer.

meaningful endeavor(Now)towards20Y-career build`#dnlg;zbs;IV

Q: what efforts go towards 20Y-career build`

This question is slightly vague, but it’s subtly different from

Q2: what efforts create social value?
Q3: what efforts are cumulative?

I feel the key question in this article is about building a deep and broad “foundation”, upon which we can start to plan (possible) social value, retirement planning, parenting/family building etc.

  • 😦 teaching and research —- Somehow I tend to feel research creates long-term value but I suspect most research efforts are not valuable!
  • 😦 open-source software —- I tend to feel OSS projects create social value, but most of them are not successful and have no lasting influence
  • 😦 Deepening your domain knowledge such as (German’s) security lending —- It does increase your value-add to some employers and help you move up to management, but I don’t call it “foundation”.
    • MSFM and other formal training —- I feel this is less effective, more ivory-tower, but it does build a firm theoretical foundation.
  • zbs such as instrumentation skills? Yes zbs helps your KPI, and also indirectly helps you with interviews. It’s portable skill, unlike local system knowledge. Can the value last 20Y? Depends on the technology churn. c++ and java are safer.
  • IV muscle building? My favorite career building effort. 20Y? 5Y yes.
  • fitness and health
  • English/Chinese writing and vocab

##orgro lens:which past accumu proved long-term # !! quant

(There’s a recoll on this accumulation lens concept…. )

This post is Not focused on IV or GTD. More like zbs.

Holy grail is orgro, thin->thick->thin…, but most of my endeavors fell short. I have no choice but keep shifting focus. A focus on apache+mysql+php+javascript would have left me with rather few options.

  • —-hall of famers
  • 1) [T] data structure theory + implementation in java, STL, c# for IV — unneeded in any project!
  • 2) [CRT] core java including java OO has seen rather low churn,
    • comparable to c++
    • much better than j2EE and c#
  • 3) [T] threading? Yes insight and essential techniques. Only for interviews. C# is adding to the churn.
  • 4) [j] instrumentation in java codebase and to a lesser extent c#. Essential for real projects and indirectly helps interviews
  • [C] core C++
  • [j] google-style algo quiz — Only for high-end tech interviews. Unneeded in any project
  • [R] SQL? yes but not a tier one skill like c++ or c#
  • regex – needed for many coding interviews and real projects
  • pure algo — for IV
  • coding IV — improved a lot at ICE
  • probability IV
  • ————————also-ran :
  • [C] low level C skills@ICE {static; array; cStr; reinterpret_cast;  enum; typedef; namespace; memcpy}
  • [!T] bond math? Not really my chosen direction, so no serious investment
  • [!T] option math?
  • SQL tuning? not much demand in the c++ interviews, but better in other interviews
  • py? increasing use in projects
  • [R] Unix instrumentation, automation? frequently used but only occasionally quizzed
  • [R] Excel + VBA? Not my chosen direction

C= churn rate is comfortable
D = has depth, can accumulate
R= robust demand
T= thin->thick->thin achieved
j|J = relevant|important to job hunting

focus+engagement2dive into a tech topic#Ashish

(Blogging. No need to reply)

Learning any of the non-trivial parts of c++ (or python) requires focus and engagement. I call it the “laser”. For example, I was trying to understand all the rules about placing definitions vs declarations in header files. There are not just “3 simple rules”. There are perhaps 20 rules, with exceptions. To please the compiler and linker you have various strategies.

OK this is not the most typical example of what I want to illustrate. Suffice to say that, faced with this complexity (or ambiguity, or “chaos”) many developers at my age simply throw up their hands. People at my age are bombarded with kids’ schooling, kids’ enrichment, baby-sitting, home repair [1], personal investment [2], home improvement… It’s hard to find a block of 3 hours to dive in/zoom in on some c++ topic.

As a result, we stop digging after learning the basics. We learn only what’s needed for the project.

Sometimes, without the “laser”, you can’t break through the stone wall. You can’t really feel you have gained any insight on that topic. You can’t connect the dots. You can’t “read a book from thin to thick, then thick to thin again”. You can’t gain traction even though you are making a real effort. Based on my experience, on most of the those tough topics the focus and engagement is a must.

I’m at my best when I have my “laser” on. Gaining that insight is what I’m good at. I relied on my “laser” to gain insights and compete on the job market for years.

Now I have the time and bandwidth, I need to capitalize on it.

[1] old wood houses give more problems than, say, condos with a management fee

[2] some spend hours every day

skill: deepen^diversify^stack up

Since 2010, I have carefully evaluated and executed 3 broad strategies:

  1. deepen – for zbs + IV
  2. diversify or branch-out. Breaking into new markets
  3. stack-up – cautiously
  • eg: Deepened my java/SQL/c++/py knowledge for IV and GTD. See post on QQ vs ZZ.
  • eg: diversified to c++, c#, quant, swing…
  • eg: diversify? west coast.
  • eg: diversify? data science
  • eg: diversify? research + teach
  • eg: stack-up to learn spring, hibernate, noSQL, GWT.

Stack-up — These skills are unlikely to unlock new markets. Lower leverage.

GTD stress/survival on the job? None of these help directly, but based on my observation GTD skill seldom advance my career as a contractor. It could create a bit of spare time, but it’s a challenge to make use of the spare time.

I feel German’s strategy goes beyond tech skills. He treats tech skills as a tool, used to implement his business ideas to create business value. I feel his target role is a mix of architect/BA but more like “product visionary”. It’s somewhat like stack-up.

in-depth^cursory study@advanced program`topics

I have seen both, and tried both.

AA) A minority of developers spend the time studying and understanding the important details of threading, generics, templates … before writing code.

BB) Majority of developers don’t have the time, vision or patience, and only learn the minimum to get the (sloppy) job done. They copy-paste sample code and rely on testing, but the test cases are limited to realistic UAT test cases and will not cover edge cases that have yet to occur. For concurrency designs, such tests are not good enough.

AA is often seen as unnecessary except:
* Interviewers often go in-depth
* In a product vendor rather than an in-house application team
* If requirement is unusual (eg: B2BTradingEngine), then BB won’t even pass basic developer self-tests. Sample code may not be available — see my post http://wp.me/p74oew-1oI. So developers must bite the bullet and take AA approach. This is my sweet spot. See
https://bintanvictor.wordpress.com/2016/10/31/strength-gtd-with-non-trivial-researchinnovation/ https://bintanvictor.wordpress.com/2016/03/30/theoretical-complexity-strength/

On Wall St projects, I learned to abandon AA and adopt BB.

On Wall St interviews, I still adopt AA.

On West Coast, I think AA is more needed.

each year grow towards a principal developer

A lead developer of any system (not just a component) needs a few years experience [1] building a similar system. Each year, I hope to accumulate relevant know-how [2]. I’m not a people manager or project manager but I can be the principal developer.

Pause ….

However, I’m not accumulating “everything relevant” because … A lot of the know-how is never quizzed in tech interviews. Here are a few examples
* build/release/test/cont-int
* version control
* peer review/QA/release process
* wiki, jira
* back-up
* system admin scripting
* job schedulers

As a wall st contractor, I used to work in kinda greenhouse environment with all of the infrastructure taken care of, so I just focused on “get-code-done” and get productive with it. It’s an easier life.

Non-lead interviews often focus on implementation details. Benny Zhao once /identified/ just 4 or 5 clusters like 1) algo 2) language details 3) OO design. This skill set is insufficient for a principal developer.

[1] but look at Saurabh, Sundip, Kirupa, Yang … they came in with only something slightly similar
[2] not useful to focus on coding year in year out.

##coding guru tricks (tools) learnt across Wall St teams

(Blogging. No need to reply.)

Each time I join a dev team, I tend to meet some “gurus” who show me a trick. If I am in a team for 6 months without learning something cool, that would be a low-calibre team. After Goldman Sachs, i don’t remember a sybase developer who showed me a cool sybase SQL trick (or any generic SQL trick). That’s because my GS colleagues were too strong in SQL.

After I learn something important about an IDE, in the next team again I become a newbie to the IDE since this team uses other (supposedly “common”) features.

eg: remote debugging
eg: hot swap
eg: generate proxy from a web service
eg: attach debugger to running process
eg: visual studio property sheets
eg: MSBuild

I feel this happens to a lesser extent with a programming language. My last team uses some c++ features and next c++ team uses a new set of features? Yes but not so bad.

Confucius said “Among any 3 people walking by, one of them could be teacher for me“. That’s what I mean by guru.

Eg: a Barcap colleague showed me how to make a simple fixed-size cache with FIFO eviction-policy, based on a java LinkedHashMap.
Eg: a guy showed me a basic C# closure in action. Very cool.
Eg: a Taiwanese colleague showed me how to make a simple home-grown thread pool.
Eg: in Citi, i was lucky enough to have a lot of spring veterans in my project. They seem to know 5 times more spring than I do.
Eg: a sister team in GS had a big, powerful and feature-rich OO design. I didn’t know the details but one thing I learnt was — the entire OO thing has a single base class
Eg: GS guys taught me remote debugging and hot replacement of a single class
Eg: a guy showed me how to configure windows debugger to kick-in whenever any OS process dies an abnormal death.
Eg: GS/Citi guys showed me how to use spring to connect jconsole to the JVM management interface and change object state through this backdoor.
Eg: a lot of tricks to investigate something that’s supposed to work
Eg: a c# guy showed me how to consolidate a service host project and a console host project into a single project.
Eg: a c# guy showed me new() in generic type parameter constraints

These tricks can sometimes fundamentally change a design (of a class, a module or sub-module)

Length of experience doesn’t always bring a bag of tricks. It’s ironic that some team could be using, say, java for 10 years without knowing hot code replacement, so these guys had to restart a java daemon after a tiny code change.

Q: do you know anyone who knows how to safely use Thread.java stop(), resume(), suspend()?
Q: do you know anyone who knows how to read query plan and predict physical/logical io statistic spit out by a database optimizer?

So how do people discover these tricks? Either learn from another guru or by reading. Then try it out, iron out everything and make the damn code work.

2kinds@essential developer tools #WallSt+elsewhere

Note: There are also “common investigative” tools, but for now i will ignore them. Reasons is that most developers have only superficial knowledge of them, so the myriad of advanced features actually fall into #2).

Note: in the get-things-done stage, performance tools are much less “useful” than logic-revealing tools. In my experience, these tools seldom shed light on perf problems in DB, java, or MOM. Performance symptoms are often resolved by understanding logical flow. Simplest and best tools simply reveal intermediate data.
I think most interview questions on dev tools largely fall into these two categories. Their value is more “real” than other tools.

1) common tools, either indispensable or for productivity
* turn on/off exception breakpoint in VS
* add references in VS
* resolve missing assembly/references in VS
* app.config in VS projects
* setting classpath in IDE, makefile, autosys…
* eclipse plugin to locate classes in jars anywhere in windows
* literally dozens of IDE pains
* cvs,
* maven, ant,
* MS excel
* junction in windows
* vi
* bash
* grep, find, tail
* browser view-source

2) Specialized investigative/analytical/instrumentation tools — offers RARE insights. RARELY needed but RARE value. Most (if not all) of the stronger developers I met have these secret weapons. However, most of us don’t have bandwidth or motivation to learn all of these (obscure) features, because we don’t know which ones are worth learning.
* JMS — browser, weblogic JMS console…
* compiler flags
* tcpdump, snoop, lsof
* truss
* core dump analysis
* sar, vmstat, perfmeter,
* query plan
* sys tables, sp_lock, sp_depend
* set statistics io
* debuggers (IDE)
* IDE call hierarchy
* IDE search
* thread dump
* jvmstart tools — visualgc, jps,
* bytecode inspector, decompiler
* profilers
* leak detector
* jconsole
* jvmstat tools — visualgc, jps,..
* http://wiki.caucho.com/Category:Troubleshooting
* any tools for code tracing Other productivity skills that no software tools can help:
* log analysis
* when I first came to autoreo, i quickly characterized a few key tables.