premium salary to compensate for intrinsic motivation@@

I recall that at more than one juncture in my job hunting career, I feel overwhelmed by a premium offer and said in my head “if I accept this offer and earn 20% more than the standard rate, then the ensuing pride, self-image boost etc would surely create a wellspring of positive motivation.”

How naive….in hind sight. The real factors affecting my job satisfaction was usually unrelated to premium salary. See the spreadsheet about job satisfaction.

I think the premium salary didn’t improve my life chances. A satisfying job does.

jav isn’t challenged by c++/c# but by high-level languages

  • I noticed at the beginning of this decade (2010-2012) c# mounted a credible challenge to dethrone java but has since lost momentum.
  • c++ had a renaissance with c++1x, but after 8 years I would say it failed to become a threat to java.

The other challengers to java seems to be domain-specific high level languages like javascript (browser and beyond), ruby-on-rails, php (web server side), perl (text processing), q (kdb). Not sure about Golang.

Python is the strongest challenger among them. Here I consider python a domain-specific language given its visible strengths in data science, machine-learning, black-box testing, and traditional process management. I sometimes consider python a general purpose language like java and c++, but i feel it’s not widely perceived as such, perhaps due to dynamic typing and efficiency.

C/c++ remains the default system programming language — that’s clear. Less clearly, Java remains the “default” application-development language on server side — but in this sentence the key words are vague.

Luckily for java developers, nowadays most of the work is on server side, including many mobile and cloud applications. In addition, java is viable for offline/batch job, data science, transaction-processing systems … Actually, I listed javascript as a java challenger because I think a lot of application functionalities are moving to client-side.

Luckily for java developers, a few internet powerhouses led by Google (and Alibaba) have chosen Java as the default language on server side. So have most Wall St firms.

G5 unexpected big wins]%%life #absorbency

should create pointer blogposts in ‘ffree’ and ‘diet’ blogs

20 uphill breakthroughs #wellness/xx is a similar list.

My unexpected success at weight improvement and diet is rather rare … the latest of top 5 achievements of my entire life. Other significant (often unexpected) successes:

  • outstanding project performance in 95G, Barcap and RTS. In other teams, I was able to do a reasonable job among bright young professionals.
  • QQ technical wins on high-end c++ positions (10+ times each), a golden ticket to high salaries on Wall St and then SG
  • [r] high absorbency for self-learning till this age.
    • MSFM
    • 😦 otherwise, the 4.5 years in SG was low.
  • [r] ffree first as bachelor and again in 2018.
  • [r=really rare]

How about my Cambodia investments? No really a personal effort per se.

retrans questions from IV+ from me

Q11 (2 x IV): do you track gaps for Line A and also track gaps in Line B?
A: No, according to Deepak and Shanyou. Deepak said we treat the seq from Line A/B as one combined sequence (with duplicates) and track gaps therein

Q2 (IV x 2): Taking parser + orderbook (i.e. rebus) as a single black box, when you notice a gap (say seq #55/56/57), do you continue to process seq # 58, 59 … or do you warehouse these #58, 59… messages and wait until you get the resent #55/56/57 messages?
A: no warehousing. We process #58 right away.

Q2b (IV): In your orderbook engine (like Rebus), suppose you get a bunch of order delete/exec/modify messages, but the orderId is unrecognized and possibly pending retrans. Rebus doesn’t know about any pending retrans. What would rebus do about those messages?
%%A: I don’t know the actual design [3], but if I were the architect I would always check the orderId. If orderId is unknown then I warehouse the message. If it is a known order Id in Rebus, I will apply the message on the order book. Risks? I can’t think of any.

[3] It’s important to avoid stating false facts, so i will add the disclaimer.

Q2c (IV): what data structures would you use to warehouse those pending messages? ( I guess this question implies warehousing is needed.)
%%A: a linked list would do. Duplicate seqNum check is taken care of by parser.

Q13 (IV): do you immediately send a retrans request every time you see a gap like (1-54, then 58,59…)? Or do you wait a while?
A: I think we do need to wait since UDP can deliver #55 out of sequence. Note Line A+B are tracked as a combined stream.

Q13b: But how long do we wait?
A: 5 ms according to Deepak

Q13c: how do you keep a timer for every gap identified?
%%A: I think we could attach a timestamp to each gap.

— The above questions were probably the most important questions in a non-tech interview. In other words, if an interview has no coding no QQ then most of the questions would be simpler than these retrans questions ! These questions test your in-depth understanding of a standard mkt data feed parser design. 3rd type of domain knowledge.

Q: after you detect a gap, what does your parser do?
A (Deepak): parser saves the gap and moves on. After a configured timeout, parser sends out the retrans request. Parser monitors messages on both Line A and B.

Q: if you go on without halting the parser, then how would the rebus cope?

  • A: if we are missing the addOrder, then rebus could warehouse all subsequent messages about unknown order IDs. Ditto for a Level 1 trade msg.

Deepak felt this warehouse could build up quickly since the ever-increasing permanent gaps could contain tens of thousands of missing sequence numbers. I feel orderId values are increasing and never reused within a day, so we can check if an “unknown” orderId is very low and immediately discard it, assuming the addOrder is permanently lost in a permanent gap.

  • A: if we are missing an order cancel (or trade cancel), i.e. the last event in the life cycle, then we don’t need to do anything special. When the Out-of-sequence message shows up, we just apply it to our internal state and send it to downstream with the OOS flag.

If a order cancel is lost permanently, we could get a crossed order book. After a few refreshes (15min interval), system would discard stale orders sitting atop a crossed book.

In general, crossed book can be fixed via the snapshot feed. If not available in integrated feed, then available in the open-book feed.

  • A: If we are missing some intermediate msg like a partial fill, then we won’t notice it. I think we just proceed. The impact is smaller than in FIX.

OOS messages are often processed at the next refresh time.

Q3b: But how long do we wait before requesting retrans?
Q3c: how do you keep a timer for every gap identified?

Q14: after you send a retrans request but gets no data back, how soon do you resend the same request again?
A: a few mills, according to Deepak. I think

Q14b: Do you maintain a timer for every gap?
%%A: I think my timestamp idea will help.

Q: You said the retrans processing in your parser shares the same thread as regular (main publisher) message processing. What if the publisher stream is very busy so the gaps are neglected? In other words, the thread is overloaded by the publisher stream.
%%A: We accept this risk. I think this seldom happens. The exchange uses sharding to ensure each stream is never overloaded.

short+efficient : holy grail ] algo search

For most of my tough algo problems (such as leetcode problems)

  1. many celebrated solutions are very short, but not so efficient
    • I think about 60% of them are easier to memorize due to length.
    • if you can understand and memorize them, they are quicker to write due to fewer keystrokes
    • About half of them are easier to understand, thanks to the source code size
    • eg: yield-based generators are a prime example
    • eg: recursive solutions are often brief but inefficient
  2. many great solutions are efficient in terms of O(), but verbose
    • eg: bottom-up DP
    • eg: top-down DP with memoization
    • eg: using a tree (when not required) can sometimes give an efficient solution

However, these two goals are often hard to harmonize. It is my holy grail to meet both criteria simultaneously, but i won”t try so hard.

  1. For real coding problems, I will prefer brevity
  2. For verbal discussions, I will concentrate on efficient solutions

pass generator output to next generator

I think this technique can be extremely time-saving in coding tests.

https://github.com/tiger40490/repo1/blob/py1/py/algo_combo_perm/1fromEachSet.py my code demos:

for myset in pool:     output = list(gen(output, myset))

The gen() function uses yield. For the first call to gen(), we exhaust all of its items and save into a list named “output”.

Then we pass this list into the second gen(), this time with a different myset

template specialization based on NDTTP=true

We know it’s possible to specialize a template for a concrete type like int or std::string, but I didn’t know that It’s also possible to

… specialize a (class or function) template for a particular compile-time const value (like “true”) of a NDTTP (like “bool flag”)

  • On [[Alexandrescu]] Page xii , Scott Meyers showed an elegant example of specializing for “true”. Note “true” is a value, not a data type !
  • P 34 has a longer example.

Note on reading TMP code — the template specialization syntax is clumsy and can add noise to the signal. Better ignore the syntax rules for now to focus on the gist.

more python mock IV questions #DeepakCM

Q (real IV): how do you verify a given string represents an integer? For example, "3e5" would be invalid.

Q (real IV): compare multi-threading vs multi-processing in python. Which one have you used in your projects?

Q: Have you used a FIFO data structure (like a queue) in python? It’s OK if you have not, but how would you create such a data structure in python?

Q: Compare and contrast deep copy vs shallow copy for a python list
Q: Compare and contrast deep copy vs shallow copy for a python dictionary

Q (advanced question from me): When is a deep copy required? Give some scenarios

Q : Have you used context managers? (I think they are useful.)

Q (advanced question from me): compare and contrast list comprehension vs generator expression

Q: write a Student class with lastName, firstName, birthDate fields. Also include a class attribute "instCnt" showing the number of student objects created. As you create a new Student instance, this instCnt should increment.

Q: have you heard of the LGB rule in python?

Q (basic question from me, never asked in interviews): what data types can’t be used a dictionary keys

Q (basic question from me, never asked in interviews): what’s the usage of "global" keyword? Give some usage scenarios
Q (basic question from me, never asked in interviews): what builtin data types, apart from tuples, are immutable? Give some examples.

##fully engaged+!ROTI still beats boredom

Every learning direction has naysayers, and virtually all learning efforts generates low ROTI. ROTI and strategic investment are holy grails, mirages or white elephants.

We can’t dismiss $ROTI but I think we really need to look beyond $ROTI — too elusive. As Mithun put it, java multithreading alone can fetch you highest salary.

Fully engaged for a few short months but without ROTI is actually not bad. It’s like satisfactory sex without simultaneous climax. I enjoyed the journey, without worrying about the destination.

I’m able to fully engaged for a few months, while my peers can’t!

I have blogposts on spare time usage. Tech learning (even the non-strategic topics) are among the most productive. There’s really nothing more worth learning.

  • — most of these are failed technology bets, but 10x better than not engaged (i.e. boredom)
  • 2012-2013 c# study
  • HFT-specific QQ topics
  • MSFM
  • 2011-2012 self-learning volatility
  • self-learning swing
  • MOM study
  • gemfire study
  • — less disappointing ROTI in terms of self-confidence, thick=>thin/zbs, mobility, broad-base
  • bond math, socket, FIX, data structure, algo, threading,
  • coding drill, including c++ language experiments
  • specific SDI and pure algo challenges

Realistically ideal next SG job: neglected factors

Essential factors discussed elsewhere: Reasonable coworker benchmark (within my grasp); Reasonable boss, salary, commute…

Here are Some of the neglected factors:

  • Ideally: enough spare time in office —- for blogging and tech exploration, AFTER I clear the initial hump.
    • wordpress access —- may be a temptation for distraction
  • Mainstream and low-churn tech —- like java, c++, py. In contrast GO, javascript, .. would create discontent and other negativity in me.
  • ideally: Mainstream dnlg —- to keep me engaged for a few months. Hopefully some math some low-level complexities
  • too much monotonous, mindless work, could get me disengaged intellectually. Rare. Any past examples? OC approval-seeking
  • gym in office —- could really make a difference
  • culture of knowledge sharing
  • respect for attention to details, but MY type of details (vague)

java primitives have no address #unlike C

In C, any variable, including those on stack, can have its address printed.

In java, the primitive variables have no address.  Every reference type object has an addresses, by definition (“reference” means address)

C# is somewhat mixed and I’m not going into it.

Python rule is extreme, simple and consistent. Every object has an address. You can print id(x) or getrefcount(x)

>>> from sys import getrefcount as rc
>>> i=40490
>>> rc(i)

##a few Advanced algo IV categories #study

By “Advanced” I mean — the average programmer without focused study will likely struggle.

  • problems that require recursion-in-loop — rather common
    • but I have analyzed quite a few. Now at a plateau. No thick->thin
  • grid or matrix problems — more common than I thought. My weakness,
    • but I have invested quite a bit. Fast early ascent is over
  • graph problems — sometimes harder than average. Hard for everyone but I often fare better.
  • DP and greedy problems — occasionally easy, but usually non-intuitive
  • single-str or two-str problems — usually hard
  • num-array problems
  • generator problems — not so common
  • problems that require backtracking — rare

However, some of the hardest problems involve nothing but an int array or a single string.

topological_sort@DAG: linear algo to assign ranks?

Q: topological sort — given a directed graph, any linear algo to assign ranks?

I prefer to use one shared rank for multiple nodes that can be simultaneously started/concretized/evaluated. This feature can increase flexibility and parallelism

terminology — avoid “dependency” — confusing. Prefer “upstream/downstream” or “ancestor/descendant”. Note ancestors and upstreams should be started/processed first.

rank table — We can use a hashtable (or pre-sized vector) to store the ranks: {rank -> list of nodes of that rank}. Assigning a node means adding the node id to the correct list, in O(1)

Assumption 1: the original graph node contains links to ancestors but no descendants. spreadsheet-model. I think this is Kahn’s assumption.

Assumption 2: the original graph nodes contain links to descendants but no ancestors. notification list or “call list”, or “listener list”. I think this model is used the DFS algo.

In most situations, One of these two assumptions would hold, but rarely both.

==== my modified version of Kahn’s algo

Scan-1 O(V+E) — build a hashtable-based two-way edgeSet representation of the graph. For each node, we maintain a hashset (or slist) of ancestors and a hashset of descendants. The duplication is needed, as described below in the Kahn context. (I think the DFS algo needs no duplication.)

Scan-2 O(V) — assign rank 0 to all top-level nodes (no precedent). Now we can use the rank table to scan rank-0 nodes

Scan-3 — Now scan the last assigned rank, rank-0 in this case. For each node in that list, check each downstream child. Unconditionally remove (O(1) thanks to hashset) the upstream link from inside the child. After that, If the child has empty hashset of ancestors it is assigned rank 1. I now believe the precedent/dependent link is never accessed again, so we can remove both.

Repeat the last scan at Rank 1, then Rank 2..

Every node is assigned only once. Every edge is checked only once or twice.

Can the ancestors hashset become an integer count?

— simplicity

Insight — In this design, I use multiple Simple passes and avoid doing too much in one pass. If needed, you can combine Scan-2 and Scan-1.

We treat the original nodes as readonly — nice simplification.

— terminology:
precedent/dependent is accurate but abstract.
“Dependency” is a confusing term. It means someone I depend on. Better avoid this word in graph problems.
uplink/downlink is visual only in a tree with root on top

— Kahn uses “incoming edge” to mean a link to an upstream/ancestor
“All nodes with no incoming edge” … implies a Node::ancestors field

When he visits downstream nodes from “current node”, he needs this->descendants field

This crucial detail is not explained in wikipedia

=== Kyle’s favorite DFS algo, as described on wikipedia, and in [[CLRS]]

Basic idea — check each remaining node and start a DFS. Whenever a leaf (downstream) node is found, Remove it from DAG, and prepend it to output list. Game over when last (most upstream) node is removed from DAG.

  • last node removed will be one of the most upstream nodes
  • first node removed will be one of the original leaf nodes
  • Output list has a meaning — a sorted list of items to “process”
  • Invariant — At any time, size of output list + size of graph == N
  • Implementation note: Actual removal of a node is tricky in either matrix representation or edge-list representation, so it’s easier to use a hashtable to hold “removedNodes”

The simple algo above will fail if cycles exist. To check cycles, we need a  small feature — “temporary mark”. I think this feature can detect cycles in any directed graph such as a tree.

[17]#1 impactful endeavor4family: IV^localSys^gym..

meaningful endeavor Now: algo^zbs^…

In the quiet hours, inevitably I would get my share of self-doubt about the value of my endeavors. See also what efforts go towards 20Y-career building

At other times, I would feel the impacts (of my effort today) on the people who depend on me — grandparents, kids, wife. There’s a lot I can do to make their lives safer, richer, easier, … For example, the interview-preparation effort looks short-term and less meaningful than zbs, but actually has more impact on family well-being such as education, health-care, housing and life-chances. Re zbs, now I realize zbs accumu is rather theoretical and actually limited. Interviews build my confidence and capacity to provide for them.

Looking at my peers … I feel their personal endeavors are not much better than mine:

  • move up to leadership positions. I think that’s a good direction if you CAN move up. I gave up long ago. So I see myself almost like a specialist consultant for hire
  • personal (property) investments
  • kids grades and top schools
accumu #not a benefit #3 mental fitness, anti-aging #2 career after 65 (RnD/teach) #1 family well-being: impact 1-5 #4 lasting social value?
good if low churn good minimal 4 minimal IV: QQ/BP #incl. algo ideas
good excellent none 4 none IV: algo practice
good excellent N.A. 5 none …cf: yoga, jog
good if low churn good possible 3 #helps PKI !! IV minimal zbs #+GTD, instrumentation
churn ask HuKun too volatile 0 none data science, machine learning
some some esp. quant none 1 {– 2 none portable dnlg(+quant)
none some none 4 #if stay`long. Can help move-up but low correlation none xx local sys
some minimal none 1 #questionable can help kids;
can’t teach
per investment analysis #unlike XR
NA minimal none 2 {–1 none … cf: family expense management
some some possible 0 some En/Ch learning
churn good minimal 0 # coding practice churn! Get Real no bandwidth! contribute to OSS??

“spread%%nlg” as a column? trivial.

Q:how much syntax2memorize4speed cod`#python

I would say 5 times more than needed on the job.

Treat it like timed coding competition.

Compare to exam preparation in China.

—- python coding drill for indeed:
small problems using lot of python data structures around chars, strings, arrays
fuxi python blog posts
list comp
define classes with fields
combine multiple lines to 1
triple quote comments

accumulated localSys Xp=brain stress relief

The most common source of everyday hazardous stress on my brain is coworker benchmark by speed of figure-things-out on localSys[1].

My focus today is on this hazardous stress.

If you stay for x years, then the localSys experience would enhance the speed of figure-things-out, and reduce the hazardous stress on the brain. This accumulation takes place naturally, but you do need to “apply yourself”, otherwise you won’t know even the basics.

In rare cases, a bright new joiner can acquire better localSys knowledge than old timers. Eg: Avichal, Yang. Viswa

bigger picture — Josh pointed out that knowledge of the bigger picture greatly reduces the hazardous stress.

[1] Note when the figure-out is NOT about localSys, then I am NOT slower than colleagues. Among other things I’m good at online research and reading documentation.

On a side note, staying too long in one system could lead to boredom and stagnation, not good for the brain, but I believe this is not as harmful as the hazardous stress. Note by “boredom” I don’t mean the initial honeymoon of engagement, but rather the boredom after 5Y. Learning some new technology is one “easy” and safe way to keep the brain active and engaged. Relatively low expectation.

simple tree applied to DP++

This is part of my thick->thin effort, to connect the dots and pattern-recognition. May or may not be highly effective, but no harm trying

By my own definition, A “simple tree” is non-recombinant and cycle-free, and every node has one uplink only. I find these trees very visual. They often help me visualize tough Dynamic Programming and other algorithms:

  • decision tree
  • (Rahul pointed out) DP algos are often recursive, and recursive algos often have a call-tree structure
  • recursive call within each loop iteration — this powerful algo can often be represented as a call-tree or decision-tree
  • backtracking — often uses a tree
  • generating all solutions (paths, formulas..) — often uses a tree
    • paths-from-root — each path often maps to a solution
    • eg: punctuation — to print out all sentences
    • eg: AQR factorization problem may be solvable using the comboSum algo.

union-find O(1) data structure

[[CLRS]] has a chapter on disjoint set. It shows a simple implementation of disjoint set using a linked list. Another implementation is a tree. I also used a hashset.

The tree implementation is described in https://en.wikipedia.org/wiki/Disjoint-set_data_structure . This data structure is probably not used in the union-find algo challenges.

The IV algo challenges are usually small scale. They can use a primitive, trivial version of disjoint-set but doesn’t really gain any meaningful performance advantage. I think this attitude towards disjoint-set is similar to the prevailing attitude towards hash table — it is just a means to achieve big-O demands. With or without this data structure, the problem is solvable but the real challenge is the big-O requirement.

Two main operations can both be optimized — 1) Union() can use by-rank or by-size and 2) Find() can use path compression. Both optimizations keep the tree height very low. High fan-out is desirable and achievable.

Using both path compression and union by rank or size ensures that the amortized time per operation is essentially O(1).

 

deadly delays@ project^feature levels

insightful article: managing tight deadlines is the background

— deadly delay at feature level, without user impact

This is real stressor primarily because there exist team colleagues who can GTD faster.

For the serious delay, user impact is … exaggerated! This is true across all my projects, but user impact doesn’t really matter. In fact, the feature may not be used much whether you deliver it in high or low quality, within or exceeding budget.

— blatant delay at project level when you are architect / team lead

In theory if you promise to delivery some new feature i.e. green field project, then it can be tough to deliver on time. In reality, project time/budget overrun is very common. You only need good explanation.

Users never care that much about any new system cf current production systems. New systems often hit various teething problems and functionally unreliable for months.

OK Users don’t really care that much, but there’s visible technology budget impact which make the technology MD look bad. MD must look for an “explanation” and may cut headcount as in StirtRisk and RTS-Trep

Whose fault? Many fingers point at you the architect, but often it’s partial fault of the user-facing manager due to immaturity and overconfidence with timeline estimate.

Delay is architect’s fault if another architect can deliver faster but I can’t imagine two architects working on two comparable projects at the same time. Therefore it’s never mostly architect’s fault. In reality, management do apply pressure and you suffer, but it’s really for political reason (like sacrificial lamb or scapegoat)

eg: RTS Trep
eg: PWM Special-Investment-management
eg: Post-Saurabh GMDS — multiple delays and scope reductions

eg@traction[def2] in GTD^IV #Sunil

Sunil is not the only one who tried but failed to break into java. Sunil was motivated by the huge java job market. I believe he had opportunities to work on (probably small) java projects and gained confidence, but that’s really the easy part. That GTD experience was completely insufficient to crack java interviews. He needs IV traction.

  • Zhurong also tried java
  • [g] Venkat actually got a java job in SCB but didn’t like it. I feel he was lacking GTD traction
  • [g] XR and the Iranian java guy had some c# projects at work but didn’t gain traction.
  • CSY had a lot to tell me about breaking into java.
  • [gi] CSY and Deepak CM both had java projects at work but no traction
  • [i=IV traction]
  • [g=GTD traction]

IV/QQ traction — I experienced better IV traction in c# than c++. I think it’s because half my c++ interviews were HFT shops.

GTD traction — I had good GTD traction with javascript, php …, much better than c++. My C# GTD traction was also better than c++. C++ is probably the hardest language in terms of GTD. As explained to Kyle, my friend Ashish experienced tremendous GTD traction but can he crack the interviews? Hiring teams can’t access your GTD but they can ask tough QQ questions.

##[19] am competent at..as a professional thanks2tsn

Background — When I listen to a professional musician or comedian from some unfamiliar country, I wonder if they are actually good. Similarly, when I consult a doctor or dentist, I wonder if they are qualified.

“Self-respecting programmer” — Yi Hai’s motivation.

I have been tested and proven on the big stage i.e. U.S. tech interviews + GTD

  • [p t w] java/c++/c#
  • [w] algo
  • [w] coding test
  • [t] SQL
  • [t] socket
  • [t] unix power user
  • swing
  • [p] web app developer across php, javascript, java
  • py, perl (and shell/javascript?) — are these professional games?
  • [t = long tradition over 20Y]
  • [w = worldwide contest]
  • [p = a well-known standard profession]

##[15] I do professionally qualify as …

See also ##[19] am competent at..as a professional

See also blogpost on “N years’ experience — unimportant” and #1 career safety enhancer@past5years

On Wall St there are “combo” jobs to my advantage, but unheard of in SG.

I can qualify as :

##spare time: what peers do to “get ahead”

Exec Summary — for all of us, strategic orgro, ROTI etc is … holy grail … elusive. My wife/in-laws often say “You spend so much family time on your studies but are you earning more?” Translating personal endeavor to income is holy grail .. frustrating..

I believe many professionals don’t have the abilities to convert spare time to tangible personal growth.

Tangibility is defined by each individual, and requires a high degree of self-knowledge.

  • investment analysis (HuKun, XR)? I doubt any of them has any ROTI
  • deeper in java for higher pay or promotion? Higher pay is basically hopeless for many of us who are already at the higher end. GTD depends more on localSys. Promotion has no correlation with deeper java knowledge.
  • coding drill
  • tsn like mobile, data science (XR), java (Sunil)
  • personal investment
  • formal education in spare time like CFA, MBA
  • Stephen Keith was able to write academic papers in his spare time .. very rare

In this realistic analysis, my c++/c#/quant/swing attempts now look commendable.

 

## candd are assessed mostly@tech skill !!dnlg

I feel mostly we as candidates are assessed on technical not domain knowledge.

Q: Among your past job interviews which one had the highest emphasis on dnlg? In the interview, which one of the 3 dnlg categories? Usually math I would assume.

I over-invested in dnlg, relying on it to stand out and differentiate. Well, it’s not “reliable” as a competitive advantage just as SQL/Unix skill isn’t such a selective screening criteria. In contrast, I would say about half of my job interviews had highly selective tech screening.

  • eg: west-coast
  • eg: HFT shops
  • pure algo challenge — About 30% are tough, including a few west-coast challenges
  • whiteboard threading — easy for me, but 60% hard for most guys
  • online tests — MCQ (30%) or coding (80% hard)
  • interactive coding, remote/onsite, whiteboard/IDE — 70% are tough
  • core java/c++/c# knowledge — 50% are tough

python closure and global variables

One way to minimize global variables is converting a regular function into nested functions. Nested functions can automatically READ local variables (like rootNode) of enclosing function outer(). No complication. This follows the usual LGB rule.

On the other hand, if nested_func() needs to rebind outer() function’s rootNode variable to another Node object, then we need to declare “global rootNode” in both nested_func() and outer (). This is tested in https://github.com/tiger40490/repo1/tree/py1/py/88lang. This “partial-global” variable is NOT used outside outer().

Another way to avoid global variables — call mutator methods on the variable, rather than reseating/rebinding. Best example is list or dict objects. Inside nested_func() , if I were to rebind myDict to an empty dict, then it has no effect on the myDict in outer(). Hence “global” needed. The alternative is to clear the content of myDict and populating it. The id(myDict) value remains unchanged. This is the standard java idiom.

resource leak due to dtor bug

[[moreEffC++]] has a chapter dedicated to resource leaks in dtor. I have yet to read it, but here are my views:

A “Resource” means a heap object as a data member, in every case I know. In such a case, “Leak” means failure to call q[ delete ] on the data member.

To guarantee the q[ delete ], I feel one of the simplest yet most reliable strategies is a smart ptr as a data member. In particular

  • If a base-class subobject is already fully constructed when subclass ctor throws, the base-class dtor would run. Any resource in base-class is released if using smart ptr.
  • If a component subobject is already fully constructed when host ctor throws, the component dtor would run.
  • In the normal case of a fully constructed subclass or host object, then obviously its dtor would trigger the base-class dtor or component class dtor, in reverse order

However, replacing every ptr field with a smart ptr is costly and impractical.

G9 asset classes,by dev-job mkt depth

Beware many big domains don’t need lots of developers.

  1. Bonds including sovereign
  2. [V] Eq (including ETF) cash and swap
  3. [V] FX cash and fwd
  4. Eq/FX options
  5. [Q] IRS
  6. IR (including bond) futures
  7. [Q] CDS
  8. [Q] MBS
  9. [d=will see higher demand for developers??]
  10. [V=volume and velocity drive the demand for developers]
  11. [Q=low volume but j4 automation is quantitative in terms of automated risk and pricing.] I believe these quantitative asset classes play to my “theoretical” strength and not too niche, but these domains aren’t growing.

## tech xx backlogS ] blog++

  • 10 most recent posts — have the highest visibility but undeserved and should be back-dated aggressively !
  • oq11 blotposts
  • fuxi_* blogposts
  • bookmarks in local browsers
  • 0zoo blogposts — recent and often needs review and quick-refresh
  • –“softer subjects” rather than mostly-tech topics
  • gmail 7day tag
  • my open blog

embed char_array ] my java object #XR

With market data it’s common to use some Message class(s) that “embeds” a fixed-length character array like 20-char for example.

Allocating an array object off-site on heap is very costly in memory footprint. One extra allocation per Message.

Also slower reading at run time due to data-cache inefficiency. Data cache favors contiguous data structures. See CPU(data)cache prefetching

c/c++ and c# (via Struct) can easily support exactly this “embedding”. I feel java also has some support. Beside JNI, I wonder if there’s another, pure-java solution.

Q: in java, how can I have embedded fixed-length char-array field in my Message or Acct object, rather than a separate array object allocated somewhere off-site?

  1. Solution: If the fixed length is small like 10, I could maintain 10 individual char fields.
  2. Solution: assuming the chars are ascii (8-bit rather than 16-bit in java), I can group first eight chars into a 64-bit long int field. Provide a translation interface when reading/writing the field. With 10 such fields I can support 80-char embedded.
  3. Solution: If not possible, I would use a gigantic singleton off-site char array to hold fixed-length “segments”. Then I need a single int “position”. Every Acct object has a field this.position, where this.position * fixedLength = offset, to identify one segment.
  4. There are two published solutions described in ringBuffer@pre_allocated objects to preempt JGC

Among them, not sure which solution is fastest in practice.

ringBuffer@pre_allocated objects to preempt JGC

Goal — to eliminate JGC completely.

Design 1: I will want Order.java to use primitive fields only and avoid reference fields [1] at all cost, so the total footprint of an Order is known in advance. Say it’s 100 bytes. I will create 10M of dummy Order instances, possibly scattered in heap, not adjacent as in c++, and hold their 10M addresses in an Order array… about 1GB footprint for the Order objects + 80M footprint for the array of 8-byte pointers.

(Note I reuse these Order instances in this object pool and never let them get garbage-collected.)

Then i need a few subscripts to identify the “activeRegion” of the ring but how about released slots enclosed therein?

[1] timestamps will be ints; symbolIDs and clientIDs are ints; short ascii strings will use 64-bit ints (8 characters/int); free-form strings must be allocated off-site:(

Design 2a: To avoid the “scatter” and to place the Order instances side by side, Can we use a serialized byte[100] array object to represent one Order? Can we use one gigantic off-heap byte array to hold all Orders, eliminating the 80M footprint? See java off-heap memory

Design 2b: https://blog.bramp.net/post/2015/08/26/unsafe-part-2-using-sun.misc.unsafe-to-create-a-contiguous-array-of-objects/ shows a contiguous array of java objects, like std::vector<MyObject>

Design 2c: https://www.ibm.com/support/knowledgecenter/en/SSYKE2_7.1.0/com.ibm.java.lnx.71.doc/user/packed_optimizing.html is a feature in IBM jvm

Ring buffer is good if the object lifetimes are roughly equal, giving us FIFO phenomenon. This occurs naturally in market data or message passing gateways. Otherwise, we may need a linked list (free list) of released slots in addition to a pair of subscript to identify the active region.

It might be better to allocate a dedicated buffer for each thread, to avoid contention. Drawback? One buffer may get exhausted when another stays unused.

##RDBMS strategies4faster read+write #widely useful#Sunil

(Further to our chat in mid Apr 2019…) Hi Sunil,

I generally agree that RDBMS such as Sybase, Oracle, DB2, MSSQL, … have good performance for Select and slower performance for Inserts, as you concluded in our last call. However, this conclusion is full of ifs and buts.

  • Read performance can be improved with partitioning — Imagine if the table is relatively large like 100 million rows. Even if the queries use an index, the index can have fairly large footprint (with implications). Often the Asia users only query the Asia records while Europe users only query Europe records.
  • Read performance can be improved with pre-computed data tables. If the same calculation is frequently requested via some frequent queries, then the calculation results can be saved in a pre-computed result table.
  • Read performance can be improved with stored procedures.
  • Read performance can benefit from de-normalization.
  • Read performance can improve if entire table is in-memory, as confirmed by my 2009 DB2 trainer in NY.

Now I think most RDBMS performance tuning techniques target Select as slow Select is the most common pain and the most severe pain.

Insert is typically much slower than read, but user expectation is also less demanding. In my observation, most RDBMS databases are either mostly-select or mostly-insert. The mostly-insert database can benefit from batch insert (bcp in sybase/MSSQL), or a background writer thread (Gemfire).

However, sometimes real-time insert is needed. I think the most common strategy is sharding (horizontal partitioning) like splitting 100 million rows into two tables of 50 millions each.

A related strategy is normalization (vertical partitioning). Normalization removes duplicate data and helps inserts.

function returning (unnamed)rvalue object

The rules are hard to internalize but many interviewers like to zero in on this topic. [[effModernC++]] explains

  • case: if return type is nonref, and in the function the thingy-to-be-returned is a …. rvr parameter [1], then by default, it goes through copy-ctor on return.
    • You should apply explicit return move(thingy)
    • [1] See P172, extremely rare except in interviews
  • case: if return type is nonref and in the function the thingy-to-be-returned is a …. stack (non-static) object behind a local variable (NOT nameless object) .. a very common scenario, then RVO optimization usually happens.
    • Such a function call is always (P175 bottom) seen by the caller as evaluating to a naturally-occurring nameless rvalue object.
    • the books says move() should never be used in the case
  • case: if return type is rvr? avoid it if possible. I don’t see any use case.
  • case: if return type is lvr (not rvr) and returned thingy is a local object, then compilation fails as explained in the blogpost above

stay]shape 4CIV+QQ till 50-55

I would say QQ remains my stronger arm. (I don’t need to care about HFT shops’ assessment of me.)

  • QQ benefits from thick->thin (and xRef) … one of the key competitive advantages I could develop through blogging and continuous refresh.
  • CIV also benefits from blogging and continuous practice … my competitive advantages. Remember David Okao’s question “what is your secret weapon?”

Note High-end CIV is only needed at top west-coast shops. I think most of the top performers are young but I could stand out among my age group.

sharding:=horizontal partition #too many rows

Assuming a 88-column, 6mil-row table, “Horizontal” means horizontally pushing a knife across all 88 columns, splitting 6 million rows into 3 millions each.

Sharding can work on noSQL too.

GS PWM positions table had 9 horizontal partitions for 9 regions.

“Partitioning” is a more generic term and can mean 1) horizontal (sharding) or 2) vertical cutting like normalization.

WallSt server-side: java gain`mkt share #Greg#Sunil

I spoke to Greg on 22 Apr 2019.

If on server-side java / c# / c++ jobs add to 100%, what percentage is java. I said about 80%. He said 70-80%.

I told him “c# is mostly client-side”. He agreed and said “some buy-sides use c# on server-side”.

I also told Greg about my OC friend Sunil. OC uses server-side c# but according to Sunil, on the job market c# is mostly on GUI and java dominates server-side job market.

I told Greg that over the last 7 years, on server-side java has gained market share relative to c# and c++. Greg agreed.

I now feel the momentum is in java’s favor. Java is clearly easier to use than c++ and not that much slower. Beside c++ and c# the old challengers, is there any new challenger to java?

xp: pure algo question often require QQ 打通脉络

  • eg: bbg 2017 pure algo question on tree serialization required a standard BFS. Just a few lines but I struggled for a long time. If you memorize the key parts and practice enough, then the *knowledge* would show as a algo skill
  • eg: facebook regex is very hard to anyone I know, unless they worked on a similar problem before, and *know* the tricks.
  • eg: generate permutations, combinations. Can be tricky unless you *know* the key points.
  • eg: DP bottom-up vs top-down with memoization
  • eg: SCB-FM IV: stack-based queue
  • .. many more examples

打通知识脉络 is the Chinese phrase

[19] assignment^rebind in python^c++j

For a non-primitive, java assignment is always rebinding. Java behavior is well-understood and simple, compared to python.

Compared to python, c++ assignment is actually well-documented .. comparable to a mutator method.

Afaik, python assignment is always rebinding afaik, even for an integer. Integer objects are immutable, reference counted.
In python, if you want two functions to share a single mutable integer variable, you can declare a global myInt.
It would be in the global idic/namespace. q[=] has special meaning like

idic[‘myInt’] =..

Alternatively, you can wrap the int in a singular list and call list mutator methods, without q[=].

See my experiment in github py/88lang and my blogpost on immutable arg-parssing

became expert via external QQ benchmark` !!localSys xx

Suppose I invest heavily and become very productive on a local c++system. Similarly, a java, or SQL, or socket, pyhton .. system, but the point is — Any local system uses only a small portion of the language features.. at most 5% of the typical high-end interview topics on that language.

Therefore the project experience won’t give me confidence to be an expert on the language, but acing multiple QQ interviews does, as these interviews compare dozens of competitive, motivated and qualified candidates across a wide field.

I grew very confident of java through this type of external benchmarking, in contrast to internal benchmarking within the local team.

required experience: QQ imt CIV

A few years into my technology career, I observed that Solaris/Oracle was harder to self-teach at home than linux/mysql/php/python/perl/javascript because even a high-school student can install and hack with the latter. Entry-barrier was non-existent.

Similarly, I now observe that Wall St QQ-IV demands longer experience than coding IV of west-coast style. Coding drill can use Leetcode over months. QQ requires not so much project experience but a lot of interview experience. It takes more than months of exploration and self-learning.

  • example — tcp/ip. My friend Shanyou preferred to read books systematically, but a book touches on hundreds of topic. He wouldn’t know what topics interviews like to dig into.
  • example — c++ TMP. A fresh grad can read about it but won’t know the favorite topics for interviewers
  • Compelling Example — java concurrency. A fresh grad can build up theoretical knowledge but won’t have my level of insight

Many inexperienced candidates were highly appreciated in west coast interviews. No such appreciation on Wall St because Wall St can VP or contract roles require work experience .. tried-n-tested.

  • Wall St interviews are selective in terms of experience
  • West Coast coding interviews are selective in terms of … speed and optimality

low-level^high-level expertise

See also my blogpost on wrapper^low-level API

Compare to laymen on the street, I have accumulated fairly deep, specific and demonstrable non-local expertise in two lucrative field i.e. software dev + finance

  • concurrency details (theory++) in real languages
  • dStruct choices and effective combinations
  • dStruct implementation details in java/c++/pyhton
  • SQL joins and tuning
  • sockets? no deep insight but my expertise beats most peers
  • Most standard algo problems are familiar to me
  • memory models, memory layout.. beneath c++ and java
  • drv pricing math, VaR statistics, bond math. My expertise goes beyond the degree

These trophies are won based on TSN, interviews and personal time investment … not automatically

— Let me focus on low-level vs high-level
Most of the dev expertise domains are low-level, consequently very specific — you either know it or don’t. I can even teach in these subjects. The more low-level, the rarer is the expertise. The laymen developers have a vague idea of the low-level details, for many reasons.

High-level understanding is often more useful (than low-level, theoretical QQ knowledge) in projects, but in job interviews, low-level knowledge is differentiator.

Practically all (98%) high-end java/c++ interviews use low-level questions as differentiator. I excluded the start-ups as I’m unfamiliar with them… Eg: Quoine.

In my dev (not system architect) interviews, they rarely asked high-level stuff like spring rationale, design patterns…
I tend to believe these questions can’t /separate the crop from the chaff/

The finance dnlg  also serves as differentiator among developers. The most specific sub-domains are quant math, the “architecture” and some of the jargon.

C++ is more low-level than java; java is more low-level than python…

Q: Paradoxically, C and assembly are even more low-level but not more valuable?
%%A: Some C knowledge is highly valued. For example, kernel knowledge is high-value and mostly at the level of C, compiler, assembly, and hardware
%%A: assembly expertise in device drivers or embedded is not really relevant to HFT, and too far from the money

MOM is not low-level and seldom asked.

c++IV=much harder than GTD #Mithun

c++ IV is much harder than c++ job GTD, as I told Mithun.

  • GTD is no different from java jobs, even though the build process can be slightly hairy. Java build can also get messy.
  • In contrast, C++ IV is a totally different game.

You need a rating of 1/10 to do a decent job, but need 7/10 to pass ibank interviews. This gap is wider in c++ than in java as java interview bar is much lower.

Most technical challenges on the job are localSys, so you can just look at existing code and 照猫画虎, 如法炮制, as AndrewYap does. Venkat of RTS said we should but we still do.

Corollary — after 3Y full time c++ job, you may still fail to pass those interviews. Actually I programed C for 2Y but couldn’t pass any C interview whatsoever.

python *args **kwargs: cheatsheet

“asterisk args” — I feel these features are optional in most cases. I think they can create additional maintenance work. So perhaps no need to use these features in my own code.

However, some codebases use these features so we had better understand the syntax rules.

— Inside the called function astFunc(),

Most common way to access the args is a for-loop.

It’s also common to forward these asterisk arguments:

def astFunc(*args, **kwargs):
anotherFunc(*args, **kwargs)

I also tested reading the q[ *args ] via list(args) or args[:]

— how to use these features when calling a function:

  • astFunc(**myDict) # astFunc(**kwa)
  • simpleFunc(**myDict) # simpleFunc(arg1, arg2) can also accept **myDict

See my github

cod`drill^quant self-study

quant knowledge used to be a halo, now replaced by coding IV skills like

  • classic generators
  • DP, recursion
  • clever data structures
  • clever graph algos
  • speed coding

In 2012 When I was studying quant in my spare time, I was on a powerful ascent. In the subsequent years, I gradually lost steam. The quant sector was shrinking and market depth was disappointing.

Since 2018, my coding drill feels very relevant, even though the gap behind the strong players continues to be a pain and a de-motivation. When I stop comparing myself with the stronger players, I would feel my improvement in a highly valuable skill

java singleton^immutable classes #enum

In c++ (and perhaps java) Most singletons are designed as “casual singletons” i.e. they are used as singletons, can be instantiated twice if you really want to, although there’s seldom a good justification.

[[effJava]] explains that an immutable class needs no copying. I think this is a performance view. (Does it apply to Optional.java? )

However, we don’t need to work hard trying to make it strict singletons.  Strict singleton is a functional requirement i.e. 2nd instance would be a functional defect.

If an immutable class happens to be used as one-instance data type, then lucky. But tomorrow it may get instantiated twice.. no worries 🙂

If boss asks you to make this class singleton, you should point out the legwork required and the numerous loopholes to fix before achieving that goal. Worthwhile?

Java enum types are special. JVM guarantees them to be immutable AND singleton, even in the face of serialization. See P 311.

%% absorbency: experiment/SDI imt speed-coding

When you find yourself in high-absorbency mood, favor (#1 most draining) speed-coding. See list of less-draining drills at the bottom.

I can read dry QQ topics for hours each day for many days, but tend to lose steam after coding for a few hours. Speed coding drill drains my “laser” energy faster than QQ reading/blogging/experiment. When my laser is weakened (by boredom), I must drill harder to go through the “brick walls”.

I guess many fellow programmers enjoy coding more than reading. I feel lucky that in my interviews, knowledge tests still outweigh coding test.

Q: What part of coding drill is worst on my absorbency?

A: speed coding implementation is worst. It drains my laser energy fastest. After coding for a few hours I always feel like a deflated balloon and discharged battery and and need a full day to recharge.

I think frustration is the key. Self-expectation (about progress and traction) and self-image create the frustration and the drain.

Instead of traction, I often feel stuck and overspent.

I feel disappointed with myself (Deepak self-identify as “annoyed”. )

Q: Can we stop comparing with others and just compare with our past? Doable sometimes. Consider Leetcode speed-coding contest #Rahul

— Just like yoga

  • if I work on easy problems I feel wasting my time
  • if I work on tough problems I feel painful, draining and want to give up. After the practice i need hours to recover.

Q: … So can we find easier coding drills that I could enjoy (as Rahul suggested)? Definitely not easy. I think the first difficult step is self-acceptance that I can’t improve much at this age.

Q (excellent question): What type of “coding” drill can I do for hours like reading/blogging?

  • pseudo-code algo on paper/whiteboard is lighter. No ECT so I am swift and efficient. Less draining/frustrating.
  • SDI is most fun, least boring, not draining/frustrating. I can spend hours on a SDI. I feel a bit of accu. More like QQ less like coding drill.
  • concurrency coding questions are less draining as other guys are not faster
  • c++/java language feature QQ experiments are more like QQ. I can spend hours on a QQ experiment. More interesting as there’s no time-line no benchmark no frustration. Also other guys are not stronger. I feel some accu exactly like reading on these me features
  • review of my previous code is much less draining (than writing new solutions) as there’s no time-line and code is already working
  • analyzing patterns and reusable techniques (very few) in past problems. Thick->thin is the holy grail. I work hard towards it.
  • reading syntax and ECT tips in books and my blog

 

#1(reusable)AuxDS for algo challenges

Here’s a Reusable data structure for many pure algo challenges:

Pre-process to construct a static data store to hold a bunch of “structs” in linked list -OR- growing vector -OR- growing RBTree , all O(1) insertion :). Then we can build multiple “indices” pointing to the nodes

Here are a few O(1) indices: (Note O(1) lookup is the best we can dream of)

  • hashtable {a struct field like a string -> iterator into the data store}
  • array indexed by a struct field like small int id, where payload is an iterator from the data store
  • If the structs have some non-unique int field like age, then we can use the same key lookup to reach a “group”, and within the group use one (or multiple) hashtable(s) keyed by another struct field

I think this is rather powerful and needed only in the most challenging problems like LRU cache.

Will tough coding IV decline]10Y@@ #Rahul

It’s possible that as easy coding IVs spread or grow, tough coding IVs decline i.e. less wide-spread. In such a case, my t-investment now will have lower ROTI than expected.

It’s possible that as easy coding IVs spread or grow, tough coding IVs also grow … for differentiation of talent.

Rahul responded to my question and pointed out the two styles of interviewers he observed:

* interested in can/can’t solve — can this candidate get the problem solved in time?
* interested in thought process

Rahul felt the first type will want to use ever tougher problems to pick outstanding candidates… selectivity

Rahul felt the second type will want to use easier problems to get more insight into the candidate’s thought process.

Rahul felt to differentiate, employers can compare completion time on easy questions

python nested function2reseat var] enclos`scope

My maxPalindromeSubstr code in https://github.com/tiger40490/repo1/tree/py1/py/algo_str demos the general technique, based on https://stackoverflow.com/questions/7935966/python-overwriting-variables-in-nested-functions

Note — inside your nested function you can’t simply assign to such a variable. This is like assigning to a local reference variable in java.

https://jonskeet.uk/java/passing.html explains the fundamental property of java reference parameter/argument-passing. Basically same as the python situation.

In c# you probably (99% sure) need to use ref-parameters. In c++, you need to pass in a double-pointer. Equivalently, you can pass in a reference to a pre-existing 64-bit ptr object.