premium salary to compensate for intrinsic motivation@@

I recall that at more than one juncture in my job hunting career, I feel overwhelmed by a premium offer and said in my head "if I accept this offer and earn 20% more than the standard rate, then the ensuing pride, self-image boost etc would surely create a wellspring of positive motivation."

How naive….in hind sight. The real factors affecting my job satisfaction was usually unrelated to premium salary. See the spreadsheet about job satisfaction.

Advertisements

too many DB-writes: sharding insufficient #Indeed

Context: each company receives many many reviews. In a similar scenario, we can say a stock receives many investor comments

Interviewer: OK you said horizontal sharding by company id can address highly concurrent data store updates. Now what if one company, say, Amazon, by itself gets 20% of the updates so sharding can’t help this one company.

me: I guess the update requests would block and possibly time out. We can use a task queue.

This is similar to WordPress import/export requests.

  • Each task takes a few seconds, so we don’t want user to wait.
  • If high server load, the wait could be longer.
  • More important — this task need not be immediate. It can be scheduled.
  • We should probably optimize for server efficiency rather than latency or throughput

So similarly, all the review submissions on Amazon can be queued and scheduled.

In FIX protocol, an order sender can receive 150=A meaning PendingNew. I call it “queued for exchange”.  The OMS Server sends the acknowledgement in two steps.

  1. Optional. Execution Report message on order placement in OMS Server’s Order Book. ExecType (150) = A (Pending New)
  1. Execution Report message on receiving acknowledgement from exchange. ExecType (150) = 0 (New). I guess this indicates placement in exchange order book

Note this optimization is completely different from HFT latency engineering.

jav isn’t challenged by c++/c# but by high-level languages

  • I noticed at the beginning of this decade (2010-2012) c# mounted a credible challenge to dethrone java but has since lost momentum.
  • c++ had a renaissance with c++1x, but after 8 years I would say it failed to become a threat to java.

The other challengers to java seems to be domain-specific high level languages like javascript (browser and beyond), ruby-on-rails, php (web server side), perl (text processing), q (kdb). Not sure about Golang.

Python is the strongest challenger among them. Here I consider python a domain-specific language given its visible strengths in data science, machine-learning, black-box testing, and traditional process management. I sometimes consider python a general purpose language like java and c++, but i feel it’s not widely perceived as such, perhaps due to dynamic typing and efficiency.

C/c++ remains the default system programming language — that’s clear. Less clearly, Java remains the “default” application-development language on server side — but in this sentence the key words are vague.

Luckily for java developers, nowadays most of the work is on server side, including many mobile and cloud applications. In addition, java is viable for offline/batch job, data science, transaction-processing systems … Actually, I listed javascript as a java challenger because I think a lot of application functionalities are moving to client-side.

Luckily for java developers, a few internet powerhouses led by Google (and Alibaba) have chosen Java as the default language on server side. So have most Wall St firms.

G5 unexpected big wins]%%life #absorbency

should create pointer blogposts in ‘ffree’ and ‘diet’ blogs

20 uphill breakthroughs #wellness/xx is a similar list.

My unexpected success at weight improvement and diet is rather rare … the latest of top 5 achievements of my entire life. Other significant (often unexpected) successes:

  • outstanding project performance in 95G, Barcap and RTS. In other teams, I was able to do a reasonable job among bright young professionals.
  • QQ technical wins on high-end c++ positions (10+ times each), a golden ticket to high salaries on Wall St and then SG
  • [r] high absorbency for self-learning till this age.
    • MSFM
    • 😦 otherwise, the 4.5 years in SG was low.
  • [r] ffree first as bachelor and again in 2018.
  • [r=really rare]

How about my Cambodia investments? No really a personal effort per se.

tough retrans questions from IV+ from me

Many mkt data interviewers ask me high-level biz logic question about retrans, without implementation details.

Q1 (IV x 2): do you track gaps for Line A and also track gaps in Line B?
%%A: no. I use both lines to fill the gaps.

Q2 (IV x 2): Taking parser + orderbook (i.e. rebus) as a single black box, when you notice a gap (say seq #55/56/57), do you continue to process seq # 58, 59 … or do you warehouse these #58, 59… messages and wait until you get the resent #55/56/57 messages?

Q2b (IV): In your orderbook engine (like Rebus), suppose you get a bunch of order delete/exec/modify messages, but the orderId is unrecognized and possibly pending retrans. Rebus doesn’t know about any pending retrans. What would rebus do about those messages?
%%A: I don’t know the actual design [3], but if I were the architect I would always check the orderId. If orderId is unknown then I warehouse the message. If it is a known order Id in Rebus, I will apply the message on the order book. Risks? I can’t think of any.

[3] It’s important to avoid stating false facts, so i will add the disclaimer.

Q2c (IV): what data structures would you use to warehouse those pending messages? ( I guess this question implies warehousing is needed.)
%%A: a linked list would do. Duplicate seqNum check is taken care of by parser.

Q3 (IV): do you immediately send a retrans request every time you see a gap like (1-54, then 58,59…)? Or do you wait a while?
A: I think we do need to wait since UDP can deliver #55 out of sequence.

The above questions were probably the most important questions in a non-tech interview. In other words, if an interview has no coding no QQ then most of the questions would be simpler than these retrans questions ! These questions test your in-depth understanding of a standard mkt data feed parser design. 3rd type of domain knowledge.

Q: after you detect a gap, what does your parser do?
A (Deepak): parser saves the gap and moves on. After a configured timeout, parser sends out the retrans request. Parser monitors messages on both Line A and B.

Q: if you go on without halting the parser, then how would the rebus cope?

  • A: if we are missing the addOrder, then rebus could warehouse all subsequent messages about unknown order IDs. Ditto for a Level 1 trade msg.

Deepak felt this warehouse could build up quickly since the eve-inceasing permanent gaps could contain tens of thousands of missing sequence numbers. I feel orderId values are increasing and never reused within a day, so we can check if an “unknown” orderId is very low and immediately discard it, assuming the addOrder is permanently lost in a permanent gap.

  • A: if we are missing an order cancel (or trade cancel), i.e. the last event in the life cycle, then we don’t need to do anything special. When the Out-of-sequence message shows up, we just apply it to our internal state and send it to downstream with the OOS flag.

If a order cancel is lost permanently, we could get a crossed order book. After a few refreshes (15min interval), system would discard stale orders sitting atop a crossed book.

In general, crossed book can be fixed via the snapshot feed. If not available in integrated feed, then available in the open-book feed.

  • A: If we are missing some intermediate msg like a partial fill, then we won’t notice it. I think we just proceed. The impact is smaller than in FIX.

OOS messages are often processed at the next refresh time.

Q3b: But how long do we wait before requesting retrans?
Q3c: how do you keep a timer for every gap identified?

Q: after you send a retrans request but gets no data back, how soon do you resend the same request again? Do you maintain a timer for every gap?

Q: You said the retrans processing in your parser shares the same thread as regular (main publisher) message processing. What if the publisher stream is very busy so the gaps are neglected? In other words, the thread is overloaded by the publisher stream.

short+efficient : holy grail ] algo search

For most of my tough algo problems (such as leetcode problems)

  1. many celebrated solutions are very short, but not so efficient
    • I think about 60% of them are easier to memorize due to length.
    • if you can understand and memorize them, they are quicker to write due to fewer keystrokes
    • About half of them are easier to understand, thanks to the source code size
    • eg: yield-based generators are a prime example
    • eg: recursive solutions are often brief but inefficient
  2. many great solutions are efficient in terms of O(), but verbose
    • eg: bottom-up DP
    • eg: top-down DP with memoization
    • eg: using a tree (when not required) can sometimes give an efficient solution

However, these two goals are often hard to harmonize. It is my holy grail to meet both criteria simultaneously, but i won”t try so hard.

  1. For real coding problems, I will prefer brevity
  2. For verbal discussions, I will concentrate on efficient solutions

pass generator output to next generator

I think this technique can be extremely time-saving in coding tests.

https://github.com/tiger40490/repo1/blob/py1/py/algo_combo_perm/1fromEachSet.py my code demos:

for myset in pool:     output = list(gen(output, myset))

The gen() function uses yield. For the first call to gen(), we exhaust all of its items and save into a list named “output”.

Then we pass this list into the second gen(), this time with a different myset

template specialization based on NDTTP=true

We know it’s possible to specialize a template for a concrete type like int or std::string, but I didn’t know that It’s also possible to

… specialize a (class or function) template for a particular compile-time const value (like “true”) of a NDTTP (like “bool flag”)

  • On [[Alexandrescu]] Page xii , Scott Meyers showed an elegant example of specializing for “true”. Note “true” is a value, not a data type !
  • P 34 has a longer example.

Note on reading TMP code — the template specialization syntax is clumsy and can add noise to the signal. Better ignore the syntax rules for now to focus on the gist.

more python mock IV questions #Deepak

Q (real IV): how do you verify a given string represents an integer? For example, "3e5" would be invalid.

Q (real IV): compare multi-threading vs multi-processing in python. Which one have you used in your projects?

Q: Have you used a FIFO data structure (like a queue) in python? It’s OK if you have not, but how would you create such a data structure in python?

Q: Compare and contrast deep copy vs shallow copy for a python list
Q: Compare and contrast deep copy vs shallow copy for a python dictionary

Q (advanced question from me): When is a deep copy required? Give some scenarios

Q : Have you used context managers? (I think they are useful.)

Q (advanced question from me): compare and contrast list comprehension vs generator expression

Q: write a Student class with lastName, firstName, birthDate fields. Also include a class attribute "instCnt" showing the number of student objects created. As you create a new Student instance, this instCnt should increment.

Q: have you heard of the LGB rule in python?

Q (basic question from me, never asked in interviews): what data types can’t be used a dictionary keys

Q (basic question from me, never asked in interviews): what’s the usage of "global" keyword? Give some usage scenarios
Q (basic question from me, never asked in interviews): what builtin data types, apart from tuples, are immutable? Give some examples.

##fully engaged+!ROTI still beats boredom

Every learning direction has naysayers, and virtually all learning efforts generates low ROTI. ROTI and strategic investment are holy grails, mirages or white elephants.

We can’t dismiss $ROTI but I think we really need to look beyond $ROTI — too elusive. As Mithun put it, java multithreading alone can fetch you highest salary.

Fully engaged for a few short months but without ROTI is actually not bad. It’s like satisfactory sex without simultaneous climax. I enjoyed the journey, without worrying about the destination.

I’m able to fully engaged for a few months, while my peers can’t!

I have blogposts on spare time usage. Tech learning (even the non-strategic topics) are among the most productive. There’s really nothing more worth learning.

  • — most of these are failed technology bets, but 10x better than not engaged (i.e. boredom)
  • 2012-2013 c# study
  • HFT-specific QQ topics
  • MSFM
  • 2011-2012 self-learning volatility
  • self-learning swing
  • MOM study
  • gemfire study
  • — less disappointing ROTI in terms of self-confidence, thick=>thin/zbs, mobility, broad-base
  • bond math, socket, FIX, data structure, algo, threading,
  • coding drill, including c++ language experiments
  • specific SDI and pure algo challenges

Realistically ideal next SG job: neglected factors

Essential factors discussed elsewhere: Reasonable coworker benchmark (within my grasp); Reasonable boss, salary, commute…

Here are Some of the neglected factors:

  • Ideally: enough spare time in office —- for blogging and tech exploration, AFTER I clear the initial hump.
    • wordpress access —- may be a temptation for distraction
  • Mainstream and low-churn tech —- like java, c++, py. In contrast GO, javascript, .. would create discontent and other negativity in me.
  • ideally: Mainstream dnlg —- to keep me engaged for a few months. Hopefully some math some low-level complexities
  • too much monotonous, mindless work, could get me disengaged intellectually. Rare. Any past examples? OC approval-seeking
  • gym in office —- could really make a difference
  • culture of knowledge sharing
  • respect for attention to details, but MY type of details (vague)

java primitives have no address #unlike C

In C, any variable, including those on stack, can have its address printed.

In java, the primitive variables have no address.  Every reference type object has an addresses, by definition (“reference” means address)

C# is somewhat mixed and I’m not going into it.

Python rule is extreme, simple and consistent. Every object has an address. You can print id(x) or getrefcount(x)

>>> from sys import getrefcount as rc
>>> i=40490
>>> rc(i)

##a few Advanced algo IV categories #study

By “Advanced” I mean — the average programmer without focused study will likely struggle.

  • problems that require recursion-in-loop — rather common
    • but I have analyzed quite a few. Now at a plateau. No thick->thin
  • grid or matrix problems — more common than I thought. My weakness,
    • but I have invested quite a bit. Fast early ascent is over
  • graph problems — sometimes harder than average. Hard for everyone but I often fare better.
  • DP and greedy problems — occasionally easy, but usually non-intuitive
  • single-str or two-str problems — usually hard
  • num-array problems
  • generator problems — not so common
  • problems that require backtracking — rare

However, some of the hardest problems involve nothing but an int array or a single string.

topological sort: given a directed graph, any linear algo to assign ranks?

Q: topological sort — given a directed graph, any linear algo to assign ranks?

I prefer to use one rank for multiple nodes that can be simultaneously started/concretized/evaluated. This feature can increase flexibility and parallelism

terminology — avoid “dependency” — confusing. Prefer “upstream/downstream” or “ancestor/descendant”. Note ancestors should be processed first.

rank table — We can use a hashtable (or pre-sized vector) to store the ranks: {rank -> list of nodes of that rank}. Assigning a node means adding the node id to the correct list, in O(1)

Assumption 1: the original graph nodes contain ancestors but no descendants. spreadsheet-model. I think this is Kahn’s assumption.

Assumption 2: the original graph nodes contain descendants but no ancestors. notification list or “call list”, or “listener list”

In most situations, One of these two assumptions would hold, but rarely both.

Scan-1 O(V+E) — build a hashtable-based two-way edgeSet representation of the graph. For each node, we maintain a hashset (or slist) of ancestors and a hashset of descendants. The duplication is needed, as described below in the Kahn context.

Scan-2 O(V) — assign rank 0 to all top-level nodes (no precedent). Now we can use the rank table to scan rank-0 nodes

Scan-3 — Now scan the last assigned rank, rank-0 in this case.
For each node in that list, check each downstream child.
Unconditionally remove (O(1) thanks to hashset) the precedent from inside the child.
After that, If the child has empty hashset of ancestors it is assigned rank 1.
I now believe the precedent/dependent link is never accessed again, so we can remove both.

Repeat the last scan at Rank 1, then Rank 2..

Every node is assigned only once. Every edge is checked only once or twice.

Can the ancestors hashset become an integer count?

— simplicity
In this design, I use multiple Simple passes and avoid doing too much in one pass. If needed, you can combine Scan-2 and Scan-1.

We treat the original nodes as readonly — nice simplification.
–terminology:
precedent/dependent is accurate but abstract.
“Dependency” is a confusing term. It means someone I depend on. Better avoid this word in graph problems.
uplink/downlink is visual only in a tree with root on top

–Kahn uses “incoming edge” to mean a precedent
“All nodes with no incoming edge” … implies a node object has this->ancestors field

When he visits downstream nodes from “current node”, he needs this->descendants field

This crucial detail is not explained in wikipedia

[17]#1 impactful endeavor(Now)4family: IV^zbs^gym..

meaningful endeavor Now: algo^zbs^…

In the quiet hours, inevitably I would get my share of self-doubt about the value of my endeavors. See also what efforts go towards 20Y-career building

At other times, I would feel the impacts (of my effort today) on the people who depend on me — grandparents, kids, wife. There’s a lot I can do to make their lives safer, richer, easier, … For example, the interview-preparation effort looks short-term and less meaningful than zbs, but actually has more impact on family well-being such as education, health-care, housing and life-chances. Re zbs, now I realize zbs accumu is rather theoretical and actually limited. Interviews build my confidence and capacity to provide for them.

Looking at my peers … I feel their personal endeavors are not much better than mine:

  • move up to leadership positions. I think that’s a good direction if you CAN move up. I gave up long ago. So I see myself almost like a specialist consultant for hire
  • personal (property) investments
  • kids grades and top schools
accumu #not a benefit #3 mental fitness, anti-aging #2 career after 65 (RnD/teach) #1 family well-being: impact 1-5 #4 lasting social value?
good if low churn good minimal 4 minimal IV: QQ/BP #incl. algo ideas
good excellent none 4 none IV: algo practice
good excellent N.A. 5 none …cf: yoga, jog
good if low churn good possible 3 #helps PKI !! IV minimal zbs #+GTD, instrumentation
churn ask HuKun too volatile 0 none data science, machine learning
some some esp. quant none 1 {– 2 none portable dnlg(+quant)
none some none 4 #if stay`long. Can help move-up but low correlation none xx local sys
some minimal none 1 #questionable can help kids;
can’t teach
per investment analysis #unlike XR
NA minimal none 2 {–1 none … cf: family expense management
some some possible 0 high En/Ch learning
churn good minimal 0 # coding practice churn! Get Real no bandwidth! contribute to OSS??

“spread%%nlg” as a column? trivial.

Q:how much syntax2memorize4speed cod`#python

I would say 5 times more than needed on the job.

Treat it like timed coding competition.

Compare to exam preparation in China.

—- python coding drill for indeed:
small problems using lot of python data structures around chars, strings, arrays
fuxi python blog posts
list comp
define classes with fields
combine multiple lines to 1
triple quote comments

simple tree applied to DP++

This is part of my thick->thin effort, to connect the dots and pattern-recognition. May or may not be highly effective, but no harm trying

By my own definition, A “simple tree” is non-recombinant and cycle-free, and every node has one uplink only. I find these trees very visual. They often help me visualize tough Dynamic Programming and other algorithms:

  • decision tree
  • (Rahul pointed out) DP algos are often recursive, and recursive algos often have a call-tree structure
  • recursive call within each loop iteration — this powerful algo can often be represented as a call-tree or decision-tree
  • backtracking — often uses a tree
  • generating all solutions (paths, formulas..) — often uses a tree
    • paths-from-root — each path often maps to a solution
    • eg: punctuation — to print out all sentences
    • eg: AQR factorization problem may be solvable using the comboSum algo.

union-find O(1) data structure

https://en.wikipedia.org/wiki/Disjoint-set_data_structure described a specific data structure, but probably not used in the union-find algo challenges.

The IV algo challenges are usually small scale. They can use a primitive, trivial version of disjoint-set but doesn’t really gain any meaningful performance advantage. I think this attitude towards disjoint-set is similar to the prevailing attitude towards hash table — it is just a means to achieve big-O demands. With or without this data structure, the problem is solvable but the real challenge is the big-O requirement.

Two main operations can both be optimized — 1) Union() can use by-rank or by-size and 2) Find() can use path compression. Both optimizations keep the tree height very low. High fan-out is desirable and achievable.

Using both path compression and union by rank or size ensures that the amortized time per operation is essentially O(1).

deadly delays@ project^feature levels

insightful article: managing tight deadlines is the background

— deadly delay at feature level, without user impact

This is real stressor primarily because there exist team colleagues who can GTD faster.

For the serious delay, user impact is … exaggerated! This is true across all my projects, but user impact doesn’t really matter. In fact, the feature may not be used much whether you deliver it in high or low quality, within or exceeding budget.

— blatant delay at project level when you are architect / team lead

In theory if you promise to delivery some new feature i.e. green field project, then it can be tough to deliver on time. In reality, project time/budget overrun is very common. You only need good explanation.

Users never care that much about any new system cf current production systems. New systems often hit various teething problems and functionally unreliable for months.

OK Users don’t really care that much, but there’s visible technology budget impact which make the technology MD look bad. MD must look for an “explanation” and may cut headcount as in StirtRisk and RTS-Trep

Whose fault? Many fingers point at you the architect, but often it’s partial fault of the user-facing manager due to immaturity and overconfidence with timeline estimate.

Delay is architect’s fault if another architect can deliver faster but I can’t imagine two architects working on two comparable projects at the same time. Therefore it’s never mostly architect’s fault. In reality, management do apply pressure and you suffer, but it’s really for political reason (like sacrificial lamb or scapegoat)

eg: RTS Trep
eg: PWM Special-Investment-management
eg: Post-Saurabh GMDS — multiple delays and scope reductions

eg@traction[def2] in GTD^IV #Sunil

Sunil is not the only one who tried but failed to break into java. Sunil was motivated by the huge java job market. I believe he had opportunities to work on (probably small) java projects and gained confidence, but that’s really the easy part. That GTD experience was completely insufficient to crack java interviews. He needs IV traction.

  • Zhurong also tried java
  • [g] Venkat actually got a java job in SCB but didn’t like it. I feel he was lacking GTD traction
  • [g] XR and the Iranian java guy had some c# projects at work but didn’t gain traction.
  • CSY had a lot to tell me about breaking into java.
  • [gi] CSY and Deepak CM both had java projects at work but no traction
  • [i=IV traction]
  • [g=GTD traction]

IV/QQ traction — I experienced better IV traction in c# than c++. I think it’s because half my c++ interviews were HFT shops.

GTD traction — I had good GTD traction with javascript, php …, much better than c++. My C# GTD traction was also better than c++. C++ is probably the hardest language in terms of GTD. As explained to Kyle, my friend Ashish experienced tremendous GTD traction but can he crack the interviews? Hiring teams can’t access your GTD but they can ask tough QQ questions.

##[19] am competent at..as a professional thanks2tsn

Background — When I listen to a professional musician or comedian from some unfamiliar country, I wonder if they are actually good. Similarly, when I consult a doctor or dentist, I wonder if they are qualified.

“Self-respecting programmer” — Yi Hai’s motivation.

I have been tested and proven on the big stage i.e. U.S. tech interviews + GTD

  • [p t w] java/c++/c#
  • [w] algo
  • [w] coding test
  • [t] SQL
  • [t] socket
  • [t] unix power user
  • swing
  • [p] web app developer across php, javascript, java
  • py, perl (and shell/javascript?) — are these professional games?
  • [t = long tradition over 20Y]
  • [w = worldwide contest]
  • [p = a well-known standard profession]

##[15] I can professionally qualify as …

See also ##[19] am competent at..as a professional

See also blogpost on “N years’ experience — unimportant” and #1 career safety enhancer@past5years

On Wall St there are “combo” jobs to my advantage, but unheard of in SG.

I can qualify as :

##spare time: what can peers do to “get ahead”

Exec Summary — for all of us, strategic orgro, ROTI etc is … holy grail … elusive. My wife/in-laws often say “You spend so much family time on your studies but are you earning more?” Translating personal endeavor to income is holy grail .. frustrating..

I believe many professionals don’t have the abilities to convert spare time to tangible personal growth.

Tangibility is defined by each individual, and requires a high degree of self-knowledge.

  • investment analysis (HuKun, XR)? I doubt any of them has any ROTI
  • deeper in java for higher pay or promotion? Higher pay is basically hopeless for many of us who are already at the higher end. GTD depends more on localSys. Promotion has no correlation with deeper java knowledge.
  • coding drill
  • tsn like mobile, data science (XR), java (Sunil)
  • personal investment
  • formal education in spare time like CFA, MBA
  • Stephen Keith was able to write academic papers in his spare time .. very rare

In this realistic analysis, my c++/c#/quant/swing attempts now look commendable.

 

## candd are assessed mostly@tech skill !!dnlg

I feel mostly we as candidates are assessed on technical not domain knowledge.

Q: Among your past job interviews which one had the highest emphasis on dnlg? In the interview, which one of the 3 dnlg categories? Usually math I would assume.

I over-invested in dnlg, relying on it to stand out and differentiate. Well, it’s not “reliable” as a competitive advantage just as SQL/Unix skill isn’t such a selective screening criteria. In contrast, I would say about half of my job interviews had highly selective tech screening.

  • eg: west-coast
  • eg: HFT shops
  • pure algo challenge — About 30% are tough, including a few west-coast challenges
  • whiteboard threading — easy for me, but 60% hard for most guys
  • online tests — MCQ (30%) or coding (80% hard)
  • interactive coding, remote/onsite, whiteboard/IDE — 70% are tough
  • core java/c++/c# knowledge — 50% are tough

python closure and global variables

One way to minimize global variables is converting a regular function into nested functions. Nested functions can automatically READ local variables (like rootNode) of enclosing function outer(). No complication. This follows the usual LGB rule.

On the other hand, if nested_func() needs to rebind outer() function’s rootNode variable to another Node object, then we need to declare “global rootNode” in both nested_func() and outer (). This is tested in https://github.com/tiger40490/repo1/tree/py1/py/88lang. This “partial-global” variable is NOT used outside outer().

Another way to avoid global variables — call mutator methods on the variable, rather than reseating/rebinding. Best example is list or dict objects. Inside nested_func() , if I were to rebind myDict to an empty dict, then it has no effect on the myDict in outer(). Hence “global” needed. The alternative is to clear the content of myDict and populating it. The id(myDict) value remains unchanged. This is the standard java idiom.

resource leak due to dtor bug

[[moreEffC++]] has a chapter dedicated to resource leaks in dtor. I have yet to read it, but here are my views:

A “Resource” means a heap object as a data member, in every case I know. In such a case, “Leak” means failure to call q[ delete ] on the data member.

To guarantee the q[ delete ], I feel one of the simplest yet most reliable strategies is a smart ptr as a data member. In particular

  • If a base-class subobject is already fully constructed when subclass ctor throws, the base-class dtor would run. Any resource in base-class is released if using smart ptr.
  • If a component subobject is already fully constructed when host ctor throws, the component dtor would run.
  • In the normal case of a fully constructed subclass or host object, then obviously its dtor would trigger the base-class dtor or component class dtor, in reverse order

However, replacing every ptr field with a smart ptr is costly and impractical.

G9 asset classes,by dev-job mkt depth

Beware many big domains don’t need lots of developers.

  1. Bonds including sovereign
  2. [V] Eq (including ETF) cash and swap
  3. [V] FX cash and fwd
  4. Eq/FX options
  5. [Q] IRS
  6. IR (including bond) futures
  7. [Q] CDS
  8. [Q] MBS
  9. [d=will see higher demand for developers??]
  10. [V=volume and velocity drive the demand for developers]
  11. [Q=low volume but j4 automation is quantitative in terms of automated risk and pricing.] I believe these quantitative asset classes play to my “theoretical” strength and not too niche, but these domains aren’t growing.

## tech xx backlogS ] blog++

  • 10 most recent posts — have the highest visibility but undeserved and should be back-dated aggressively !
  • oq11 blotposts
  • fuxi_* blogposts
  • bookmarks in local browsers
  • 0zoo blogposts — recent and often needs review and quick-refresh
  • –“softer subjects” rather than mostly-tech topics
  • gmail 7day tag
  • my open blog

embed char_array ] java object: %%ideas #XR

With market data it’s common to use some Message class(s) that “embeds” a fixed-length character array like 20-char for example.

Allocating an array object off-site on heap is very costly in memory footprint. One extra allocation per Message.

Also slower reading at run time due to data-cache inefficiency. Data cache favors contiguous data structures. See CPU(data)cache prefetching

c/c++ and c# (via Struct) can easily support exactly this “embedding”. I feel java also has some support. Beside JNI, I wonder if there’s another, pure-java solution.

Q: in java, how can I have embedded fixed-length char-array field in my Message or Acct object, rather than a separate array object allocated somewhere off-site?

  1. Solution: If the fixed length is small like 10, I could maintain 10 individual char fields.
  2. Solution: assuming the chars are ascii (8-bit rather than 16-bit in java), I can group first eight chars into a 64-bit long int field. Provide a translation interface when reading/writing the field. With 10 such fields I can support 80-char embedded.
  3. Solution: If not possible, I would use a gigantic singleton off-site char array to hold fixed-length “segments”. Then I need a single int “position”. Every Acct object has a field this.position, where this.position * fixedLength = offset, to identify one segment.

Among them, not sure which solution is fastest in practice.

homemade ringBuffer@pre_allocated objects to preempt JGC

Goal — to eliminate JGC completely.

Design 1: I will want Order.java to use primitive fields only and avoid reference fields [1] at all cost, so the total footprint of an Order is known in advance. Say it’s 100 bytes. I will create 10M of dummy Order instances, possibly scattered in heap, not adjacent as in c++, and hold their 10M addresses in an Order array… about 1GB footprint for the Order objects + 80M footprint for the array of 8-byte pointers.

(Note I reuse these Order instances in this object pool and never let them get garbage-collected.)

Then i need a few subscripts to identify the “activeRegion” of the ring but how about released slots enclosed therein?

[1] timestamps will be ints; symbolIDs and clientIDs are ints; short ascii strings will use 64-bit ints (8 characters/int); free-form strings must be allocated off-site:(

Design 2a: To avoid the “scatter” and to place the Order instances side by side, Can we use a byte[100] array object to represent one Order? Can we use one gigantic byte array to hold all Orders, eliminating the 80M footprint?

Design 2b: https://blog.bramp.net/post/2015/08/26/unsafe-part-2-using-sun.misc.unsafe-to-create-a-contiguous-array-of-objects/ shows a contiguous array of java objects, like std::vector<MyObject>

Design 2c: https://www.ibm.com/support/knowledgecenter/en/SSYKE2_7.1.0/com.ibm.java.lnx.71.doc/user/packed_optimizing.html is a feature in IBM jvm

Ring buffer is good if the object lifetimes are roughly equal, giving us FIFO phenomenon. This occurs naturally in market data or message passing gateways. Otherwise, we may need a linked list (free list) of released slots in addition to a pair of subscript to identify the active region.

It might be better to allocate a dedicated buffer for each thread, to avoid contention. Drawback? One buffer may get exhausted when another stays unused.

##RDBMS strategies4faster read+write #widely useful#Sunil

(Further to our chat in mid Apr 2019…) Hi Sunil,

I generally agree that RDBMS such as Sybase, Oracle, DB2, MSSQL, … have good performance for Select and slower performance for Inserts, as you concluded in our last call. However, this conclusion is full of ifs and buts.

  • Read performance can be improved with partitioning — Imagine if the table is relatively large like 100 million rows. Even if the queries use an index, the index can have fairly large footprint (with implications). Often the Asia users only query the Asia records while Europe users only query Europe records.
  • Read performance can be improved with pre-computed data tables. If the same calculation is frequently requested via some frequent queries, then the calculation results can be saved in a pre-computed result table.
  • Read performance can be improved with stored procedures.
  • Read performance can benefit from de-normalization.
  • Read performance can improve if entire table is in-memory, as confirmed by my 2009 DB2 trainer in NY.

Now I think most RDBMS performance tuning techniques target Select as slow Select is the most common pain and the most severe pain.

Insert is typically much slower than read, but user expectation is also less demanding. In my observation, most RDBMS databases are either mostly-select or mostly-insert. The mostly-insert database can benefit from batch insert (bcp in sybase/MSSQL), or a background writer thread (Gemfire).

However, sometimes real-time insert is needed. I think the most common strategy is sharding (horizontal partitioning) like splitting 100 million rows into two tables of 50 millions each.

A related strategy is normalization (vertical partitioning). Normalization removes duplicate data and helps inserts.

function returning rvr or rvalue object

The rules are hard to internalize but many interviewers like to zero in on this topic. [[effModernC++]] explains

  • case: if return type is nonref and in the function the thingy-to-be-returned is a …. rvr parameter [1], then by default, it goes through copy-ctor on return.
    • You should apply explicit return move(thingy)
    • [1] extremely rare except in interviews
  • case: if return type is nonref and in the function the thingy-to-be-returned is a …. stack (non-static) object behind a local variable (NOT nameless object) .. a very common scenario, then RVO optimization usually happens.
    • Such a function call is always (P175 bottom) seen by the caller as evaluating to a naturally-occurring nameless rvalue object.
    • the books says move() should never be used in the case
  • case: if return type is rvr? avoid it if possible. I don’t see any use case.
  • case: if return type is lvr (not rvr) and returned thingy is a local object, then compilation fails as explained in the blogpost above

sharding=horizontal partition #too many rows

Assuming a 88-column, 6mil-row table, “Horizontal” means horizontally pushing a knife across all 88 columns, splitting 6 million rows into 3 millions each.

Sharding can work on noSQL too.

GS PWM positions table had 9 horizontal partitions for 9 regions.

“Partitioning” is a more generic term and can mean 1) horizontal (sharding) or 2) vertical cutting like normalization.

WallSt server-side: java gain`mkt share #Greg#Sunil

I spoke to Greg on 22 Apr 2019.

If on server-side java / c# / c++ jobs add to 100%, what percentage is java. I said about 80%. He said 70-80%.

I told him “c# is mostly client-side”. He agreed and said “some buy-sides use c# on server-side”.

I also told Greg about my OC friend Sunil. OC uses server-side c# but according to Sunil, on the job market c# is mostly on GUI and java dominates server-side job market.

I told Greg that over the last 7 years, on server-side java has gained market share relative to c# and c++. Greg agreed.

I now feel the momentum is in java’s favor. Java is clearly easier to use than c++ and not that much slower. Beside c++ and c# the old challengers, is there any new challenger to java?

fear@large codebase #web/script coders

One Conclusion — my c++ /mileage/ made me a more confident, and slightly more competent programmer, having “been there; done that”

For half my career I avoided enterprise technologies like java/c++/c#/SQL/storedProc/MOM/Corba/sockets/turboC++…

Until GS, I was scared of the technical jargon, complexities, low-level API’s debuggers/linkers/IDE, compiler errors and opaque failures in java/SQL … (even more scared of C and Windows). Scared of the larger, more verbose codebases in these languages (cf the small php/perl programs)… so scared that I had no appetite to study these languages.

Look around your office. Many people have at most a single (rarely two) project involving a large codebase. Large like 50k to 100k lines of code (excluding comments and empty lines). I feel the RTB/DBA or BA/PM roles within dev teams usually don’t require the individual to take on those large codebases. Since it’s no fun, time-consuming and possibly impenetrable, few of them would take it on. In other words, most people who try would give up sooner or later.

Searching in a large codebase is a first challenge. Even figuring out a variable’s actual type can be a challenge in a compiled language.

Compiling can be a challenge esp. with C/c++.

Tracing code flow is a common complexity across languages but worse in compiled languages.

In my experience, perl,php,py,javascript codebases are usually small like pets. When they grow to big creatures they are daunting and formidable just like compiled language projects. Some personal experiences —
* Qz? Not a python codebase at all
* pwm comm? I would STILL say codebase would be bigger if using a compiled language

  • Analogy — if you have not run marathons you would be afraid of it.
  • Analogy — if you have not coached a child on big exams you would be afraid of it.

I feel web (or batch) app developers using standard tools lack the hardcore experience. They operate at a higher level, cleaner and simpler.

Note Java is much cleaner than c++

 

xp: pure algo question often require QQ 打通脉络

  • eg: bbg 2017 pure algo question on tree serialization required a standard BFS. Just a few lines but I struggled for a long time. If you memorize the key parts and practice enough, then the *knowledge* would show as a algo skill
  • eg: facebook regex is very hard to anyone I know, unless they worked on a similar problem before, and *know* the tricks.
  • eg: generate permutations, combinations. Can be tricky unless you *know* the key points.
  • eg: DP bottom-up vs top-down with memoization
  • eg: SCB-FM IV: stack-based queue
  • .. many more examples

打通知识脉络 is the Chinese phrase

[19] assignment^rebind in python^c++j

For a non-primitive, java assignment is always rebinding. Java behavior is well-understood and simple, compared to python.

Compared to python, c++ assignment is actually well-documented .. comparable to a mutator method.

Afaik, python assignment is always rebinding afaik, even for an integer. Integer objects are immutable, reference counted.
In python, if you want two functions to share a single mutable integer variable, you can declare a global myInt.
It would be in the global idic/namespace. q[=] has special meaning like

idic[‘myInt’] =..

Alternatively, you can wrap the int in a singular list and call list mutator methods, without q[=].

See my experiment in github py/88lang and my blogpost on immutable arg-parssing

became expert via external QQ benchmark` !!localSys xx

Suppose I invest heavily and become very productive on a local c++system. Similarly, a java, or SQL, or socket, pyhton .. system, but the point is — Any local system uses only a small portion of the language features.. at most 5% of the typical high-end interview topics on that language.

Therefore the project experience won’t give me confidence to be an expert on the language, but acing multiple QQ interviews does, as these interviews compare dozens of competitive, motivated and qualified candidates across a wide field.

I grew very confident of java through this type of external benchmarking, in contrast to internal benchmarking within the local team.

required experience: QQ imt CIV

A few years into my technology career, I observed that Solaris/Oracle was harder to self-teach at home than linux/mysql/php/python/perl/javascript because even a high-school student can install and hack with the latter. Entry-barrier was non-existent.

Similarly, I now observe that Wall St QQ-IV demands longer experience than coding IV of west-coast style. Coding drill can use Leetcode over months. QQ requires not so much project experience but a lot of interview experience. It takes more than months of exploration and self-learning.

  • example — tcp/ip. My friend Shanyou preferred to read books systematically, but a book touches on hundreds of topic. He wouldn’t know what topics interviews like to dig into.
  • example — c++ TMP. A fresh grad can read about it but won’t know the favorite topics for interviewers
  • Compelling Example — java concurrency. A fresh grad can build up theoretical knowledge but won’t have my level of insight

Many inexperienced candidates were highly appreciated in west coast interviews. No such appreciation on Wall St because Wall St can VP or contract roles require work experience .. tried-n-tested.

  • Wall St interviews are selective in terms of experience
  • West Coast coding interviews are selective in terms of … speed and optimality

low-level^high-level expertise

Compare to laymen on the street, I have accumulated fairly deep, specific and demonstrable non-local expertise in two lucrative field i.e. software dev + finance

  • concurrency details (theory++) in real languages
  • dStruct smart choices and nesting
  • dStruct implementation details in java/c++/pyhton
  • SQL joins and tuning
  • sockets? no deep insight but beats most peers
  • Most standard algo problems are familiar to me
  • drv pricing math, VaR statistics, bond math. My expertise goes beyond the degree

These trophies are won based on tsn, interviews and personal time investment … not automatically

— Let me focus on low-level vs high-level
Most of the dev expertise domains are low-level (consequently very specific). I can even teach in these subjects. The more low-level, the rarer is the expertise. The laymen developers have a vague idea of the low-level details, for many reasons.

High-level understanding is often more useful (than low-level, theoretical QQ knowledge) in projects, but in job interviews, low-level knowledge is differentiator.

Practically all (98%) high-end java/c++ interviews use low-level questions as differentiator. I excluded the start-ups as I’m unfamiliar with them… Eg: Quoine.

In my dev (not system architect) interviews, they rarely asked high-level stuff like spring rationale, design patterns…
I tend to believe these questions can’t /separate the crop from the chaff/

The finance dnlg  also serves as differentiator among developers. The most specific sub-domains are quant math, the “architecture” and some of the jargon.

C++ is more low-level than java, which is more low-level than python…

Q: Paradoxically, C and assembly are even more low-level but not more valuable?
%%A: Some C knowledge is highly valued. For example, kernel knowledge is high-value and mostly at the level of C, compiler, assembly, and hardware
%%A: assembly expertise in device drivers or embedded is not really relevant to HFT, and too far from the money

MOM is not low-level and seldom asked.

c++IV=much harder than GTD #Mithun

c++ IV is much harder than c++ job GTD, as I told Mithun.

  • GTD is no different from java jobs, even though the build process can be slightly hairy. Java build can also get messy.
  • In contrast, C++ IV is a totally different game.

You need a rating of 1/10 to do a decent job, but need 7/10 to pass ibank interviews. This gap is wider in c++ than in java as java interview bar is much lower.

Most technical challenges on the job are localSys, so you can just look at existing code and 照猫画虎, 如法炮制, as AndrewYap does. Venkat of RTS said we should but we still do.

Corollary — after 3Y full time c++ job, you may still fail to pass those interviews. Actually I programed C for 2Y but couldn’t pass any C interview whatsoever.

python *args **kwargs: cheatsheet

“asterisk args” — I feel these features are optional in most cases. I think they can create additional maintenance work. So perhaps no need to use these features in my own code.

However, some codebases use these features so we had better understand the syntax rules.

— Inside the called function astFunc(),

Most common way to access the args is a for-loop.

It’s also common to forward these asterisk arguments:

def astFunc(*args, **kwargs):
anotherFunc(*args, **kwargs)

I also tested reading the q[ *args ] via list(args) or args[:]

— how to use these features when calling a function:

  • astFunc(**myDict) # astFunc(**kwa)
  • simpleFunc(**myDict) # simpleFunc(arg1, arg2) can also accept **myDict

See my github

cod`drill^quant self-study

quant knowledge used to be a halo, now replaced by coding IV skills like

  • classic generators
  • DP, recursion
  • clever data structures
  • clever graph algos
  • speed coding

In 2012 When I was studying quant in my spare time, I was on a powerful ascent. In the subsequent years, I gradually lost steam. The quant sector was shrinking and market depth was disappointing.

Since 2018, my coding drill feels very relevant, even though the gap behind the strong players continues to be a pain and a de-motivation. When I stop comparing myself with the stronger players, I would feel my improvement in a highly valuable skill

java singleton^immutable classes #enum

[[effJava]] explains that an immutable class needs no copying.

However, we don’t need to work hard trying to make it singletons.

If an immutable class happens to be used as one-instance data type, then lucky. But tomorrow it may get instantiated twice.. no worries 🙂

If boss asks you to make this class singleton, you should point out the legwork required and the numerous loopholes to fix before achieving that goal. Worthwhile?

Java enum types are special. JVM guarantees them to be immutable AND singleton, even in the face of serialization. See P 311.

%% absorbency: experiment/SDI imt speed-coding

When you find yourself in high-absorbency mood, favor (#1 most draining) speed-coding. See list of less-draining drills at the bottom.

I can read dry QQ topics for hours each day for many days, but tend to lose steam after coding for a few hours. Speed coding drill drains my “laser” energy faster than QQ reading/blogging/experiment. When my laser is weakened (by boredom), I must drill harder to go through the “brick walls”.

I guess many fellow programmers enjoy coding more than reading. I feel lucky that in my interviews, knowledge tests still outweigh coding test.

Q: What part of coding drill is worst on my absorbency?

A: speed coding implementation is worst. It drains my laser energy fastest. After coding for a few hours I always feel like a deflated balloon and discharged battery and and need a full day to recharge.

I think frustration is the key. Self-expectation (about progress and traction) and self-image create the frustration and the drain.

Instead of traction, I often feel stuck and overspent.

I feel disappointed with myself (Deepak self-identify as “annoyed”. )

Q: Can we stop comparing with others and just compare with our past? Doable sometimes. Consider Leetcode speed-coding contest #Rahul

— Just like yoga

  • if I work on easy problems I feel wasting my time
  • if I work on tough problems I feel painful, draining and want to give up. After the practice i need hours to recover.

Q: … So can we find easier coding drills that I could enjoy (as Rahul suggested)? Definitely not easy. I think the first difficult step is self-acceptance that I can’t improve much at this age.

Q (excellent question): What type of “coding” drill can I do for hours like reading/blogging?

  • pseudo-code algo on paper/whiteboard is lighter. No ECT so I am swift and efficient. Less draining/frustrating.
  • SDI is most fun, least boring, not draining/frustrating. I can spend hours on a SDI. I feel a bit of accu. More like QQ less like coding drill.
  • concurrency coding questions are less draining as other guys are not faster
  • c++/java language feature QQ experiments are more like QQ. I can spend hours on a QQ experiment. More interesting as there’s no time-line no benchmark no frustration. Also other guys are not stronger. I feel some accu exactly like reading on these me features
  • review of my previous code is much less draining (than writing new solutions) as there’s no time-line and code is already working
  • analyzing patterns and reusable techniques (very few) in past problems. Thick->thin is the holy grail. I work hard towards it.
  • reading syntax and ECT tips in books and my blog

 

#1(reusable)AuxDS for algo challenges

Here’s a Reusable data structure for many pure algo challenges:

Construct a data store to hold a bunch of “structs” in linked list OR growing vector (O(1) insertion). Then we can build multiple “indices” pointing to the nodes

Here are a few average-O(1) indices:
(Note O(1) lookup is the best we can dream of)

  • hashtable keyed by a struct field like a string
  • array indexed by a struct field like small int id
  • If there’s a non-unique int field, then we can use the same array lookup to reach a “group”, and within it use one (or multiple) hashtable(s) keyed by another field

Will tough coding IV decline]10Y@@ #Rahul

It’s possible that as easy coding IVs spread or grow, tough coding IVs decline i.e. less wide-spread. In such a case, my t-investment now will have lower ROTI than expected.

It’s possible that as easy coding IVs spread or grow, tough coding IVs also grow … for differentiation of talent.

Rahul responded to my question and pointed out the two styles of interviewers he observed:

* interested in can/can’t solve — can this candidate get the problem solved in time?
* interested in thought process

Rahul felt the first type will want to use ever tougher problems to pick outstanding candidates… selectivity

Rahul felt the second type will want to use easier problems to get more insight into the candidate’s thought process.

Rahul felt to differentiate, employers can compare completion time on easy questions

python nested function2reseat var] enclos`scope

My maxPalindromeSubstr code in https://github.com/tiger40490/repo1/tree/py1/py/algo_str demos the general technique, based on https://stackoverflow.com/questions/7935966/python-overwriting-variables-in-nested-functions

Note — inside your nested function you can’t simply assign to such a variable. This is like assigning to a local reference variable in java.

https://jonskeet.uk/java/passing.html explains the fundamental property of java reference parameter/argument-passing. Basically same as the python situation.

In c# you probably (99% sure) need to use ref-parameters. In c++, you need to pass in a double-pointer. Equivalently, you can pass in a reference to a pre-existing 64-bit ptr object.