Am I driven2master localSys@@

I think a dose (not fixation) of negative motivation is needed, including fear.

I think I wasn’t at mvea. At RTS I lost the steam after a while.

What’s my vague definition of “master”? Perhaps I did master the pspc module of mvea?

I would say yes at catcha, empworld, zed, Strategem, RTS. Perhaps yes at Chartered,

[20] slow track, again

This blogpost is another critique of the slow track.

My Macq managers Kevin A and Stephen Keith are fairly successful old-timers. Such an individual would hold a job for 5-10 years, grow in know-how, effectiveness (influence,,,). Once a few years they move up the rank. In their eyes, a job hopper or contractor like me is hoplessly left on the slow track — rollling stone gathers no moss.

I would indeed have felt that way if I had not gained the advantages of burn rate + passive incomes. No less instrumental are my hidden advantages like

  • frugal wife
  • SG citizenship
  • stable property in HDB
  • modest goal of an average college for my kids

A common mistake is missing the silent majority of old timres who don’t climb up. See also

[20]random list@”protections” close2%%heart

See also layers of buffer

The focus on the word “protection” is likely to uncover some threats I perceive consciously or subconsciouly.

  • [c] SG citizenship — protects against healthcare inflation, long-term nursing co, housing cost, general inflation. Some protective benfits of SG citizenship are not measured by money —
    • protection against poor service (common in many countries) when I’m in need of help
    • protective environment for up-bringing of kids — drug-free,,
    • SG government has a large past reserve to protect Singapore against many threats.
  • [c] good burn rate as family — protects against many cashflow threats.
  • [c] bx plans
  • sound education for kids — protects against parents’ worries
  • [w] good sleep — protects against workload, ,,,
  • [w] good diet
  • [w] bmi greenzone
  • stable marriage — is a cornerstone and protects against many threats
  • harmony at home
  • [j] WallSt contract job market — protects against early decline of demand
  • [j] IV competitiveness including theoretical skills
  • [j] English skill — protects against job market competition from overseas newcomers
  • [j=threat of job loss or job pain]
  • [c=cashflow stress or distress]
  • [w=welness protections]

live-free-of restrictions #LS

I can live with the restrictions of diet, burn rate, workout routine, perhaps strict work hours … but not LiuShuo style self-restraint in office context. Why?

  • I feel the suffering is not worthwhile.
  • I don’t feel self-confident I can “improve” myself.

I want to live free of this last category of restrictions. For this, I would let go of any leadership career.

git commit –am # risky

Consider the harmless-looking command git commit –am

Most of the time, I rely on this command as a safe way to amend commit message without touching the original files in the commit.

However, if there’s further changes on file1 in staging, then this command would silently introduce this change into the commit. Once you finalize the commit message and save it, this change inadvertently becomes part of the commit !

spare time as SlackResource #blogg`,parent`,localSys

In 2017 I wrote to Gerald Robinson that spare time depletion would become one of the key challenges/pains of my forthcoming/upcoming adjustments.

For many (probably most) of my peers, if they don’t feel the drive to keep learning in their spare time, the resource depletion doesn’t feel so harmful, hazardous, stressful.. Beside this “learning”, I also have a need for quiet time for reflection.

— commute — Bayonne commute is tolerable only when I have plenty of spare time as “slack resource”.

— Parenting — (esp. my son) became a huge commitment and strain on my slack resources.

— Additional paid annual leave is supposed to be a slack resource but I think it doesn’t reduce the pressure to perform

… but I did spend these hours (weekends + evenings) in office to learn localSys, provided I’m engaged, in the mood.

Need to think harder about how to turn these large quantities of high-quality spare resource into stress-relief. These resources feel like untapped oil reserve.

git tag with annotation

I always want a long description for my git branch. Not sure how best to do it. https://stackoverflow.com/questions/4750845/how-do-you-annotate-a-branch has some suggestions.

— Git tag can have an annotation:

git tag -a newTag some_commit #create
git show newTag #reveal the annotation
git tag -d newTag #deletes the tag

Two ways to edit annotation:

  1. We can remove and re-tag with a modified annotation.
  2. git tag <tag name> <tag name> -f -a

3stressors: FOMO^PIP^livelihood

  • PIP
  • FOMO including brank envy
  • burn rate stress esp. the dreaded job-loss

jolt: FSM dividend has never delivered the de-stressor as /envisioned/. In contrast, my GRR has produced /substantial/ nonwork income, but still insufficient to /disarm/ the threat of PIP !

In my experience, Job-loss stressor is indeed alleviated by this income or the promise thereof 🙂

Excluding the unlucky (broken, sick,,) families, I feel most “ordinary” people’s stress primarily come from burn rate i.e. making ends meet, including job-loss fear. I feel the middle class families around me could survive  at a much lower theoretical burn rate of SGD 3.5-4.5k (or USD 6k perhaps… no 1st-hand experience) but they choose the rat race of keeping up with the Jones’s (FOMO). Therefore, their burn rate becomes 10k. See also SG: bare-bones ffree=realistic #WBank^wife^Che^3k and covid19$$handout reflect`Realistic burn rate

For some, FOMO takes them to the next level — bench-marking against the high-flyers.

— jolt: PIP^job-loss fear

For the middle class, any burn rate exceeding 3k is a real (albeit subconscious) stressor because the working adults now need to keep a job and build up a job-loss contingency reserve. Remember Davis Wei….3-month is tough for him? How about 30 years?

With a full time job, salaried workers experience a full spectrum of stressors including PIP. PIP would be impotent/toothless if the job is like a hobby. I would say very few people have such a job.

Aha .. Contract career is free of PIP.

For me, job loss is a lighter stressor than PIP fear. In fact, I don’t worry about end of contract and bench time. I worry more about humiliating bonus.

I’d rather lose a job than receiving a token bonus on PIP.

Most people assume that my PIP fear was fundamentally related to cashflow worry, but I am confident about job hunting. So my PIP fear is all about ego and unrelated to cashflow.

[18]semi-retirement jobs:Plan B #if !! U.S.

I had blogged about this before, such as the blogger-pripri post on “many long term career options”

Hongzhi asked what if you can’t go to the US.

* Some Singapore companies simply keep their older staff since they still can do the job, and the cost of layoff is too high.
* Support and maintenance tech roles
* Some kind of managerial role ? I guess I am mediocre but could possibly do it, but i feel hands-on work is easier and safer.
* Teach in a Poly or private school ? possibly not my strength
* Run some small business such as Kindergarten with wife

##strongER trec taking up large codebases

I have grown from a sysAdmin to a dev
I have grown from web and scripting pro into a java pro then c++ pro !
Next, I hope to grow my competence with large codebase.

I feel large codebase is the #1 diffentiator separating the wheat from the chaff — “casual” vs hardcore coders.

With a large codebase, I tend to focus on the parts I don’t understand, regardless that’s 20% or 80% of the code I need to read. I can learn to live with that ambiguity. I guess Rahul was good at that.

In a few cases, within 3M I was able to “take up” a sizable brown field codebase and become somewhat productive. As I told Kyle, I don’t need bottom-up in-depth knowledge to be competent.

In terms of count of successes taking up sizable brown-field codebase, I am a seasoned contractor, so I would score more points than a typical VP or an average old timer in a given system.

  • eg: Guardian — don’t belittle the challenge and complexity
  • eg: mvea c++ codebase — My changes were localized, but the ets ecosystem codebase was huge
  • eg: mvea pspc apportionment/allocation logic + price matching logic
  • eg: mtg comm (small green field), integrated into a huge brown field codebase
  • eg: StirtRisk personalization — I made it work and manager was impressed
  • eg: StirtRisk bug fixes on GUI
  • eg: Macq build system — many GTD challenges
  • eg: Moodles at NIE
  • eg: Panwest website contract job at end of 2016
  • ~~~ the obvious success stories
  • eg: AICE — huge stored proc + perl
  • eg: Quest app owner
  • eg: error memos
  • eg: RTS

burn-out: intensity^overtime

For 99.5% of us, high intensity is unsustainable almost by definition. I believe Intensity doesn’t equal GTD output, which is usually measured by some numbers, or subjectively by observers. As a student, I had a strong foundation (“abilities”), so a moderate intensity was able to create high output. Intensity factors include (not limited to)

  • non-stop mental focus
  • truly high-pace workplace
  • creative engine or analytical engine
  • deadlines
  • multi-tasking and context switching
  • quick recall of huge amount of details
  • GS style time booking

https://www.channelnewsasia.com/news/commentary/working-relentless-pace-wont-help-career-prospects-10596240?cid=h3_referral_inarticlelinks_24082018_cna is a good online article

During summer, the firm works Just four days — a total of 32 hours. The company’s summer workload must fit reduced hours, Mr Fried insists. This employer doesn’t offer “flexible” hours in exchange for intensity

Employers and policymakers focus a lot on the excessive hours, but compared with overtime, work intensity predicts much greater reductions in well-being and career-related outcomes.

lazy evaluation: a common lambda usage

http://dobegin.com/lambda-functions-everywhere/

Lambda function fits nice to do lazy evaluation. It gives a special syntax to capture some lambda expression or a block to be evaluated later:

// Java: delay counting lines of an input text
Supplier<Integer> createLineCounter(String text) {
    return () -> countLines(text);
}

Here, the method creates and returns a Supplier-of-integer, a callable object that produces an integer not now, but at a later time.

Q: at registration time, the string text is received but used only at counting time, so where is this string stored?
%%A: probably “captured” in some hidden field of the Supplier object

I suspect our dropcopy framework uses lambda mostly for this purpose.

json format: simple rules

Basic rule — name:value pairs.

  • Name is (always?) a primitive string.
  • Value can be array (horizontal), jsonObj (vertical) or primitives.
    • (As the ancestor of javascript, C language introduced array and struct. )

— array of jsonObjects are enclosed in []; jsonObj is enclosed in {} and consists of name:value pairs

Note array can only appear as a value in a name:value pair.

— A jsonObj can be a field of an enclosing jsonObj, or can be an array element. In both casee, it is the value of a name:value pair.

[20]what protects family livelihood: IV skill^GTD #Ashish

Hi Ashish,

After talking to you about PIP (i.e. performance improvement process) in my past and in future scenarios, and considering my financial situation (wife not working, 2 young kids + 2 elderly grandparents) over the 20Y horizon , I came up with this question —

Q: Between two factors: AA) my competitive position on the tech hiring market, BB) job security at MLP, which factor has more impact on my family livelihood

Answer: I remain as convinced now as 10 years ago: AA is the dominant factor. I won’t allow myself to rely on some employer to provide my family a stable income for 20Y, even if I do a good job. There are countless stories among my peers who worked hard but lost jobs.

If I do a good job in the current team, what’s the chance of keeping this job for 10Y? 10%? There are individuals (like my manager) who stay in one team for 10+ years, but I think anyone like him has witnessed dozens of coworkers who have no choice but leave, for various reasons (not counting those who had a choice to stay but hopped higher elsewhere.)

That’s the basic answer to my opening question, but there are a few important sub-factors to point out.

Family livelihood includes housing, medical and education. In the U.S., I would incur $3k/M rental + 2k/M health insurance. Therefore, livelihood in the U.S. is more precarious, less secured.

My Health — is a big hidden factor. Stamina, mental capacity has a direct impact on our “performance” in the competition, both on job market and on-the-job. I think you probably need a lot of physical and mental energy, stamina,,, to deep dive into an unfamiliar local system or codebase, to become so confident, right?

company stability — is a sub-factor of BB. Some investment banks (GS, Barclays, MS) are known to aggressively cut headcount even in profitable years, to stay lean and mean.

Aging — is a factor affecting AA slightly more than BB. Age discrimination is something I seem to notice more as I grow older. So far I’m able to maintain my “cometptive fitness” on job market. If I rely on BB too much as I age, then I believe I would pay less attention to AA, and grow older and weaker. To strengthen the foundation of my family livelihood as I age, I tell myself to see the reality — as I age I would face a less friendly job market + instability on any job. Therefore I need to give priority to AA, by maintaining/improving my tech skills for competitive interviews.

Demand — for developers continue to grow in the job markets. This is a fundamental reason why AA is so valuable and reliable. This robust demand doesn’t help BB at all.

Overall, my view is biased in favor of AA. This is deliberate. With PIP or without PIP, any high-paying tech job (like yours or mine) comes with an expectation and risk of job loss. AA is the parachute.

CONST parameter of copy-ctor: j4

Background: in Apr 2020 during the covid19 lockdown, my friend Deepak get a question from MS — why copy ctor parameter is a reference. Interviewer didn’t ask for the CONST. On the CONST, here’s my letter to Deepak.

If you have a const Trade object named bigTrade, and you try to copy this Trade object, then you would need the CONST in

Trade(Trade CONST &)

I’m 99% confident that if you remove that CONST, then this copy ctor would not work with the const object bigTrade.

As to your doubts on move + temp object, your understanding of pre-c++11 language rule was probably incomplete. As a result, your understanding of c++11 is questionable.

int id=123;
Trade aClone( Trade(id) );

.. would create a temp object having id=123. This temporary can’t bind to the CONSTless copy-ctor parameter. Before or after c++11, such a copy-ctor is always a mistake. Even if there are other hacks to fix it, the only correct fix is adding the missing CONST to the copy-ctor. See temp object bind`preferences: rvr,lvr,, #SCB

git reset: 3 steps

In general, a commit procedure involves two transitions between three “trees”:

  • from local tree -> staging
  • from staging -> index

As a “treatment”, git-reset comes in three strengths:

  1. soft: reset the index tree only
  2. mixed: reset the staging tree too
  3. hard: reset local tree and all other trees

StampedLock.java

  • StampedLocks were introduced in java8 only for internal/low-level utilities in the development of thread-safe components.
  • main advantage over RWlock — optimisticRead, which can reduce writer starvation by readers
  • main drawback — non-reentrant and prone to self-deadlock

— optimistic reading using a StampedLock object
A thread can hold this object in optRead mode. This mode can be thought of as an extremely weak version of a readLock, that can be broken by a writer at any time.

Your thread would first tryOptimisticRead() to get a temporary stamp. Then validate(theStamp) would pass iFF still “free”.

After first successful validation, you should quickly read just a few data items, and validate() again to confirm availability.

Remember that, with zero notice a writeLock can be created from this object, invalidating your optRead lock.

OptRead lock is designed for quick small reads, like stealing…

This class offers too many API methods. I wonder which ones are most often used. I hope the javadoc example helps.

— stamps are long integers

https://dzone.com/articles/a-look-at-stampedlock explains the stamps superficially.

Most if not all grab attempts return a stamp that is needed for unlocking or lock level upgrade/downgrade.

— why did they design this lock as non-reentrant? Must be some performance reason.

This construct is definitely designed for experts not the casual programmer.

— relation to the (interview-wise) hotter AtomicStampedReference

I don’t see any relation.

competitors to x86 instruction set

Interviewers often focus on x86 instruction sets, but I could point out competing instruction sets.

  • IBM’s PowerPC and Oracle’s Sparc — were a pair of once-popular instruction sets in HighPerformanceComputing. Even if they are less popular now, the still have a presence.
  • AMD64 — instruction set (i.e. x86-64) was created at AMD, as an extension on x86
  • IA-64 — instruction set was created at HP then Intel, widely deployed in data centers.
  • ARM — If interview is not 100% focused on server-side, then I would point out ARM instruction set is dominant in handheld.

brank ≈ luxury car #white elephant

What’s in common between a luxury car and a brank like “Director”?

  • enviable, glamorous, glorifying
  • high maintenance, high visibility
  • you need some minimum personal capacity (“performance“) to maintain it over a long time
  • can wear out quickly, esp. when you age
  • Not a passive income generator,
  • perhaps not a reliable cash-cow asset,

My contractor job is like a mid-range reliable Japanese car.

job hopper^stagnant mediocre#CNA

https://www.channelnewsasia.com/news/commentary/career-mobility-new-normal-career-stability-job-hopping-11469760 is a Singapore perspective:

Long gone is the notion of the career ladder, where the ideal CV looks like a narrow, vertical progression. Today’s gold-standard CV looks like a career matrix, with horizontal and vertical moves signifying depth and breadth of experience, skills and exposure to different cultures.

Employers have gone from being cynical about hiring job-hoppers to becoming accustomed to seeing diverse CVs from top talent who are in frequent demand.

most U.S.job seekers shun contracts

I often say that the young, bright candidates basically ignore contract jobs — no benefit, no leave, no promotion prospect.

https://www.channelnewsasia.com/news/commentary/covid-19-coronavirus-job-work-tip-skills-government-support-hire-12509712 is a Singapore perspective:

“many jobseekers turn away such contracts, choosing to hold out for something long-term that may or may not come”

 

 

mixing log4j logger.info + System.out.println

log4j.rootCategory=INFO, sysout
log4j.category.package1=INFO, logfile

Given this log4j.cfg content,

  • package1 classes’s logger output would go to log file specified
  • other packages’s logger output would go to sysout
  • System.out and System.err would go to sysout
  • In my dev system, sysout messages are saved to a separate sysout log file.

I have seen that System.out is less likely buffered.

car park concurrency design #volatile field#AshS

Design a car park class for concurrent access given

  • a single sensor A at the entry
  • a single sensor B at the exit

Without using locks, Provide enter() and leave() API methods that can be safely invoked on multiple threads.

— If at any time there’s only one thread calling enter (and leave), then volatile-based solution might work. I think this assumption is reasonable. Given there’s only one physical entry point, it’s not physically possible for two threads to run enter() simultaneously.

volatile int sensorA, sensorB;

int attemptToPark(){
count = this.senserA – this.sensorB;
if (count > CAPACITY) throw IllegalStateException;
if (count == CAPACITY) return 0;
this.senserA ++ ;
return count+1;
}
void leave(){
//concurrent access disallowed for this R/W operation
this.sensorB ++ ;
}

— If multiple threads are allowed to increment this.senserB, then we need (either a lock or) atomicInteger. ABA problem is not a concern here given the integer never decrements.

denigrate%%intellectual strength #ChengShi

I have a real self-esteem problem as I tend to belittle my theoretical and low-level technical strength. CHENG, Shi was the first to point out “你就是比别人强”.

  • eg: my grasp of middle-school physics was #1 strongest across my entire school (a top Beijing middle school) but I often told myself that math was more valuable and more important
  • eg: my core-java and c++ knowledge (QQ++) is stronger than most candidates (largely due to absorbency++) but i often say that project GTD is more relevant. Actually, to a technical expert, knowledge is more important than GTD.
  • eg: I gave my dad an illustration — medical professor vs GP. The Professor has more knowledge but GP is more productive at treating “common” cases. Who is a more trusted expert?
  • How about pure algo? I’m rated “A-” stronger than most, but pure algo has far lower practical value than low-level or theoretical knowledge. Well, this skill is highly sought-after by many world-leading employers.
    • Q: Do you dismiss pure algo expertise as worthless?
  • How about quant expertise? Most of the math has limited and questionable practical value, though the quants are smart individuals.

Nowadays I routinely trivialize my academic strength/trec relative to my sister’s professional success. To be fair, I should say my success was more admirable if measured against an objective standard.

Q: do you feel any IQ-measured intelligence is overvalued?

Q: do you feel anything intellectual (including everything theoretical) is overvalued?

Q: do you feel entire engineering education is too theoretical and overvalued? This system has evolved for a century in all successful nations.

The merit-based immigration process focus on expertise. Teaching positions require expertise. When laymen know you are a professional they expect you to have expertise. What kind of knowledge? Not GTD but published body of jargon and “bookish” knowledge based on verifiable facts.

count allocation strategies of $K into N funds #70%

!! how is this related to the CSY staircase problem?

 

Q: given K dollars and N distinct mutual funds (including a cash fund), how many unique allocation strategies are there? It’s permissible to allocate $0 to any fund, but total amount in a strategy must add up to exactly K dollars

Note K can be higher or lower than N, both positive integers. Allocation amount must be whole dollars.

Without loss of generality, let’s say K is $5 and N is 3.

====Analysis

I think this is more a mathematical than an algorithm question. Each strategy can be represented by an N-element array. [0,5,0] is one strategy, allocation $5 to 2nd fund and $0 to the other funds. If we are required to enumerate all the strategies, we can print out the arrays.

My solution starts by defining a mathematical function f(k, n) := count of allocation strategies given k dollars and n funds. Based on such a definition, we can see

  • f(any_integer, 1) == 1 because there’s only one fund to receive the full amount. One strategy only.
  • f(0, any_integer) == 1 because all funds must receive $0 each. No other strategy.

Without loss of generality, let’s say K is 33 and N is 21. Using bottom-up DP, I assert that when we get a “new” fund, the 22nd fund, we can use the earlier answers to find f(33,22):

f($33,22) = f($33,21)f($0,1) + f($32,21)f($1,1) + f($31,21)f($2,1) +… +f($1,21)f($32,1)+f($0,21)f($33,1)

Let me explain the 3rd term on the right-hand-side. This term means the count for “$31 into the 21 existing funds” multiplied by the count for “$2 into the new fund”. This describes the scenario of first allocating $2 to the new fund, and then allocating the remaining $31 to the existing funds. The above formula simplifies to

f($33,22) = f($33,21) + f($32,21) + f($31,21) +… + f($1,21) +1

This formula for f(33,22) uses earlier results for f(smaller or equal amounts, one fund fewer). A bottom-up DP algorithm is straightforward. Generalization of this example gives

f(K,n+1) = f(K,n) + f(K-1,n) + f(K-2,n) + f(K-3,n)+..+ f(2,n) + f(1,n) + f(0,n)

 

zbs=vague,cf to QQ+GTD

Many zbs learning experiences (me or DQH) were arguably irrelevant if never quizzed.

–iv questions on malloc, tmalloc and the facebook malloc
In latency benchmarks, c++ should simply 1) avoid DMA or 2) pre-allocate. Therefore, if the iv question touches on benchmark, then the malloc question become moot.

–iv questions on lockfree in c++
messier and more low-level than java lockfree. I would say lockfree is seldom written by a c++ app developer. Therefore not a GTD topic at all. Purely QQ halo topic.

–iv question on volatile
Again, many interviewers use volatile as concurrency feature in c++ but it’s non-standard ! Even if it has a side effect under gcc, the practice is “not something I would recommend”, though I won’t criticize hiring team’s current practice.

Not a GTD skill, not zbs either.

–iv question on throwing dtor
c++ standard says UDB but still some interviewers ask about it. So this topic is not zbs not GTD but really QQ

–iv question on debugger breakpoint
Not GTD, not even zbs

–iv question (GS-HK) on CAS no slower than uncontended lock acquisition
I feel I don’t need to know how much faster. My knowledge is good enough for GTD

–iv question on how many clock cycles per instruction
no GTD but zbs for a latency enthusiast

 

unspecified^undefined behavior

Some interviewers really really go into UDB, so to impress them it is very very useful to understand UnSpecifiedBehavior (USB):

Compliant compilers are not required to document the behavior but the behavior should be consistent and valid such as crash. For a given program containing a USB, where possible the Standard provides two or more possibilities (i.e. permissible outcomes), such that an executable from a compliant compiler should produce one of these outcomes.

Compared to USB, UDB gives compilers more freedom. The actual result could be anything including invalid results in one instance and in another instance turning a blind eye.

3gradual changes]SG job market #cautious optimism

  1. c++ (and c#) is /conceding/ market share to java, partly due to the two categories above. Apparently, java is growing more dominant than before. I guess java is more proven, better supported, by a bigger ecosystem and have bigger talent pool. In contrast, c++ skill is harder to find in Singapore?
    1. Overall good news for me since my java arm is still stronger than c++ arm
  2. remote hiring — more Singapore teams are willing to hire from overseas. Lazada said “mostly over skype”
  3. Many non-finance companies now can pay 150k base or higher for a senior dev role. In my 2015 job search, I didn’t find any
  4. Many smaller fintech companies (not hedge funds) can pay 150k base or higher
  5. contracts becoming slightly more common
  6. lighter-blue-collar — programmer used to be blue-collar support staff for the revenue staff. Some of the companies listed above treat programmers as first-class citizens.

Reality warning — every time I try the SG job market, recruiters would tell me there are so many “new” employers or new markets, but invariably, i need to focus on the old guards .. mostly ibanks, as the new market is not open to me. This is similar to Deepak, Shanyou trying the wall St c++ job market.

I must stop /romanticizing/ about the “improvement” in SG job market.

  • Singapore tech shops are mostly not keen about my profile. U.S.? Not sure.
  • Singapore fintech shops ? zero interest shown, even when I asked 150k
  • Singapore buy-sides are interested but way too selective and kinda slow.
  • Note except GS I didn’t try the ibank jobs this time round.

Basically no change in the landscape since 2015. The jobs available to me are mostly ibanks. Cherish the MLP job but beware attachment. If this job goes sour, I would have to consider WallSt, rather than another perm job in SG.

read/write volatile var=enter/exit sync block

As explained in 3rd effect@volatile introduced@java5

  • writing a volatile variable is like exiting a synchronized block, flushing all temporary writes to main memory;
  • reading a volatile variable is like entering a synchronized block, reloading all cached shared mutables from main memory.

http://tutorials.jenkov.com/java-concurrency/volatile.html has more details.

https://stackoverflow.com/questions/9169232/java-volatile-and-side-effects also addresses “other writes“.

java≠a natural choice 4 latency #DQH

I think java could deliver similar latency numbers to c/c++, but the essential techniques are probably unnatural to java:

  • STM — Really low latency systems should use single-threaded mode. STM is widely used and well proven. Concurrency is the biggest advantage of java but unfortunately not effective in low-latency.
  • DAM — (dynamically allocated memory) needs strict control, but DAM usage permeates mainstream java.
  • arrays — Latency engineering favors contiguous memory arrays, rather than object graphs including hash tables, lists, trees, or array of heap pointers,,. C pointers were designed based on tight integration with array, and subsequent languages have all moved away from arrays. Programming with raw arrays in java is unnatural.
    • struct — Data structures in C has a second dimension beside arrays – namely structs. Like arrays, structs are very compact, wasting no memory and can live on heap or non-heap. In java, this would translate to a class with only primitive fields. Such a class is unnatural in java.
  • GC — Low latency doesn’t like a garbage collector thread that can relocate objects. I don’t feel confident discussing this topic, but I feel GC is a handicap in the latency race. Suppressing GC is unnatural for a GC language like java.

My friend Qihao commented —

There are more management barriers than technical barriers towards low latency java. One common example is with “suppressing gc is unnatural”.

mgr role stress: ownership #heavy

I feel some people are good at (or trained for) juggling big responsibilities that affect other people as “They depend on my system as a service provider”. I’m not so good at it so I feel these responsibilities are very heavy, and a type of cancer-stress.

Someone like Josh, Larry … need to be on the ball and keep track of the large number of (big or small [1]) changes affecting “his baby”.

In contrast, as a foot soldier I only have obligation (professional responsibility) to keep an eye on my project or my module. Some foot soldiers are eager to learn, but usually her scope of responsibility is much smaller.

Stay on top of “everything” — as a parent I have to be on top of everything about my kids and my house.

LocalSys — If I don’t understand one of many things in and outside my team’s applications, I don’t need to struggle to find out and clarify my understanding.

[1] it’s not always straightforward to determine if a change is big or small. The app owner need to quickly estimate the impacts of each change. Sometimes the estimate is inaccurate. There’s a bit of risk.

scan array{both ends, keeping min/max #O(N)

I feel a reusable technique is

  • scan the int array from Left  and keep track of the minimum so far. Optionally save it in an array
  • scan the int array from right and keep track of the maximum so far. Optionally save it in an array
  • save the difference of these two arrays

With these three shadow arrays, many problems can be solved visually and intuitively, in linear  time.

eg: max proceeds selling 2 homes #Kyle

How about the classic max profit problem?

How about the water container problem?

sorting J integers, each in [1,N]

Q: What’s the time complexity of sorting J integer scores which are all [1, N], possibly non-unique?

This is classic counting sort. My book [[algo in a nutshell]] incorrectly says O(J). Incorrect when N is large.

My analysis below uses a total of three dimensions N, J and K, where N and J can be vastly different, and K <=min(N, J), but K could be much smaller.

====analysis

Textbook counting sort is listed at the bottom. If N is smaller than J, then O(N+J) is dominated by O(J). Now suppose N is huge (like the revenue figure of a company).

Suppose there are K distinct scores among the J scores. I would want a constant-time translation from each distinct score to a distinct counter. I wish there’s a perfect hashing from the K scores to K buckets but I think it’s impossible since the K distinct values are not known in advance. Even with imperfect hashing, I can get O(1) translation from each score value to a distinct counter. I will iterate over the J scores. For each,

  • look up (on hash table) the corresponding counter.
  • If the score is new i.e. missing from the hash table,
    • then create a counter
    • add the “pair” {score -> counter} into the hash table.
    • Also insert the score into a min-heap
  • increment the counter

O(K logK) : Now pop the min-heap K times, each time to get a distinct score in ascending order. Lookup the score in hash table to get the counter. If counter for score 55 the count is 3, then output 55 three times. This output would be a sorted sequence of the original J scores.

— comparing with alternatives: Mine is O(J + K logK). If K*logK exceeds N, then I would fall back to standard counting sort.

The comparison-based sort would be O(J logJ), inferior to mine if K is much smaller than J.

The textbook counting sort would be O(J + N) , using N independent counters.

range bump-up@intArray 60% #AshS

https://www.hackerrank.com/challenges/crush/problem Q: Starting with a 1-indexed array of zeros and a list of operations, for each operation add a “bump-up” value to each of the array element between two given indices, inclusive. Once all operations have been performed, return the maximum in your array.

For example, given array 10 of zeros . Your list of 3 operations is as follows:

    a b k
    1 5 3
    4 8 7
    6 9 1

Add the values of k between the indices a and b inclusive:

index->	 1 2 3  4  5 6 7 8 9 10
	[0,0,0, 0, 0,0,0,0,0, 0]
	[3,3,3, 3, 3,0,0,0,0, 0]
	[3,3,3,10,10,7,7,7,0, 0]
	[3,3,3,10,10,8,8,8,1, 0]

The largest value is 10 after all operations are performed.

====analysis

This (contrived) problem is similar to the skyline problem.

— Solution 1 O(minOf[N+J, J*logJ ] )

Suppose there are J=55 operations. Each operation is a bump-up by k, on a subarray. The subarray has left boundary = a, and right boundary = b.
Step 1: Sort the left and right boundaries. This step is O(N) by counting sort, or O(J logJ) by comparison sort. A conditional implementation can achieve O(minOf[N+J, J*logJ ] )

In the example, after sorting, we get 1 4 5 6 8 9.

Step 2: one pass through the sorted boundaries. This step is O(J).
Aha — the time complexity of this solution boils down to the complexity of sorting J small positive integers whose values are below n.

3overhead@creating a java stackframe]jvm #DQH

  • additional assembly instruction to prevent stack overflow… https://pangin.pro/posts/stack-overflow-handling mentions 3 “bang” instructions for each java method, except some small leaf methods
  • safepoint polling, just before popping the stackframe
  • (If the function call receives more than 6 arguments ) put first 6 args in register and the remaining args in stack. The ‘mov’ in stack involves more instructions than registers. The subsequent retrieval from stack is likely L1 cache, slower than register read.

age40-50career peak..really@@stereotype,brainwash,,

stereotype…

We all hear (and believe) that the 40-50 period is “supposed” to be the peak period in the life of a professional man. This expectation is created on the mass media (and social media such as Linkedin) brainwash that presents middle-aged managers as the norm. If not a “manager”, then a technical architect or a doctor.

[[Preparing for Adolescence]] illustrates the peer pressure (+self-esteem stress) felt by the adolescent. I feel a Deja-vu. The notion of “normal” and “acceptable” is skewed by the peer pressure.

Q: Out of 100 middle-aged (professional or otherwise) guys, how many actually reach the peak of their career in their 40’s?
A: Probably below 10%.

In my circle of 40-somethings, the norm is plateau or slow decline, not peak. The best we could do is keep up our effort and slow down the decline, be it wellness, burn rate, learning capacity, income,,,

It’s therefore hallucinatory to feel left behind on the slow track.

Q: at what age did I peak in my career?
A: I don’t want to overthink about this question. Perhaps towards the end of my first US era, in my late 30s.

I think middle-aged professional guys should read [[reconciliations]] by Theodore Rubin. The false expectation creates immense burden.

const data member initialization: simple on the surface

The well-known Rule 1 — a const data member must be initialized exactly once, no more no less.

The lesser-known Rule 2 — for class-type data member, there’s an implicit default-initialization feature that can kick in without us knowing. This default-init interacts with ctor initializer in a strange manner.

On a side note, [[safeC++]] P38 makes clever use of Rule 2 to provide primitive wrappers. If you use such a wrapper in place of a primitive field (non-const), then you eliminate the operational risk of “forgetting to initialize a non-const primitive field

The well-known Rule 3 — the proper way to explicitly initialize a const field is the ctor initializer, not inside ctor body.

The lesser-known Rule 4 — at run-time, once control passes into the ctor body, you can only modify/edit an already-initialized field. Illegal for a const field.

To understand these rules, I created an experiment in https://github.com/tiger40490/repo1/blob/cpp1/cpp/lang_misc/constFieldInit.cpp

— for primitive fields like int, Rule 2 doesn’t apply, so we must follow Rule 1 and Rule 3.

— for a class-type field like “Component”,

  • We can either leave the field “as is” and rely on the implicit Rule 2…., or
  • If we want to initialize explicitly, we must follow Rule 3. In this case, the default-init is suppressed by compiler.

In either case, there’s only one initialization per const field (Rule 1)

joinable instance of std::thread

[[effModernC++]] P 252 explains why in c++ joinable std::thread objects must not get destroyed. Such a destruction would trigger std::terminate(), therefore, programmers must make their std::thread objects non-joinable before destruction.

The key is a basic understanding of “joinable”. Informally, I would say a joinable std::thread has a real thread attached to it, even if that real thread has finished running. https://en.cppreference.com/w/cpp/thread/thread/joinable says “A thread that has finished executing code, but has not yet been joined is still considered an active thread of execution and is therefore joinable.”

An active std::thread object becomes unjoinable

  • after it is joined, or
  • after it is detached, or
  • after it is be “robbed” via move()

The primary mechanism to transition from joinable to unjoinable is via join().

std::thread key points

For a thread to actually become eligible, a Java thread needs start(), but c++ std::thread becomes eligible immediately after initialization i.e. after it is initialized with its target function.

For this reason, [[effModernC++]] dictates that between an int field and a std::thread field in a given class Runner, the std::thread field should be the last initialized in constructor. The int field needs to be already initialized if it is needed in the new thread.

Q1: Can you initialize the std::thread field in the constructor body?
A: yes unless the std::thread field is a declared const field

Now let’s say there’s no const field.

Q2: can the Runner copy ctor initialize the std::thread field in the ctor body, via move()?
A: yes provided the ctor parameter is non-const reference to Runner.
A: no if the parameter is a const reference to Runner. move(theConstRunner) would evaluate to a l-value reference, not a rvr. std::thread ctor and op= only accept rvr, because std::thread is move-only

See https://github.com/tiger40490/repo1/tree/cpp1/cpp/sys_thr for my experiments.

##[18] 4 qualities I admire ] peers #!!status

I ought to admire my peers’ [1] efforts and knowledge (not their STATUS) on :

  1. personal wellness
  2. parenting
  3. personal finance, not only investment and burn rate
  4. mellowness to cope with the multitude of demands, setbacks, disappointments, difficulties, realities about the self and the competition
  5. … to be Compared to
    • zbs, portable GTD, not localSys
    • how to navigate and cope with office politics and big-company idiosyncrasies.

Even though some of my peers are not the most /accomplished/ , they make a commendable effort. That attitude is admirable.

[1] Many people crossing my path are … not really my peers, esp. those managers in China. Critical thinking required.

I don’t have a more descriptive title..

2011 white paper@high-perf messaging

https://www.informatica.com/downloads/1568_high_perf_messaging_wp/Topics-in-High-Performance-Messaging.htm is a 2011 white paper by some experts. I have saved the html in my google drive. Here are some QQ  + zbs knowledge pearls. Each sentence in the article can expand to a blogpost .. thin->thick.

  • Exactly under what conditions would TCP provide low-latency
  • TCP’s primary concern is bandwidth sharing, to ensure “pain is felt equally by all TCP streams“. Consequently, a latency-sensitive TCP stream can’t have priority over other streams.
    • Therefore, one recommendation is to use a dedicated network having no congestion or controlled congestion. Over this network, the latency-sensitive system would not be victimized by the inherent speed control in TCP.
  • to see how many received packets are delayed (on the receiver end) due to OOS, use netstat -s
  • TCP guaranteed delivery is “better later than never”, but latency-sensitive systems prefer “better never than late”. I think UDP is the choice.
  • The white paper features an in-depth discussion of group rate. Eg: one mkt data sender feeding multiple (including some slow) receivers.

 

analyzing my perception of reality

Using words and numbers, am trying to “capture” my perceptions (intuitions + observations+ a bit of insights) of the c++/java job market trends, past and future. There’s some reality out there but each person including the expert observer has only a limited view of that reality, based on limited data.

Those numbers look impressive, but actually similar to the words — they are mostly personal perceptions dressed up as objective measurements.

If you don’t use words or numbers then you can’t capture any observation of the “reality”. Your impression of that reality [1] remains hopelessly vague. I now believe vague is the lowest level of comprehension, usually as bad as a biased comprehension. Using words + numbers we have a chance to improve our perception.

[1] (without words you can’t even refer to that reality)

My perceptions shape my decisions, and my decisions affect my family’s life chances.

My perceptions shape my selective listening. Gradually, actively, my selective listening would modify my “membrane” of selective listening! All great thinkers, writers update their membrane.

Am not analyzing reality. Instead, am basically analyzing my perception of the reality, but that’s the best I could do. I’m good at analyzing myself as an object.

Refusing to plan ahead because of high uncertainty is lazy, is pessimistic, is doomed.

latency zbs in java: lower value cf c++@@

Warning — latency measurement gotchas … is zbs but not GTD or QQ

— My tech bet — Demand for latency QQ will remain higher in c++ than java

  • The market’s perception would catch up with reality (assuming java is really no slower than c++), but the catch-up could take 30 years.
  • the players focused on latency are unused to the interference [1] by the language. C++ is more free-wheeling
  • Like assembly, c++ is closer to hardware.
  • In general, by design Java is not as a natural a choice for low latency as c++ is, so even if java can match c++ in performance, it requires too much tweaking.
  • related to latency is efficiency. java is a high-level language and less efficient at the low level.

[1] In the same vein, (unlikely UDP) TCP interferes with data transmission rate control, so even if I control both sender and receive, I still have to cede control to TCP, which is a kernel component.

— jvm performance tuning is mainstream and socially meaningful iFF we focus on
* machine saturation
* throughput
* typical user-experience response time

— In contrast, a narrow niche area is micro-latency as in HFT

After listening to FPGA, off-heap memory latency … I feel the arms race of latency is limited to high-speed trading only. latency technology has limited economic value compared to mobile, cloud, cryptocurrency, or even data science and machine learning.

Churn?

accu?

 

find all subsums divisible by K #Altonomy

Q: modified slightly from Leetcode 974: Given an array of signed integers, print all (contiguous, non-empty) subarrays having a sum divisible by K.

https://github.com/tiger40490/repo1/blob/py1/py/algo_arr/subarrayDivisibleByK.py is my one-pass, linear time solution. I consider this technique an algoQQ. Without prior knowledge, a O(N) solution is inconceivable.

I received this problem in an Altonomy hackerrank. I think Kyle gave me this problem too.

===analysis

Sliding window? I didn’t find any use.

Key idea — build data structure to remember cumulative sums, and remember all the positions hitting a given “cumsum level”. My homegrown solution kSub3() shows more insight !

enumerate()iterate py list/str with idx+val

The built-in enumerate() is a nice optional feature. If you don’t want to remember this simple simple syntax, then yes you can just iterate over xrange(len(the_sequence))

https://www.afternerd.com/blog/python-enumerate/#enumerate-list is illustrated with examples.

— to enumerate backward,

Since enumerate() returns a generator and generators can’t be reversed, you need to convert it to a list first.

for i, v in reversed(list(enumerate(vec)))

c++nlg pearls: xx new to refresh old 知新而后温故

Is this applicable in java? I think so, but my focus here is c++.

— 温故而知新 is less effective at my level. thick->thin, reflective.

— 知新而后温故 — x-ref, thin->thick->thin learning.

However, the pace of learning new knowledge pearls could appear very slow and disappointing. 5% new learning + 95% refresh. In such a case, the main benefit and goal is the refresh. Patience and Realistic expectation needed.

In some situations, the most effective learning is 1% new and 99% refresh. If you force yourself to 2% new and 98% refresh, learning would be less effective.

This technique is effective with distinct knowledge PEARLS. Each pearl can be based on a sentence in an article but developed into a blogpost.

 

non-volatile field can have volatile behavior #DQH

Unsafe.getObjectVolatile() and setObjectVolatile() should be the only access to the field.

I think for an integer or bool field (very important use cases), we need to use Unsafe.putIntVolatile() and Unsafe.getIntVolatile()

Q: why not use a volatile field?
A: I guess in some designs, a field need not be volatile at most access points, but at one access point it needs to behave like volatile field.  Qihao agrees that we want to control when we want to insert a load/store fence.

Non-volatile behavior usually has lower latency.

 

half%%peers could be forced into retirement

Reality — we are living longer and healthier.

Observation — compared to old men, old women tend to have more of a social life and more involvement with grandchildren.

I suspect that given a choice, half the white-collar guys in my age group actually wish to keep working past 65 (or 70), perhaps at a lower pace. In other words, They will decide to retire not by choice. My reasoning for the suspicion — Beside financial needs, many in this group do not have enough meaningful, “engaging” things to do. Many would suffer.

It takes long-term planning to stay employed past 65.

I think most of the guys in this category do not prepare well in advance and will find themselves unable to find a suitable job. (We won’t put it this way, but) They will be kinda forced into early retirement. The force could be health or in-demand skillset or …

[19] keen observer@SG workers in their 40s

“Most of them in the 40s are already stable and don’t want to quit. Even though the pay may not be so good, they’re willing to work all the way[1]. It’s an easy-going life.”

The observer was comparing SG (white or blue collar) employees across age groups, and this is the brief observation of the 40-something. This observation is becoming increasingly descriptive of me… Semi-retired on the back of my passive income streams.

[1] I interpret “all the way” as all the way to retirement age, no change of direction, not giving in to boredom, sticking to the chosen career despite occasional challenges (pains, disappointments, setbacks).

 

local variables captured in nested class #Dropcopy

If a (implicitly final) local variable [1] is captured inside a nested class, where is the variable saved?

https://stackoverflow.com/questions/43414316/where-is-stored-captured-variable-in-java explains that the anonymous or local class instance has an implicit field to hold the captured variable !

[1] local variable can be an arg passed into the enclosing function. Could a primitive type or a reference type i.e. heapy thingy

The java compiler secretly adds this hidden field. Without this field, a captured primitive would be lost and a captured heapy would be unreachable when the local variable goes out of scope.

A few hours later, when the nested class instance need to access this data, it would have to rely on the hidden field.

 

lambda^anon class instance ] java

A java lambda expression is used very much like an instance of an anonymous class. However, http://tutorials.jenkov.com/java/lambda-expressions.html#lambda-expressions-vs-anonymous-interface-implementations pointed out one interesting difference:

The anonymous instance in the example has a field named. A lambda expression cannot have such fields. A lambda expression is thus said to be stateless.

get collection sum after K halving operations #AshS

Q: given a collection of N positive integers, you perform K operations like “half the biggest element and replace it with its own ceiling”. Find the collection sum afterwards.

Note the collection size is always N. Note K(like 5) could exceed N(like 2), but I feel it would be trivial.

====analysis====

This is a somewhat contrived problem.

I think O(N + K log min(N,K)) is pretty good if feasible.

git | merge-commits and pull-requests

Key question — Q1: which commit would have multiple parents?

— scenario 1a:

  1. Suppose your feature branch brA has a commit hash1 at its tip; and master branch has tip at hashJJ, which is the parent of hash1
  2. Then you decide to simply q[ git merge brA ] into master

In this simple scenario, your merge is a fast-forward merge. The updated master would now show hash1 at the tip, whose only parent is hashJJ.

A1: No commit would have multiple parents. Simple result. This is the default behavior of git-merge.

Note this scenario is similar to https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/about-pull-request-merges#rebase-and-merge-your-pull-request-commits

However, github or bit-bucket pull-request flow don’t support it exactly.

— scenario 1b:

Instead of simple git-merge, what about pull request? A pull-request uses q[ git merge –no-ff brA ] which (I think) unconditionally creates a merge-commit hashMM on maser.

A1: now hashMM has two parents. In fact, git-log shows hashMM as a “Merge” with two parent commits.

Result is unnecessarily complex. Therefore, in such simple scenarios, it’s better to use git-merge rather than pull request.

https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/about-pull-request-merges explains the details.

— Scenario 2: What if ( master’s tip ) hashJJ is Not parent of hash1?

Now maser and brA have diverged. I think you can’t avoid a merge commit hashMM.

A1: hashMM

— Scenario 3: continue from Scenario 1b or Scenario2.

3. Then you commit on brA again , creating hash2.

Q: What’s the parent node of hash2?
A: I think git actually shows hash1 as the parent, not hashMM !

Q: is hashMM on brA at all?
A: I don’t think so but some graphical tools might show hashMM as a commit on brA.

I think now master branch shows  hashMM having two parents (hash1+hashMM), and brA shows hash1 -> hash2.

I guess that if after the 3-way-merge, you immediately re-create (or reset) brA from master, then hash2’s parent would be hashMM.


Note

  • direct-commit on master is implicitly fast-forward, but merge can be fast-forward or non-fast-forward.
  • fast-forward merge can be replaced by a rebase as in Scenario 1a. Result is same as direct-commit.
  • fast-forward merge-commit (Scenario 1b) and 3way merge (Scenario 2) both create a merge-commit.
  • git-pull includes a git-merge without –no-ff

Optiver coding hackathon is like marathon training

Hi Ashish,

Looking back at the coding tests we did together, I feel it’s comparable to a form of “marathon training” — I seldom run longer than 5km, but once a while I get a chance to push myself way beyond my limits and run far longer.

Extreme and intensive training builds up the body capacity.

On my own, it’s hard to find motivation to run so long or practice coding drill at home, because it requires a lot of self-discipline.

Nobody has unlimited self-discipline. In fact, those who run so much or takes on long-term coding drills all have something beside self-discipline. Self-discipline and brute force willpower is insufficient to overcome the inertia in every one of these individuals. Instead, the invisible force, the wind beneath their wings is some forms of intrinsic motivation. These individuals find joy in the hard drill.

( I think you are one of these individuals — I see you find joy in lengthy sessions of jogging and gym workout. )

Without enough motivation, we need “organized” practice sessions like real coding interviews or hackathons. This Optiver coding test could probably improve my skill level from 7.0 to 7.3, in one session. Therefore, these sessions are valuable.

[18] latency in a typical broker DMA box

(This topic is not GTD not zbs, but relevant to some QQ interviewers.)

https://www.youtube.com/watch?v=BD9cRbxWQx8 is a 2018 presentation.

  1. AA is when a client order hits a broker
  2. Between AA and BB is the entire broker DMA engine in a single process, which parses client order, maintains order state, consumers market data and creates/modifies the outgoing FIX msg
  3. BB is when the broker ships the FIX msg out to exchange.

Edge-to-edge latency from AA to BB, if implemented in a given language:

  • python ~ about 50 times longer than java
  • java – can aim for 10 micros if you are really really good. Dan recommends java as a “reasonable choice” iFF you can accept 10+ micros. Single-digit microsecond shops should “take a motorbike not a bicycle”.
  • c# – comparable to java
  • FPGA ~ about 1 micro
  • ASIC ~ 400 ns

— c/c++ can only aim for 10 micros … no better than java.

The stronghold of c++, the space between java and fpga, is shrinking … “constantly” according to Dan Shaya. I think “constantly” is like the growth of Everest.. perhaps by 2.5 inches a year

I feel c++ is still much easier, more flexible than FPGA.

I feel java programming style would become more unnatural than c++ programming in order to compete with c++ on latency.

— IPC latency

Shared memory beats TCP hands down. For an echo test involving two processes:

Using an Aeron-based messaging application, 50th percentile is 250 ns. I think NIC and possibly kernel (not java or c++) are responsible for this latency.

sponsored DMA

Context — a buy-side shop (say HRT) uses a DMA connection sponsored by a sell-side like MS (or Baml or Instinet) to access NYSE. MS provides a DMA platform like Speedway.

The HRT FIX gateway would implement the NYSE FIX spec. Speedway also has a FIX spec for HRT to implement. This spec should include minor customization on the NYSE spec.

I have seen the HPR spec. (HPR is like an engine running in Baml or GS or whatever.) HPR spec seems to talks about customization for NYSE, Nsdq etc …re Gary chat.

Therefore, the HRT FIX gateway to NYSE must implement, in a single codebase,

  1. NYSe spec
  2. Speedway spec
  3. HPR spec
  4. Instinet spec
  5. other sponsors’ spec

The FIX session would be provided (“sponsored”) by MS or Baml, or Instinet. I think the HRT FIX gateway would connect to some IP address belonging to the sponsor like MS. Speedway would forward the FIX messages to NYSE, after some risk checks.

VWAP=bmark^execAlgo

In the context of a broker algos (i.e. an execution algo offered by a broker), vwap is

  • A benchmark for a bulk order
  • An execution algo aimed at the benchmark. The optimization goal is to minimize slippage against this benchmark. See other blogposts about slippage.

The vwap benchmark is simple, but the vwap algo implementation is non-trivial, often a trade secret.

Avichal: too-many-distractions

Avichal is observant and sat next to me for months. Therefore I value his judgment. Avichal is the first to point out I was too distracted.

For now, I won’t go into details on his specific remarks. I will simply use this simple pointer to start a new “thread”…

— I think the biggest distraction at that time was my son.

I once (never mind when) told grandpa that I want to devote 70% of my energy to my job (and 20% to my son), but now whenever I wanted to settle down and deep dive into my work, I feel the need and responsibility to adjust my schedule and cater to my son. and try to entice him to study a little bit more.

My effort on my son is like Driving uphill with the hand-brake on.

As a result, I couldn’t have a sustained focus.

gradle: dependency-jar refresh, cache, Intellij integration..

$HOME/.gradle holds all the jars from all previous downloads.

[1] When you turn on debug, you can see the actual download : gradle build –debug.

[2] Note IDE java editor can use version 123 of a jar for syntax check, but the command line compilation can use version 124 of the jar. This is very common in all IDEs.

When I make a change to a gradle config,

  • Intellij prompts for gradle import. This seems to be unnecessary re-download of all jars — very slow.
  • Therefore, I ignore the import. I think as a result, Intellj java editor [2] would still use the previous jar version as the old gradle config is in effect. I live with this because my focus is on the compilation.
  • For compilation, I use the grade “build” action (probably similar to command line build). Very fast but why? Because only one dependency jar is refreshed [3]
  • Gary used debug build [1] to prove that this triggers a re-download of specific jars iFF you delete the jars from $HOME/.gradle/caches/modules-2/files-2.1

[3] For a given dependency jar, “refresh” means download a new version as specified in a modified gradle config.

— in console, run

gradle build #there should be a ./build.gradle file

Is java/c# interpreted@@ #JIT

category? same as JIT blogposts

Q: are java and c# interpreted? QQ topic — academic but quite popular in interviews.

https://stackoverflow.com/questions/8837329/is-c-sharp-partially-interpreted-or-really-compiled shows one explanation among many:

The term “Interpreter” referencing a runtime generally means existing code interprets some non-native code. There are two large paradigms — Parsing: reads the raw source code and takes logical actions; bytecode execution : first compiles the code to a non-native binary representation, which requires much fewer CPU cycles to interpret.

Java originally compiled to bytecode, then went through an interpreter; now, the JVM reads the bytecode and just-in-time compiles it to native code. CIL does the same: The CLR uses just-in-time compilation to native code.

C# compiles to CIL, which JIT compiles to native; by contrast, Perl immediately compiles a script to a bytecode, and then runs this bytecode through an interpreter.

bone health for dev-till-70 #CSY

Hi Shanyou,

I have a career plan to work as a developer till my 70’s. When I told you, you pointed out bone health, to my surprise.

You said that some older adults suffer a serious bone injury and become immobile. As a result, other body parts suffer, including weight, heart, lung, and many other organs. I now believe loss of mobility is a serious health risk.

These health risks directly affect my plan to work as a developer till my 70’s.

Lastly, loss of mobility also affects our quality of life. My mom told me about this risk 20 years ago. She has since become less vocal about this risk.

Fragile bones become more common when we grow older. In their 70’s, both my parents suffered fractures and went through surgeries.

See ## strengthen our bones, reduce bone injuries #CSY for suggestions.

available time^absorbency[def#4]:2 limiting factors

see also ## identify your superior-absorbency domains

Time is a quintessential /limiting factor/ — when I try to break through and reach the next level on some endeavor, I often hit a /ceiling/ not in terms of my capacity but in terms of my available time. This is a common experience shared by many, therefore easy to understand. In contrast, a more subtle experience is the limiting factor of “productive mood” [1].

[1] This phrase is vague and intangible, so sometimes I speak of “motivation” — not exactly same and still vague. Sometimes I speak of “absorbency” as a more specific proxy.

“Time” is used as per Martin Thompson.

  • Specific xp: Many times I took leaves to attend an IV. The time + absorbency is a precious combination that leads to breakthrough in insight and muscle-building. If I only provide time to myself, most of the time I don’t achieve much.
    • I also take leave specifically to provide generic “spare time” for myself but usually can’t achieve the expected ROTI.
  • Specific xp: yoga — the heightened absorbency is very rare, far worse than jogging. If I provide time to myself without the absorbency, I won’t do yoga.
  • the zone — (as described in my email) i often need a block of uninterrupted hours. Time is clearly a necessary but insufficient condition.
  • time for workout — I often tell my friends that lack of time sounds like an excuse given the mini-workout option. Well, free time still helps a lot, but motivation is more important in this case.
  • localSys — absorbency is more rare here than coding drill, which is more rare than c++QQ which is more rare than java QQ
  • face time with boy — math practice etc.. the calm, engaged mood on both sides is very rare and precious. I tend to lose my cool even when I make time for my son.
  • laptop working at train stations — like MRT stations or 33rd St … to capture the mood. Available time by itself is useless

exec algo: with-volume

— WITH VOLUME
Trade in proportion to actual market volume, at a specified trade rate.

The participation rate is fixed.

— Relative Step — with a rate following a step-up algo.

This algo dynamically adjusts aggressiveness(participation rate) based on the
relative performance of the stock versus an ETF. The strategy participates at a target percentage of overall
market volume, adjusting aggressiveness when the stock is
significantly underperforming (buy orders) or outperforming (sell orders) the reference security since today’s open.

An order example: “Buy 90,000 shares 6758.T with a limit price of ¥2500.
Work order with a 10% participation rate, scaling up to 30%
whenever the stock is underperforming the Nikkei 225 ETF (1321.OS)
by 75 basis points or more since the open.”

If we notice the reference ETF has a 2.8% return since open and our 6758.T has a 2.05% return, then the engine would assume 6758.T is significantly underperforming its peers (in its sector). The engine would then step up the participation to 30%, buying more aggressively, perhaps using bigger and faster slices.

What if the ETF has dropped 0.1% and 6758.T has dropped 0.85%? This would be unexpected since our order is a large order boosting the stock. Still, the other investors might be dumping this stock. The engine would still perceive the stock as underperforming its peers, and step up the buying speed.

Y alpha-geeks keep working hard #speculation

Based on my speculation, hypothesis, imagination and a tiny bit of observation.

Majority of the effective, efficient, productive tech professionals don’t work long hours because they already earn enough. Some of them can retire if they want to.

Some percentage of them quit a big company or a high position, sometimes to join a startup. One of the reasons — already earned enough. See my notes on envy^ffree

Most of them value work-life balance. Half of them put this value not on lips but on legs.

Many of them still choose to work hard because they love what they do, or want to achieve more, not because no-choice. See my notes on envy^ffree

by-level traversal: isBST #AshS

Q: given a sequence of payload values produced from a by-level traversal of a binary tree, could the tree be a BST?

Ashish gave me this question. We can assume the values are floats.

====analysis

(Not contrived, not practical)

— idea 1:

Say we have lined up the values found on level 11. We will split the line-up into sections, each split point being a Level-10 value.

Between any two adjacent values on level 10 (previous level), how many current-level values can legally fit in? I would say up to 2.

In other words, each section can have two nodes or fewer. I think this is a necessary (sufficient?) condition.

–idea 2: One-pass algo

Construct a BST as we consume the sequence.

Aha — there’s only one possible BST we can build.

 

(unordered)map erase: prefer by-key !! itr #AshS

Relevant in coding tests like speed-coding, take-home, onsite. Practical knowledge is power !

As shown in https://github.com/tiger40490/repo1/blob/cpp1/cpp/lang_misc/mapEraseByVal.cpp

.. by-key is cleaner, not complicated by the iterator invalidation complexities.

You can save all the “bad” keys, and later erase one by one, without the invalidation concerns. You can also print the keys.

if you were to save the “bad” iterators, then once you erase one iterator, are the other iterators affected? No, but I don’t want to remember.

STL iterator invalidation rules, succinctly has a succinct summary, but I don’t prefer to deal with these complexities when I have a cleaner alternative solution.

std::pair mvctor = field-by-field

std::pair has no pointer field so I thought it needs no meaningful mvctor, but actually std::pair mvctor is defaulted i.e. field-wise move i.e. each field is moved.

If a pair holds a vector and a string, then the vector would be move-constructed, and so does the string.

Q1: So what kind of simple class would have no meaningful mvctor?
%%A: I would say a class holding no pointer whatsoever. Note it can embed another class instance as a field.

Q2: so why is std::pair not one of them?
A: std::pair is a template, so the /concrete/ field type can be a dynamic container including std::string.

All dynamic containers use pointers internally.

fine print ] source code

Q1: “Here is 95% of the logic” — In what contexts is such a documentation considered complete? Answered at the end.

When we programmers read source code and document the “business logic” implemented thereby, we are sometimes tempted to write “My write-up captures the bulk of the business logic. I have omitted minor details, but they are edge cases. At this stage we don’t need to worry about them”. I then hope I have not glossed over important details. I hope the omitted details are just fine prints. I was proven wrong time and again.

Sound byte: source code is all details, nothing but details.

Sound byte: Everything in source code is important detail until proven otherwise. The “proof” takes endless effort, so in reality, Everything in source code is important detail.

The “business logic” we are trying to capture actually consists of not only features and functionalities, but functional fragments i.e. the details.

When we examine source code, a visible chunk of code with explicit function names, variable names, or explicit comments are hard to miss. Those are the “easy parts”, but what about those tiny functional fragments …

  • Perhaps a short condition buried in a complicated if/while conditional
  • Perhaps a seemingly useless catch block among many catches.
  • Perhaps a break/continue statement that seems serve no purpose
  • Perhaps an intentional switch-case fall-through
  • Perhaps a seemingly unnecessary sanity check? I tend to throw in lots of them
  • Perhaps some corner case error handling module that look completely redundant and forgettable, esp. compared to other error handlers.
  • Perhaps a variable initialization “soon” before an assignment
  • Perhaps a missing curly brace after a for-loop header

( How about the equal sign in “>=” … Well, that’s actually a highly visible fragment, because we programmers have trained vision to spot that “=” buried therein. )

Let me stress again. The visibility, code size … of a functional fragment is not indication of its relative importance. The low-visibility, physically small functional fragments can be equally important as a more visible functional fragment.

To the computer, all of these functional fragments are equally significant. Each could have impact on a production request or real user.

Out of 900 such “functional fragments”, which ones deal with purely theoretical scenarios that would never arise (eg extraneous data) .. we really don’t know without analyzing tons of production data. One minor functional fragment might get activated by a real production data item. So the code gets executed unexpectedly, usually with immediate effect, but sometimes invisibly, because its effect is concealed by subsequent code.

I would say there are no fine-prints in executable source code. Conversely, every part of executable source code is fine print, including the most visible if/else. Every executable code has a real impact, unless we use real production data to prove otherwise.

A1: good enough if you have analyzed enough production data to know that every omitted functional fragment is truly unimportant.

intellij=cleaner than eclipse !

intellij (the community version) is much cleaner than eclipse, and no less rich in features.

On a new job, My choice of java ide is based on
1) other developers in the team, as I need their support

2) online community support — as most questions are usually answered there
I think eclipse beats intellij

3) longevity — I hate to learn a java ide and lose the investment when it loses relevance.
I think eclipse beats intellij, due to open-source

)other factors include “clean”

The most popular tools are often vastly inferior for me. Other examples:
* my g++ install in strawberryPerl is better than all the windows g++ installs esp. msvs
* my git-bash + strawberryPerl is a better IDE than all the fancy GUI tools
* wordpress beats blogger.com hands down
* wordpad is a far simpler rich text editor than msword or browsers or mark-down editors

value-based type as mutex #Optional.java

Value-based type is a new concept in java. Optional.java is the only important example I know, so I will use it as illustration.

One of the main ideas about value types is the lack of object identity (or perhaps their identity is detectable only to the underlying implementation i.e. JVM not Java applications). In such a world, how could we tell whether variables aa and bb “really” are the same or different?

Q: why avoid locking on value-based objects?
%%A: locking is based on identity. See why avoid locking on boxed Integers
A: https://stackoverflow.com/questions/34049186/why-not-lock-on-a-value-based-class

Side note — compared to java, c++ has a smaller community and collective brain power so discussions are more limited.

Y avoid us`boxed Integer as mutex

https://stackoverflow.com/questions/34049186/why-not-lock-on-a-value-based-class section on “UPDATE – 2019/05/18” has a great illustration

Auto-boxing of “33” usually produces distinct objects each time, but could also produce the same object repeatedly. Compiler has the leeway to optimize, just as in c++.

Remember: locking is based on object identity.

wage+cost levels: biased views@China colleagues

— salary

Many China colleagues (YH, CSY, CSDoctor, Jenny as examples) say their classmates earn much more than them in the U.S.  These colleagues seem to claim their Typical Chinese counterpart earns higher than in the U.S.

Reality — those classmates are outliers. The average salary in China is much lower than U.S. Just look at statistics.

Some of these guys (like CSY) feel inferior and regret coming to the U.S.

— cost of reasonable lifestyle

Many Chinese friends complain that cost level is higher in China than U.S. and Singapore. A young MLP colleague (Henry) said a RMB 500k/year feels insufficient to a Chinese 20-something.

In reality, per-sqft property price is indeed higher in some cities than in the U.S. Everything else in China, cost level is much lower than in the U.S. Just look at statistics.

success in long-term learning: keen!=interest

For both my son and my own tech learning over the long term, “interest” is not necessarily the best word to capture the key factor.

I was not really interested in math (primary-secondary) or physics (secondary). In college, I tried to feel interested in electronics, analog IC design etc but unsuccessful. At that level, extrinsic motivation was the only “interest” and the real motivation in me. Till today I don’t know if I have found a real passion.

Therefore, the strongest period of my life to look at is not college but before college. Going through my pre-U schools, my killer strength was not so much “interest” but more like keenness — sharp, quick and deep, absorbency…

Fast forward to 2019, I continue to reap rewards due to the keenness — in terms of QQ and zbs tech learning. Today I have stronger absorbency than my peers, even though my memory, quick-n-deep, sharpness .. are no longer outstanding.

Throughout my life, Looking at yoga, karaoke, drawing, sprinting, debating, piano, .. if I’m below-average and clueless, then I don’t think I can maintain “interest”.

Optional.java notes

Q: if an optional is empty, will it remain forever empty?

— An Optional.java variable could but should never be null, as this instance need a state to hold at least the boolean isPresent.

If a method is declared to return Optional<C>, then the author need to ensure she doesn’t return a null Optional ! This is not guaranteed by the language.

https://dzone.com/articles/considerations-when-returning-java-8s-optional-from-a-method illustrates a simple rule — use a local var retVal throughout then, at the very last moment, return Optional.ofNullable(retVal). This way, retVal can be null but the returned reference is never null.

If needed, an Optional variable should be initialized to Optional.empty() rather than null.

https://dzone.com/articles/considerations-when-returning-java-8s-optional-from-a-method shows the pitfalls of returning Optional from a method.

I feel this is like borrowing from a loan shark to avoid accidental credit card interest charge.

If you are careful, it can help you avoid the NPE of the traditional practice, but if you are undisciplined (like most of us), then this new stragey is even worse —

Traditional return type is something like String, but caller has to deal with nulls. New strategy is supposed to remove the risk/doubt of a null return value, but alas, caller still needs to check null first, before checking against empty!

–immutability is tricky

  1. the referent object is mutable
  2. the Optional reference can be reseated, i.e. not q[ final ]
  3. the Optional instance itself is immutable.
  4. Therefore, I think an Optional == a mutable ptr to a const wrapper object enclosing a regular ptr to a mutable java object.

Similarity to String.java — [B/C]

Compare to shared_ptr instance — [A] is true.

  • C) In contrast, a shared_ptr instance has mutable State, in terms of refcount etc
  • B) I say Not-applicable as I seldom use a pointer to a shared_ptr

— get() can throw exception if not present

— not serializable

— My motivation for learning Optional is 1) QQ 2) simplify my design in a common yet simple scenario

https://www.mkyong.com/java8/java-8-optional-in-depth/ is a demo, featuring … flatMap() !!

OO-modeling: c++too many choices

  • container of polymorphic Animals (having vtbl);
  • Nested containers; singletons;
  • class inheriting from multiple supertypes ..

In these and other OO-modeling decisions, there are many variations of “common practices” in c++ but in java/c# the best practice usually boils down to one or two choices.

No-choice is a Very Good Thing, as proven in practice. Fewer mistakes…

These dynamic languages rely on a single big hammer and make everything look like a nail….

This is another example of “too many variations” in c++.

dev jobs ] Citi SG+NY #Yifei

My friend Yifei spent 6+ years in ICG (i.e. the investment banking arm) of Citi Singapore.

  • Over 6Y no layoff. Stability is Yifei’s #1 remark
  • Some old timers stay for 10+ years and have no portable skill. This is common in many ibanks.
  • Commute? Mostly in Changi Biz Park, not in Asia Square
  • Low bonus, mostly below 1M
  • VP within 6 years is unheard-of for a fresh grad

I feel Citi is rather profitable and not extremely inefficient, just less efficient than other ibanks.

Overall, I have a warm feeling towards Citi and I wish it would survive and thrive. It offers good work-life balance, much better than GS, ML, LB etc

[17] deadlock involving 1 lock(again) #StampedLock

— Scenario 3: If you app uses just one lock, you can still get deadlock. Eg: ThreadM (master) starts ThreadS (slave). S acquires a lock. M calls the now-deprecated suspend() method on S, then tries to acquire the lock, which is held by the suspended S. S is waiting to be resumed by M — deadlock.

Now the 4 conditions of deadlock:
* wait-for cycle? indeed
* mutually exclusive resource? Just one resource
* incremental acquisition? indeed
* non-Preemptive? Indeed. But if there’s a third thread, it can restart ThreadS.

— Scenario 2: a single thread using a single non-reentrant lock (such as StampedLock.java and many c++ lock objects) can deadlock with itself. A common mis-design.

— Scenario 1:

In multithreaded apps, I feel single-lock deadlock is very common. Here’s another example — “S gets a lock, and wait for ConditionA. Only M can create conditionA, but M is waiting for the lock.”

Here, ConditionA can be many things, including a Boolean flag, a JMS message, a file, a count to reach a threshold, a data condition in DB, a UI event where UI depends partly on ThreadM.

P90 [[eff c#]] also shows a single-lock deadlock. Here’s my recollection — ThrA acquires the lock and goes into a while-loop, periodically checking a flag to be set by ThrB. ThrB wants to set the flag but need the lock.

 

debugger stepping into library

I often need my debugger to step into library source code.

Easy in java:

c++ is harder. I need to find more details.

  • in EclipseCDT, STL source code is available to IDE ( probably because class templates are usually in the form of header files), and debugger is able to step through it, but not so well.

Overall, I feel debugger support is significantly better in VM-based languages than c++, even though debugger was invented before these new languages.

I guess the VM or the “interpreter” can serve as an “interceptor” between debugger and target application. The interceptor can receive debugger commands and suspend execution of the target application.

complacent guys]RTS #DeepakCM

Deepak told me that Rahul, Padma etc stayed in RTS for many years and became “complacent” and uninterested in tech topics outside their work. I think Deepak has sharp observations.

I notice many Indian colleagues (compared to East European, Chinese..) uninterested in zbs or QQ topics. I think many of them learn the minimum to pass tech interviews. CSY has this attitude on coding IV but the zbs attitude on socket knowledge

–> That’s a fundamental reason for my QQ strength on the WallSt body-building arena.

If you regularly benchmark yourself externally, often against younger guys, you are probably more aware of your standing, your aging, the pace of tech churn, … You live your days under more stress, both negative and positive stress.

I think these RTS guys may benchmark internally once a while, if ever. If the internal peers are not very strong, then you would get a false sense of strength.

The RTS team may not have any zbs benchmark, since GTD is (99.9%) the focus for the year-end appraisal.

These are some of the reasons Deepak felt 4Y is the max .. Deepak felt H1 guys are always on our toes and therefore more fit for survival.

fear@large codebase #web/script coders

One Conclusion — my c++ /mileage/ made me a slightly more confident, and slightly more competent programmer, having “been there; done that”, but see the big Question 1 below.

— Historical view

For half my career I avoided enterprise technologies like java/c++/c#/SQL/storedProc/Corba/sockets… and favored light-weight technologies like web apps and scripting languages. I suspect that many young programmers also feel the same way — no need to struggle with the older, harder technologies.

Until GS, I was scared of the technical jargon, complexities, low-level API’s debuggers/linkers/IDE, compiler errors and opaque failures in java/SQL … (even more scared of C and Windows). Scared of the larger, more verbose codebases in these languages (cf the small php/perl/javascript programs)… so scared that I had no appetite to study these languages.

— many guys are unused to large codebases

Look around your office. Many developers have at most a single (rarely two) project involving a large codebase. Large like 50k to 100k lines of code excluding comments.

I feel the devops/RTB/DBA or BA/PM roles within dev teams don’t require the individual to take on those large codebases. Since it’s no fun, time-consuming and possibly impenetrable, few of them would take it on. In other words, most people who try would give up sooner or later. Searching in a large codebase is perhaps their first challenge. Even figuring out a variable’s actual type can be a challenge in a compiled language.

Compiling can be a challenge esp. with C/c++, given the more complex tool chain, as Stroustrup told me.

Tracing code flow is a common complexity across languages but worse in compiled languages.

In my experience, perl,php,py,javascript codebases are usually small like pets. When they grow to big creatures they are daunting and formidable just like compiled language projects. Some personal experiences —
* Qz? Not a python codebase at all
* pwm comm perl codebase? I would STILL say codebase would be bigger if using a compiled language

Many young male/female coders are not committed to large scale dev as a long-term career, so they probably don’t like this kinda tough, boring task.

— on a new level

  • Analogy — if you have not run marathons you would be afraid of it.
  • Analogy — if you have not coached a child on big exams you would be afraid of it.

I feel web (or batch) app developers often lack the “hardcore” experience described above. They operate at a higher level, cleaner and simpler. Note Java is cleaner than c++. In fact I feel weaker as java programmer compared to a c++ programmer.

Q1: I have successfully mastered a few sizable codebases in C++, java, c#. So how many more successful experiences do I need to feel competent?
A: ….?

Virtually every codebase feels too big at some time during the first 1-2 years, often when I am in a low mood, despite the fact that in my experience, I was competent with many of these large codebases.
I think Ashish, Andrew Yap etc were able to operate well with limited understanding.
I now see the whole experience as a grueling marathon. Tough for every runner, but I tend to start the race assuming I’m the weakest — impostor syndrome.
Everyone has to rely on log and primitive code browsing tools. Any special tools are usually marginal value. With java, live debugger is the most promising tool but still limited pain-relief. Virtually all of my fellow developers face exactly the same challenges so we all have to guess. I mean Yang, Piroz, Sundip, Shubin, … virtually all of them, even the original authors of the codebase. Even after spending 10Y with a codebase, we could face opaque issues. However, these peers are more confident against ambiguity.

[19] 4 Deeper mtv2work4more$ After basic ffree

Note sometimes I feel my current ffree is so basic it’s not real ffree at all. At other times I feel it is real, albeit basic, ffree. After achieving my basic ffree, here are 3 deeper motivations for working hard for even more money:

  • am still seeking a suitable job for Phase B. Something like a light-duty, semi-retirement job providing plenty of free time (mostly for self-learning, blogging, helping kids). This goal qualifies as a $-motivation because … with more financial resources, I can afford to take some desirable Phase-B jobs at lower pay. In fact, I did try this route in my 2019 SG job search.
  • I wish to spend more days with grandparents — need more unpaid leaves, or work in BJ home
  • more respect, from colleagues and from myself
  • stay relevant for 25 years. For next 10 years, I still want more upstream yet churn-resistant tech skills like c++.

–Below are some motivations not so “deep”

  • better home location (not size) — clean streets; shorter commute; reasonable school.. Eg Bayonne, JC
  • Still higher sense of security. Create more buffers in the form of more diversified passive incomes.

— Below are some secondary $-motivations

  • * more time with kids? Outside top 10 motivations.
  • * better (healthy) food? usually I can find cheaper alternatives
  • * workout classes? Usually not expensive

profilers for low-latency java

Most (simple) java profilers are based on jvm safepoint. At a safepoint, they can use JVM API to query the JVM. Safepoint-based profiling is relatively easy to implement.

s-sync (Martin) is not based on safepoint.

Async profiler is written for openJDK, but some features are usable on Zinc JVM. Async profiler is based on process level counters. Can’t really measure micro-latency with any precision.

Perf is an OS-level profiler, probably based on kernel counters.

“Strategic” needs a re-definition #fitness

“Strategic” i.e. long-term planning/t-budgeting needs a re-definition. quant and c# were two wake-up calls that I tragically missed.

For a long time, the No.1 strategic t-expense was quant, then c#/c++QQ, then codingDrill (the current yellowJersey).

Throughout 2019, I considered workout time as inferior to coding drill .. Over weekends or evenings I often feel nothing-done even though I push myself to do “a bit of” yoga, workout, or math-with-boy, exp tracking,

Now I feel yoga and other fitness t-spend is arguably more strategic than tech muscle building. I say this even though fitness improvement may not last.

Fitness has arguably the biggest impact on brain health and career longevity

git | reword historical commit msg

Warning — may not work if there’s a recent merge-in on your branch

Find the target commit and its immediate parent commit.

git rebase -i the.parent.commit

First commit in the list would be your target commit. Use ‘r’ for the target commit and don’t change other commits. You will land in vim to edit the original bad commit msg. Once you save and quit vim, the rebase will complete, usually without error.

Now you can reword subsequent commit messages.

c++low-^high-end job market prospect

As of 2019, c++ low-end jobs are becoming scarce but high-end jobs continue to show robust demand. I think you can see those jobs across many web2.0 companies.

Therefore, it appears that only high-end developers are needed. The way they select a candidates is … QQ. I have just accumulated the minimum critical mass for self-sustained renewal.

In contrast, I continue to hold my position in high-end coreJava QQ interviews.

conclusions: mvea xp

Is there an oth risk? Comparable to MSFM, my perception of the whole experience shapes my outlook and future decision.

  • Not much positive feedback beside ‘providing new, different viewpoints’, but Josh doesn’t give positive feedback anyway
  • should be able to come back to MS unless very stringent requirement
  • Josh might remember Victor as more suitable for greenfield projects.
  • I think Josh likes me as a person and understands my priorities. I did give him 4W notice and he appreciated.
  • I didn’t get the so-called “big picture” that Josh probably valued. Therefore I was unable to “support the floor” when team is out. The last time I achieved that was in Macq.
  • work ethic — A few times I worked hard and made personal sacrifices. Josh noticed.
  • In the final month, I saw myself as fairly efficient to wrap up my final projects including the “serialization” listed below

Q: I was brought in as a seasoned c++ old hand. Did I live up to that image? Note I never promised to be an expert
A: I think my language knowledge (zbs, beyond QQ) was sound
A: my tool chain GTD knowledge was as limited as other old hands.

Q: was the mvea c++ codebase too big for me?
A: No, given my projects are always localized. and there were a few old hands to help me out.

I had a few proud deliveries where I had some impetus to capture the momentum (camp out). Listed below. I think colleagues were impressed to some extent even though other people probably achieved more. Well, I don’t need to compare with those and feel belittled.

This analysis revealed that Josh is not easily impressed.  Perhaps he has high standard as he never praised anyone openly.

  • * I identified two stateless calc engines in pspc. Seeing the elegant simplicity in the design, I quickly zoomed in, stepped over and documented the internal logic and replicated it in spreadsheet.
  • * my pspc avg price sheet successfully replicated a prod “issue”, shedding light into a hitherto murky part of the codebase
  • * I quickly figure out serialization as root cause of the outage, probably within a day.
  • * I had two brave attempts to introduce my QOT innovation
  • * My 5.1 Brazil pspc project was the biggest config project to date. I single-handedly overcame many compilation (gm-install) and startup errors. In particular, I didn’t leave the project half-cooked, even though I had the right to do so.
  • I make small contributions to python test set-up

##With time2kill..Come2 jobjob blog

  • for coding drill : go over
    • [o] t_algoClassicProb
    • [o] t_commonCodingQ22
    • t_algoQQ11
    • open questions
  • go over and possibly de-list
    1. [o] zoo category — need to clear them sooner or later
    2. [o] t_oq tags
    3. t_nonSticky tags
    4. [o] t_fuxi tags
    5. Draft blogposts
    6. [o] *tmp categories? Sometimes low-value
    7. remove obsolete tags and categories
  • Hygiene scan for blogposts with too many categories/tags to speed up future searches? Low value?
  • [o=good for open house]

c++screening Questions for GregR

Similar to https://bintanvictor.wordpress.com/2017/09/15/more-mcq-questions-on-core-java-skills/, hopefully these questions can be given over phone.

Other topics easy to quiz over phone: smart pointers, std::thread, rvr/move, big4

  • — Q3: Suppose you have a simple class “Account” with only simple data fields like integers and strings, we can get an Account object constructed on 1) heap or in 2) data segment. Where else can it be constructed? For clarity, Data segment holds things like global variables.  [on stack]
  • Q3b: Suppose Account class has a default constructor Only. In which cases above is this constructor used to instantiate the Account object? [all three cases]
  • Q3c: in which one of the three cases must we use pointer to access the constructed object? [the Heap case ]
  • Q3d: in which one of the three cases above do we have a risk of memory leak? [the Heap case]
  • Q3e: in which of the three cases can the Account object construction happen before main() function starts? Hint: dynamic vs static allocation [ data segment case ]
  • Q3e2 (advanced): for a static Account object allocated in data segment, is the construction always before main()? [Not always. Consider local static variables]
  • Q3f (advanced): RAII relies on which of the 3 types of allocation? Hint: RAII invokes the destructor automatically and reliably. [stack allocation]
  • Q3g: can you have an array of Account objects constructed on heap? [yes]
  • Q3h: YES we can get such an array of Account objects, using array-new, but do we get a pointer or an array or do we get the first array element in an Account variable? [pointer]
  • Q3i: what happens when one of these Account objects is destructed? [destructor would run]
  • Q3j: OK let’s assume Account destructor runs. In which cases above is the destructor Guaranteed to run? Note we have four destruction scenarios so far — on stack, in data-segment, on heap and after array-new. [all cases, including static objects]
  • Q3k: what if we don’t define any destructor in Account class, in which cases above is destructor skipped? [Never skipped in any  case]
  • Q3L: In the array-new case, suppose 100 Account objects are constructed on heap, when we destruct the array via array-delete, how many times would the Account destructor run? [ 100 times]
  • — Q4: I have a variable var1 whose type is integer pointer, and I pass var1 directly into a function, what types of functions below can accept this argument?
    • A: funcA takes a parameter of non-constant reference to integer
    • B: funcB takes a parameter of constant reference to integer
    • * C: funcC takes a parameter of pointer to integer
    • D: funcD takes a parameter of bare integer (not pointer or reference)
  • Q4b: I want to call funcX(100+200+300), what types of function can accept this argument? [B/D]
  • Q4c: I have a variable var2 as a integer variable, and I want to pass it in like funcX(var2), what types of functions below can accept this argument? [A/B/D]
  • — Q5: which casts below are platform-specific? [correct answer is marked with *]
    • A: static_cast
    • B: down_cast
    • C: const_cast
    • D: dynamic_cast
    • * E: reinterpret_cast
  • Q5b: which casts are Not part of the c++ standard?
  • Q5c: which casts are closest to the old-school cast in C language? [A]
  • Q5d: which casts are likely to require the vtable? [D]
  • Q5e: which casts usually work with pointers and references Only? [D/E]
  • Q5f (advanced): when applied on pointers, which casts could produce an address different from the input pointer? [D]
  • — Q6: which standard data types below usually require heap allocation [see asterisks]
    • A: integer
    • * B: the string in standard library
    • C: plain vanilla array of 20 timestamps
    • * D: the list in standard library
    • * E: the vector in standard library
    • * F: the map in standard library
    • * G: unordered_map
    • H: C-style string
  • Q6b: which data types are available in C? [A C H]
  • Q6c: which data types by design won’t grow to accommodate more elements? [C H] A is a trivial case.
  • Q6d (advanced): which implementation classes for these data types must include code to allocate an array on heap? [B E G]
  • Q6d2 (more advanced, if last question was correctly answered): which one among these three classes may skip array allocation in a common optimization? [B in the form of small-string optimization]
  • Q6e: which data types offer random-access iterators capable of jumping by an arbitrary offset? [B C E H]
  • Q6f: which data types offer amortized constant-time lookup? [BCEH and G]
  • Q6g (advanced): which data type(s) offer unconditional guaranteed performance for insertion/deletion at any position and in all scenarios ? [D F]
  • Q6h: (advanced): which data structures are allocated on heap but doesn’t require reallocation of existing elements? [list and map]

##command line c++toolchain: Never phased out

Consider C++ build chain + dev tools  on the command line. They never get phased out, never lost relevance, never became useless, at least till my 70s. New tools always, always keep the old features. In contrast, java and newer languages don’t need so many dev tools. Their tools are more likely to use GUI.
  • — Top 5 examples similar things (I don’t have good adjectives)
  • unix command line power tools
  • unix shell scripting for automation
  • C API: socket API, not the concepts
  • — secondary examples
  • C API: pthreads
  • C API: shared memory
  • concepts: TCP+UDP, http+cookies
Insight — unix/linux tradition is more stable and consistent. Windows tradition is more disruptive.

Note this post is more about churn (phased-out) and less about accumulation (growing depth)

finding1st is easier than optimizing

  • Problem Type: iterate all valid choices without duplicates — sounds harder than other types, but usually the search space is quite constrained and tractable
    • eg: regex
    • eg: multiple-word search in matrix
  • Problem Type: find best route/path/combo, possibly pruning large subtrees in the search space — often the hardest type
  • Problem Type: find fist — usually easier

In each case, there are recurring patterns.

automation scripts for short-term GTD

Background — automation scripts have higher values in some areas than others

  1. portable GTD
    • Automation scripts often use bash, git, SQL, gradle, python/perl… similar to other portable GTD skills like instrumentation know-how
  2. long-term local (non-portable) GTD
  3. short-term local (non-portable) GTD

However, for now let’s focus on short-term local GTD. Automation scripts are controversial in this respect. They take up lots of time but offer some measurable payoff

  • they could consolidate localSys knowledge .. thick -> thin
  • They can serve as “executable-documentation”, verified on every execution.
  • They reduce errors and codify operational best practices.
  • They speed up repeated tasks. This last benefit is often overrated. In rare contexts, a task is so repetitive that we get tired and have to slow down.

— strength — Automation scripts is my competitive strength, even though I’m not the strongest.

— respect – Automation scripts often earn respect and start a virtuous cycle

— engagement honeymoon — I often experience rare engagement

— Absorbency — sometimes I get absorbed, though the actual value-add is questionable .. we need to keep in mind the 3 levels of value-add listed above.

jGC heap: 2 unrelated advantages over malloc

Advantage 1: faster allocation, as explained in other blogposts

Advantage 2: programmer can "carelessly" create an "local" Object in any method1, pass (by reference) the object into other methods and happily forget about freeing the memory.

In this extremely common set-up, the reference itself is a stack variable in method1, but the heapy thingy is "owned" by the GC.

In contrast, c/c++ requires some "owner" to free the heap memory, otherwise memory would leak. There’s also the risk of double-free. Therefore, we absolutely need clearly documented ownership.

max-sum non-adjacent subSequence: signedIntArr#70%

Q: given a signed int array a[], find the best sub-sequence sum. Any Adjacent elements in a[] should not both show up. O(N) time and O(1) space.

====analysis

— O(1) idea 2:

Two simple variables needed: m_1 is the m[cur-1] and m_2 is m[cur-2]. So compare a[cur] + m_2 vs m_1.

https://github.com/tiger40490/repo1/blob/py1/py/algo_arr/nonAdjSeqSum.py is self-tested DP solution.

FB 2019 ibt Indeed onsite

  • coding rounds — not as hard as the 2011 FB interview — regex problem
    • Eric gave positive feedback to confirm my success but perhaps other candidates did even better
    • No miscommunication as happened in the VersionedDict.
    • 2nd Indeed interviewers failed me even though I “completed” it. Pregnant interviewer may follow suit.
  • data structure is fundamental to all the problems today.
  • SDI — was still my weakness but I think I did slightly better this time
  • Career round — played to my advantage as I have multiple real war stories. I didn’t prepare much. The stories just came out raw and hopefully authentic
  • the mini coding rounds — played to my advantage as I reacted fast, thanks to python, my leetcode practice …

So overall I feel getting much closer to passing. Now I feel One interview is all it takes to enter a new world

* higher salary than HFT
* possibly more green field
* possibly more time to research, not as pressed as in ibanks

sorting collection@chars|smallIntegers

Therefore, given a single (possibly long) string, sorting it should Never be O(N logN) !

Whenever an algo problem requires sorting a bunch of English letters, it’s always, always better to use counting sort. O(N) rather O(N logN)

Similarly, in more than one problems, we are given a bunch of integers bound within a limited range. For example, the array index values could be presented to us as a collection and we may want to sort them. Sorting such integers should always , always be counting sort.

GTD fear-graph: delays^opacity^benchmark..

  • — manifestations first, fundamental, hidden factors last
  • PIP, stigma, damagedGood. Note I’m not worried about cashflow
  • [f] project delays
  • [f] long hours
  • [f] long commute
  • large codebase in brown field. Am slower than others at reading code.
  • [f] opacity — is worse than complexity or codebase size
  • figure-out speed benchmark — including impostor’s syndrome
  • [f = faced by every team member]
The connections among these “nodes” look complex but may not be really complex.
PIP is usually the most acute, harmful item among them. I usually freak out, though in hind sight I always feel I tried my best and should not feel ashamed. Josh is one manager who refuses to use PIP, so under Josh I had no real fear.

dev career]U.S.: fish out of water in SG

Nowadays I feel in-demand on 1) Wall St, 2) with the web2.0 shops. I also feel welcome by 3) the U.S. startups. In Singapore, this feeling of in-demand was sadly missing. Even the bank hiring managers considered me a bit too old.

Singapore banks only has perm jobs for me, which feel unsuitable, unattractive and stressful.

In every Singapore bank I worked, I felt visibly old and left behind. Guys at my age were all managers… painful.

expertise: Y I trust lecture notes more than forums

Q: A lot of times we get technical information in forums, online lecture notes, research papers. why do I trust some more than others?

  1. printed books and magazines — receive the most scrutiny because once printed, mistakes are harder to correct. Due to this fundamental reason, there is usually more scrutiny.
  2. research papers — receive a lot of expert peer review and most stringent editorial scrutiny, esp. in a top journal and top conference. There’s also internal review within the authors’ organization, who typically cares about its reputation.
  3. college lecture notes — are more “serious” than forums,
    • mostly due to the consequence of misinforming students.
    • When deciding what to include in lecture notes, many professors are conservative and prudent when adding less established, less proven research findings. The professor may mention those findings verbally but more prudent about her posted lecture notes.
    • research students ask lots of curious questions and serve as a semi-professional scrutiny of the lecture notes
    • lecture notes are usually written by PhD holders
  4. Stackoverlow and Quora — have a quality control via voting by accredited members.
  5.  the average forum — Least reliable.

[11]concurrent^serial allocation in JVM^c++

–adapted from online article (Covalent)

Problem: Multithreaded apps create new objects at the same time. During object creation, memory is locked. On a multi CPU machine (threads run concurrently) there can be contention

Solution: Allow each thread to have a private piece of the EDEN space. Thread Local Allocation Buffer
-XX:+UseTLAB
-XX:TLABSize=
-XX:+ResizeTLAB

You can also Analyse TLAB usage -XX:+PrintTLAB

Low-latency c++ apps use a similar technique. http://www.facebook.com/notes/facebook-engineering/scalable-memory-allocation-using-jemalloc/480222803919 reveals insight of lock contention in malloc()

Q: When do U choose c++over python4personal coding #FB

A facebook interviewer asked me “I agree that python is great for coding interviews. When do you choose c++?” I said

  1. when I need to use template meta programming
  2. when I need a balanced tree

Now I think there are other reasons

  1. when I practice socket programming
  2. when I practice memory mgmt techniques — even java won’t give me the low-level controls
  3. when I parse binary data using low-level constructs like reinterpret_cast, endianness conversion
  4. when I practice pthreads — java is easier

I didn’t showcase c++ but I felt very confident and strong about my c++, which probably shined through in front of the last interviewer (from the architect team)

I think my c++/python combo was good combo for the FB onsite, even though only one interviewer noticed my c++ strength.

%%FB onsite coding/SDI questions

— Q: design type-ahead i.e. search suggestion. Scalability is key.

— Q: innerProduct2SparseArray (Table aa, Table bb). First you need to design the undefined “Table” class to represent a sparse array.
Then you need to write real code for innerProduct2SparseArray(), assuming the two Tables are already populated according to your definition.

— Q: add2DecimalsSameLength (string decimal1, string decimal2) to return a string version of the sum. Can’t use the python integer to hold the sum, as in java/c++ integers have limited size. You must use string instead.

Aha — carry can only be 0 or 1
I forget to add the last carry as an additional digit beyond the length of the input strings 😦

— Q: add2longBinary(string binaryNum1, string binaryNum2). I told interviewer that my add2BigDecimal solution should work, so we skipped this problem.

— Q: checkRisingFalling(listOfInts) to return 1 for rising, -1 for falling and 0 for neither. Rising means every new num is no lower than previous

— Q: checkDiameter(rootOfBinTree)

save iterators and reuse them without invalidation risk

For a static STL container, the iterator objects can be safely stored and reused.

The dreaded iterator invalidation is a risk only under structural changes.

Many coding interview questions allow (even require) me to save those iterators and store them in a vector, a hash table …

Afterwards, we retrieve an iterator value, and visit the next/previous of it.

SDI: 3 ways to expire cached items

server-push update ^ TTL ^ conditional-GET # write-through is not cache expiration

Few Online articles list these solutions explicitly. Some of these are simple concepts but fundamental to DB tuning and app tuning. https://docs.oracle.com/cd/E15357_01/coh.360/e15723/cache_rtwtwbra.htm#COHDG198 compares write-through ^ write-behind ^ refresh-ahead. I think refresh-ahead is similar to TTL.

B) cache-invalidation — some “events” would trigger an invalidation. Without invalidation, a cache item would live forever with a infinity TTL, like the list of China provinces.

After cache proxies get the invalidation message in a small payload (bandwidth-friendly), the proxies discard the outdated item, and can decide when to request an update. The request may be skipped completely if the item is no longer needed.

B2) cache-update by server push — IFF bandwidth is available, server can send not only a tiny invalidation message, but also the new cache content.

IFF combined with TTL, or with reliability added, then multicast can be used to deliver cache updates, as explained in my other blogposts.

T) TTL — more common. Each “cache item” embeds a time-to-live data field a.k.a expiry timestamp. Http cookie is the prime example.

In Coherence, it’s possible for the cache proxy to pre-emptively request an update on an expired item. This would reduce latency but requires a multi-threaded cache proxy.

G) conditional-GET in HTTP is a proven industrial strength solution described in my 2005 book [[computer networking]]. The cache proxy always sends a GET to the database but with a If-modified-since header. This reduces unnecessary database load and network load.

W) write-behind (asynchronous) or write-through — in some contexts, the cache proxy is not only handling Reads but also Writes. So the Read requests will read or add to cache, and Write requests will update both cache proxy and the master data store. Drawback — In distributed topology, updates from other sources are not visible to “me” the cache proxy, so I still rely one of the other 3 means.

TTL eager server-push conditional-GET
if frequent query, in-frequent updates efficient efficient frequent but tiny requests between DB and cache proxy
if latency important OK lowest latency slower lazy fetch, though efficient
if in-frequent query good waste DB/proxy/NW resources as “push” is unnecessary efficient on DB/proxy/NW
if frequent update unsuitable high load on DB/proxy/NW efficient conflation
if frequent update+query unsuitable can be wasteful perhaps most efficient

 

@outage: localSys know-how beats generic expertise

When a production system fails to work, do you contact

  • XX) the original developer or the current maintainer over last 3Y (or longer) with a no-name college degree + working knowledge of the programming language, or
  • YY) the recently hired (1Y history) expert in the programming language with PhD and published papers

Clearly we trust XX more. She knows the localSys and likely has seen something similar.

Exception — what if YY has a power tool like a remote debugger? I think YY may gain fresh insight that XX is lacking.

XX may be poor at explaining the system design. YY may be a great presenter without low-level and hands-on know-how.

If you discuss the language and underlying technologies with XX he may show very limited knowledge… Remember Andrew Yap and Viswa of RTS?

Q: Who would earn the respect of teammates, mgr and external teams?

XX may have a hard time getting a job elsewhere .. I have met many people like XX.

lockfree stack with ABA fix #java AtomicStampedRef

http://15418.courses.cs.cmu.edu/spring2013/article/46 explains ABA in the specific context of a … lockfree stack. To counter ABA problem, It uses CAS2 i.e. CAS on a group of two objects — the original object + a stamp.

Java CAS2? I believe AtomicStampedReference is designed specifically for it.

http://tutorials.jenkov.com/java-util-concurrent/atomicstampedreference.html#atomicstampedreference-and-the-a-b-a-problem explains the AtomicStampedReference solving the ABA problem but in the last section it doesn’t clearly explain the benefit of get().

Also, the retry need a loop.

====Actually, many ABA illustrations are simplistic. Consider this typically simplistic illustration of a CAS stack:

  1. Thread 0 begins a POP and sees “A” as the top, followed by “B”. Thread saves “A” and “B”, before committing anything.
  2. Thread 1 begins and completes a POP , returning “A”.
  3. Thread 1 begins and completes a Push of “D”.
  4. Thread 1 begins and completes a Push of the same “A”. So now the actual stack top has A above D above B, i.e. with D inserted just below the original top-dog “A”
  5. Thread 0 sees that “A” is “still” on top and returns “A”, setting the new top to “B”. Therefore, Node D is lost.

— With a vector-of-pointer implementation, Thread 0 needs to save integer position within the stack. At Step 5, it should then detect that the A sitting at stack top is now at a higher position than observed earlier, detecting a symptom of ABA.

The rest is simple. Thread 0 should then query the new item (D) below A. Lastly, the CAS would compare current stack top position with the saved position, before committing.

However, in reality I think a vector-of-ptr is non-trivial to implement if we have two shared mutable things to update via CAS: 1) the stackTopPosition integer variable and 2) the (null) ptr in the next slot of the vector.

However, if Thread 1 replaces B with D, keeping the stack size unchanged, then the ABA would be undetectable by position check alone.

— with a linked list implementation, I think we only have node addresses, so at Step 5, Thread 0 can’t tell that D has been inserted between A and B.

You may consider using stack size as a second check, but it would be similar complexity as CAS2 but less reliable.

array-based order book: phrasebook

For HFT + mkt data + .. this is a favorite interview topic, kind of similar to retransmission.

Perhaps look at … https://web.archive.org/web/20141222151051/https://dl.dropboxusercontent.com/u/3001534/engine.c has a brief design doc, referenced by https://quant.stackexchange.com/questions/3783/what-is-an-efficient-data-structure-to-model-order-book

  • TLB?
  • cache efficiency