denigrate%%intellectual strength #ChengShi

I have a real self-esteem problem as I tend to belittle my theoretical and low-level technical strength. CHENG, Shi was the first to point out “你就是比别人强”.

  • eg: my grasp of middle-school physics was #1 strongest across my entire school (a top Beijing middle school) but I often told myself that math was more valuable and more important
  • eg: my core-java and c++ knowledge (QQ++) is stronger than most candidates (largely due to absorbency++) but i often say that project GTD is more relevant. Actually, to a technical expert, knowledge is more important than GTD.
  • eg: I gave my dad an illustration — medical professor vs GP. The Professor has more knowledge but GP is more productive at treating “common” cases. Who is a more trusted expert?
  • How about pure algo? I’m rated “A-” stronger than most, but pure algo has far lower practical value than low-level or theoretical knowledge. Well, this skill is highly sought-after by many world-leading employers.
    • Q: Do you dismiss pure algo expertise as worthless?
  • How about quant expertise? Most of the math has limited and questionable practical value, though the quants are smart individuals.

Nowadays I routinely trivialize my academic strength/trec relative to my sister’s professional success. To be fair, I should say my success was more admirable if measured against an objective standard.

Q: do you feel any IQ-measured intelligence is overvalued?

Q: do you feel anything intellectual (including everything theoretical) is overvalued?

Q: do you feel entire engineering education is too theoretical and overvalued? This system has evolved for a century in all successful nations.

The merit-based immigration process focus on expertise. Teaching positions require expertise. When laymen know you are a professional they expect you to have expertise. What kind of knowledge? Not GTD but published body of jargon and “bookish” knowledge based on verifiable facts.

val@pattern_recognition imt reusable_technique

Reusable coding techniques include my classic generators, DP, augmented trees, array-element swap, bigO tricks like hash tables and medium-of-3 pivot partitioning.

  • One of the best examples of reusable technique — generate “paths/splits” without duplicates. Usually tough.

More useful than “reusable techniques” are pattern-recognition insight into the structure and constraints in a given coding problem. Without these insights, the problem is /impenetrable/intractable/.

Often we need a worst input to highlight the the hidden constraint. The worst input can sometimes quickly eliminate many wrong pathways through the forest and therefore help us see the right pathway.

However, in some context, a bigO trick could wipe out the competition, so much so that pattern recognition, however smart, doesn’t matter.

 

## Y revisit old algo Qn #Kyle

I find it hard to be interested in previously solved problems. I think many fellow developers (not only those in a hurry) simply skip such problems. However, we are not so “good at” these problems actually.

  1. Reason – there are often insights and recurring patterns in an old problem, that require slow and repeated digestion. Solving it quickly is usually not enough… 囫囵吞枣.
  2. reason — our solutions are often sub-optimal.
  3. reason — even if our solutions are optional, there are still very smart alternative solutions out there we don’t know.
  4. reason — we forget.
  5. reason — with a simple tweak, the problem can become a new problem. How well and how fast we can solve the modified problem depends on our insight into the original problem.

Therefore, I feel folks tend to trivialize some well-known problems and fail to learn enough from them.

  • Well-known doesn’t mean well-understood.
  • Well-known doesn’t mean simple.
  • Well-known doesn’t inflexible and restricted to its known form.

Revisiting old problems is boring to many programmers, but can often feels easier than tackling new problems. I don’t get tired easily when revisiting old problems. If you are like me, then it could be more beneficial than tackling new problems.

However, my visible progress is much slower in terms of #problems solved per week. I think Kyle also pointed out something related to this.

bigO CIV: clarify_requirement means..worst_data

———- Forwarded message ———
Date: Tue, 9 Jul 2019 at 23:10

Big-O means “worst” case complexity. We need to be doubly sure what “worst” case data look like. (Note it doesn’t mean ridiculously rare data like integer overflow.)

On high-end coding interviews, these “worst data set” frequently shed light on the structure of the original problem, and often point to the correct direction for an efficient solution. (Note a nice-looking solution is not efficient if it exhibits poor time complexity in the face of a worst yet realistic data set. Such a system may not scale well in practice, given its Achilles’ heel.)

If interviewer wants a solution optimized for bigO time complexity, you can waste precious time tackling the wrong problem. The wrong problem is the problem defined by your idea of “worst data set”, but the real worst data set is very different.

Sometimes, the worst data is _subtly_ different but it points to a different algorithm direction. It may point to an iterative solution rather than recursion, for example.

The original interview question may not be that hard iFF we identify the worst data set early on, but sadly, we run out of time.

For some tough problems, the first (sometimes the only) challenge is quickly identifying worst data set. Interviewer always gives you the simple data set, NEVER the worst data set. It’s your job to identify the worst data set… often the key insight into the problem.

It’s no exaggeration to say — identifying the worst data set early or too late can make-or-break your chance at this employer. You may kiss good-bye to this job opportunity exactly because you are too slow to identify the worst data set. I know what this means. Other candidates probably got shot by this arrow on the heel, without knowing what happened.

Time is ticking as soon as the problem is presented to you. If interviewer says time is not a factor… take your time.. we want quality and thought process … I just ignore it. Our speed is always part of assessment. Therefore, a very common failure scenario is —

.. tractable problem but candidate runs out of time after wasting time on preliminaries like using incorrect worst data set to reason about the (wrong) problem.

##[19]Y WallStContract=%%best Arena #Grandpa

Competition arenas … we all CHOOSE the arena to compete in. It’s a choice, either implicit choice or explicit choice. I would say better to be conscious about this choice.

Not much new content in this blogpost. I feel very convinced to stick with WallSt contract market. Here is a ranking of the reasons why I consider it a rational decision, though our decisions are shaped by our deeply personal experiences and inherently irrational.

Beware of attachment !

  1. low stress, low expectation — my #1 reason as of 2019
  2. low-caliber competitors, mostly due to the “offputting” below
  3. age friendly
  4. I get to practice interviews and keep a precious burning-pleasure for a month each year on average.
    1. In contrast, If I were an ibank VP I would have multiple obstacles on that front.
  5. — other reasons to prefer Wall St contract
  6. higher probability of greenfield projects
  7. leverage on domain knowledge
  8. I can easily explain my job hopping profile
  9. ?? a number of firms to hop around

Now the downside, off-putting factors. ## Y most young dev shun contracts — Many bright or young competitors are put off by these factors, reducing the competition.

CIV: subOptimal $$valuable solutions#CSY

My friend Shanyou consistently holds a position like “I don’t need to find those better solutions (forget the “optimal”). If my solution is brute force or non-optimal, at least it works, so it will be good enough for many interviews and good enough for my coding drill. Many candidates can’t find even a brute force solution in the given time, so if I strengthen my capabilities to this level it’s valuable.

Admirable !

That’s inner strength — 定力
That’s much-needed focus
That’s undervalued realism
That’s celebration of every progress
That’s a pat on our own back

The all-or-nothing, single-minded pursuit of optimal solution can often wipe out my motivation, joy, satisfaction, self-esteem.

It’s a common and fatal de-motivation to tell ourselves (like XR … ) “There’s no value in pursuing non-optimal solution.” Well there ARE a few:

  1. pride and achievement from completion of a tough challenge
  2. ECT, speed
  3. syntax idioms
  4. self-mastery, internal locus of control, similar to my yoga practice
  5. absorbency — demonstrate to myself; self confidence in absorbency
  6. joy, self-satisfaction — see exactly what I want{cod`drill beside immediate IV #Rahul
  7. … Other blogs provide other perspectives

!! Many published solutions are non-optimal in O()
!! Most solutions out there are non-optimal in terms of concise implementation, even if matching the best solutions in big-O.

Some interviewers only care about big-O and don’t care about too many loops; Other interviews want an elegant, simple solution; The most extreme interviews (Indeed, FB?) score candidates on syntax and code quality too. Which one do you target? Realistically, I would take Shanyou’s target.

Q: your real goals, motivations for coding drill? Absorbency? Burning pleasure? Satisfaction (as Rahul puts it)? Maintaining a hobby?
So defocus can help. The focus on “optimal” creates negativity.

— Many non-optimal solutions are worthwhile.

  • Some of them offer insight into the structure, constraints… but may not offer enough insight to solve the problem
  • Some of them are non-optimal but optimal solutions to modified problems.
  • Some of them are non-optimal but break down the original problem into smaller, familiar problems. This process builds the xRef I need for thick->thin. The breakdown technique can be reusable.
  • Some of them offer reusable techniques
  • Some of them are non-optimal but very short and clever
  • Some of them (my homemade) are unique and innovative … worthwhile achievements.
  • Some of them are good enough for “2nd-tier” coding interviews

 

demanding mental power: localSys imt QQ

I told grandpa that I have noticed signs that my mental power is declining, mostly in localSys and local codebase absorbency. However, localSys has always been a weakness in me. The decline, if any, is very gradual.

QQ learning is still in good progress. Experience, accumulation, deepening.. thick->thin.

Note A.Brooks’ article talks about creativity, innovation, analytical … My issue is mostly memory capacity and speed. Recall the Indeed interview.

##strongER trec taking up large codebases

I have grown from a sysAdmin to a dev
I have grown from web and scripting pro into a java pro then c++ pro !
Next, I hope to grow my competence with large codebase.

I feel large codebase is the #1 diffentiator separating the wheat from the chaff — “casual” vs hardcore coders.

With a large codebase, I tend to focus on the parts I don’t understand, regardless that’s 20% or 80% of the code I need to read. I can learn to live with that ambiguity. I guess Rahul was good at that.

In a few cases, within 3M I was able to “take up” a sizable brown field codebase and become somewhat productive. As I told Kyle, I don’t need bottom-up in-depth knowledge to be competent.

In terms of count of successes taking up sizable brown-field codebase, I am a seasoned contractor, so I would score more points than a typical VP or an average old timer in a given system.

  • eg: Guardian — don’t belittle the challenge and complexity
  • eg: mvea c++ codebase — My changes were localized, but the ets ecosystem codebase was huge
  • eg: mvea pspc apportionment/allocation logic + price matching logic
  • eg: mtg comm (small green field), integrated into a huge brown field codebase
  • eg: StirtRisk personalization — I made it work and manager was impressed
  • eg: StirtRisk bug fixes on GUI
  • eg: Macq build system — many GTD challenges
  • eg: Moodles at NIE
  • eg: Panwest website contract job at end of 2016
  • ~~~ the obvious success stories
  • eg: AICE — huge stored proc + perl
  • eg: Quest app owner
  • eg: error memos
  • eg: RTS

##G3 uncertainties10Y out: #Demand^otherCandd

Let’s be specific — on the competitive landscape over next 10Y, what specific factors are we trying to predict

  • G3: predicting my health, energy, motivation, absorbency, “retention rate” (MSFM…), coding drill visible-progress ..
    • I have 3x more insight on this factor than other factors
  • G5: predicting strength of competing candd (i.e. candidates), often younger
    • one note — my accumulated theoretical and low-level QQ strength will continue to be rare among the young competitors
  • G9: predicting market demand evolution, including churn. Coding IV is the prime example.
    • Note this evolution is not so fast. Read my analysis of MOM and RDBMS

Since 2017, I have tentatively started to shift more focus towards west coast. As a result, the Competition and Demand factors felt too fast-changing, which makes my painstaking analysis look naive and futile.

Critical thinking needed — Suppose I keep my focus on ibanks. In such a case, my analysis and prediction is actually relevant, even valuable. The Competition/Demand factors have not changed drastically.

I’m pretty good at this kind of competitive landscape analysis , but It’s truly tough to predict any of these 3 factors. However, as Warren Buffett (?) observed, those who decide to give up and submit to mindless investing is likely (not guaranteed) to lose out to those who try and analyze/predict.

 

## good-looking designs/ideas #white elephant

I think a true architect knows the difference. The best design is often not so good-looking and hopelessly outdated.

Not all of them are classified “white elephants”. Not all of them are “designs”.

  1. lock-free — worthwhile?
  2. multi-threading — not always significantly faster than multi-Processing. I find single-threaded mode fastest and cleanest
  3. sharedMem — not always significantly faster than sockets. I briefly discussed these two choices with my friend Deepak M, in the parser/rebus context. I feel it may not be faster.
    1. other fancy IPC techniques? I am most familiar with sockets …
  4. java generic module (beyond collections) — look impressive, can be hard to maintain but doesn’t buy us much
  5. converting java system to c++ — not always brings significant performance gains
  6. forward() and move() instead of cloning
  7. hash-table — not always faster than RBTree
  8.  noSQL — not always significantly faster than REBMS with lots of RAM.  I find rdbms much more reliable and well understood. The indices, temp tables, joins, column constraints, triggers, stored procs add lots of practical value that can dramatically simplify the main application. I understand the limitations of rdbms, but most of my data stores are not so big.
  9. RPC and web services? Probably necessary, but I still don’t know how reliable they are
  10. thick client? I still feel web UI is simplest

     

##[15] 20 low-level IV topics : c++imt Java

This is collected after i stopped active interviewing in Aug 2012.

Low-level domains are my strength, proven again in 2017-2018 interviews. Beside the obvious domains — threading,  data structures, c++/java/c#..

  • exception in c++/java
  • concurrent singleton
  • GC
  • big-O analysis in west coast interviews
  • linux/compiler/C++ interviews are even more low-level than java. Despite serious “weak-joints”[1], I generally excel at the lower-level
    • shared mem — pointer therein!
    • sockets .. like kernel bypass
    • unix signals
    • cpu cache – instruction and data
    • inline
    • rvr, the most important c++0x feature, is very low-level
    • big-4
    • pbclone^pbref^pbptr, slicing
    • undefined behavior – usually are low level
    • internals of deque, vector/hashtable reallocation…
    • smart pointer internals
    • reinterpet_cast vs move()
    • pass a functor to a thread
    • [1] See items of the same color
  • —-On the other hand, interviewers don’t bother to go low level on …
  • thread pool? somehow never low-level
  • Unix commands, python? never low level
  • (noSQL and) SQL? no
  • FIX? not so low-level
  • SOA, MOM
  • asynch designs
  • design patterns

long term planning can be demoralizing

My father often tells me I plan ahead too much…

Q: where will I be, what job will I have 5 years from now?

Such questions can be demoralizing and sometimes can dampen a precious spirit of optimism. I sometimes perform better by focusing on here and now.

I think the reality may be quite bland and uninspiring — same job, with declining income, not much “offensive” to mount …

lockfree usage in high speed trading system@@

Hi Deepak,

Further to our conversation, do high frequency shops use lock-free? I would guess that the most demanding high-speeding data processing system (such as the matching engine in an exchange) are single-threaded, rather than lock-free multithreaded.

I hope to hear a typical high-speed data processing system that has lots of shared mutable data. I have not found one.

· Order matching

· Airline booking

· Vote counting in real time?

If 99% of the data is not shared mutable, then single-threaded design is probably better.

· I feel one reason is complexity vs simplicity. Lock-free designs can be very tricky, according to experts.

· Another reason is performance. Lock-free is, in my view, definitely slower than single-threaded. The memory fence required on those atomic objects impeded compiler optimizations.

· Most important reason, in my view — it’s always possible to achieve the same functionality without using locks or lock-free designs. Single-threaded designs are possible, simpler and faster.

If we have 64 cores, just run 64 processes, each single-threaded. However, in reality these systems often do have some shared mutable data between two threads. There are various solutions I’m not qualified to compare and illustrate. These solutions could use lock-free or locks.

peers’priority^%%priority: G5 #stayVague

In a nut shell, Some of my peers’ priorities I have decidedly given up, and some of my priorities are fairly unique to me.

Actually I only spoke to a small number of (like 10) peers, mostly Chinese and Indian. There are big differences among their views. However, here’s a grossly oversimplified sample of “their” priorities.

  • theirs — top schools, top grades
  • theirs — move up (long-term career growth) in a good firm
    • build long-term relationships with big bosses
  • theirs — early retirement
  • theirs — bonus
  • theirs — green card
  • ———-
  • mine — diversify (instead of deepen or stack-up) and avoid stagnation
  • mine — stay technical and marketable, till 70
  • mine — multiple properties and passive income
  • mine — shorter commute
  • mine — smaller home price tag

 

in-depth^cursory study@advanced program`topics

I have seen both, and tried both.

AA) A minority of developers spend the time studying and understanding the important details of threading, generics, templates … before writing code.

BB) Majority of developers don’t have the time, vision or patience, and only learn the minimum to get the (sloppy) job done. They copy-paste sample code and rely on testing, but the test cases are limited to realistic UAT test cases and will not cover edge cases that have yet to occur. For concurrency designs, such tests are not good enough.

AA is often seen as unnecessary except:
* Interviewers often go in-depth
* In a product vendor rather than an in-house application team
* If requirement is unusual (eg: B2BTradingEngine), then BB won’t even pass basic developer self-tests. Sample code may not be available — see my post http://wp.me/p74oew-1oI. So developers must bite the bullet and take AA approach. This is my sweet spot. See
https://bintanvictor.wordpress.com/2016/10/31/strength-gtd-with-non-trivial-researchinnovation/ https://bintanvictor.wordpress.com/2016/03/30/theoretical-complexity-strength/
https://bintanvictor.wordpress.com/2016/03/03/tech-specializations-no-such-luxury-in-singapore/

On Wall St projects, I learned to abandon AA and adopt BB.

On Wall St interviews, I still adopt AA.

On West Coast, I think AA is more needed.

how likely are two threads sharing a CHM were to clash@@

I would also like to point out the java ConcurrentHashMap lets two threads (possibly a writer thread and a reader thread) access the shared data structure concurrently, probably without locking or CAS, when the two threads happen to access two distinct segments.

Note there can be a large number of segments, up to a few hundred at least, so the chance of two threads hitting distinct segments is very high (99.999% chance for a 333-segments map). Therefore, contrary to what Jun said, for reader/writer threads to access the same data structure,  we don’t always need locking in every read and write access.

concurrencyLevel – the estimated number of concurrently updating threads. The implementation may use this value as a sizing hint.

python shortcomings for large enterprise app

– performance, efficiency. Compare — C statements translate to brief machine codes.
-? scalability
– threading – GIL, no precise control
– industrial strength commercial support. Only c++/java/c# has big tech powerhouses. Linux is widely adopted by enterprises, but is an exception that proves the rule.
– ? commercial auxiliary software to support the ecosystem, including IDE, debuggers, jars, code generators, profilers, … I feel python has these.
-? most enterprise apps are now in java or c# as these are proven, in terms of performance, resource scalability, commercial support etc.

##what bad things could happen to a thread in a pool

Do you ever wonder why threads in a thread pool can go bad and become unusable? Weblogic has long realized that. Hard to diagnose.

However, I don’t believe “voodoo” exists in a threading system. A stuck thread must be stuck in a useless loop or waiting for cpu, IO or memory. It won’t be stuck for “no reason”. Pretend to be a spy looking at the program counter inside any thread — why not moving? There are only a limited (not UNlimited) number of reasons. Here are various bad things that can happen to an innocent thread.

– in java, any uncaught exception can fall on a thread (like a rotten apple:). These are extremely common so a thread in a pool should handle these /gracefully/.

– a thread may be seen doing useful work in normal conditions, but in an edge case get into an endless loop but doing no useful work — just wasting cpu
** Note endless loop is fine if the thread is doing useful work.

– divide by zero (perhaps terminating the thread)
– access wrong memory location and trigger a segfault. In C++ there are many pointer/memory-related errors.
– java thread can run out of memory too
– in java, a thread can be stopped due to the deprecated but supported stop() and suspend() methods. Such a thread might emerge in bad shape.
– starvation due to low thread priority.
– stack overflow, due to deep recursion (like a recursive equals() or hashCode()).

– thread could get stuck in a normal blocking operation like accept(), read(), write(), lock()/synchronized keyword, wait(), join(), long sleep. Many of these aren’t even interruptible.
** IO (disk or network) wait is notorious culprit. In this state, everything including those “kill -9” signals are ignored (see [[Unix Power Tools]]). Entire process may appear stuck in mud.
** my java thread book also says a thread blocked on a socket may not respond to interrupt.
– deadlock

[12]call`same method@unrelated types: template outshines java

Suppose pure abstract class Animal has a die() method, and so does Project, Product, Plant and Planet, but they don’t share a base class. How would you write a reusable function that invokes this method on a generic input object, whose type could be any of them?

Java can’t do this. In C++ you create

template<typename T>  f1(T input){ input.die(); }

If you pass an int into f1(), then you get compile time error. Probably a linker error. Is this SFINAE ? I doubt it.

STL algorithms routinely take an iterator argument and then call operator>() on the iterator. Now, this operator is undefined for a lot of iterator INSTANCES. I think only RandomAccessIterator category supports it.

Q: So how does STL make sure you don’t pass an object of ForwardInterator category into such a function?
A: use the template parameter type name (dummy type name) as a hint. Instead of the customary “T”, they put a suggestive dummy type name like “BidirectionayInterator” or “InputIterator”. If you ignore the hint you get compile-time error.

Now we understand that STL iterators have no inheritance hierarchy, but “stratification” among them.

upstream, knowledge-intensive, high-margin sectors for S’pore

Upstream/strategic, knowledge-intensive, high-margin, high-barrier sectors for Singapore

Singapore is traditionally not a high-tech economy…. To compete, Singapore must seek greener pastures and keep a steady foothold therein. This is what every advanced economy does.

Role models? Switzerland, Hongkong, Taiwan, Silicon Valley, Boston

Entry barriers due to large capital, specialized know-how, talent shortage… Singapore EDB likes a sector with such barriers.

RnD – upstream, knowledge-intensive and entry barrier.

— Now the sectors —
$ finance — #1 most special (“sacred”) sector. One of the pillars of the economy but finance is first among equals [1]. Unlike other pillars, finance is part of the life support of this nation. Singapore gov needs long term strategies to protect (its currency and) financial health, even survival, of the nation. It grows a deep foreign reserve and invests it long term.

Not sure if finance is the most high-margin sector. Commercial banking is lower margin; AssetMgmt, security trading and investment banking create few jobs.

[1] if you shortlist the pillar industries of { oil, logistics}, where Singapore competes successfully on the global arena.

$ oil — perhaps the #2 sacred sector.
$ medical
($ life science? not sure what this includes)
$ electronics
$$ semiconductor fabrication – at the higher end of electronic sector
$$ IC design, esp. microprocessor design – still higher margin

$ telecom equipment and telecom operator — 2 big sectors by revenue
$ enterprise IT solutions, including software production and distribution
$ consumer IT product creation. Not a big sector, but look at Creative Lab
$ aviation — at the higher end of the logistic sector. Servicing, component design, research, airport ..
$ higher education and training

————
Now a small sample of the opposite list. Many traditional sectors don’t meet all the criteria in the title but do support a lot of jobs for Singaporeans —
– logistics?? Singapore’s traditional bread-and–butter economic contributor (commerce is another), but margin is deteriorating.
— distribution, warehousing
– commerce
— traditional retail – i.e. consumer shopping
– construction
– tourism and hospitality, casino
– entertainment – online/electronic gaming, music and film
– marketing and advertising, media, publishing, exhibition, conferencing

Many of these above sectors are domestic. They don’t directly contribute to the Singapore Team on the international front — contrast electronics, aviation, medical, tourism… Every advanced economy must show competitiveness and high value-add on a number of major global markets.