java 2017 MS java threading IV#phone

See also https://bintanvictor.wordpress.com/2009/03/21/realtime-inter-vm-communication-in-front-desk-trading-sys/

These are in the QQ category i.e. skills required for QnA IV only.

Q1: 3 threads to print the numbers 1,2,3,4,5… in deterministic, serial order. Just like single-threaded.

Q1b: what if JVM A has T1, T2, and JVM B has T3? How do they coordinate?
%%A: in C++ shared memory is the fastest IPC solution for large data volume. For signaling, perhaps a semaphore or named pipe
%%A: I feel the mutex is probably an kernel object, accessible by multiple processes.

On Windows, mutex, wait handle, … are all accessible cross-process, but java (on Windows or unix) is designed differently and doen’t have these cross-process synchronization devices.

%%A: use a database table with one row one column. Database can notify a JVM.
AA: The java.nio.file package provides a file change notification API, called the Watch Service API. The registered JVM has a thread dedicated to watching.AA: in java, the JDK semaphore is NOT a wrapper of the operation system semaphore so not usable for IPC
A: java Semaphore? Probably not an IPC construct in java.

Q2: have you used any optimized Map implementations outside the JDK?

Q3: to benchmark your various threading solutions how do you remove the random effects of GC and JIT compilation?
%%A: allocate enough memory to avoid GC. Turn off JIT to compile every code path. Perhaps give the JVM some warm-up period to complete the JIT compilation before we start the benchmark.

5 thrusts/directions over next5Y in U.S.

1) buy 1st home as soon as financially feasible. Before that consider REITs.

2) shift more focus to academic parenting

3) start PhD if everything works out. Will pave the way to a research/teaching career till age 75

4) consider some Chinese-language-teaching business for wife

5) may need to change gear to a relaxed job, but maintain competitiveness on job market (I didn’t say “on the job”)

I feel my health and job market competitiveness (#5) are the foundation.

noSQL landscape is fragmented;SQL is standardized

Reality — Many people spend more time setting up SQL infrastructure than writing query. Set-up includes integration with a host application. They then use _simple_ queries including simple joins, as I saw in ALL my jobs except GS.

The advanced SQL programmers (like in GS) specialize in joins, stored procs, table and index designs. For the large tables in PWM, every query need to be checked. By default they will run forever. In terms of complex joins, a lot of business logic is implemented in those joins.

Good thing is, most of this skill is portable and long-lasting, based on a consistent and standard base technology.

Not the case with noSQL. I don’t have real experience, but I know there are a few distinct types such as distributed hashmap, document stores (mongo) and columnar. So if there’s a query language it won’t have the complexity of SQL joins. Without the complexity it can’t implement the same amount of business logic. So GS type of SQL expertise is not relevant here.

SQL is slow even if completely in-memory

Many people told me flat-file data store is always faster than RDBMS.

For writing, I guess one reason is — transactional. REBMS may have to populate the redo log etc.

For reading, I am less sure. I feel noSQL (including flat-file) simplify the query operation and avoid scans or full-table joins. So is that faster than a in-memory SQL query? Not sure.

Instead of competing with RDBMS on the traditional game, noSQL products change the game and still get the job done.

2017 orgro^ unconnecteDiversify: tech xx ROTI

Background – my learning capacity is not unlimited. In terms of QQ and ZZ (see post on tough topics with low leverage), many technical subjects require substantial amount of laser energy, not a few weeks of cram reading — remember FIX, tibrv… With limited resources, we have to economize and plan long term with a vision, not shooting in all directions.

Example – my time spent on XAML now looks not organic growth, so the effort is likely wasted.

Similarly, I always keep a distance from the new web stuff — javascript, mobile apps, cloud technologies…

However, on the other extreme, staying in my familiar zone of java/SQL/perl/Linux is not strategic. I feel stagnant and left behind by those who branch out (see https://bintanvictor.wordpress.com/2017/02/22/skill-deependiversifystack-up/). More seriously, I feel my GTD capabilities are possibly diminishing as I age, so I feel a need to find new “cheese station”.

My Initial learning curves were steeper and exciting — cpp, c#, SQL.

Since 2008, this has felt like a fundamental balancing act in my career.

Unlikely most of my peers, I like learning new things. My learning capacity is 7/10 or 8/10 but I don’t enjoy staying in one area too long.

How about data science? I feel it’s kind of organic based on my pricing knowledge and math training. Also it could become a research/teaching career.

I have a habit of “touch and go”. Perhaps more appropriately, “touch, deep dive and go”. I deep dived on 10 to 20 topic and decided to move on: * java threading
* perl
* SQL joins and tuning
* algorithms
* swing

Following such a habit I could spread out too thin.

construct graph from list of connections #BGC

Given an input file showing a list of {string, string} pairs, build a connection graph.

If you have a correct connection graph, then you can easily determine the connectedness (bool) of any 2 nodes. In a social-network, this bool flag indicates whether 2 individuals are completely unconnected or somehow connected.

—-analysis:
I see this as a social-network. Any pair represents an edge connecting 2 nodes.  At any time there are a number of disconnected islands. The next pair could 1) merge 2 islands or 2) add a node to an existing island or 3) create a new island 4) do nothing, if the 2 nodes are already in some existing island

  • Any known node appears exactly once in the entire graph, in exactly one of the islands.
  • All nodes are contained in a lookup table or hashmap  {node -> island}
  • Each island can be a implemented as a hashset of nodes.

So here’s a proposed algo to process a new pair {A, B}. Look for A and B in the  graph. 3 scenarios + a dummy scenario:

  • (Scenario 3) If both A an B are new comers, then they form a new island.
  • if both A and B are already in the graph,
    • (Scenario 4) if they are in the same island, then exit. Nothing to do
    • (Scenario 1) else we can merge the 2 islands
  • (Scenario 2) If A is in island 3 but B is new comer, then B joins island 3

The merge operation is expensive. The big lookup table needs update but here’s an alternative:

  • At merge time, the smaller island would have all the nodes moved to the bigger island. When the island is empty, it gets a pointer “this.redirect” to the bigger island.
  • lookup table needs no update, avoiding locking a global object.
  • At query time, we look up the table to get the original island, then we follow its pointer (defaults to null) until the island is non-empty.
  • endless loop? would only be a programming error.

stress@jobHunt^GTD

Both mental stress and physical stress. Let’s take a step back and compare the stress intensity during job hunt vs GTD stress on the job.

Many people say it’s too stressful and tiring to keep interviewing compared to a long-term job in a big company. Well, I blogged many times that I need to keep interviewing…. The stress is so far manageable.

On a regular job, the GTD stress levels range from 5 to 7 on a scale of 10 (Donald Trump on women;). Often rise to 8.

Became 9 or 10 whenever (as FTE) boss gave a negative feedback. I know from several past experiences. In contrast, contract projects felt much better.

(To be fair, I did improve after the negative feedback.)

During my job hunt including the challenging Singapore lap, my stress level felt like 4 to 7, but often positive stress, perhaps due to positive progress and positive energy.

Conclusion — I always felt more confident on the open market than recovering from setback on a job.

skill: deepen^diversify^stack up

Since 2010, I have carefully evaluated and executed 3 broad strategies:

  1. deepen – for zbs + IV
  2. diversify or branch-out. Breaking into new markets
  3. stack-up – cautiously
  • eg: Deepened my java/SQL/c++/py knowledge for IV and GTD. See post on QQ vs ZZ.
  • eg: diversified to c++, c#, quant, swing…
  • eg: diversify? west coast.
  • eg: diversify? data science
  • eg: diversify? research + teach
  • eg: stack-up to learn spring, hibernate, noSQL, GWT.

Stack-up — These skills are unlikely to unlock new markets. Lower leverage.

GTD stress/survival on the job? None of these help directly, but based on my observation GTD skill seldom advance my career as a contractor. It could create a bit of spare time, but it’s a challenge to make use of the spare time.

I feel German’s strategy goes beyond tech skills. He treats tech skills as a tool, used to implement his business ideas to create business value. I feel his target role is a mix of architect/BA but more like “product visionary”. It’s somewhat like stack-up.