accumulation: contractor vs FTE #XR

XR,

You said that we contractors don’t accumulate (积累) as FTE do.

I do agree that after initial 2Y of tough learning, some FTE could reap the monetary rewards whereas consultants are often obliged, due to contract, to leave the team. (Although there are long-term contracts, they don’t always work out as promised.)

Here’s my experience in GS for 2.5Y. My later months had much lower “bandwidth” tension i.e. the later months required less learning and figure-things-out. Less stress, fewer negative feedbacks, less worry about my own competence, more confidence , more in-control because more familiar with the local system. If my compensation had become 150k I would say that money amounts to “reaping the reward”. In reality, the monetary accumulation was an empty promise.

As a developer stays longer, the accumulation in terms of his value-add to the team is natural and likely [1]. Managers like to point out that after a FTE stays in the team for 2Y her competence, her design, her solutions, her suggestions, her value-add per year grows higher every year. If her initial value-add to the company can be quantified as $100k, every year it grows by 30%. Alas, that doesn’t always translate to compensation.

That’s accumulation in personal income. How about accumulation in tech skill? Staying in one system usually means less exposure to other, or newer, technologies. Some developers prefer to be shielded from newer technologies. I embrace them. I feel my technical accumulation is higher when I keep moving from company to company.

[1] There are exceptions. About 5% of the old timers are, in my view, organization /dead-weights/. Their value-add doesn’t grow and is routinely surpassed within a year by bright new joiners. Often company can’t let them go due to political, legal or ethical reasons.

You said IV questions change over time so much (ignoring the superficial changes) that the IV skills we acquire today is useless in 5Y and we have to again learn new IV skills. This is not intuitive to me. Please give one typical example if you can without a lot of explaining (I understand your time constraints). I guess you mean technology churn? If I prepare for a noSQL or big data interview, then I will probably face technology churn.

On the other hand, in my experience, many interview topics remain ever-green including some hard topics — algorithms (classic algos and creative algos), classic data structures, concurrency, java OO, pass-by reference/value, SQL, unix commands, TCP/UDP sockets, garbage collection, asynchronous/synchronous concepts, pub/sub, producer/consumer, thread pool concepts… In the same vein, most coding tests are similar to 10 yeas ago when I first received them. So the study of these topics do accumulate to some extent.

edit1file]big python^c++ prod system

Q1: suppose you work in a big, complex system with 1000 source files, all in python, and you know a change to a single file will only affect one module, not a core module. You have tested it + ran a 60-minute automated unit test suit. You didn’t run a prolonged integration test that’s part of the department-level full release. Would you and approving managers have the confidence to release this single python file?
A: yes

Q2: change “python” to c++ (or java or c#). You already followed the routine to build your change into a dynamic library, tested it thoroughly and ran unit test suite but not full integration test. Do you feel safe to release this library?
A: no.

Assumption: the automated tests were reasonably well written. I never worked in a team with a measured test coverage. I would guess 50% is too high and often impractical. Even with high measured test coverage, the risk of bug is roughly the same. I never believe higher unit test coverage is a vaccination. Diminishing return. Low marginal benefit.

Why the difference between Q1 and Q2?

One reason — the source file is compiled into a library (or a jar), along with many other source files. This library is now a big component of the system, rather than one of 1000 python files. The managers will see a library change in c++ (or java) vs a single-file change in python.

Q3: what if the change is to a single shell script, used for start/stop the system?
A: yes. Manager can see the impact is small and isolated. The unit of release is clearly a single file, not a library.

Q4: what if the change is to a stored proc? You have tested it and run full unit test suit but not a full integration test. Will you release this single stored proc?
A: yes. One reason is transparency of the change. Managers can understand this is an isolated change, rather than a library change as in the c++ case.

How do managers (and anyone except yourself) actually visualize the amount of code change?

  • With python, it’s a single file so they can use “diff”.
  • With stored proc, it’s a single proc. In the source control, they can diff this single proc
  • with c++ or java, the unit of release is a library. What if in this new build, beside your change there’s some other change , included by accident? You can’t diff a binary 😦

So I feel transparency is the first reason. Transparency of the change gives everyone (not just yourself) confidence about the size/scope of this change.

Second reason is isolation. I feel a compiled language (esp. c++) is more “fragile” and the binary modules more “coupled” and inter-dependent. When you change one source file and release it in a new library build, it could lead to subtle, intermittent concurrency issues or memory leaks in another module, outside your library. Even if you as the author sees evidence that this won’t happen, other people have seen innocent one-line changes giving rise to bugs, so they have reason to worry.

  • All 1000 files (in compiled form) runs in one process for a c++ or java system.
  • A stored proc change could affect DB performance, but it’s easy to verify. A stored proc won’t introduce subtle problems in an unrelated module.
  • A top-level python script runs in its own process. A python module runs in the host process of the top-level script, but a typical top-level script will include just a few custom modules, not 1000 modules. Much better isolation at run time.

There might be python systems where the main script actually runs in a process with hundreds of custom modules (not counting the standard library modules). I have not seen it.

##transparent^semi-transparent^opaque languages

With a transparent language, I am very likely (high correlation) to have higher GTD/productivity/KPI.

Stress — correlates with opacity. Would I take a job in php, SQL, javascript, perl?

Confidence to take on a gig — The more transparent, the higher my confidence — like py, javascript

Bootstrap — with a transparent language, I’m confident to download an open source project and hack it (Moodle …). With an opaque language like C++, I can download, make and run it fine, but to make changes I often face the opacity challenge. Other developers are often more competent at this juncture.

Learning — The opaque parts of a language requires longer and more “tough” learning, but sometimes low market value or low market depth.

Competitiveness — I usually score higher percentiles in IV, and lower percentiles in GTD. The “percentile spread” is wider and worse with opaque languages. Therefore, I feel 滥竽充数 more often

In this context, transparency is defined as the extent to which you can use __instrumentation__ (like debugger or print) to understand what’s going on.

  • The larger systems tend to use the advanced language features, which are less transparent.
  • The more low-level, the less transparent.

–Most of the items below are “languages” capable of expressing some complexity:

  • [T] SQL, perl, php, javascript, ajax, bash
  • [T] stored proc unless complex ones, which are uncommon
  • [T] java threading is transparent to me, but not other developers
  • [S] java reflection-based big systems
  • [T] regular c++, c# and java apps
  • [O]… but consider java dynamic proxy, which is used in virtually every non-trivial package
  • [T] most python scripts
  • [S] … but look at python import and other hacks. Import is required in large python systems.
  • [O] quartz
  • [S] Spring underlying code base. I initially thought it was transparent. Transparent to Piroz
  • [O] Swing visual components
  • [O] C# WCF, remoting
  • [T] VBA in Excel
  • —-below are not “languages” even in the generalized sense
  • [S] git .. semi-transparency became stressor cos KPI!
  • [O] java GC … but NO STRESS cos so far this is not a KPI
  • [O] MOM low level interface
  • [S] memory leak detectors in java, c#, c++
  • [O] protobuf. I think the java binding uses proxies
  • [T] XML + simple SAX/DOM
  • [S =semi-transparent]
  • [O =opaque]

 

zero sum game #my take

“Zero sum game” is a vague term. One of my financial math professors said every market is a zero sum game. After the class I brought up to him that over the long term, the stock (as well as gov bond) market grows in value [1] so the aggregate “sum” is positive. If AA sells her 50 shares to BB who later sells them back to AA, they can all become richer. With a gov bond, if you buy it at par, collect some coupons, sell it at par, then everyone makes money. My professor agreed, but he said his context was the very short term.

Options (if expired) and futures look more like ZSG to me, over any horizon.

If an option is exercised then I’m not sure, since the underlier asset bought (unwillingly) could appreciate next day, so the happy seller and the unwilling buyer could both grow richer. Looks like non-zero-sum-game.

Best example of ZSG is football bet among friends, with a bookie; Best example of NZSG is the property market. Of course we must do the “sum” in a stable currency and ignore inflation.

[1] including dividends but excluding IPO and delisting.

self-degrad`habit: compare with higher flyers

This post is Relevant to parenting, in sports and also in a dev team…

eg: I was very strong in physics at high-school level in a top secondary school. Then other students rose up to be as strong as me. I had a deep sense that my edge was gone and I would no longer be far and away the strongest in my school. One day I suddenly saw myself as a nobody since I was no longer the strongest in at least one major field. The fact that I was still one of the strongest in that Beijing district, and also across China was completely wiped off the table and trashed. At the same time, my math was also in the top 0.1% and I had no weakness academically.

eg: 王傑 (Dave Wong), one of my all-time favorite Taiwan singers became less popular after more than a decade, and lost further ground in terms of “mind share” as he aged past 50. There have been too many new talents. Many of them come and go, and few had the same longevity as Dave. I don’t know how he felt, but a fact that’s easy to lose is, he has a distinct style. Not being the top 20 most popular singer doesn’t wipe out that fact. Many fans still love him even though he can’t sing like 30 years ago.

eg: A great pastor (or preacher) goes to Australia and was wowed to witness another preacher of a similar style, but more powerful (or more eloquent). He soon decided to learn and improve himself, and developed his distinct style. He has unique talent as an individual, and he would sooner or later develop some feature of his own.

eg: A solid movie can be upstaged by a movie of a similar genre, but higher box office. However, every serious movie is unique. If two very similar movies are both good, then invariably the differences will be significant.

I know more than one individual with a habit to degrade, belittle, and dismiss her/his own achievements, strengths and talents. The opposite of “celebrate every little improvement“. An irrational, unfair, self-destructive habit. Because most grown-ups have developed some immunity, doing this thing to other people is not as harmful as doing it to ourselves and loved ones. We will have more to say about parenting…

It’s impractical to dictate “If I’m good at this field than I have to be #1 in the entire school or nothing.” If you are indeed #1, then sooner or later you would meet someone with a higher talent in some part of this field, so your #1 position becomes questionable. Does it diminish your achievements? Possibly, arguably yes, but does it zero out your achievement? No. You are still a valuable player in the team.

There’s a related patten. An individual would still degrade himself if he is indeed the #1 local champion but in a less glamorous field, like a less popular art or sport, or less important domain. Well, that field is valuable and worthwhile. It’s a real fact. For example, my dad is a top researcher in several less glamorous domains.

I think [[Compassion and self-hate]] has something relevant here.

tech advantages of c# over java#le2XR

Hi XR,

Based on whatever little I know, here are some technical advantages of c# over java.

(Master these c# feature and mention them in your next java interview 🙂

  • C# has many more advantages on desktop GUI, but today let’s focus on server side.
  • [L] generics —- c# generics were designed with full knowledge of java/c++ shortcomings. Simpler than c++ (but less powerful), but more complete than java (no type erasure). For example see type constraints.
  • [L] delegates —- Rather useful. Some (but not all) of its functionalities can be emulated in java8.
  • [L] c# can access low-level windows concurrency constructs such as event wait handles. Windows JVM offers a standardized, “reduced-fat” facade. If you want optimal concurrency on windows, use VC++, or c#.
  • [L] reflection —- is more complete than java. Over the years java reflection proved to be extremely powerful. Not sure if c# has the same power, but c# surely added a few features such as Reflection.Emit.
  • concurrency —- dotnet offers many innovative concurrency features. All high level features, so probably achievable in java too.
  • tight integration with COM and MS Office. In fact, there are multiple official and unofficial frameworks to write Excel add-ins in c#
  • tight integration with high-level commercial products from Microsoft like MSSQL, sharepoint
  • tight integration with windows infrastructure like Windows Services (like network daemons), WCF, Windows networking, Windows web server, windows remoting, windows registry, PowerShell, windows software installation etc
  • c# gives programmers more access to low-level windows system API, via unmanaged code (I don’t have examples). In contrast, Java programmers typically use JNI, but I guess the java security policy restricts this access.
  • probably higher performance than JVM on windows
  • CLR offers scripting languages VB.net, F#, IronPython etc, whereas JVM supports scripting languages javascript, scala, groovy, jython etc.

[L = low-level feature]

If you want highest performance on Windows, low-level access to windows OS, but without the complexity of VC++ and MFC, then c# is the language of choice. It is high-level, convenient like java but flexible enough to let you go one level lower when you need to.

Another way to address your question — listen to the the complaints against java. (Put aside the complaints of GUI programmers.)

Even if a (rational, objective) architect doesn’t recognize any of these as important advantages, she may still favor c# over java because she is familiar and competent ONLY in the Microsoft ecosystem. She could point out countless features in Visual Studio and numerous windows development tools that are rather different from the java tool set, so different that it would take months and years to learn.

Also, there are many design trade-off and implementation techniques built on and for Dotnet. If she is reliant on and comfortable in this ecosystem, she would see the java ecosystem as alien, incomplete, inconvenient and unproductive. Remember when we first moved to U.S. — everything inconvenient.

On a more serious note, her design ideas may not be achievable using java. So java would appear to be missing important features and tools. In a nutshell, for her java is a capable and complete ecosystem theoretically, but in practice an incomplete solution.

readLock without writeLock – same as no lock@@

Q9: With regard to shared-mutable access semantics, using a readLock and discarding the writeLock (i.e. unable to use it) is similar to using no lock, but are they semantically the same?

This is a theoretical discussion. The implementations are not familiar to me and not relevant. Before we answer  Q9, let’s look at

Q1: in the absence of shared-mutable, do we need any lock?
A1: no

Q2: what if we use readLock without writeLock?
A2: should be fine.

Q3″ what if we do use a lock in a shared function?
A3: system behavior is modified — serialized access, which is unnecessary

A9: should be semantically identical. Everyone gets a free ticket at entrance.

java exception passing between threads

Many people ask how to make a child thread’s exception “bubble up” to the parent thread.

Background — A Runnable task is unsure how to handle its own exception. It wants to escalate to parent thread. Note parent has to block for the entire duration of the child thread (right after child’s start()), blocked either in wait() or some derivative of wait().

This question is not that trivial. Here are my solutions:

1) Callable task and Futures results — but is the original exception escalated? Yes. P197 [[java threads]]

2) Runnable’s run() method can temporarily catch the exception, save the object in a global variable such as a blocking queue, and notifyAll(). Parent thread could check the global variable after getting notified. Any thread can monitor the gloabal.

If you don’t have to escalate to parent thread, then

3) setUncaughtExceptionHandler() – I think the handler method is called in the same exception thread — single-threaded. In the handler, you can give the task to a thread pool, so the exception thread can exit, but I don’t know how useful.

4) adopt invokeAndWait() design — invokeAndWait() javadoc says “Note that if the Runnable.run() method throws an uncaught exception (on EDT) it’s caught and rethrown, as an InvocationTargetException, on the callers thread”

In c#, there are various constructs similar to Futures.get() — seems to be the standard solutions for capturing child thread exception.
* Task.Wait()
* Task.Result property
* EndInvoke()

IRS trading system@@ – Eric of Citi

IRS is not transferable. IRS contract can be re-assigned in some cases, but the original 2 counter parties and the new party must all agree.

Both parties must scrutinize the other’s credit worthiness. Libor rate is for top-credit borrowers. If the floating-payer is lower, then the spread on Libor (or the fixed rate?) will reflect that – a.k.a. credit spread. Alternatively, the counter party (floating receiver) can demand collateral.

There’s no secondary market for IRS like there are in listed securities.

Q: Is there an IRS trading system?
%%A: Most needed system might be a deal management system that tracks all our unexpired IRS contracts. Since each deal is bespoke, volume is not high. The basic entity in the system is known not as a position, but a deal. It’s treated like a trade as there are 2 accounts involved, and multiple settlement dates.

Q: Is IRS market regulated?
A: Regulators set limits on total exposure. Participant’s quarterly balance sheets include these IR swaps. One big swap could push a company above the limit.

coding IV – favored by the smartest companies

XR,

I see a few categories of IV questions:

1a) fundamentals — Some Wall St (also in Asia) tough interviews like deep, low-level (not obscure) topics like threading, memory, vtbl, RB-tree ..

1b) lang — Some (IMO mediocre) interviewers (like programming language lawyers) focus on language details unrelated to 1a)
2) Some (IMO mediocre) interviewers like architecture or high level design questions (excluding algo designs) like OO rules and patterns but I feel these are not so tough.

3) algo — Some tough interviews focus on algo problem solving in pseudo-code. See http://bigblog.tanbin.com/2007/09/google-interviews-apply-comp-science.html. I got these at Google and Facebook. Some quants get these too.

4) compiler coding question — is another tough interview type, be it take-home, onsite or webex.
With some exceptions (like easy coding questions), each of these skills is somewhat “ivory tower” i.e. removed from everyday reality, often unneeded in commercial projects. However these skills (including coding) are heavily tested by the smartest employers, the most respected companies including Google, Facebook, Microsoft… You and I may feel like the little boy in “emperor’s new dress”, but these smart guys can’t all be wrong.
I will again predict that coding questions would grow more popular as the logistic cost is driven down progressively.
Candidate screening is tough, time consuming and, to some extent, hit-and-miss. With all of these screening strategies, the new hire still can fail on TECHNICAL ground. Perhaps she/he lacks some practical skills — Code reading; debugging using logs; automation scripts; tool knowledge (like version control, text search/replace tools, build tools, and many linux commands)
Needless to say, new hire more often fail on non-technical ground like communication and attitude — another topic altogether.

In terms of real difficulty, toughest comp science problems revolve around algorithm and data structures, often without optimal solutions. Algo interview questions are the mainstay at big tech firms, but not on Wall St. Some say “Practice 6 months”. Second toughest would be coding questions —
* Often there’s too little time given
* Sometimes interviewer didn’t like our style but onsite coding tends to focus on substance rather than style.

Tan bin

P.S. I had webex style coding interviews with 1) ICE Feb 2011, 2) Barclays (swing), 3) Thomson Reuters, 4) Bloomberg

P.S. I tend to have more success with algo interviews and onsite coding than take-home coding. See blog posts.. (
http://tigertanbin2.blogspot.com/2015/05/sticky-weakness-revealed-by-interviews-c.html
http://tigertanbinpripri.blogspot.com/2015/03/high-end-developer-interviews-tend-to.html
)