zbs/GTD/KPI/effi^productivity #succinctly

See zbs^GTD^QQ

  • zbs — real, core strength (内功) of the tech foundation for GTD and IV.
  • GTD —
  • KPI — boss’s assessment often uses productivity as the most important underlying KPI, though they won’t say it.
  • productivity — GTD level as measured by __manager__, at a higher level than “effi”
  • effi — a lower level measurement than “Productivity”

##most used(GTD+)Generic know-how4WallSt developers

#1 java — the ecosystem
#2 c/c++ — not yet my chosen direction
#3 dotnet including c#, WCF, remoting, windows debugging …
#4 py — purely automation in most places(easy instrumentation); advanced py systems in a few places

# unix power user tools – including scripting, instrumentation… More widespread but depth is seldom required compared to SQL and python. More well-defined as a tech skill than windows dev.

# SQL/stored proc — losing pole position in low latency and big data environments, still dominant in traditional apps

# Excel add-in and automation

# javascript

establish tech stronghold(GTD+nlg)]each team

Note – focus is job performance, not IV.

To survive in a team,need unique technical specialty in a key area.

With the possible exception of the Citi Muni team, I have not seen a banking IT team to tolerate a “dead weight” guy. To do reasonably well in any financial IT team, I often need some “hard” strength (like an subject matter expertise), not just soft skill. If within your team you have a unique technical strength in a complex and important sub-system, then you can “name your price”, and you are kind of irreplaceable. Perhaps a go-to person. Some team members don’t have any expertise but survive on strong relationships. I’m unable to do that. In the past teams, I think many team members have such a strength. If I don’t have it, my position would be less established, less solid.

  • 😦 😦 stirt? Worst experience. Was good at the insignificant preferences framework; canceled trades
  • 😦 GS? Uncomfortable position. My Perl was not top notch. My java was below Yang. I became good at Error Memos (EOS) and trailers/upfronts
  • 🙂 OC? was good at Bloomberg adapter and Guardian builtin web server to show the log files
  • 🙂 🙂 95 Green? My DB design was convincing to Ravi. My wait-notify based design was pretty hard to match.
  • 🙂 Barc? FMD, fmath
  • 😦 Macquarie? In the quant team, whatever I accomplish seems trivial to manager, since he’s too strong. I’m seen as very slow on small tasks.

IV^GTD – grow as techie@WS

I want to grow stronger/broader/versatile as a survivor, not necessarily grow my value-add. Actually I don’t have to grow.
— IV skills – Compared to GTD skills, these skills give me more confidence, more protection, more stress-relief. It works in big job markets like Wall St.

Lots of theory, which is my sweet spot.
Selective on what IV topics to learn!
coding IV + algo — favored by SiV
— (portable) GTD skills
Lots of tools….
Selective on what GTD topics to learn!

Needed for a lead developer, but such a role is stressful. I guess some good lead developers are also good at IV, but I’m not sure and I assume I’ll not be good at both.

Warning – a lot of projects don’t provide real GTD enrichment. Eg: Quartz, tibrv wrappers, Gemfire wrappers, FIX wrappers
Macquarie environment lets me learn lots of GTD skills.
OC gave me IV (c#) and GTD enrichment.
Stirt – none!

A java environment would give me some additional GTD enrichment but less IV enrichment

In SG, I continued my previous strategy, learning IV skills + GTD skills. Not effective so far. I feel my c# IV skills improved a lot but still not there. C++ IV skills didn’t improve much partly due to distractions.


2 focuses when learning a new tech skill: IV^GTD

When we learn a new tech skill, we need very, very specific goals and guidelines to help us choose what NOT to dig into too early.

Focus #1: IV topics, a well-defined focus. Basically textbook knowledge. Go deep enough but not too deep on a particular subtopic. Examples — op-new, pure virtual, const ref param of copier, recursive locks, OOM calling new,

Focus #2: GTD i.e. dev productivity — as measured by managers, bench-marked against other developers. Incidentally, perceived productivity sometimes depends so heavily on local system knowledge that a new hire always lags behind no matter what. I feel productivity and GTD has low dependency on #1, as most of the textbook knowledge is irrelevant.

Focus #2b: tools — see post on 2 categories of tools.

Focus #2c? tips on StackOverflow. No such books. No way to

Focus #?: best practices — or Acceptable practices and well-trodden paths

Focus #?: learn all the syntax details to read any source code on books, interviews and online

Focus #: high-level design decisions, for technical leadership roles. Myriads of decisions are made every hour during development, some at high some at low levels. Apparently, a lot of these decisions involve theoretical understanding.

tech zbs( !!GTD) outweigh other(soft)job skills #%%belief

label: big20

In many teams (including …), I feel technical zbs/GTD capabilities

are still the #1 determinant of job security, comfort level,

competence level, work-life balance. Secondary yet important factors


– relative caliber within the team

– boss relationship

– criticality of the role within the team, within the firm

– reliable funding of the project, team or role

– closeness-to-money

– long work hours

– fire-n-hire culture

So zbs is key to keeping a job, but for finding a job, something else needed.

sys instrumentation: win32 murkier than linux #c++@@

See also

I recently browsed [[inside windows debugging]]. In the past I also browsed a few *nix system programming books. I’m more afraid of windows than *nix. Windows is more dark and murky; linux is more open and consistent. As a beginner, I feel in linux the challenge is interpreting tool output but on windows it’s getting past the installation and security restrictions.

  • complication: GUI — As explained in this blog (http://bigblog.tanbin.com/2013/04/for-administration-prefer-command-line.html), GUI tends to hide a lot of features in layers and compartments. Console tools provide the same amount of power features …
    • if there’s a security or registry or other issues that reduce the provided feature set to below 100%, a console tool is more likely to report it to the user but GUI tools often “intelligently” handle it as if it’s normal.
    • MSVS is the primary windows instrumentation tool – very complicated.
  • complication: tool installation – Linux tools are not easy to install, but Windows tools tend to be even harder to install.
    • ** registry.
    • ** On any platform GUI tools are, on average, harder to install. Linux tools are less likely GUI.
  • complication: security restrictions – for a novice, 1/3 of the problems with windows tools are related to security/permission. Even under an administrator account, there are many security restrictions we may need to painstakingly disable! I don’t have any example now. I feel windows was criticized as vulnerable so Microsoft added a lot of security features.
    • ** some security features seem to be controlled by the domain administrators.
  • complication: environment factors – If you run the same tool in 2 machines with different results it’s often due to environment factors. In Linux, environment variables and some utilities are the only environment factors I know. In windows there seem to be many more places to check.
    • ** registry
    • ** certificates
    • ** control panel items
    • ** secpol.msc
  • complication: tool availability – many windows tools are commercial. On windows, I generally find the open-source tools better-quality, but for some tasks there’s only one tool – shipped by Microsoft. If I feel it is often crippled. On linux the best tools are invariably open-source.
  • complication: undocumented features — Even if you follow the all documented procedures, still you may hit something unexpected, due to undocumented features. This happens more often on windows then linux.

Note undocumented features affect not only instrumentation tools, but other components of the platform, which affect our debugging.

##windows GTD skillset, my take

(master copy? Recrec)

Dotnet is like java, right? Dotnet developers operate in a sandbox similar to the java developer?

Wrong! I feel a windows developer team operates in an ecosystem that’s fundamentally different from a java development ecosystem. Differences lie beneath the surface.

Since all of my java servers run in linux (or unix), I also need some linux platform knowledge, but nothing in-depth.

A windows developer needs a lot of specific (platform) skills. The list below is what I feel I need to pick up. [[Inside windows debugging]] by a MS veteran points out loads of essential know-how required for dotnet (or pre-dotnet) hacking beyond the MSVS debugger. From this book i get the sense that every serious windows developer need (non-trivial) windows know-how.

As an analogy, serious sql coders need (non-trivial) knowledge of optimizer and query plan.

I feel the average windows developer don’t have the capacity to know all of these. Perhaps some guys specialize in winforms some in wpf, some specialize in web development, some specialize in VB, some specialize in c# server side… Yet it’s not inconceivable for some lead developer/architect to have some experience with all of these, and some deep experience in a few topics.

* windows TCP/IP – Microsoft has its own implementation
* windows threading – many small differences from java threading, even though the concepts match.
* windows web service (SOAP etc) – Microsoft has its own implementation
* debugger – remote debugger
* xml parsing – Microsoft has its own implementation
* office integration (COM), excel addin
* IDE – many many must-know 
* project configuration, 
* application configuration
* GUI dev – silverlight
* GUI dev – wpf
* set up WCF service and host
* c# loggers
* powershell, DOS scripting

* iis
* messaging server
* sharepoint server 
* exchange server 
* GUI dev – winforms
* GUI dev – MFC
* web development – silverlight
* web development – ASP, ASP.net
* web development – vbscript, jscript etc

The blog title says “windows” because I believe a serious dotnet developer always knows windows well beyond dotnet. Many __dotnet__ dev skills require windows platform knowledge. (In java projects platform knowledge is useful but not as essential as in dotnet.) I believe a dotnet developer has to become a windows developer to be truly competent.

– COM integration
– MS-Office integration
– interop with VB, VC, both rich-history technologies retrofit into dotnet
– windows debuggers
– sysinternals tools
– registry
– application installer/uninstaller
– event viewer
– performance counters
– dotnet threading is based on win32
– WCF is based on a earlier Microsoft remoting technologies

important IV/GTD skills for algo trading developer

Traditional (“real”) algo trading is still confined to buy side. (Sell side algo trading includes things like RFQ auto pricer, limit order auto pricer etc. Dr. Chou points out execution cost reduction algos…) Such shops look for the “strategist” role which is often (usually?) a different role from the implementation engineer’s role. Allow me to focus on the latter.

In my opinion, the really important skills are all about optimization, latency, performance Engineering

#1 [IV] algorithm/data structure optimization
#2 instrumentation (incl. debugging) — profilers, memory leak detector
# [IV] threading — for multiprocessor
# network/socket programming, including NIO
# fast messaging – like RV or solace
# memory management (c++), GC tweaking (java)
# OS tuning including memory, net and CPU
# cache products – like coherence
# exchange conn beyond FIX – understand its idiosyncrasies

Other skills are important to the team but less important to the engineer
– market data gateway
– back testing
– architectures
– OO designs
– automated pricing. No need to understand BS or bond math, or hedging techniques
– basic trading algorithms
– scripting for system automation
– statistics software

## low latency – key expertise4GTD

I talked to a few big and small low-latency shops. I feel latency optimization is “low-logic, high speed, high throughput”. Below are the Expertise required, but in any real system, you will see diminishing return on each aspect, so remember to prioritize the cost-effective areas to optimize. See my post on 80/20 (http://bigblog.tanbin.com/2012/03/8020-rule-dimishing-return.html)

* threading
** parallel lock free threading but sometimes we can’t avoid inter-thread-comm (condVar)
** [L] try every technique to avoid locking (such as immmutables)

* [L] low-level memory management — custom allocators, custom (or disabled) garbage collectors
** care and feeding of java GC
** try every way to avoid non-deterministic garbage collection

* in-memory data stores — KDB etc
** This is how secDB compensates for some of the performance drawbacks
** state maintenance in memory — important in many OMS, DMA, exchange connectivity engines.

* try every techniques to avoid RDBMS
** use flat files

* low-latency messaging — 29West, tibrv messaging appliance …
** [H] async MOM is the only choice for high volume systems
** multicast is the de-facto standard

* OS monitoring and optimization
** dtrace
** network utilization
** paging activity
** cpu utilization

* [L] socket programming customized
* avoid ethernet collisions
* [H] avoid choke points, promote multi-lane highways
* [H] connectivity, collocation
* [L] serialization – customized
* [L] powerful hardware, FPGA
* [H] scale out – important
* 64 bit
* realtime java vs C++
[L] = low-level
[H] = high-level architectural feature

marathon – strengthen your(GTD+)lead in the pack#sg high-end

Y Lin pointed out those high-end jobs tend require specialized skills. I now feel concurrency is an example, in addition to —

latency – c++ is the default for extreme latency
latency – sockets
latency – market data
latency – java?
dnlg – math — better entry barrier, closer to business.
dnlg – jargons in FX, Rates…
dnlg – arch

real projects are completed in lowly tools like sql, java, C, scripts, but interviewers need to see much much more than those.

knowledge@vendor products is far from enough4GTD

Consider typical non-trivial financial app based on 1 or a couple primary vendor products + some database + some JMS server + some app server + some web server. The #1 “primary vendor software” could be a distributed cache, spring, ETL, or something like ION or something else.

Q: How much complexity is in the vendor software, and how much is in the site-specific code including localized customizations of the vendor software?
A: 10% vs 90%

How about the ML Edge b2b trade engine? 70/30% at this stage (March).

harmless productivity tools ^ frameworks like ..Spring

[[The Productive Programmer]] by Neal Ford advocates learning a large number of _simple_ tools for productivity. Tools could be a complex but we use them in a non-intrusive, simple manner. Many other tools promise to help productivity but are traps – spring, hibernate, jaxb, xslt. Once you employ them they tend to become your master.

I have no such issues with perl/shell scripting, testing tools like junit, unix utilities, text editors, eclipse, maven/ant (?) A critical feature is, the option to use the tool in part of my project, and use alternative tools for the remainder. The tool should not be possessive and exclusive.

Hibernate is more gregarious than Spring, as you can mix Hibernate and straight JDBC in different modules of your system. Spring, being a framework, tends to engulf your entire project. You are then constrained by that framework developers’ decisions.

c++^perl zbs insights: vastly different values

Looking at my blog posts on c++ vs perl, the value of my insights in these 2 languages are very different.

* c/c++ is the foundation underneath java, c#, OS, DB and most infrastructure software. perl is something most systems can make do without
* c++ performance is still the envy of most languages in the context of many high-value systems
* compared to c++, perl and php are niche languages, even if ranked in the top 10 languages
* there’s a large following and corporate endorsement for any popular language like ruby, groovy, pike, LISP, Tcl, but it doesn’t mean these are as important as c# or c++. Be critical. Be comparative. Be objective. Take a perspective.
* If you look at wall street salary, some perl or mainframe jobs could pay $100/hour, but be very critical and look beyond short term.

##front office technical know-how

–Most imp to hiring and productivity–

swing, wpf — more complex than all other tools
pricing math
pnl rollup, MTD, unrealized.. is imp to traders
fire fighting
c++ …rv
mastery over db query 

general low latency techniques
gemfire — imp to both risk and real time pre-trade. Let’s xx2set up
trade booking, cancel, MOD
java OOM — simple techniques
position mgmt is important to traders
trade blotter basics

young, productive developers #GS

Regarding my young GS colleague … He learns the production codebase very fast, so

* he can understand, in-depth, how and where all those complicated logic is implemented in production system
* he can understand what users mean, which is usually condensed in very terse emails, or quick phone calls
* he can quickly come up with quick solutions
* he knows what test scenarios are extremely unlikely so we need not handle them or test them
* he is faster navigating the code base from java to perl/shell scripts, to stored proc and big queries, to autosys and javascript, where business logic lives.

(I was weaker on all these aspects, for a long time.)

He’s more confident fixing existing codebase than architecting a rewrite. Limited experience. In this aspect, employer would see limited value in him.

##growing in java: where to focus when learning c++/c#

I feel stronger than a rookie java developer because we are comfortable with both GTD and IV:
1) IDE
** (remote) debugging. Also consider my Perl sss()
2) threading — idioms, principals
** thread creation, synchronized, wait…
** threadlocal
** thread pool
** immutable
3) reflection + proxy
) syntax of overriding, constructors, final, singletons, generics
) design patterns — large idioms
) unit test

other signs of strength:
) generics — creating generic classes
) serialization
) garbage collections
) inner classes
) collections and data structures — choosing

What about spring, hibernate …? I feel these fads come and go.

##differentiation ] a commodity skill market

Problem: java used in finance is becoming a commodity skill. In fact, any software skill tends to spread very quickly (esp. to the very young developers). What are the exceptions and what are the differentiations we can build up, differentiations that are recognizable by hiring managers?

  • Differentiation: c++, c#, python, perl
  • Differentiation: DB tuning. Most developers don’t face this challenge.
  • Differentiation: jvm tuning including GC
  • Differentiation: practical java performance problem-solving. Compare — I had the opportunity to deal with large DB.
  • differentiation: low latency
  • Differentiation: swing – has unique paradigm and  challenges liks EDT, models, layout … that the regular java developer won’t face

Exception: googlers and facebook developers are probably used to deal with large volume and higher performance. Outside developers face no such challenge.

Exception: Oracle employees probably know their own db much better so few developers can match their skill.

%%tech strength/weak among Sr trad’dev candd ]NY

^ threading?
^ java overall?
^ c++ overall?
^ generic algorithms and data structures
^ generic OO
^ SQL?
^ Unix?
^ DB tuning?
^ scripting?
design patterns?

Imagine you are an interviewer comparing candidates “feature-by-feature”. Some areas outweigh others. For example, many interviewers feel fundamental knowledge outweigh experience using 3rd-party tools.

%%tech strengths for fin IT jobs

— ranked
java + c# + c++
perl, unix — still widely used in financial but not many people know in-depth
SQL — outer join, self-join, sub-query and other complex joins, case expressions, group-by. See the t-sql book.

# eor
index design and utilization — banks heavy usage
milestoning pitfalls, limitations — banks heavy usage
complex data analysis using SQL — banks use it all the time.
query plan

real stories illustrat`%%debugging expertise — focus on java

see other stories in j,ora.xls

DNS reload
freebsd NIC driver
custom classloader breaking EJB casting
weblogic upgrade max connection exceeded — ulimit for the shell,
max_tcp_connections, file descriptors

jdbc failures — snoop

weblogic jms XA conn factory; disable catch clause to trigger rollback

2 (custom) class loaders loading the same class — cast exception

nbc firefox and ie gets different dates from the same url. we reduced
the code to the simplest. then i managed to get a
Runtime.getRuntime().exec( “ls /tmp” );

log4j appenders in assignment are from 2 classloaders, but “murderer”
(classloader bug) looks like an accomplice — log4j errors are usually

real&&rare value-add to developers@weblogic

Some developers have 3-5 yeras experience on weblogic but offer no valud-add relative to “thinking” develoeprs coming from, say, websphere backgrounds. Value-add ranked in terms of value:

* 1) insight into weblogic trace logs (=> tuning, TR). Each app server’s developers have their preference what to log. Perhaps a neglected aspect of J2EE standardization.
* key limitations of weblogic vs websphere, jboss. There are probably dozens of limitations but real veterans know a few serious ones
* weblogic server tuning
* monitoring (=> tuning, TR)
* weblogic cluster
* JRockit Runtime Analyzer and esp. mem leak detection
* weblogic support for web services — different from other vendors

other value-add without ranking:

* weblogic domain
* weblogic deployment tools
* weblogic JNDI idiosyncrasy
* AR
* dev^production mode
* when not to use weblogic jms
* when not to use weblogic conn pool
* idiosyncrasies of weblogic thread pool