deadly delays@ project^feature levels

insightful article: managing tight deadlines is the background

— deadly delay at feature level, without user impact

This is real stressor primarily because there exist team colleagues who can GTD faster on this same task.

For the serious delay, user impact is … exaggerated! This is true across all my projects — user impact doesn’t really matter. In fact, the feature may not be used much whether you deliver it in high or low quality, within or exceeding budget.

— blatant delay at project level when you are architect / team lead

In theory if you promise to delivery some new feature i.e. green field project, then it can be tough to deliver on time. In reality, project time/budget overrun is very common. You only need good explanation.

Users never care that much about any new system cf current production systems. New systems often hit various teething problems and functionally unreliable for months.

OK , Users don’t really care that much, but there’s visible technology budget impact which make the technology MD look bad. MD must look for an “explanation” and may cut headcount as in StirtRisk and RTS-Trep

Whose fault? Many fingers point at you the architect, but often it’s partial fault of the user-facing manager due to immaturity and overconfidence with timeline estimate.

Delay is mostly architect’s fault if another architect can deliver faster but I can’t imagine two architects working on two comparable projects at the same time. Therefore it’s never “mostly architect’s fault”. In reality, management do apply pressure and you suffer, but it’s really for political reason (like sacrificial lamb or scapegoat)

eg: RTS Trep
eg: PWM Special-Investment-management
eg: Post-Saurabh GMDS — multiple delays and scope reductions

c++IV=much harder than GTD #Mithun

c++ IV is much harder than c++ job GTD, as I told Mithun.

  • GTD is no different from java jobs, even though the build process can be slightly hairy. Java build can also get messy.
  • In contrast, C++ IV is a totally different game.

You need a rating of 1/10 to do a decent job, but need 7/10 to pass ibank interviews. This gap is wider in c++ than in java as java interview bar is much lower.

Most technical challenges on the job are localSys, so you can just look at existing code and 照猫画虎, 如法炮制, as AndrewYap does. Venkat of RTS said we should but we still do.

Corollary — after 3Y full time c++ job, you may still fail to pass those interviews. Actually I programed C for 2Y but couldn’t pass any C interview whatsoever.

replacing auto_ptr with unique_ptr #IViewer’s focus

I spent lots of time cleaning up hundreds of auto_ptr usages but interviewers were more interested in the theoretical differences between the two.

  • On the job, it’s an operational challenge. You don’t need the knowledge.
  • On the interview, it’s a knowledge challenge… theoretical, bookish. You don’t need the GTD capacities.

Proven GTD: worthless in candidate ranking #JackZ

I feel that Jack Zhang is competent with localSys GTD but weak on c++ and comp science.

Does he have working knowledge of c++? I assume so. Working knowledge is attainable in a couple of months for a clean language, and up to a year for c++

The required level of working knowledge and basic skill is very low for localSys GTD.

His c++ knowledge is probably barely enough to do the job. Remember I didn’t know what things live on java heap vs stack.

Based on my guesstimate, he would fail any c++ interview and any algo interview. He can write simple SQL in an interview, but I am not sure if he can write complex joins.

The fact that Jacn and DeepakM are are proven on GTD is useless and lost in the conversation.

How about CSY? He can solve many algo problems without practice, but he is reluctant to practice.

I think the self-sense of on-the-job competency is misleading. Many in their positions might feel GTD competency is more important than IV skills. They are so afraid of the benchmark that they don’t want to study for it.

When the topic of tech interview comes up, I think they wish to escape or cover their ears.

RTS: Don’t rely@QA2test your code@@

insightful article: managing tight deadlines is the background

My ICE manager said "Don’t rely on QA team to test your code. Do your thorough testing before passing to QA."

This is Impractical given the thousands of detailed items in a 50-page spec. Thorough developer testing would take more time and I didn’t do it. A colleague was able to give QA team a well-tested rpm and received only 30 bug reports (mine received 60 to 100), but did it win him any award? No.

  • I could explain my requirement is more complex. Manager may or may not accept.
  • Sometimes we could explain one of us is new
  • Which system has more crashes in QA/production? Each crash can take a long time to figure out. Thorough developer testing doesn’t help much here.
  • In the end, which system goes live faster? That’s the visible yardstick. 2nd visible yardstick is how quickly we start QA, so by cutting corners on developer testing, I was able to start QA earlier. A more thorough developer test won’t help with any of these visibles.

%%churn OK, accu bad] quantDev: Sg+U.S.

My past experiences are underwhelming. I thought that once I become experienced and proven in quant dev domain, things will be easier and I could move from one job to another. Wrong!

  • Barclays? helped me get into OC since OC interviewers are interested in how things are done in Barclays. Didn’t really help me go anywhere else
  • Stirt? helped me a bit with Mac interview
  • MSFM? didn’t help me get anywhere, partly because I didn’t try.

OC/Stirt/Mac gave me no insight no breakthrough in my understanding no thick->thin->thick

The number of quant dev positions is much fewer than in market data!

multicast: IV care only about bookish nlg !!practical skills

Hi friends,

I recently used multicast for a while and I see it as yet another example of the same pattern — technical interviewers care about deep theoretical knowledge not practical skills.

Many new developers don’t know multicast protocol uses special IP addresses. This is practical knowledge required on my job, but not asked by interviewers.

Unlike TCP, there’s not a “server” or a “client” in a multicast set-up. This is practical knowledge in my project but not asked by interviewers.

When I receive no data from a multicast channel, it’s not obvious whether nobody is sending or I have no connectivity. (In contrast, with TCP, you get connection error if there’s no connectivity. See tcp: detect wire unplugged.) This is practical knowledge, but never asked by interviewers.

I never receive a partial message by multicast, but I always receive partial message by TCP when the message is a huge file. This is reality in my project, but never asked by any interviewer.

So what do interviewers focus on?

  • packet loss — UDP (including multicast) lacks delivery guarantee. This is a real issue for system design, but I seldom notice it.
  • higher efficiency than TCP — I don’t notice it, though it’s a true.
  • socket buffer overflow — should never happen in TCP but could happen in UDP including multiast. This knowledge is not needed in my project.
  • flow control — TCP receiver can notify sender to reduce sending speed. This knowledge is not needed in many projects.
  • non-blocking send/receive — not needed in any project.

So what can we do? Study beyond what’s needed in the project. (The practical skills used is only 10% of the interview requirements.) Otherwise, even after 2 years using multicast in very project, I would still look like as a novice to an interviewer.

Without the job interviews, it’s hard to know what theoretical details are required. I feel a multicast project is a valuable starting point to get me started. I can truthfully mention multicast in my resume. Then I need to attend interviews and study the theoretical topics.

##c++QQ topics discovered ONLY from IV

I didn’t know these were important when I was reading on my own.

  • socket details
    • tcp handling of OOS
    • tcp flow control, AWS
  • smart ptr api and control-block manipulation
    • make_shared details
    • enable_shared_from_this
    • auto_ptr vs unique_ptr
  • multiple inheritance, casting
  • template techniques
  • std::forward
  • exception catching/re-throwing
  • q[ … ] variadic template params
  • inline: impact on performance
  • throwing dtor
  • details of pure virtual

effi^instrumentation ] new project

I always prioritize instrumentation over effi/productivity/GTD.

A peer could be faster than me in the beginning but if she lacks instrumentation skill with the local code base there will be more and more tasks that she can’t solve without luck.

In reality, many tasks can be done with superficial “insight”, without instrumentation, with old-timer’s help, or with lucky search in the log.

What if developer had not added that logging? You are dependent on that developer.

I could be slow in the beginning, but once I build up (over x months) a real instrumentation insight I will be more powerful than my peers including some older timers. I think the Stirt-tech London team guru (John) was such a guy.

In reality, even though I prioritize instrumentation it’s rare to make visible progress building instrumentation insight.

in-depth^cursory study@advanced program`topics

I have seen both, and tried both.

AA) A minority of developers spend the time studying and understanding the important details of threading, generics, templates … before writing code.

BB) Majority of developers don’t have the time, vision or patience, and only learn the minimum to get the (sloppy) job done. They copy-paste sample code and rely on testing, but the test cases are limited to realistic UAT test cases and will not cover edge cases that have yet to occur. For concurrency designs, such tests are not good enough.

AA is often seen as unnecessary except:
* Interviewers often go in-depth
* In a product vendor rather than an in-house application team
* If requirement is unusual (eg: B2BTradingEngine), then BB won’t even pass basic developer self-tests. Sample code may not be available — see my post So developers must bite the bullet and take AA approach. This is my sweet spot. See

On Wall St projects, I learned to abandon AA and adopt BB.

On Wall St interviews, I still adopt AA.

On West Coast, I think AA is more needed.

##most used(GTD+)Generic know-how4WallSt developers

#1 java — the ecosystem
#2 c/c++ — not yet my chosen direction
#3 dotnet including c#, WCF, remoting, windows debugging …
#4 py — purely automation in most places(easy instrumentation); advanced py systems in a few places

# unix power user tools – including scripting, instrumentation… More widespread but depth is seldom required compared to SQL and python. More well-defined as a tech skill than windows dev.

# SQL/stored proc — losing pole position in low latency and big data environments, still dominant in traditional apps

# Excel add-in and automation

# javascript

killer talent2compensate4many shortcomings

The scope is professional life and financial security.

update — to keep the killer sharp, you need sustained focus and continuous improvement.

eg: my dad is a top researcher in his fields. His research papers are heavy weight and top quality. Someone like him may have many limited skills on many other aspects (tech) but that one killer talent compensates for everything. Crucially, his research output has a financial reward. Otherwise it’s hard to sustain the effort.

eg: sales dragons are simply good at hitting sales numbers. A sales dragon may not understand the products, or remain loyal to the firm, or take care of the clients… Still this is a killer talent to keep her successful, that (kind of) compensates for everything.

eg: I know many IT professionals who are sub-standard technically, but can survive or even thrive in a job for a long time because they keep the boss happy — Killer talent… However, compared to other killer talents this one isn’t as weather-proof as other killer talents. The next boss may not be so easy to please. The company may go under…

eg: at least in my first 10 years of my career, I always could get my jobs done. I was always technically up to the job. This talent did compensates for everything.

eg: On Wall St, my job hunting skill is a killer talent. This is one of the most effective kill talents known to me.

eg: Some people manage to amass a small fortune and invest successfully, to get a decent return (like $5k/m). I guess Anthony Lin, Lian Zhong, Chun Tih .. all had multiple properties.

IV^GTD – grow as techie@WS

I want to grow stronger/broader/versatile as a survivor, not necessarily grow my value-add. Actually I don’t have to grow.
— IV skills – Compared to GTD skills, these skills give me more confidence, more protection, more stress-relief. It works in big job markets like Wall St.

Lots of theory, which is my sweet spot.
Selective on what IV topics to learn!
coding IV + algo — favored by SiV
— (portable) GTD skills
Lots of tools….
Selective on what GTD topics to learn!

Needed for a lead developer, but such a role is stressful. I guess some good lead developers are also good at IV, but I’m not sure and I assume I’ll not be good at both.

Warning – a lot of projects don’t provide real GTD enrichment. Eg: Quartz, tibrv wrappers, Gemfire wrappers, FIX wrappers
Macquarie environment lets me learn lots of GTD skills.
OC gave me IV (c#) and GTD enrichment.
Stirt – none!

A java environment would give me some additional GTD enrichment but less IV enrichment

In SG, I continued my previous strategy, learning IV skills + GTD skills. Not effective so far. I feel my c# IV skills improved a lot but still not there. C++ IV skills didn’t improve much partly due to distractions.


overvalue ^ undervalued in tech IV – random picks

–overvalued – 10 items unranked:
problem solving with comp science constructs — mostly algos
fast coding
code quality – in those take-home coding tests
corner cases
OO design theories, principles
–undervalued practical skills:
stackoverflow type of know-how, including syntax details. These aren’t quizzed in high-end interviews.
tools including IDEs
[T] bug hunting
[T] large code base reading
[T] reading lots of documentation of various quality to figure out why things not working
[T = hard to test in IV]

##job spec + %%observations: tech lead

#1 [SY,pwmcore] make architecture decisions for the team(s)
#2 [S] give software development direction to developer teams, including sample code
#3 [S,pwmcore] maintain and bug fix key components and infrastructure of the architecture.
#4 [S,pwmcore] code reviews
#5 policeman — ensure implementation follows firmwide standards. But I feel this is often impractical under tight deadlines. Wild west is the norm under tight deadlines
#6 whip up interfacing code to downstream (upstream too?) systems.
# buildmaster
# automated blackbox testing

Above are based on a well-written job spec — a perspective from the senior management. Below are a few more job duties I have observed

[S=Sundip of Stirt Risk]

# [S, pwmcore] go-to person and problem solver on just about anything technical, but (bandwidth of this person is limited so) particularly the components that’s not owned by any single application.
# automated test framework
# automated build and deploy
# [SY] represent the department and interface with firmwide architect councils on decisions affecting the entire firm
# [SY] long-term planning; long-term bets; monitor long-term trends
# evaluate new “not-invented-here” software products proposed for adoption by application teams

The same job spec also said “bring a strong sense of ownership and must be driven to deliver tasks to completion”. So this is a doer not talker role. I guess more problem solving, getting-things-done, less (but still a great deal of) presentation, persuasion, explaining.

How about debugging? A software architect (see other posts) must be quick with debuggers etc.

##shallow technical xp is uaually enough for GTD

(See also post on theory^how-to explaining threading and eclipse.)

Here’s a list of sexy technical skills hiring managers look for, but when you spend a few months in it you realize there’s no real *accumulation* or depth in it. 1 month’s systematic learning is enough then you need to “meet and greet” the creatures in it. That’s how you become productive with each tool.
– spring, hibernate
– RV
– gemfire
– perl, python because most places don’t use advanced features
– web service

? threading requires some theoretical and analytical skill.
? jvm tuning may require accumulation but needed in only low-latency systems

In contrast,
+ DB tuning requires accumulation
+ c++ requires accumulation
+ c#
+ swing, wpf
+ unix has depth but no job requires it
+ reflection, but no job spec mention it
+ eclipse has, but few job specs mention that

## know what “nice” solutions work only on paper

We are talking about real technical value-adding, not interviews. Context — non-trivial trading system design. Team of 5 – 20 developers.

If you can get at least one credible [1] dose of non-trivial first-hand experience in pricing / booking / real time mark2market -> real time pnl -> real time VaR / market data feed / execution … then you can give real inputs to design discussions. Real inputs, not the typical speculative inputs from an “armchair quarterback”. In other words, if you have done it, then you can speak from experience. Speculators can’t be sure if their solutions will work. If it’s a top brain, she can be fairly confident, but actual development invariably requires many implementation decision. Even for a genius, It’s impossible to make an educated guess at every juncture and be right every time.

Most of the tools are made by “strangers”. No one can be sure about the hidden limitations esp. at integration time. Some experience is better than no experience.

Here are a small sample of those tools that came to mind. These aren’t trivial tools, so real experience will teach you what to *avoid*. In short, every solutions works on paper, but Speculators don’t know what don’t work.

  • temp queue
  • lock free
  • rv
  • cache triggers and gemfire updaters
  • gemfire
  • camel
  • large scale rmi
  • large scale web service
  • DB-polling as a notification framework

(I have a fundamental bias for time-honored tools like MOM, SQL, RMI..)

[1] need to be a *mainstream* system, not a personal trading system. Each trading system could have quite different requirements, so ideally you want to sample a good variety of, say, pricing engines.

##java value-add` GTD/KPI skill when u join a trading team

What skills make you a more resourceful and *productive* developer to a fast-paced trading system? These usually have little to do with arch design. Almost never asked in interviews, ironically.

See also my post on 2 types of tools.

1) IDE remote debugging
** breakpoint on exception
** conditional breakpoint
** compile with local vars
2) IDE refactor — rename, search

) reflection, dynamic proxy
) profiling
) jconsole, JMX
) requesting Runtime.gc()
) generics for algorithm abstraction but this tends to be too complex
) code gen? cache triggers

Threading? nope it seldom adds too much biz value