val@pattern_recognition imt reusable_technique

Reusable coding techniques include my classic generators, DP, augmented trees, array-element swap, bigO tricks like hash tables and medium-of-3 pivot partitioning.

  • One of the best examples of reusable technique — generate “paths/splits” without duplicates. Usually tough.

More useful than “reusable techniques” are pattern-recognition insight into the structure and constraints in a given coding problem. Without these insights, the problem is /impenetrable/intractable/.

Often we need a worst input to highlight the the hidden constraint. The worst input can sometimes quickly eliminate many wrong pathways through the forest and therefore help us see the right pathway.

However, in some context, a bigO trick could wipe out the competition, so much so that pattern recognition, however smart, doesn’t matter.

 

Advertisements

## Y revisit old algo Qn #Kyle

I find it hard to be interested in previously solved problems. I think many people (not only those in a hurry) simply skip such problems. However, we are not so “good at” previously solved problems actually.

  • Reason – there are often insights and repeating patterns in an old problem, that require slow and repeated digestion. Solving it quickly is usually not enough.
  • reason — our solutions may not be optimal.
  • reason — there are very smart solutions out there we don’t know.
  • reason — we forget.
  • reason — with a simple tweak the problem can become a new problem. How well and how fast we can solve the modified problem depends on our insight into the original problem.

Therefore, I feel folks tend to trivialize some well-known problems and fail to learn enough from them.

Well-known doesn’t mean well-understood.
Well-known doesn’t mean simple.

Revisiting old problems is boring to many programmers, but can often feels easier than tackling new problems. I don’t get tired easily when revisiting old problems. If you are like me, then it could be more beneficial than tackling new problems.

However, my visible progress is much slower in terms of #problems solved per week. I think Kyle also pointed out something related to this.

bigO CIV: clarify_requirement means..worst_data

———- Forwarded message ———
Date: Tue, 9 Jul 2019 at 23:10

Big-O means “worst” case complexity. We need to be doubly sure what “worst” case data look like. (Note it doesn’t mean ridiculously rare data like integer overflow.)

On high-end coding interviews, these “worst data set” frequently shed light on the structure of the original problem, and often point to the correct direction for an efficient solution. (Note a nice-looking solution is not efficient if it exhibits poor time complexity in the face of a worst yet realistic data set. Such a system may not scale well in practice, given its Achilles’ heel.)

If interviewer wants a solution optimized for bigO time complexity, you can waste precious time tackling the wrong problem. The wrong problem is the problem defined by your idea of “worst data set”, but the real worst data set is very different.

Sometimes, the worst data is _subtly_ different but it points to a different algorithm direction. It may point to an iterative solution rather than recursion, for example.

The original interview question may not be that hard iFF we identify the worst data set early on, but sadly, we run out of time.

For some tough problems, the first (sometimes the only) challenge is quickly identifying worst data set. Interviewer always gives you the simple data set, NEVER the worst data set. It’s your job to identify the worst data set… often the key insight into the problem.

It’s no exaggeration to say — identifying the worst data set early or too late can make-or-break your chance at this employer. You may kiss good-bye to this job opportunity exactly because you are too slow to identify the worst data set. I know what this means. Other candidates probably got shot by this arrow on the heel, without knowing what happened.

Time is ticking as soon as the problem is presented to you. If interviewer says time is not a factor… take your time.. we want quality and thought process … I just ignore it. Our speed is always part of assessment. Therefore, a very common failure scenario is —

.. tractable problem but candidate runs out of time after wasting time on preliminaries like using incorrect worst data set to reason about the (wrong) problem.

demanding mental power: localSys imt QQ

I told grandpa that I have noticed signs that my mental power is declining, mostly in localSys and local codebase absorbency. However, localSys has always been a weakness in me. The decline, if any, is very gradual.

QQ learning is still in good progress. Experience, accumulation, deepening.. thick->thin.

Note A.Brooks’ article talks about creativity, innovation, analytical … My issue is mostly memory capacity and speed. Recall the Indeed interview.

max salary: simple game-plan

The strategy — “Whether I ask for a big base or modest base, there’s a chance I may have problem with manager expectation. So let’s just go for the max salary and forget about learning/tsn

    • algo trading? tend to pay a premium, but I wonder how they assess my trec.
    • java/c++ combo role? will Not pay lower
    • Some quant skill? tend to pay a premium
    • If a HFT shop makes a real offer at S$150k base I will decline — no real upside for me. Similarly, if a quant dev job pays $170k base I will decline — the promised accu (across jobs) is a big mirage. Accu can happen within a single job, but so is the technical accu on a single job.

Max-salary game plan must not ignore :

  • correlation between salary and expectation — as observed in some past jobs but not in every lucrative role. My Barclays and 95G roles were great.
  • the stigma, damagedGoods and high expectations in Stirt and Macq…. Ashish’s view — just earn the money for 6 months and leave if not happy.
  • commute
  • reputation risk at the major banks.

Am i still a survivor? I would say YES in OC and GS, and yes in Macq based on the internal transfer offer.

Mithun suggested — Are we traumatized/scarred and fixated on the stigma? I said the same to Deepak CM.

## G3 uncertainties for 10Y: #Demand^otherCandd

Let’s be specific — on the competitive landscape over next 10Y, what specific factors are we trying to predict

  • G3: predicting my healthy, energy, motivation, absorbency, “retention” (MSFM…), coding drill visible-progress ..
    • I have 3x more insight on this factor than other factors
  • G5: predicting strength of competing candd (i.e. candidates), often younger
    • one note — my accumulated theoretical and low-level QQ strength will continue to be rare among the young competitors
  • G9: predicting market demand evolution, including churn. Coding IV is the prime example.
    • Note this evolution is not so fast. Read my analysis of MOM and RDBMS

Since 2017, I have tentatively started to shift more focus towards west coast. As a result, the Competition and Demand factors felt too fast-changing, which makes my painstaking analysis look naive and futile.

Critical thinking needed — Suppose I keep my focus on ibanks. In such a case, my analysis and prediction is actually relevant, even valuable. The Competition/Demand factors have not changed drastically.

I’m pretty good at this kind of competitive landscape analysis , but It’s truly tough to predict any of these 3 factors. However, as Warren Buffett (?) observed, those who decide to give up and submit to mindless investing is likely (not guaranteed) to lose out to those who try and analyze/predict.

 

## good-looking designs/ideas #white elephant

I think a true architect knows the difference. The best design is often not so good-looking and hopelessly outdated.

Not all of them are classified “white elephants”. Not all of them are “designs”.

  1. lock-free — worthwhile?
  2. multi-threading — not always significantly faster than multi-Processing. I find single-threaded mode fastest and cleanest
  3. sharedMem — not always significantly faster than sockets. I briefly discussed these two choices with my friend Deepak M, in the parser/rebus context. I feel it may not be faster.
    1. other fancy IPC techniques? I am most familiar with sockets …
  4. java generic module (beyond collections) — look impressive, can be hard to maintain but doesn’t buy us much
  5. converting java system to c++ — not always brings significant performance gains
  6. forward() and move() instead of cloning
  7. hash-table — not always faster than RBTree
  8.  noSQL — not always significantly faster than REBMS with lots of RAM.  I find rdbms much more reliable and well understood. The indices, temp tables, joins, column constraints, triggers, stored procs add lots of practical value that can dramatically simplify the main application. I understand the limitations of rdbms, but most of my data stores are not so big.
  9. RPC and web services? Probably necessary, but I still don’t know how reliable they are
  10. thick client? I still feel web UI is simplest

     

[19] problems+!published solutions: better4me

Nowadays I feel problems without published solutions, without extensive test cases are better for me, since I don’t worry about … wipe-out, like bloodshed.

-> better avoid Leetcode. Careercup and bbg codecon are fine. Hackerrank?

I used to worry about not knowing correct solutions -> wasting my time. Now I know from extensive experience that coming up with my homemade solutions is good enough and rewarding, even if incorrect/incomplete.

I don’t always need or want to know the correct solution, not every time.

Further, I often wish there’s no known solution, since they always wipe out my satisfaction and self-esteem.

 

##[15] 20 low-level IV topics : c++imt Java

This is collected after i stopped active interviewing in Aug 2012.

Low-level domains are my strength, proven again in 2017-2018 interviews. Beside the obvious domains — threading,  data structures, c++/java/c#..

  • exception in c++/java
  • concurrent singleton
  • GC
  • big-O analysis in west coast interviews
  • linux/compiler/C++ interviews are even more low-level than java. Despite serious “weak-joints”[1], I generally excel at the lower-level
    • shared mem — pointer therein!
    • sockets .. like kernel bypass
    • unix signals
    • cpu cache – instruction and data
    • inline
    • rvr, the most important c++0x feature, is very low-level
    • big-4
    • pbclone^pbref^pbptr, slicing
    • undefined behavior – usually are low level
    • internals of deque, vector/hashtable reallocation…
    • smart pointer internals
    • reinterpet_cast vs move()
    • pass a functor to a thread
    • [1] See items of the same color
  • —-On the other hand, interviewers don’t bother to go low level on …
  • thread pool? somehow never low-level
  • Unix commands, python? never low level
  • (noSQL and) SQL? no
  • FIX? not so low-level
  • SOA, MOM
  • asynch designs
  • design patterns

long term planning can be demoralizing

My father often tells me I plan ahead too much…

Q: where will I be, what job will I have 5 years from now?

Such questions can be demoralizing and sometimes can dampen a precious spirit of optimism. I sometimes perform better by focusing on here and now.

I think the reality may be quite bland and uninspiring — same job, with declining income, not much “offensive” to mount …

## HFT c++knowhow is distinct like .. swing

I figured out long ago that java swing is a distinct skill, distinct from mainstream java.

Now I feel HFT c++ is also such a skill — distinct from mainstream c++. Interviewers tend to ask questions (almost) irrelevant to mainstream c++. Shanyou said focus is on system-level not application-level. I feel superficial knowledge is often enough.

  1. [u] avoid “virtual” for latency
  2. CPU cache management
  3. translation lookaside buffer
  4. [P] kernel bypass
  5. [uP] socket tuning
  6. [P] memory and socket performance monitoring
  7. shared memory
  8. [u] placement new and custom allocator
  9. system calls
  10. [u] in-lining
  11. [u = used in my systems to some extent]
  12. [P = no programming, pure system knowledge]

–These topics are relevant to mainstream c++ but more relevant to HFT

  • setjmp
  • IPC mutex
  • reference counting

c++QnA interviews(HFT): !! pickier than Facebook

I feel my HFT interviews are very picky on low-level kernel or compiler optimizations or network card engineering, but such QQ knowledge is not widely needed.

Don’t feel inferior to them.

  • Q: what if a HFT programmer goes to a Facebook interview?
  • Q: what if a coding contest champion from Facebook goes to an HFT interview?
  • Q: what if these guys go to a tough logic or probability quiz?

 

lockfree usage in high speed trading system@@

Hi Deepak,

Further to our conversation, do high frequency shops use lock-free? I would guess that the most demanding high-speeding data processing system (such as the matching engine in an exchange) are single-threaded, rather than lock-free multithreaded.

I hope to hear a typical high-speed data processing system that has lots of shared mutable data. I have not found one.

· Order matching

· Airline booking

· Vote counting in real time?

If 99% of the data is not shared mutable, then single-threaded design is probably better.

· I feel one reason is complexity vs simplicity. Lock-free designs can be very tricky, according to experts.

· Another reason is performance. Lock-free is, in my view, definitely slower than single-threaded. The memory fence required on those atomic objects impeded compiler optimizations.

· Most important reason, in my view — it’s always possible to achieve the same functionality without using locks or lock-free designs. Single-threaded designs are possible, simpler and faster.

If we have 64 cores, just run 64 processes, each single-threaded. However, in reality these systems often do have some shared mutable data between two threads. There are various solutions I’m not qualified to compare and illustrate. These solutions could use lock-free or locks.

peers’priority^%%priority: G5 #income+..

In a nut shell, Some of my peers’ priorities I have decidedly given up, and some of my priorities are fairly unique to me.

Actually I only spoke to a small number of (like 10) peers, mostly Chinese and Indian. There are big differences among their views. However, here’s a grossly oversimplified sample of “their” priorities.

  • theirs — top school district
  • theirs — move up (long-term career growth) in a good firm?
    • build long-term relationships with big bosses
  • theirs — green card
  • theirs — early retirement
  • ———-
  • mine — diversify (instead of deepen or stack-up) and avoid stagnation
  • mine — stay technical and marketable, till 70
  • mine — multiple properties and passive income
  • mine — shorter commute
  • mine — smaller home price tag

 

MOM+threading Unwelcome ] low latency@@ #FIX/socket

Piroz told me that trading IT job interviews tend to emphasize multi-threading and MOM. Some use SQL too. I now feel all of these are unwelcome in low latency trading.

A) MOM – see also HFT mktData redistribution via MOMFor order processing, FIX is the standard. FIX can use MOM as transport, but not popular and unfamiliar to me.

B) threading – Single-Threaded-Mode is generally the fastest in theory and in practice. (I only have a small observed sample size.) I feel the fastest trading engines are STM. No shared mutable. Nsdq new platform (in java) is STM

MT is OK if they don’t compete for resources like CPU, I/O or locks. Compared to STM, most lockfree systems introduce latency like retries, and additional memory barrier. By default compiler optimization doesn’t need such memory barriers.

C) SQL – as stated elsewhere, flat files are much faster than relational DB. How about in-memory relational DB?

Rebus, the order book engine, is in-memory.

Y more threads !! help`throughput if I/O bound

To keep things more concrete. You can think of the output interface in the I/O.

The paradox — given an I/O bound busy server, the conventional wisdom says more thread could increase CPU utilization [1]. However, the work queue for CPU gets quickly /drained/, whereas the I/O queue is constantly full, as the I/O subsystem is working at full capacity.

[1] In a CPU bound server, adding 20 threads will likely create 20 idle, starved new threads!

Holy Grail is simultaneous saturation. Suggestion: “steal” a cpu core from this engine and use it for unrelated tasks. Additional threads or processes basically achieve that purpose. In other words, the cpu cores aren’t dedicated to this purpose.

Assumption — adding more I/O hardware is not possible. (Instead, scaling out to more nodes could help.)

If the CPU cores are dedicated, then there’s no way to improve throughput without adding more I/O capacity. At a high level, I clearly see too much CPU /overcapacity/.

in-depth^cursory study@advanced program`topics

I have seen both, and tried both.

AA) A minority of developers spend the time studying and understanding the important details of threading, generics, templates … before writing code.

BB) Majority of developers don’t have the time, vision or patience, and only learn the minimum to get the (sloppy) job done. They copy-paste sample code and rely on testing, but the test cases are limited to realistic UAT test cases and will not cover edge cases that have yet to occur. For concurrency designs, such tests are not good enough.

AA is often seen as unnecessary except:
* Interviewers often go in-depth
* In a product vendor rather than an in-house application team
* If requirement is unusual (eg: B2BTradingEngine), then BB won’t even pass basic developer self-tests. Sample code may not be available — see my post http://wp.me/p74oew-1oI. So developers must bite the bullet and take AA approach. This is my sweet spot. See
https://bintanvictor.wordpress.com/2016/10/31/strength-gtd-with-non-trivial-researchinnovation/ https://bintanvictor.wordpress.com/2016/03/30/theoretical-complexity-strength/
https://bintanvictor.wordpress.com/2016/03/03/tech-specializations-no-such-luxury-in-singapore/

On Wall St projects, I learned to abandon AA and adopt BB.

On Wall St interviews, I still adopt AA.

On West Coast, I think AA is more needed.

how likely are two threads sharing a CHM were to clash@@

I would also like to point out the java ConcurrentHashMap lets two threads (possibly a writer thread and a reader thread) access the shared data structure concurrently, probably without locking or CAS, when the two threads happen to access two distinct segments.

Note there can be a large number of segments, up to a few hundred at least, so the chance of two threads hitting distinct segments is very high (99.999% chance for a 333-segments map). Therefore, contrary to what Jun said, for reader/writer threads to access the same data structure,  we don’t always need locking in every read and write access.

concurrencyLevel – the estimated number of concurrently updating threads. The implementation may use this value as a sizing hint.

c++IV: importance: knowledge imt dev xp

1) Many hard-core tech interviewers (Yaakov, Jump, 3Arrows, Bbg, nQuants …) often asked me to explain a language feature, then drill in to see if I really do understand the key ideas, including the rationale, motivation and history. This knowledge is believe to … /separate the wheat from the chaff/

This knowledge can’t be acquired simply by coding. In fact, a productive coder often lacks such knowledge since it’s usually unnecessary theoretical knowledge.

2) West Coast always drills in on algo (+ data structure). No way to pick up this skill in projects…

1+2 —> many interviewers truly believe a deep thinker will always learn faster and design better.

%% priorities in a take-home cod`IV

A lot of times the requirements are way too numerous or stringent, given the time limit. Must give up some. Here are my priorities:

  1.  basic functionality. The essential problem should be solved, or shown to be solvable.
    • pass the essential test cases only
  2. On that foundation, add as many big features (in the problem statement) as I can within the time limit. Leave out the small features.
  3. simplify. Simplicity is the first element of clean , readable code.

I guess some hiring managers are very particular about code quality and readability. I feel it’s like beauty contest. I would give up in those cases. I was told JPM follows [[Clean Code]] when judging/scoring/ranking code submitted.

Need to ensure file timestamps are all within a short window. Prefer to use fewer files. Java solutions require more files :(. If I claim 2H, then better leave some rough edges in the code like input validations.

python shortcomings for large enterprise app

– performance, efficiency. Compare — C statements translate to brief machine codes.
-? scalability
– threading – GIL, no precise control
– industrial strength commercial support. Only c++/java/c# has big tech powerhouses. Linux is widely adopted by enterprises, but is an exception that proves the rule.
– ? commercial auxiliary software to support the ecosystem, including IDE, debuggers, jars, code generators, profilers, … I feel python has these.
-? most enterprise apps are now in java or c# as these are proven, in terms of performance, resource scalability, commercial support etc.

plowback for zbs(+GTD)@@

For many years I was deeply convinced and motivated by the “Shanyou” notion that I ought to “plowback” for zbs and, to a lesser extent, GTD, not just aim at IV as the Mithuns would.

After working for 20 years, I now believe ANY tech knowledge, accu, deepening/stack-up, sustained focus … has extremely low leverage and basically worthless if not relevant to IV

  • GTD skills? dominated by localSys. Tool knowledge can help but localSys is 10x more important.
    • localSys xx is almost always cursory, rather than in-depth since unnecessary digging (pwm-comm bottom-up xx plan) is invariably unsustainable — The local systems are patchworks , destined for rewrite, illustrating few best practices
  • BestPractice design? Almost never mattered in my career. After EMPWorld, I have never risen to decision-maker level. This zbs is overrated by 10 times.
  • BestPractice implementation? What’s “best” is mostly a matter of taste (personal preference) of the manager
  • zbs? Consider the most practical books like [[Pearls]] — the classics. If not relevant to IV then this zbs is only for self-satisfaction

This is one reason why my reinforcement loop completely broke down in Singapore.

… In contrast, my NY years were filled with joys of self improvement, driven by … interviews.

arch xp is overrated as KPI #instrumentation, tools…

See also other blog posts —

// “in the initial dev stage, instrumentation is my #1 design goal”.
// http://bigblog.tanbin.com/2010/09/2-important-categories-of-software.html

Going out on a limb, I’d say that in high-pace projects GTD (Getting Things Done) is more important than many things which are supposed to be more important.

GTD is more important than … integrating deep domain knowledge insight into the system. I think such domain knowledge can sometimes be valuable. It can help dev team avoid wasting time on “the wrong things”.

GTD is more important than … quality of code. For a few months at least, the only guy looking at the code is the original author.

GTD is more important than … “quality of design”, which is seldom well-defined.

GTD is more important than … being nice to others. Many times other people can tolerate a bit of personality if you can GTD. However, you must not offend the people who matter.

GTD is more important than … impressive presentations that address many real customer pains. A user may die for a guy who “understands my pains”, but when it is time to deliver, selling/understanding is an irrelevant distraction.

GTD is more important than … in-depth knowledge of the language or of a software platform/product. Such knowledge is more important than GTD during interviews though. Threading, data structure…

GTD is more important than … uncovering the root casue of (intermittent) problems/errors/surprises — the Toyota drill-down root-cause investigation. Regex engine creator may need to fully characterise every unexpected behavior; AutomateTellerMachine error probably deserve a drill-down investigation but in enterprise apps drill-down investigation simply takes too much time. We developers are paid to build a usable tool, not a fully-understood tool. Live with ambiguity and move on.

GTD is more important than … adding an important test to the automated test suite. Sure that mistake we just found may happen again so we really appreciate adding that test, but in reality, most high-pace environments don’t have an automated test suite. If we are lucky enough to have documented manual test plan, then yes add it, but such a plan seldom is comprehensive, so after a while people won’t follow it religiously. So in reality we just hope developers learn the lesson and avoid making the same mistake. If they do, then we just need someone who can GTD. Any system to incorporate such heard-learned lesson is likely an imperfect system, and whoever investing in such a system is often wasting his/her time.  If a GTD guy doesn’t bother with that system, he will still be respected and loved by manager and users. Basically, any long-term investment is unappreciated. GTD is all about short-term results. This is the reality of quality control in fast-paced teams.

GTD is more important than … adding meaningful error messages. Anyone debugging the system would love the guy who added the meaningful error message, but he is an unsung hero. Manager love the guy who GTD fast. Code quality is invisible and therefore ignored and unrewarded.

To achieve GTD, you must solve tech problems. Tech problems could (occasionally) call for architectural perspective, but less than tool knowledge, debugging experience, or low-level system insight.

In defense of architects, architectural track record is more important in sales contexts, including selling to internal clients and business users.

I should probably get input from Raja, Xiao An, Yi Hai …

##what bad things could happen to a thread in a pool

Do you ever wonder why threads in a thread pool can go bad and become unusable? Weblogic has long realized that. Hard to diagnose.

However, I don’t believe “voodoo” exists in a threading system. A stuck thread must be stuck in a useless loop or waiting for cpu, IO or memory. It won’t be stuck for “no reason”. Pretend to be a spy looking at the program counter inside any thread — why not moving? There are only a limited (not UNlimited) number of reasons. Here are various bad things that can happen to an innocent thread.

– in java, any uncaught exception can fall on a thread (like a rotten apple:). These are extremely common so a thread in a pool should handle these /gracefully/.

– a thread may be seen doing useful work in normal conditions, but in an edge case get into an endless loop but doing no useful work — just wasting cpu
** Note endless loop is fine if the thread is doing useful work.

– divide by zero (perhaps terminating the thread)
– access wrong memory location and trigger a segfault. In C++ there are many pointer/memory-related errors.
– java thread can run out of memory too
– in java, a thread can be stopped due to the deprecated but supported stop() and suspend() methods. Such a thread might emerge in bad shape.
– starvation due to low thread priority.
– stack overflow, due to deep recursion (like a recursive equals() or hashCode()).

– thread could get stuck in a normal blocking operation like accept(), read(), write(), lock()/synchronized keyword, wait(), join(), long sleep. Many of these aren’t even interruptible.
** IO (disk or network) wait is notorious culprit. In this state, everything including those “kill -9” signals are ignored (see [[Unix Power Tools]]). Entire process may appear stuck in mud.
** my java thread book also says a thread blocked on a socket may not respond to interrupt.
– deadlock

[12]call`same method@unrelated types: template outshines java

Suppose pure abstract class Animal has a die() method, and so does Project, Product, Plant and Planet, but they don’t share a base class. How would you write a reusable function that invokes this method on a generic input object, whose type could be any of them?

Java can’t do this. In C++ you create

template<typename T>  f1(T input){ input.die(); }

If you pass an int into f1(), then you get compile time error. Probably a linker error. Is this SFINAE ? I doubt it.

STL algorithms routinely take an iterator argument and then call operator>() on the iterator. Now, this operator is undefined for a lot of iterator INSTANCES. I think only RandomAccessIterator category supports it.

Q: So how does STL make sure you don’t pass an object of ForwardInterator category into such a function?
A: use the template parameter type name (dummy type name) as a hint. Instead of the customary “T”, they put a suggestive dummy type name like “BidirectionayInterator” or “InputIterator”. If you ignore the hint you get compile-time error.

Now we understand that STL iterators have no inheritance hierarchy, but “stratification” among them.

##Just what are(not) part of Architecture (too-late-to-change)

Dogfight – When you develop a dogfight jet over 30 years, you stick to your initial design. You try to build on the strengths of your design and protect the weaknesses. You don’t suddenly abandon your design and adopt your rival’s design because you would be years behind on that path. Software architecture is similar.

Anyone with a few years experience can specify an architecture in some details, but those decisions are flexible and could be modified later on, so they aren’t make-or-break decisions, though they could waste time.

Q1: what Features define a given architecture? I’m a fellow architect. If you must describe in 10 phrases your system architecture and provide maximum Insight, which 10 aspects would you focus on?

Q2: (another side of the same coin) What technical Decisions must be made early when creating a system from scratch?
Q2b: in other words, what design decisions will be expensive to change later? Why expensive?

Disclaimer — In this context, “later” means somewhere during those pre-UAT phases, not LATE into the release process. Obviously after release any config/DB/code change requires regression test and impact analysis, but that’s not our topic here.

A colleague of mine liked to discuss in terms of layering. In OO architecture, A layer consists of a bunch of similar classes (or interfaces) dedicated to a logically coherent high-level task. 2 interacting layers can exist in the same OS process or (less common but very critical) across processes. Intra-process layering basically means the 2 interacting classes (belonging to 2 layers) have well-defined responsibilities and a well-defined contract between, where one is always a Service-provider (“dependency”) to the service-consumer. Though popular among architects, I’m not sure these are the most important architectural features.

Here are my answers to the question “Is the decision costly to change later in SDLC?”

? anything in the wsdl/schema of a web service? yes if i’m not the only developer of this web service’s clients. A change would need these fellow programmers to change their code and test, test, test
? fundamental data constraints like “if this is positive then that must be non-null”? Yes because other developers would assume that and design code flows accordingly.
? database schema, the ER between a bunch of tables? Yes since it usually requires code change “upstairs”. DB schema is usually a Foundation layer, often designed up front.
? choose a DB vendor? yes if you happen to have truly skillful database developers in the team. They will use proprietary vendor extensions behind the manager’s back when the temptation is too strong. Such non-portable business logic will be expensive to re-implement in another database.
? use MOM or RPC-style communication? yes
? Serialization format? yes. “Later” means implementation effort wasted
? XML or binary or free text format? yes probably
? make a bunch of objects immutable/read-only/const? yes. Adding this feature later can be hard. Look into C++ literature.
? shared vs unshared cache? yes. effort is non-trivial
? What logic to put into SQL vs java/C#? sometimes No. You can change things later on, but usually wasting sizable implementation effort
? Choose programming language? Yes. Implementation effort wasted if switching late in the game.
? make a batch task re-runnable? yes. Adding this feature later on can be hard

? MVC (the highest-level design pattern there is)? well… yes, though you can adjust this design later on
? where to implement authentication, entitlement? Many say yes, but I feel no. Make the damn system work first, then worry about securing it. On fast-paced trading floor, Business usually can tolerate this feature being not so perfect. Business and upper management worry more about ….. You got it — Functionality.

? Which commercial(or free) product to choose? sometimes NO. You can use mysql or activemq and then change to commercial products. In fast projects, We can often delay this decision. Start coding without this particular commercial product.

? make a module thread safe? no, even though adding this feature can be hard
? how many staging tables? no
? choose a decorator/factor/singleton (or any of the classic) design pattern? no. They are usually too  low-level to be among the “10 architectural features”. I don’t know why a lot of architect interview questions hit this area.

upstream, knowledge-intensive, high-margin sectors for S’pore

Upstream/strategic, knowledge-intensive, high-margin, high-barrier sectors for Singapore

Singapore is traditionally not a high-tech economy…. To compete, Singapore must seek greener pastures and keep a steady foothold therein. This is what every advanced economy does.

Role models? Switzerland, Hongkong, Taiwan, Silicon Valley, Boston

Entry barriers due to large capital, specialized know-how, talent shortage… Singapore EDB likes a sector with such barriers.

RnD – upstream, knowledge-intensive and entry barrier.

— Now the sectors —
$ finance — #1 most special (“sacred”) sector. One of the pillars of the economy but finance is first among equals [1]. Unlike other pillars, finance is part of the life support of this nation. Singapore gov needs long term strategies to protect (its currency and) financial health, even survival, of the nation. It grows a deep foreign reserve and invests it long term.

Not sure if finance is the most high-margin sector. Commercial banking is lower margin; AssetMgmt, security trading and investment banking create few jobs.

[1] if you shortlist the pillar industries of { oil, logistics}, where Singapore competes successfully on the global arena.

$ oil — perhaps the #2 sacred sector.
$ medical
($ life science? not sure what this includes)
$ electronics
$$ semiconductor fabrication – at the higher end of electronic sector
$$ IC design, esp. microprocessor design – still higher margin

$ telecom equipment and telecom operator — 2 big sectors by revenue
$ enterprise IT solutions, including software production and distribution
$ consumer IT product creation. Not a big sector, but look at Creative Lab
$ aviation — at the higher end of the logistic sector. Servicing, component design, research, airport ..
$ higher education and training

————
Now a small sample of the opposite list. Many traditional sectors don’t meet all the criteria in the title but do support a lot of jobs for Singaporeans —
– logistics?? Singapore’s traditional bread-and–butter economic contributor (commerce is another), but margin is deteriorating.
— distribution, warehousing
– commerce
— traditional retail – i.e. consumer shopping
– construction
– tourism and hospitality, casino
– entertainment – online/electronic gaming, music and film
– marketing and advertising, media, publishing, exhibition, conferencing

Many of these above sectors are domestic. They don’t directly contribute to the Singapore Team on the international front — contrast electronics, aviation, medical, tourism… Every advanced economy must show competitiveness and high value-add on a number of major global markets.

threading features – c# improving over java

C# “adapted” many java designs, but by different magnitudes.
+ Where c# creators found _robust_ designs in java, they made minimal adjustments — like in threading. Basically wholesale borrow.
+ Where they found problematic/controversial designs, they made bigger adaptations.

Java’s *language* support for threading is rather clean and comes in only 3 foundation building blocks – creation, locking and signal. So in these 3 areas what are the adjustments by c# creators?

– signaling (wait/notify) – identical
– locking – identical
** synchronized keyword replaced by lock keyword
** static synchronized similarly adapted

– thread creation – Fundamentally identical. Superficial changes
** delegate to replace Runnable interface

Beyond these 3, java language includes some important (but less central) features, largely borrowed wholesale by C# creators.
– join
– interrupt
– sleep

Other java threading support is largely “superstructure”.
+ read-write lock
+ thread pool
+ callable tasks and future results — http://bigblog.tanbin.com/2010/09/callable-tasks-future-results.html

4 distinct sub-domains of financial math

Even though many real world scenarios involve more than a single topic below, it’s still practically useful if you understand one of the topics well.

* bond math including IRS, FRA…
* option
* VAR — correlation, simulation
* credit risk — essential to commercial banks, consumers, corporate borrowers… though I don’t know how much math

Every single topic is important to risk management (like the OC quant team); Half the topics are important to pre-trade pricing. For example, a lot of bond math (duration, OAS, term structure) is not that critical to pre-trade quote pricing.

All derivatives always involve more math than the underlying instrument do, partly because of the expiration built into every derivative instrument. Among derivatives, option math complexity tends to exceed swaps, futures, and forwards.

y trading IT pay so high in US and also ] sg

I asked a few friends —

Q: finance IT pays 2 to 3 times the salary compared to other sectors. Initially i thought the person requirement must be much higher, i.e. the average non-finance developer probably can’t easily handle the job. To my dismay, just about any java developer can handle the job. You can learn multi-threading, MOM, … on the job. So why do they keep paying such high salaries?

A: profit margin is much higher in finance. Having qualified developers means faster time to market and more profit. Qualified means track record.
A: trading systems prefer a elite team of experienced guys, not large army of rookies. Average salary is therefore higher. Root cause is time-to-market. Tiny team of elites deliver faster.

A: At the senior developer level, trading system experience (track record) is very rare, partly because of the low head count in a lot of successful desks.

A: manager must spend the budget or get a reduced budget next year.
A: employer must pay high enough to keep the talent

A: Will (recruiter) said trading sys architect talent is scarce in S’pore. Globally, Battle tested architects are rare. Battle tested on trading system is ..rarer. Trading systems have additional characteristics such as (but not limited to) extreme time-to-market, extreme quick-and-dirty, instrumentation, manual override, “explain what happened”….

A: Y. Lin feels many of those high-end jobs are very specialized, with very few qualified local candidates

A: MS commodity trading interviewer said most local candidates don’t have the design experience — they only assist London or NY

A: Raymond feels some sectors (like web, or php) are over-supplied due to inflow of immigrants. System requirement is fairly simple compared to …. say Oracle or SAP — mostly used by large, well-funded enterprises. System is expensive and mission critical.

A: Raymond T also feels an IT service provider is bound by the price clients are willing to pay. Raymond gave examples of construction industry. These clients have a budget and won’t pay extraordinary prices, so the IT guys serving that industry get an average salary. Raymond said finance industry is unique IT serves internal business units and business unit decides how much (possibly huge) budget to allocate to IT. I would add that headcount is much smaller than a team in other industry. So finance IT budget is probably higher, and head count is probably much lower, therefore salary is so high.

A: Raymond T feels in other industries, IT salary is kept “reasonable” by specialized outsourcing suppliers who specialize in certain high-value sectors. In trading, I only know IBM and Sapient have such “practices” and they probably cost a bank more than hiring directly.

y java is dominant in enterprise app

What's so good about OO? Why are the 3 most “relevant” enterprise app dev languages all happen to be OO – java, c# and c++?

Why is google choosing java, c++ and python?

(Though this is not really a typical “enterprise app”) Why is apple choosing to promote a compiled OO language — objective C?

Why is microsoft choosing to promote a compiled OO language more vigorously than VB.net?

But why is facebook (and yahoo?) choosing php?

Before c++ came along, most enterprise apps were developed in c, cobol, fortran…. Experience in the field show that c++ and java require more learning but do offer real benefits. I guess it all boils down to the 3 base OO features of encapsulation, inheritance and polymorphism.

##[11]upstreams(sg gov)for next 10Y #wall St tech

(See also blog post on 2010 GS annual report.) Singapore government is smart to always look out for upstream domains. Similarly, in financial IT, there are some upstream *domains* too.

Upstream: real time risk — upstream in derivative risk … but poor Market Depth
Upstream: grid — Memory virtualization, grid computing
Upstream: Eq(and FX) HFT and algo trading is upstream to other asset classes
Upstream: Option-related (including mtg) analytics tend to be more advanced, and probably upstream. Any derivative with optinality tends to be dominated by vol in terms of pricing complexity.
Upstream: connectivity — collocation, streaming,
Upstream: latency — sub-m, multi-core, non-blocking…
Upstream: voluminous market data processing — storage, distribution…FX and eq option has the highest volume, and often dominates entire infrastructure. Many asset classes are far behind
Upstream: SecDB — firmwide singleton mkt risk engine. Upstream in m-risk technology. Most shops have data silos.

However, upstream technologies often become fads.
– C++? not really upstream. A strong c++ guy isn’t necessarily strong in java or C#.
– web service
– solace-like messaging hardware
– CEP? How much real value does it add?
– rule engines
– MortgageBackedSecurity pricing? Not really upstream, though most complex analytically.
– ETL?
– hibernate?
– python?

## [11] y no dotnet on sell-side server side@@

(A fairly sketchy, limited, amateurish write-up.)
I was recently asked “dotnet has formidable performance and other strengths compared to java, but in trading engines space, why is dotnet making inroads only on the user-interface, never on the server side?

Reason — as an casual observer, I feel Windows was designed as a GUI operating system with a single user at any given time. Later WinNT tried to extend the kernel to support multiple concurrent users. In contrast, Unix/Linux was designed from Day 1 to be multi-user, with the command line as the primary UI. (Personally I used to feel GUI is a distraction to high volume data processing OS designers.) A trading server needs no GUI.

Reason — Java and c/c++ were created on Unix; dotnet runs only on a windowing operating system. I feel web server is a light weight application, so both java and dotnet (and a lot of scripting languages) are up to the job[1], but truly demanding server-side apps need Unix AND java/c++. I guess Windows is catching up. In terms of efficiency, I guess java and c# are comparable and below C++.

Reason — Sell-side trading system is arms race. (Perhaps same among hedge funds.) Banks typically buy expensive servers and hire expensive system engineers, and then try to push the servers to the max. C/C++ makes the most efficient use of system resources, but Java offers many advantages over C++. Since the late 90’s, trading servers have progressively migrated from C++ to Java. Java and C++ are proven on the high-performance server side. Not dotnet.

Reason — I still feel *nix are more stable than Windows under high load. See http://efreedom.com/Question/1-214362/Java-Large-Heap-Sizes. However, I think you can create big clusters of windows servers

Reason — (from a friend) — *nix is considered more secure than windows. A GUI desktop can affect one trader if compromised, but a sell-side trading server affects all the traders from all the institutional and retail clients if compromised. So security risk is more serious on server side than GUI side.

The reasons below are arguments for java over dotnet in general, but don’t really explain why java is NOT on the GUI side and dotnet is still chosen on the GUI side.

Reason — big banks need stronger support than a single vendor company. What if Microsoft makes a mistake and dotnet loses technical direction and momentum? Java and *nix enjoy broader industry support.

[1] unless you are google or facebook, who needed c++ for their demanding requirements.

hard-to-replace talent in financial IT: tech^dnlg^mgmt@@ #GS

What’s valuable and hard to replace in financial IT? Is it

1) technical know-how or
2) management skill or
3) domain knowledge?

I feel managers usually earn more salary. The more hands-off, the more salary. Problem is the political skill required, which I don’t have. Leadership is a combination of technical, domain knowledge and ….what I call people skill.

Technical and domain knowledge is real “knowledge”. These are like history or geography knowledge — if you know it, then everyone agrees you know it. If you don’t know it, you can’t pretend for too long (like Emperor’s new dress). As you can sense, I’m a big fan of tech and domain knowledge.

In GS, tech know-how is considered easily replaced. Find a fast learner. Throw him into the water and he becomes an expert in 6 – 12 months. I took 1.5 years. In other I-banks, tech know-how helps you keep your job. If you are the only system expert then employer won’t touch you.

In GS (my department) , 2) and 3) make a valuable combination. They feel such a person is hard to find. If necessary they would keep such a “manager” and replace all the hands-on developers under him. They believe this manager can then train up a new crop of developers.

major forces shaping future of java (+software industry)

* B — software vendors like Bbbill Gates
* C — open source Cccommunity, a new force significant only in the last 20 years.
* D — Dddownstream industries like Wall Street and web 2.0
* A — Aaaacademic, research including commercial R&D, and hardware innovation. This force represents the pushers of the frontier of technology
* government? not really a major force except in S’pore;). This game is largely market-driven and innovation-driven.

There are non-trivial overlaps and straddles, yet it’s good to keep the big picture simple without being simplistic.