deadly delays@ project^feature levels

insightful article: managing tight deadlines is the background

— deadly delay at feature level, without user impact

This is real stressor primarily because there exist team colleagues who can GTD faster on this same task.

For the serious delay, user impact is … exaggerated! This is true across all my projects — user impact doesn’t really matter. In fact, the feature may not be used much whether you deliver it in high or low quality, within or exceeding budget.

— blatant delay at project level when you are architect / team lead

In theory if you promise to delivery some new feature i.e. green field project, then it can be tough to deliver on time. In reality, project time/budget overrun is very common. You only need good explanation.

Users never care that much about any new system cf current production systems. New systems often hit various teething problems and functionally unreliable for months.

OK , Users don’t really care that much, but there’s visible technology budget impact which make the technology MD look bad. MD must look for an “explanation” and may cut headcount as in StirtRisk and RTS-Trep

Whose fault? Many fingers point at you the architect, but often it’s partial fault of the user-facing manager due to immaturity and overconfidence with timeline estimate.

Delay is mostly architect’s fault if another architect can deliver faster but I can’t imagine two architects working on two comparable projects at the same time. Therefore it’s never “mostly architect’s fault”. In reality, management do apply pressure and you suffer, but it’s really for political reason (like sacrificial lamb or scapegoat)

eg: RTS Trep
eg: PWM Special-Investment-management
eg: Post-Saurabh GMDS — multiple delays and scope reductions

pick java if you aspire 2be arch #py,c#

If you want to be architect, you need to pick some domains.

Compared to python.. c#.. cpp, Java appears to be the #1 best language overall for most enterprise applications.

• Python performance limitations seem to require proprietary extensions. I rarely see pure python server that’s heavy-duty.
• c#is less proven less mature. More importantly it doesn’t work well with the #1 platform — linux.
• cpp is my 2nd pick. Some concerns:
  • much harder to find talents
  • Fewer open-source packages
  • java is one of the cleanest languages. cpp is a blue-collar language, rough around the edges and far more complex.

mainstream^specialist jobs: positioning in Sg+U.S.

I feel this is all about positioning — accumulation, body-building contest…. The high-end jobs are always limited and contested.

• I tend to see algoTrading and quantDev as high-end specialist roles.
• I tend to see app owner or lead developer (a.k.a. architect) in mainstream domains not as “low-level specialist“, because I tend to believe “devil is in the details”. I feel better with the low-level specialist role
• There are also many senior dev roles in mainstream (non-specialist) domains, such as market data, risk,…

In the U.S. there are more specialist type of roles, some even open to contractors.

In Singapore, I sought after HFT and quantDev specialist roles but as I told a Bbg interviewer, these attempts never worked out. The quant related roles offered no accumulation and poor market depth. Therefore, for my next Singapore job I will consider more mainstream.

##[09] ideas2reduce emotional cost@design discussions

* Do not give tech design
* Do not write down details in black and white
* When asked, give minimal details
* If u r reasonably sure, then just build a prototype, but don’t show too early
* Under separate cover, get 2nd opinion on your design from known experts. Establish feasibility
* Get out when debates get sour

 

 

As CTO, I’d favor transparent langs, wary of outside libraries

If I were to propose a system rewrite, or start a new system from scratch without constraints like legacy code, then Transparency (+instrumentation) is my #1 priority.

• c++ is the most opaque. Just look at complex declarations, or linker rules, the ODR…
• I feel more confident debugging java. The JVM is remarkably well-behaving (better than CLR), consistent, well discussed on-line
• key example — the SOAP stub/skeleton hurts transparency, so does the AOP proxies. These semi-transparent proxies are not like regular code you can edit and play with in a sandbox
• windows is more murky than linux
• There are many open-source libraries for java, c++, py etc but many of them affects transparency. I think someone like Piroz may say Spring is a transparent library
• SQL is transparent except performance tuning
• Based on my 1990’s experience, I feel javascript is transparent but I could be wrong.
• I feel py, perl are still more transparent than most compiled languages. They too can become less transparent, when the codebase grows. (In contrast, even small c++ systems can be opaque.)

This letter is about which language, but allow me to digress briefly. For data store and messaging format (both require serialization), I prefer the slightly verbose but hugely transparent solutions, like FIX, CTF, json (xml is too verbose) rather than protobuf. Remember 99% of the sites use only strings, numbers, datetimes, and very rarely involve audio/visual data.

architect’s helicopter view of a system

This (pearl) post is not only describing some role models. This is a career tip, possibly survival tip.

Bottom line — my capabilities (code reading…) is not very high, so as architect/owner of any large codebase I would tend to struggle. There are some ways to survive, but not easy. I would think a high volume, high latency batch+web/quant system in java/SQL/python/javascript would be easier for me and harder for my peers.

Prime eg: Yang. Nikhil calls it “his system knowledge” – Without knowing the source code, Yang would ask sharp questions that baffles the actual coders. If the answer smells bad, then either answer is not exactly truthful, or existing system design is imperfect. Yang is quick to notice “inconsistency with reality”, a very sharp observation. This pattern is critical in troubleshooting and production support as well. I told XR that production support can be challenging and require logical, critical thinking about the systems as a whole.

An __effective__ geek needs more than high-level system knowledge. Somehow, every one of them soon gets down to the low level, unless her role is a purely high-level role … like presales?

Not everyone has the same breadth and width of helicopter view. Often I don’t need it in the first few years. You would think after many years, each would acquire enough low-level and high-level knowledge, but I imagine some individuals would refuse to put in the extra effort, and keep her job by taking care of her own “territory” and never step out.

— my own experience —

At Citi muni, I largely avoided the complexity. I did try and trace through the autoreo codebase but stopped when I was given other projects. In hind sight, I feel it’s too slow and probably not going to work. Mark deMunk pointed out the same.

At 95 Green and Barcap, I was a contractor and didn’t need to step out of my territory. This might be one reason I had an easier job and exceed expectations. See post on “slow decline” and http://tigertanbinpripri.blogspot.com/2015/09/fwdsome-reasons-why-oc-boss-complains.html, both in the pripri blog.

At OC, the Guardian codebase I did take up fully. Quest was given to me after I /fell out/ with boss, so I did enough research to implement any required feature. No extra effort to get a helicopter view.

Stirt was tough. I spent a few months trying to learn some but not all of the nuts and bolts in Quartz. This knowledge is fundamental and crucial to my job and I didn’t have too much time learning the Stirt system, which I did learn but didn’t master. In fact, I asked many good system-level questions about Sprite and Stirt-Risk, but failed to gain Yang’s insight.

 

I can qualify as tech lead on___(system)

Q: what kinda systems can you be an architect or lead developer? See also ##a few projects technically too tough4me

A: In an optimistic mood …

	domain		firm	tech used		cfd	personal advantage
	trade execution		b2b	sql, tibrv		4
	blotter + backend #essential to any		citi	sql		3
	mkt data parser		ICE	socket		3
	bond repricing/risk engine		reo	sql, async		4	bond math, curve-based pricer
	billing, commission		PWM	SQL, batch		5
	STP		none	sql, batch, MOM		3	SQL
	java web …		PWM	sql, batch		5	many
	web …		Catcha, NIE	lamp, javascript		5	many

I can get these systems to work, although

• I’m not the fastest
• i may not use latest, most efficient technology
• reliability and stability ?
  • Look at GMDS .. MMF is an unstable base technology
  • LOok at Gauardian ssh? not a mature , well adopted base technology

(arch) xp I hope to accummulate

Architecture is all about design trade-off. Each time I start on a new technology, I want to grow as a hands-on architect. I hope to gain a bit of insight on the real drawbacks of the most common designs in that technology. I am not satisfied with the glossy brochures. They tend to play up many “shortcomings” that are irrelevant to my projects.

In the Quartz system, I don’t feel I learned what I wanted. I didn’t notice some common design I could *reuse* when I design my next system…

Guardian actually gave me such an opportunity, because it’s so simple, but I didn’t get to play with multiple working designs. The one component that actually worked well is the embedded web server I whipped up.

I foresee that with swing and WPF I would have more opportunities to “play with” various designs and get some insight.

I foresee that with javascript and php I would have that chance too.

Python?

app arch – civil engineer or a salesman, sg^U.S.

I feel in Singapore context, “architect” is more likely to refer to a hands-off role. In the US, I never meet an architect who doesn’t writes code hands-on at least half his time.

A real app architect in finance (different from professional software product vendors) really needs hands-on capabilities. Her output is not just on paper, but in code. If her design (not the document but the “key ideas” implemented) doesn’t work, she must roll up her sleeves and make it work. Most of those ideas are low level, like some serialization or some threading construct, and require just one good developer. In that sense it’s not unlike a library module.

In this context, architect is the name of the technical lead of ANY dev team. Often known as lead developer, in a smaller team.

Any dev team needs a technical leader. Half the time, the team manager or the project manager is technical enough to lead a suppor team but not a dev team,  so an architect is needed.  Often, the architect is the only leader.

The pre-sales architect is very different. Castle in the sand. Imaginary buildings.

Update: I feel in the US I could become a lead developer, once I become familiar with a codebase, but any role that requires a lot of persuasion I’m not too sure. I feel if my technical grasp of everything is higher than the rest then it’s possible. It’s all relative.

Software arch vs enterprise arch

In many companies architects (are perceived to) operate at a high level of abstraction — high-level system modules/interfaces, high-level concepts, high-level decisions. Their audiences include non-technical users, who speak business language. I guess some architects absolutely avoid low-level tech jargon almost like profanity — it’s all about image of an architect. The wrong word on the wrong occasion would creates a permanent bad impression.

To many people, “architecture” implies some network layer, networking protocol, network nodes. These architects might feel less than comfortable if their system has just one machine or just one jvm instance [1], without messaging, database, remote-procedure-call, web service, distributed caching, IPC, shared memory, shared data format (like XML) or communication protocol.

In contrast, software architects can be extremely low-level, too. Their audience are developers. (Examples below include a lot of c/c++…)

– Consider the creator of STL — an architect?
– connectivity engine team lead?
– FIX engine architect?
– Any low-latency trading engine architect?
– Creator of perl/python/boost regex engines?
– All boost libraries I know are low-level. Are the authors architects?
– Tomcat is a single-jvm system. Authors qualify as architects?
– Hibernate isn’t a big system with big modules.  so the authors are low-level architects?
– JNI component designers — single-VM
– java GC innovators
– Creators of any reusable component like a jar, a dll — Just a pluggable module — not even a standalone “server”. Do they qualify as architects?
– The pluggable look-and-feel framework in Swing is designed by someone — an architect?
– spring framework, before all the add-on modules were added
– the tcp/ip/udp software stack was designed by some architect, but that person probably worked mostly on 1 machine, occasionally on 2 machines for testing only.
– GCC creator
– apache creator
– mediawiki creator
– creators of the dotnet CLR and the crown-jewel of Sun, the JVM.

I feel majority of the really important software [2] used in the world today were created by truly powerful developers who focused on a single module in a single system, without messaging, database, remote-procedure-call, web service, distributed caching, shared data format (like XML) or communication protocol.

Now I think there’s a difference between software architect vs enterprise system architect (ESA). To the software/internet industry, the software architect adds more value, whereas to a non-tech company (like finance/ telecom/logistics…), the ESA is the master mind.

I feel some ESA may need to solve performance problems but not implementation/debugging problems.

ESA often covers network/hardware configuration and design too. Security is often part of the “network piece”. If the database is small, then the ESA often covers that too. Tunin/optimization is cross-discipline and touches all the pieces, but in most of my enterprise systems, hardware/network tuning tends to require fewer specialists than DB or application (including MOM and cache). Therefore, ESA sometimes becomes the optimization lead.

(I recently encountered some concepts, methodologies and paradigms at architect level but not sure where to place them — design patterns, TDD/BDD, closures/lambda, meta-programming, functional programming, aspect-oriented… and other programming paradigms — http://en.wikipedia.org/wiki/Programming_paradigm.)

[2] By the way, these so-called “really important” software products are predominantly in C.
[1] Many of the items listed can add real value in a single process or multiple

##Just what are(not) part of Architecture (too-late-to-change)

Dogfight – When you develop a dogfight jet over 30 years, you stick to your initial design. You try to build on the strengths of your design and protect the weaknesses. You don’t suddenly abandon your design and adopt your rival’s design because you would be years behind on that path. Software architecture is similar.

Anyone with a few years experience can specify an architecture in some details, but those decisions are flexible and could be modified later on, so they aren’t make-or-break decisions, though they could waste time.

Q1: what Features define a given architecture? I’m a fellow architect. If you must describe in 10 phrases your system architecture and provide maximum Insight, which 10 aspects would you focus on?

Q2: (another side of the same coin) What technical Decisions must be made early when creating a system from scratch?
Q2b: in other words, what design decisions will be expensive to change later? Why expensive?

Disclaimer — In this context, “later” means somewhere during those pre-UAT phases, not LATE into the release process. Obviously after release any config/DB/code change requires regression test and impact analysis, but that’s not our topic here.

A colleague of mine liked to discuss in terms of layering. In OO architecture, A layer consists of a bunch of similar classes (or interfaces) dedicated to a logically coherent high-level task. 2 interacting layers can exist in the same OS process or (less common but very critical) across processes. Intra-process layering basically means the 2 interacting classes (belonging to 2 layers) have well-defined responsibilities and a well-defined contract between, where one is always a Service-provider (“dependency”) to the service-consumer. Though popular among architects, I’m not sure these are the most important architectural features.

Here are my answers to the question “Is the decision costly to change later in SDLC?”

? anything in the wsdl/schema of a web service? yes if i’m not the only developer of this web service’s clients. A change would need these fellow programmers to change their code and test, test, test
? fundamental data constraints like “if this is positive then that must be non-null”? Yes because other developers would assume that and design code flows accordingly.
? database schema, the ER between a bunch of tables? Yes since it usually requires code change “upstairs”. DB schema is usually a Foundation layer, often designed up front.
? choose a DB vendor? yes if you happen to have truly skillful database developers in the team. They will use proprietary vendor extensions behind the manager’s back when the temptation is too strong. Such non-portable business logic will be expensive to re-implement in another database.
? use MOM or RPC-style communication? yes
? Serialization format? yes. “Later” means implementation effort wasted
? XML or binary or free text format? yes probably
? make a bunch of objects immutable/read-only/const? yes. Adding this feature later can be hard. Look into C++ literature.
? shared vs unshared cache? yes. effort is non-trivial
? What logic to put into SQL vs java/C#? sometimes No. You can change things later on, but usually wasting sizable implementation effort
? Choose programming language? Yes. Implementation effort wasted if switching late in the game.
? make a batch task re-runnable? yes. Adding this feature later on can be hard
—

? MVC (the highest-level design pattern there is)? well… yes, though you can adjust this design later on
? where to implement authentication, entitlement? Many say yes, but I feel no. Make the damn system work first, then worry about securing it. On fast-paced trading floor, Business usually can tolerate this feature being not so perfect. Business and upper management worry more about ….. You got it — Functionality.

? Which commercial(or free) product to choose? sometimes NO. You can use mysql or activemq and then change to commercial products. In fast projects, We can often delay this decision. Start coding without this particular commercial product.

? make a module thread safe? no, even though adding this feature can be hard
? how many staging tables? no
? choose a decorator/factor/singleton (or any of the classic) design pattern? no. They are usually too  low-level to be among the “10 architectural features”. I don’t know why a lot of architect interview questions hit this area.

(archtect IV) what I wish java to have

— big wishes —
* NullPointerException — too many of these are thrown in production systems and can take hours of wild goose chase. Developers must be very very thorough, and adopt a lot of defensive coding habits. Java won’t help you.
* easier tools for byte code engineering
* easier reflection — Look at dynamic scripting languages
* programmatic class creation; runtime class creation.
* memory leak — hard to detect
* easier immutable objects — String is great but we need more

–small wishes
* simpler getter/setter — look at C# properties
* Bags as collections.
* serialization — is a bit murky. I feel this is an important area neglected by many developers, perhaps because it’s murky. Perhaps java can support a special debug-serialization so we can see what it does to a complicated object graph
* checked exception — is a mistake in many developers’ opinion. Narrow the scope of this construct.

##job spec + %%observations: tech lead

#1 [SY,pwmcore] make architecture decisions for the team(s)
#2 [S] give software development direction to developer teams, including sample code
#3 [S,pwmcore] maintain and bug fix key components and infrastructure of the architecture.
#4 [S,pwmcore] code reviews
#5 policeman — ensure implementation follows firmwide standards. But I feel this is often impractical under tight deadlines. Wild west is the norm under tight deadlines
#6 whip up interfacing code to downstream (upstream too?) systems.
# buildmaster
# automated blackbox testing

Above are based on a well-written job spec — a perspective from the senior management. Below are a few more job duties I have observed

[S=Sundip of Stirt Risk]
[Y=Yang]

# [S, pwmcore] go-to person and problem solver on just about anything technical, but (bandwidth of this person is limited so) particularly the components that’s not owned by any single application.
# automated test framework
# automated build and deploy
# [SY] represent the department and interface with firmwide architect councils on decisions affecting the entire firm
# [SY] long-term planning; long-term bets; monitor long-term trends
# evaluate new “not-invented-here” software products proposed for adoption by application teams

The same job spec also said “bring a strong sense of ownership and must be driven to deliver tasks to completion”. So this is a doer not talker role. I guess more problem solving, getting-things-done, less (but still a great deal of) presentation, persuasion, explaining.

How about debugging? A software architect (see other posts) must be quick with debuggers etc.

arch: skills needed^skill-tests in IV@@

Most items below are relevant for a senior developer too, but this discussion is about software architects. By “Skill” I mean something practical, relevant, value-adding… An architect should bring some of these skills to a team. Think of these as job duties, if you like.

skill – insight on the limitations of a large number of impractical/discredited designs. Everyone can come up with some solutions on-paper, but most of them won’t work.
skill – know a large number of common errors and solutions — usually available on google but way too many. You don’t need insight. Just awareness is valuable enough. This comes from “mileage”.
skill – insight on weakness and limitations of dominant, default solutions (like Oracle, linux, WCF, protobuf…)
skill – insight on project best practices and their limitations
skill – know a large number of FOSS and commercial solutions
skill – write library component for others. The more developers use the component, the more senior this author is
skill – multiple technologies (non-trivial skill with each). Most financial/non-financial apps need more than one of DB, java/c#/c++, javascript, GUI, MOM, so an architect needs proficiency with more than one technology.
skill – practical testing strategy, automated or not. Junior guys may be too busy to worry about these. Testing can be tricky.
skill – tuning, optimization. Often overrated relative to GettingThingsDone.
** memory mgmt
** data structures nitty-gritty
** threading

Those are the skills interviewers like. Here are some evidence interviewers look for —

evidence – ability to clearly describe past systems, their design trad-offs, architecture, challenges, limitations,
evidence – ability to draw simple (but effective!) diagrams to illustrate any technical design
evidence – contribution to FOSS
evidence – know algorithms and their computational complexities. Google? Often overrated.
evidence – can identify and apply comp science constructs to programming problems, independently, without guidance — tough. Often overrated.
evidence(?) – know the “gory” details of popular languages
evidence – brain teaser mastery
evidence – design patterns. This is one of the laziest interview tests — ask the candidate to describe how he employed DP.

eg@3rd domain knowledge #Apama

I-bank may hire someone from a software vendor (like mysis, sunguard, murex …) but more likely they prefer the core engine developer not a presales architect according to Lin Y. The PS architect’s insight and value-add is tied to that core product (hopefully long-lasting), not very generic.

If you remove the “unremarkable” code, I guess the __core__ of the product is a kind of winning architecture + some patented algo/optimizations. They are integrated but each is an identifiable feature. Together they constitute an entry barrier. If you don’t have these insights like the core Apama developers have, then you can’t help break the entry barrier and build a similar trading engine at an ibank. The ibank is more likely to hire someone with core engine development experience from other ibanks

Q: What features can qualify for a patent?

%%A: Not the high level architecture, but some mid-level techniques, or “tricks” in Shaun’s words. I feel the high-level architecture may not be an entry barrier (or qualify as a patent) since in the world of trading platform there are only a small number of common architectures at the high level. However, as a job candidate and a team lead you are expected to communicate in high-level architectural lingo. The low-level jargon is harder to understand by management or by some fellow developers.

insist on better (less painful) design@@ Don’t !

(another blog. You need not read…)

It’s possible to be less passionate about technical wisdom, and wise decisions.

As you pointed out, software dev is much more flexible than most industries like /putting a nut on a screw/. There are few large industries where every day presents challenging choice to each worker. (Counseling, radio hosts come to mind.)

With the multitude of choices come the criteria of wisdom. Clever choices, regrettable choices, elegant choices, wise choices. After working for 10+ years in software, we all have truckloads of bitter experiences, so to avoid the pains, many of us slowly become purist, meticulous, opinionated, even emotional against lousy choices imposed on us.

Such choices cover not only implementation, coding style/convention, tooling, build/deploy, file layout, but also project teamwork. I’d say Wall St dev teams are commanding not democratic.

Our industry is littered with jargon related to these large number of small choices and wisdom — “DRY – Don’t Repeat Yourself”, code-smells, TDD, Agile, Extensible, Reusable, Readability, Maintenance cost/TCO, defensive (and non-defensive) design, Future-proof design, code cloning…

For example, a TDD/BDD purist would see every developer colleague of mine as amateurish, non-professional, tactical… I have seen some architects insisting on some design principles but it made the code base harder to understand, bloated and … inevitably class proliferation. EJB was a great best-practice rejected by the community/market and replaced by simpler design principles. I was told all 23 GoF design patterns require additional classes (and likely additional code complexity, though the advocates may argue complexity actually reduces at some mysterious higher level.) Now I feel most best-practice principles have non-trivial drawbacks or costs i.e. they can make someone’s life harder.

Some developer colleagues (like my Taiwanese colleague) are less stubborn, less lazy, less /purist/, more accommodating. Perhaps they see the code as nothing but a way to get a pay check. Perhaps they don’t feel threatened by the risk of losing job. They don’t struggle. They do understand the possibility of losing job, so they do things quick and dirty, try to please the mgr and please others who they depend on, and avoid confrontation. Way to go..

## y CTO choosing java over dotnet/python, again

• proven track record in “my” industry. If python offers a track record then more CTO’s will choose it.
  • php was chosen by wikipedia, Yahoo and facebook (Hack) but remains unproven to many enterprises CTO’s
• better integration with enterprise-grade infrastructure. Large systems usually need the power of Oracle/DB2/Sybase, Solaris/AIX, Weblogic/Websphere, perhaps beyond microsoft platform
• better support offered by “my” approved vendors — ibm, oracle, tibco, weblogic…
• competent developers in java^.NET? I think at least 50% more, given the differing maturity. C++ is even worse.
• java offers more capabilities and products including open-source. Even if a single needed capability is available in java but not .NET, this one factor alone could be a big factor.
• cost of programmer, software tools, hardware… Java is not cheaper but Linux is probably cheaper than windows.
• large scale deployment. c# and python are less proven IMO.
• avoid lock-in by microsoft. JVM is now offered by IBM and BEA.
• dotnet is proven only on windows, but linux is the overwhelming favorite operating system

##[11] y no dotnet on sell-side server side@@

(A fairly sketchy, limited, amateurish write-up.)
I was recently asked “dotnet has formidable performance and other strengths compared to java, but in trading engines space, why is dotnet making inroads only on the user-interface, never on the server side?”

Reason — as an casual observer, I feel Windows was designed as a GUI operating system with a single user at any given time. Later WinNT tried to extend the kernel to support multiple concurrent users. In contrast, Unix/Linux was designed from Day 1 to be multi-user, with the command line as the primary UI. (Personally I used to feel GUI is a distraction to high volume data processing OS designers.) A trading server needs no GUI.

Reason — Java and c/c++ were created on Unix; dotnet runs only on a windowing operating system. I feel web server is a light weight application, so both java and dotnet (and a lot of scripting languages) are up to the job[1], but truly demanding server-side apps need Unix AND java/c++. I guess Windows is catching up. In terms of efficiency, I guess java and c# are comparable and below C++.

Reason — Sell-side trading system is arms race. (Perhaps same among hedge funds.) Banks typically buy expensive servers and hire expensive system engineers, and then try to push the servers to the max. C/C++ makes the most efficient use of system resources, but Java offers many advantages over C++. Since the late 90’s, trading servers have progressively migrated from C++ to Java. Java and C++ are proven on the high-performance server side. Not dotnet.

Reason — I still feel *nix are more stable than Windows under high load. See http://efreedom.com/Question/1-214362/Java-Large-Heap-Sizes. However, I think you can create big clusters of windows servers

Reason — (from a friend) — *nix is considered more secure than windows. A GUI desktop can affect one trader if compromised, but a sell-side trading server affects all the traders from all the institutional and retail clients if compromised. So security risk is more serious on server side than GUI side.

The reasons below are arguments for java over dotnet in general, but don’t really explain why java is NOT on the GUI side and dotnet is still chosen on the GUI side.

Reason — big banks need stronger support than a single vendor company. What if Microsoft makes a mistake and dotnet loses technical direction and momentum? Java and *nix enjoy broader industry support.

[1] unless you are google or facebook, who needed c++ for their demanding requirements.

technically challenging role on the WS job market

Imagine a tech lead role that requires substantial architecture + implementation capabilities.

Q1: are there such roles on the market?
A1: yes I have seen some people playing such roles.

In our discussion, we often say “this job spec seems to be tough, but once hired, just about anyone can do the job.” or “Other people in the team are probably mediocre so the new hire need not be as good as the job spec says.”

In SG, a trading sys developer is paid higher than an Oracle DBA, or a manager at HP, or a system architect at some government companies (I know such friends), so it’s easy to think that trading dev skill set requirement must be significantly higher than the average java developer. But I believe just about any smart java developer can do the job.

Now back to my answer A1 above. Some positions on the market are indeed that challenging. How do we tell which one is? If I know which one is, I could either avoid such a “hot” job, or grab such a “hot” job.

y many java architects avoid static methods #threading

Hi XR,

Not sure why my architect doesn’t like static methods. I have some stateless method. It takes an object, pass to a static JNI method, then gets an output object, and returns it. I created a simple wrapper class around this single method. I feel making the method non-static requires callers to instantiate the class first. It would give the illusion that there’s state in the instance. It can also lead to unnecessary object proliferation.

My colleague mentioned threading as a reason. He’s not a newbie on threading, but I feel a stateless method is safest when it’s static. It is possible that he worries that this static method may one day access some static field? In that case, making it non-static and requiring every caller to have its own instance can help a bit.

Another colleague (elsewhere) also felt static methods are bad in threading. He’s also experienced in threading.

In threading, My habit is to keep things simple. If a method is stateless, i keep it static and have zero static field in that class. When people look at the code they immediately see it’s stateless, just like Math.abs().

##common IV questions@arch xp

To build and strengthen a lead over non-trading developers, you need to demonstrate architectural experience. Shuo described how to demonstrate management experience. Same idea.

I think a “mainstream” [2] hiring manager in a trading system often looks at in-depth understanding, fundamentals, architecture experience. Now, what kind of questions test your arch?

• * your favorite pattern — u r expected to know it inside out
• * describe some difficult design discussion in your team
• * ….How would you design a solution to this problem?
• * How did you improve performance (or throughput or latency)
• * When did you choose async over sync?
• * how did you use xml?
• * When in your experience is DB not a good data source?
• * pros/cons of stored proc?
• * How did you systematically handle exceptions in your system?

Some of these questions don’t sound like arch-questions. Be careful. I believe some hiring teams actually divide interview questions into categories, with one of them on arch. They actually want to hear your “presentation” on arch. If you talk about other things, they might quietly give you a 0 mark — “unable to operate at a higher level”.

Compared to the average developer, an architect thinks with clarity and focus, and zooms in on the key factors quickly, and probably communicates the same. I remember Yang, Chad (Billing)… Arch questions open up the candidate and reveal design thought process and technical communication style.

[2] neither the sharpest nor the laziest.

##[10]result-oriented lead developer skills

(… that are never tested in interviews)

Many buy-side shops need a lead developer not to manage people but get things done.

In many of my past teams, Site-specific local system knowledge is 95% of the required knowledge. Generic knowledge pales in comparison including portable GTD and zbs.

[t] diagnosis
[t] Unravel and explain (high AND low-level) data flow and business logic. Requests come from business and other teams.
[t] how-to-achieve (H2A) a requirement — knowledge about existing system is far more relevant than generic techniques. Highly valued by employers. Budget is often allocated to requirements.

I feel the Lab49 consultants and the productive GS developers, and Sundip Jangi are good at all these areas.

[t=tracing large amounts of code is required]

trading sys arch IV : cod`IV@best practice

Most of the SG/HK employers (UBS, RBS, Mdaq) perceive a wall street veteran (like me) more as an architect/tech-lead candidate than a senior dev. However, the tough tests are coding tests.

You would think an architect interview would not focus so much on the fine details. Well, many details (by definition of the word) are not important to an arch interview, but some (1%?) of the details are. Fundamentally, this is because any large of small software architecture is always conceived and ultimately implemented using __basic_comp_science_constructs, such as stackVar/heapVar/staticVar, thread safe iterators (loops), queues (linked list), circular array, concurrency primitives …

Another reason — the team lead would define and enforce low-level design and coding standard.

Fact is, these architect roles are extremely hands-on. In low-latency architect roles, a key value-add is the experience, insight, techniques, tools on low-level optimizations.

Among the 3 types of domain knowledge (see my other blog), your jargon/math experience is not relevant in a coding test. How about your trading arch experience? Relevant to some extent.

arch design xp]job candd #trading sys

(Some personal observations, to be posted on my blog)

If a job description says … architectural design experience desired…, but the role is NOT the architect, then I bet the new hire is NOT really welcome to comment on architectural decisions.

I have seen enough to be convinced that each small development team (3 – 10 developers) only has one architect. “No 2 tigers in 1 mountain.” as the Chinese saying goes.

If a new hire has a lot of architect experience, he (or she) had better keep a low profile on that. Whether the system is in early or final design stage, the designated architect inevitably has some good ideas how she (or he) wants it done. Most software designs have non-trivial flaws and painful trade-offs, so it’s easy to question her design.

( Requirements are best understood after system roll-out, and very vague in the design stage. Requirements are always subject to change, so every architect endeavors to future-proof her design, which is always a hit and miss — we always overshoot there and under-design here. )

The system architect won’t admit it but she seldom enjoys challenging questions on her design. If the question gives her a chance to show off her design, fine. Otherwise, those questions waste more time and adds more confusion than value. Some questions may bring rare insight and help the design, but i’m no expert on the communication process required for such questions to be openly asked. There are many wrong ways to ask such a right question.

As job descriptions put it, employers genuinely “desire” some amount of architect experience in a new hire. Perhaps such an experience helps the new hire work with the existing architect. A soccer team desires players with center-midfield experience. Doesn’t mean you and the existing center midfielder should both play that position.

design patterns adding value in a real world project@@

I think DP can sometimes add real values to an app architect.

Most of time I apply DP on small scales, but you can apply a DP to a large project. Even though it may look unnatural and introduce many additional classes, you can have some confidence such a pattern is proven to last.

[09] towards an app (!!SYS) arch across domains

Hi friends,

Self-appraisal on the numerous areas for improvement towards an application architect. You don’t have to reply, but your comments are definitely welcome.

First off, a shortlist of key metrics (among many metrics) for an enterprise application architecture. In fact, an architecture is evaluated on these criteria
* flexible + extensible + adaptable + resilient, for changes in volatile environments where requirements change frequently and need quick implementation. Not rigid or fragile. Will we be forced to scrap our current architecture and rebuild from scratch, or our current architecture can adapt to changes?
* performance — and throughput, capacity, and cluster support. Frequently, these are absolutely essential requirements an architecture must meet, or else the architect is disqualified.
* testability — quality assurance
* Cost — hardware/software cost often pale in comparison to labor cost. Cost improves when an architecture is more resource-efficient, easier to Learn, more Adaptable to changes, faster to Test, or offers faster Speed-to-market
* speed to market — Speed improves when an architecture becomes more Flexible/Adaptable, more Testable, or easier to Learn. Any experience/knowledge can help an architect make faster and better decisions.
* ease of learning (and maintenance) — simplicity helps learning.

Now, my own room4improvement:

– UML.– helps Learning, and perhaps Speed, since it facilitates communication.

– unit testing — for both OO and batch/scripting. I think there are many tools, real-world challenges/solutions.

– OO design patterns

Helps Flexibility, Learning,

– IDE — Not sure about small companies, but many large company (my focus for now) developers use IDE.
These tools affect Learning and Speed-to-market, but a non-IDE programmer may understand troubleshooting a bit better.

– API design — A poorly-defined skill. I think an experienced architect can somehow design better API interfaces between API-team and “client developers” who use the API. “Better” in terms of Flexibility, Testability, Learning, Speed-to-market, and simplified communications.

– DB schema design — The territory of data architects. Large companies (my focus for now) may or may not separate data architect and software architect roles.

– DB system design — stored programs, constraints, views, indices, locking, tuning … DBA’s duty. Software architects depends on and work closely with DBA, so deep knowledge is not absolutely required but important for a good software architecta — basic knowledge suffices. In reality, “basic” knowledge in this area is a rather high expectation.

– reporting — enterprise reporting, management reporting … There are many tools and common solutions. Reporting is a common functionality, found in perhaps 30% of current enterprise projects. An Architect may not need BI (Business Intelligence) but need a decent knowledge of reporting. Otherwise his credentials are somewhat in doubt.
This skill helps Speed to market, since it probably results in a more proven design — less trial-and-error.

– mem leak — tools, nlg, experience. I think an architect may be called to solve such problems.

Helps Performance.

– prototyping — A poorly-defined composite skill. Rapid prototyping and proof-of-concept. Personally I tend to favor these more than UML and product brochures.
This skill helps Speed to market, and can improve Cost by reducing unnecessary purchases.

– capacity planning — for hardware and software. I had a long-time friend, a SAP capacity planner and performance tuner. CP seems to be such a niche that few architects have expertise. I think in large companies (my focus for now), there are designated experts/approaches to decide how much cpu/bandwidth/DB-license… to buy. An architect is often involved in cost-estimation and need CP knowledge.
This skill helps Performance, Cost and Speed.

– code generation — I think some java architects are not knowledgeable about code generators like xdoclet. Code generators can be very effective. However, lack of such knowledge may not affect an architect’s performance appraisal, even though it can dramatically affect the team and the system built.
This skill helps Speed, Flexibility

– profiling/benchmarking — for performance-sensitive systems. There are many tools

– app-servers — knowledge of their relative strengths and limitations, and when to avoid each. Also how to work around their weaknesses, when you have to deal with them.

– OS features — that affect applications, such as OS tuning, threading, CPU/RAM allocation. One of my strengths.

[09]implementation skill ] a leadDeveloper/softwareArch

Update — compared to 2009, I am now much less confident because I see many mismatches in my profile vs arch job requirement, such as

• presentation and persuasion – eg Mac
• code reading in a large codebase
• debugging opaque systems

Hi XR,
(another blog)

I might do well in interviews, but still need know-how to be a senior architect. (Senior developer fine. Junior architect fine. Team lead perhaps ok.) As i confessed to you in our recent call, there are too many (dozens of times a week) practical problem-solving situations [2] i will encounter, research and try various wrong solutions, overcome, and then internalize.

I feel this vast amount of know-how is a must-have for a software architect in large project, but probably not needed by an EnterpriseSystemArchitect — see other posts.

[2] See post on [[## result-oriented lead developer skills]] for some of the problem-solving situations.

I guess your spring/hibernate performance problem was one of those encounters. We need to encounter problems to learn. Whenever i go through a project without implementation[1] difficulties, i feel like time wasted. Therefore, it’s not how many years that count. For example, I improved my SQL skill only in GS, not the other 2-3 years of SQL experience before.

[1] I don’t want to say “design” difficulties. If I face a difficult design issue, i don’t always learn. Most of my tech learning takes place after overcoming implementation challenges.

All the architects I know keep their skills up to date by doing hands-on coding.

* my GS team lead
* my GS mentor, a 40-something senior manager in charge of a 10-people team
* Lab49 team lead and the java developer
* Guo Qiao
* Xiao An

These individuals do hands-on work on a weekly basis — sometimes daily, but seldom go without coding for 3 months. That means

– their code is in CVS and they can’t afford to produce poor code
– they set coding standard for others, by example
– they force themselves to stay productive and churn out code fast. A hands-off architect tends to be a slow coder, i would imagine
– As Guo Qiao said, if you don’t code, gradually you lose your voice in tech discussions. I feel that’s true in high-level discussions just as in implementation discussions.
– i feel a hands-on architect can estimate man-day effort better than a PM.

[07] questioning accumulation as sys architect

You have repeatedly focused on accu.

@55, if u accumulate 5 years of lead-architect xp, it’s like to help tanbin’s sy1 (tanbin is suitable for it) even though

* even though In 2029 sg AR outside finance may earn less than many non-AR roles in US

* even though many of your peers might be in more glamorous roles in other countries earning more
* even though process differs from telco to telco — xp
* even though once-dominant technologies subside. Consider Raymond, Software AG, novell, Notes, Infomix…
* even though in many jobs and in many domains, accu is /elusive/
* even though — xp — you were frequently distracted by dotcoms, sunrise domains …

Q: how fast to become an large sys AR

If it takes years, then let’s accumulate now. Don’t diversify into PHP, DBA or SAP

Q: if not in US, then in SG?

perhaps lower pay than a non-AR role in US. Not greed.

Q: u feel AR adds more value than other roles, but it could be an internal system or a new, unproven service?

A: Still, AR tend to add more value.

2 key skills for development: TR^AR

TR = TRace, TRoubleshoot
AR = ARchitecture, design

TRoubleshooting requires TR
Customizing (moodle, mambo) requires TR and AR

learning design patterns

Initially, reduce the number of classes/interfaces to the essential participating objects, ie “the players”.

The #1 type of knowledge-pearl to *notice* is the HAS-A relationships among the players. Simple! But authors don’t always highlight them enough.

Whenever possible, look at methods (in a run-time context) as “calls” instead of “methods”.

Interesting [1] calls are mostly instance-methods rather than static methods.

[1] interesting to the pattern.

Key quetions to always bear in mind:

– {method} hosting object of the method?
– {method} intended client objects, according to the pattern? Not always intuitive.
– – } How would the client object get a pointer to the hosting object? Via a setter/constructor or arg?
– {call} objects in the argument? Not always easy to remember
– – } How did the client object get a pointer to the arg object? Via a setter/constructor or arg?
– For each object, instantiated by which creator object?