As CTO, I’d favor transparent langs, wary of outside libraries

If I were to propose a system rewrite, or start a new system from scratch without constraints like legacy code, then Transparency (+instrumentation) is my #1 priority.

  • c++ is the most opaque. Just look at complex declarations, or linker rules, the ODR…
  • I feel more confident debugging java. The JVM is remarkably well-behaving (better than CLR), consistent, well discussed on-line
  • key example — the SOAP stub/skeleton hurts transparency, so does the AOP proxies. These semi-transparent proxies are not like regular code you can edit and play with in a sandbox
  • windows is more murky than linux
  • There are many open-source libraries for java, c++, py etc but many of them affects transparency. I think someone like Piroz may say Spring is a transparent library
  • SQL is transparent except performance tuning
  • Based on my 1990’s experience, I feel javascript is transparent but I could be wrong.
  • I feel py, perl are still more transparent than most compiled languages. They too can become less transparent, when the codebase grows. (In contrast, even small c++ systems can be opaque.)

This letter is about which language, but allow me to digress briefly. For data store and messaging format (both require serialization), I prefer the slightly verbose but hugely transparent solutions, like FIX, CTF, json (xml is too verbose) rather than protobuf. Remember 99% of the sites use only strings, numbers, datetimes, and very rarely involve audio/visual data.


architect’s helicopter view of a system

This (pearl) post is not only describing some role models. This is a career tip, possibly survival tip…


Prime eg: Yang. Nikhil calls it “his system knowledge” – Without knowing the source code, Yang would ask sharp questions that baffles the actual coders. If the answer smells bad, then either answer is not exactly truthful, or existing system design is imperfect. Yang is quick to notice “inconsistency with reality”, a very sharp observation. This pattern is critical in troubleshooting and production support as well. I told XR that production support can be challenging and require logical, critical thinking about the systems as a whole.


An __effective__ geek needs more than high-level system knowledge. Somehow, every one of them soon gets down to the low level, unless her role is a purely high-level role … like presales?


Not everyone has the same breadth and width of helicopter view. Often I don’t need it in the first few years. You would think after many years, each would acquire enough low-level and high-level knowledge, but I imagine some individuals would refuse to put in the extra effort, and keep her job by taking care of her own “territory” and never step out.


— my own experience —

At Citi muni, I largely avoided the complexity. I did try and trace through the autoreo codebase but stopped when I was given other projects. In hind sight, I feel it’s too slow and probably not going to work. Mark deMunk pointed out the same.


At 95 Green and Barcap, I was a contractor and didn’t need to step out of my territory. This might be one reason I had an easier job and exceed expectations. See post on “slow decline” and, both in the pripri blog.


At OC, the Guardian codebase I did take up fully. Quest was given to me after I /fell out/ with boss, so I did enough research to implement any required feature. No extra effort to get a helicopter view.


Stirt was tough. I spent a few months trying to learn some but not all of the nuts and bolts in Quartz. This knowledge is fundamental and crucial to my job and I didn’t have too much time learning the Stirt system, which I did learn but didn’t master. In fact, I asked many good system-level questions about Sprite and Stirt-Risk, but failed to gain Yang’s insight.


(arch) xp I hope to accummulate

Architecture is all about design trade-off. Each time I start on a new technology, I want to grow as a hands-on architect. I hope to gain a bit of insight on the real drawbacks of the most common designs in that technology. I am not satisfied with the glossy brochures. They tend to play up many “shortcomings” that are irrelevant to my projects.

In the Quartz system, I don’t feel I learned what I wanted. I didn’t notice some common design I could *reuse* when I design my next system…

Guardian actually gave me such an opportunity, because it’s so simple, but I didn’t get to play with multiple working designs. The one component that actually worked well is the embedded web server I whipped up.

I foresee that with swing and WPF I would have more opportunities to “play with” various designs and get some insight.

I foresee that with javascript and php I would have that chance too.


app architect – civil engineer or a salesman, SG^US

I feel in Singapore context, “architect” is more likely to refer to a hands-off role. In the US, I never meet an architect who doesn’t writes code hands-on at least half his time.

A real app architect in finance (different from professional software product vendors) really needs hands-on capabilities. Her output is not just on paper, but in code. If her design (not the document but the “key ideas” implemented) doesn’t work, she must roll up her sleeves and make it work. Most of those ideas are low level, like some serialization or some threading construct, and require just one good developer. In that sense it’s not unlike a library module.

In this context, architect is the name of the technical lead of ANY dev team. Often known as lead developer, in a smaller team.

Any dev team needs a technical leader. Half the time, the team manager or the project manager is technical enough to lead a suppor team but not a dev team,  so an architect is needed.  Often, the architect is the only leader.

The pre-sales architect is very different. Castle in the sand. Imaginary buildings.

Update: I feel in the US I could become a lead developer, once I become familiar with a codebase, but any role that requires a lot of persuasion I’m not too sure. I feel if my technical grasp of everything is higher than the rest then it’s possible. It’s all relative.

arch xp is overrated as KPI#instrumentation,tools…

See also other blog posts —

// “in the initial dev stage, instrumentation is my #1 design goal”.

Going out on a limb, I’d say that in high-pace projects GTD (Getting Things Done) is more important than many things which are supposed to be more important.

GTD is more important than … integrating deep domain knowledge insight into the system. I think such domain knowledge can sometimes be valuable. It can help dev team avoid wasting time on “the wrong things”.

GTD is more important than … quality of code. For a few months at least, the only guy looking at the code is the original author.

GTD is more important than … “quality of design”, which is seldom well-defined.

GTD is more important than … being nice to others. Many times other people can tolerate a bit of personality if you can GTD. However, you must not offend the people who matter.

GTD is more important than … impressive presentations that address many real customer pains. A user may die for a guy who “understands my pains”, but when it is time to deliver, selling/understanding is an irrelevant distraction.

GTD is more important than … in-depth knowledge of the language or of a software platform/product. Such knowledge is more important than GTD during interviews though. Threading, data structure…

GTD is more important than … uncovering the root casue of (intermittent) problems/errors/surprises — the Toyota drill-down root-cause investigation. Regex engine creator may need to fully characterise every unexpected behavior; AutomateTellerMachine error probably deserve a drill-down investigation but in enterprise apps drill-down investigation simply takes too much time. We developers are paid to build a usable tool, not a fully-understood tool. Live with ambiguity and move on.

GTD is more important than … adding an important test to the automated test suite. Sure that mistake we just found may happen again so we really appreciate adding that test, but in reality, most high-pace environments don’t have an automated test suite. If we are lucky enough to have documented manual test plan, then yes add it, but such a plan seldom is comprehensive, so after a while people won’t follow it religiously. So in reality we just hope developers learn the lesson and avoid making the same mistake. If they do, then we just need someone who can GTD. Any system to incorporate such heard-learned lesson is likely an imperfect system, and whoever investing in such a system is often wasting his/her time.  If a GTD guy doesn’t bother with that system, he will still be respected and loved by manager and users. Basically, any long-term investment is unappreciated. GTD is all about short-term results. This is the reality of quality control in fast-paced teams.

GTD is more important than … adding meaningful error messages. Anyone debugging the system would love the guy who added the meaningful error message, but he is an unsung hero. Manager love the guy who GTD fast. Code quality is invisible and therefore ignored and unrewarded.

To achieve GTD, you must solve tech problems. Tech problems could (occasionally) call for architectural perspective, but less than tool knowledge, debugging experience, or low-level system insight.

In defense of architects, architectural track record is more important in sales contexts, including selling to internal clients and business users.

I should probably get input from Raja, Xiao An, Yi Hai …

Software architect vs enterprise architect

In many companies architects (are perceived to) operate at a high level of abstraction — high-level system modules/interfaces, high-level concepts, high-level decisions. Their audiences include non-technical users, who speak business language. I guess some architects absolutely avoid low-level tech jargon almost like profanity — it’s all about image of an architect. The wrong word on the wrong occasion would creates a permanent bad impression.

To many people, “architecture” implies some network layer, networking protocol, network nodes. These architects might feel less than comfortable if their system has just one machine or just one jvm instance [1], without messaging, database, remote-procedure-call, web service, distributed caching, IPC, shared memory, shared data format (like XML) or communication protocol.

In contrast, software architects can be extremely low-level, too. Their audience are developers. (Examples below include a lot of c/c++…)

– Consider the creator of STL — an architect?
– connectivity engine team lead?
– FIX engine architect?
– Any low-latency trading engine architect?
– Creator of perl/python/boost regex engines?
– All boost libraries I know are low-level. Are the authors architects?
– Tomcat is a single-jvm system. Authors qualify as architects?
– Hibernate isn’t a big system with big modules.  so the authors are low-level architects?
– JNI component designers — single-VM
– java GC innovators
– Creators of any reusable component like a jar, a dll — Just a pluggable module — not even a standalone “server”. Do they qualify as architects?
– The pluggable look-and-feel framework in Swing is designed by someone — an architect?
– spring framework, before all the add-on modules were added
– the tpc/ip/udp software stack was designed by some architect, but that person probably worked mostly on 1 machine, occasionally on 2 machines for testing only.
– GCC creator
– apache creator
– mediawiki creator
– creators of the dotnet CLR and the crown-jewel of Sun, the JVM.

I feel majority of the really important software [2] used in the world today were created by truly powerful developers who focused on a single module in a single system, without messaging, database, remote-procedure-call, web service, distributed caching, shared data format (like XML) or communication protocol.

Now I think there’s a difference between software architect vs enterprise system architect (ESA). To the software/internet industry, the software architect adds more value, whereas to a non-tech company (like finance/ telecom/logistics…), the ESA is the master mind.

I feel some ESA may need to solve performance problems but not implementation/debugging problems.

ESA often covers network/hardware configuration and design too. Security is often part of the “network piece”. If the database is small, then the ESA often covers that too. Tunin/optimization is cross-discipline and touches all the pieces, but in most of my enterprise systems, hardware/network tuning tends to require fewer specialists than DB or application (including MOM and cache). Therefore, ESA sometimes becomes the optimization lead.

(I recently encountered some concepts, methodologies and paradigms at architect level but not sure where to place them — design patterns, TDD/BDD, closures/lambda, meta-programming, functional programming, aspect-oriented… and other programming paradigms —

[2] By the way, these so-called “really important” software products are predominantly in C.
[1] Many of the items listed can add real value in a single process or multiple

##Just what are(not) part of Architecture (too-late-to-change)

Dogfight – When you develop a dogfight jet over 30 years, you stick to your initial design. You try to build on the strengths of your design and protect the weaknesses. You don’t suddenly abandon your design and adopt your rival’s design because you would be years behind on that path. Software architecture is similar.

Anyone with a few years experience can specify an architecture in some details, but those decisions are flexible and could be modified later on, so they aren’t make-or-break decisions, though they could waste time.

Q1: what Features define a given architecture? I’m a fellow architect. If you must describe in 10 phrases your system architecture and provide maximum Insight, which 10 aspects would you focus on?

Q2: (another side of the same coin) What technical Decisions must be made early when creating a system from scratch?
Q2b: in other words, what design decisions will be expensive to change later? Why expensive?

Disclaimer — In this context, “later” means somewhere during those pre-UAT phases, not LATE into the release process. Obviously after release any config/DB/code change requires regression test and impact analysis, but that’s not our topic here.

A colleague of mine liked to discuss in terms of layering. In OO architecture, A layer consists of a bunch of similar classes (or interfaces) dedicated to a logically coherent high-level task. 2 interacting layers can exist in the same OS process or (less common but very critical) across processes. Intra-process layering basically means the 2 interacting classes (belonging to 2 layers) have well-defined responsibilities and a well-defined contract between, where one is always a Service-provider (“dependency”) to the service-consumer. Though popular among architects, I’m not sure these are the most important architectural features.

Here are my answers to the question “Is the decision costly to change later in SDLC?”

? anything in the wsdl/schema of a web service? yes if i’m not the only developer of this web service’s clients. A change would need these fellow programmers to change their code and test, test, test
? fundamental data constraints like “if this is positive then that must be non-null”? Yes because other developers would assume that and design code flows accordingly.
? database schema, the ER between a bunch of tables? Yes since it usually requires code change “upstairs”. DB schema is usually a Foundation layer, often designed up front.
? choose a DB vendor? yes if you happen to have truly skillful database developers in the team. They will use proprietary vendor extensions behind the manager’s back when the temptation is too strong. Such non-portable business logic will be expensive to re-implement in another database.
? use MOM or RPC-style communication? yes
? Serialization format? yes. “Later” means implementation effort wasted
? XML or binary or free text format? yes probably
? make a bunch of objects immutable/read-only/const? yes. Adding this feature later can be hard. Look into C++ literature.
? shared vs unshared cache? yes. effort is non-trivial
? What logic to put into SQL vs java/C#? sometimes No. You can change things later on, but usually wasting sizable implementation effort
? Choose programming language? Yes. Implementation effort wasted if switching late in the game.
? make a batch task re-runnable? yes. Adding this feature later on can be hard

? MVC (the highest-level design pattern there is)? well… yes, though you can adjust this design later on
? where to implement authentication, entitlement? Many say yes, but I feel no. Make the damn system work first, then worry about securing it. On fast-paced trading floor, Business usually can tolerate this feature being not so perfect. Business and upper management worry more about ….. You got it — Functionality.

? Which commercial(or free) product to choose? sometimes NO. You can use mysql or activemq and then change to commercial products. In fast projects, We can often delay this decision. Start coding without this particular commercial product.

? make a module thread safe? no, even though adding this feature can be hard
? how many staging tables? no
? choose a decorator/factor/singleton (or any of the classic) design pattern? no. They are usually too  low-level to be among the “10 architectural features”. I don’t know why a lot of architect interview questions hit this area.

(archtect IV) what I wish java to have

— big wishes —
* deadlock — extremely hard to avoid, detect, or resolve. (consider thread dump.)
* programmatic class creation; runtime class creation.
* memory leak — hard to detect
* easier tools for byte code engineering
* easier reflection — Look at dynamic scripting languages
* easier performance profiling to help compare against C++ code
* NullPointerException — too many of these are thrown in production systems and can take hours of wild goose chase. Developers must be very very thorough, and adopt a lot of defensive coding habits. Java won’t help you.
* immutable objects — String is great but we need more and easier immutables
–small wishes
* simpler getter/setter — look at C# properties
* Bags as collections.
* serialization — is a bit murky. I feel this is an important area neglected by many developers, perhaps because it’s murky. Perhaps java can support a special debug-serialization so we can see what it does to a complicated object graph
* checked exception — is a mistake in many developers’ opinion. Narrow the scope of this construct.

##typical job spec: senior in-house architect

#1 [SY,pwmcore] make architecture decisions for the team(s)
#2 [S] give software development direction to developer teams, including sample code
#3 [S,pwmcore] maintain and bug fix key components and infrastructure of the architecture.
#4 [S,pwmcore] code reviews
#5 policeman — ensure implementation follows firmwide standards. But I feel this is often impractical under tight deadlines. Wild west is the norm under tight deadlines
#6 whip up interfacing code to downstream (upstream too?) systems.
# buildmaster
# automated blackbox testing

Above are based on a well-written job spec — a perspective from the senior management. Below are a few more job duties I have observed

[S=Sundip of Stirt Risk]

# [S, pwmcore] go-to person and problem solver on just about anything technical, but (bandwidth of this person is limited so) particularly the components that’s not owned by any single application.
# automated test framework
# automated build and deploy
# [SY] represent the department and interface with firmwide architect councils on decisions affecting the entire firm
# [SY] long-term planning; long-term bets; monitor long-term trends
# evaluate new “not-invented-here” software products proposed for adoption by application teams

The same job spec also said “bring a strong sense of ownership and must be driven to deliver tasks to completion”. So this is a doer not talker role. I guess more problem solving, getting-things-done, less (but still a great deal of) presentation, persuasion, explaining.

How about debugging? A software architect (see other posts) must be quick with debuggers etc.

learning design patterns #letter to Mithun

Another thing about design patterns – each author has a different description.

Some describe it in 2 paragraphs or a 10-line example using one dog/cat class. Others write 3 pages and 5 classes. Yet others write a whole chapter on it in a design pattern book.

If I go by the simplistic interpretation to describe a pattern, then the interviewer may think I don’t really know it.

If I give an in-depth example, then it could be too complicated to describe (I’m not extremely good at describing complexity) and interviewer may not be an expert on that pattern to understand me.

My suggestion is to focus on 1 or 2 patterns. Understand them inside out and also remember some simple examples in your own app. Our knowledge of a pattern should grown from thin to thick to thin, yes back to thin. Only when we have a streamlined understanding can we describe it with clarity.

Here are some complex patterns I have struggled with for years – visitor, bridge, strategy, composite, chain of command, memento, command, observable …. Each of them can be summarized in 1 paragraph, but let’s be honest – these aren’t simple.