fear@large codebase #web/script coders

One Conclusion — my c++ /mileage/ made me a slightly more confident, and slightly more competent programmer, having “been there; done that”, but see the big Question 1 below.

— Historical view

For half my career I avoided enterprise technologies like java/c++/c#/SQL/storedProc/Corba/sockets… and favored light-weight technologies like web apps and scripting languages. I suspect that many young programmers also feel the same way — no need to struggle with the older, harder technologies.

Until GS, I was scared of the technical jargon, complexities, low-level API’s debuggers/linkers/IDE, compiler errors and opaque failures in java/SQL … (even more scared of C and Windows). Scared of the larger, more verbose codebases in these languages (cf the small php/perl/javascript programs)… so scared that I had no appetite to study these languages.

— many guys are unused to large codebases

Look around your office. Many developers have at most a single (rarely two) project involving a large codebase. Large like 50k to 100k lines of code excluding comments.

I feel the devops/RTB/DBA or BA/PM roles within dev teams don’t require the individual to take on those large codebases. Since it’s no fun, time-consuming and possibly impenetrable, few of them would take it on. In other words, most people who try would give up sooner or later. Searching in a large codebase is perhaps their first challenge. Even figuring out a variable’s actual type can be a challenge in a compiled language.

Compiling can be a challenge esp. with C/c++, given the more complex tool chain, as Stroustrup told me.

Tracing code flow is a common complexity across languages but worse in compiled languages.

In my experience, perl,php,py,javascript codebases are usually small like pets. When they grow to big creatures they are daunting and formidable just like compiled language projects. Some personal experiences —
* Qz? Not a python codebase at all
* pwm comm perl codebase? I would STILL say codebase would be bigger if using a compiled language

Many young male/female coders are not committed to large scale dev as a long-term career, so they probably don’t like this kinda tough, boring task.

— on a new level

  • Analogy — if you have not run marathons you would be afraid of it.
  • Analogy — if you have not coached a child on big exams you would be afraid of it.

I feel web (or batch) app developers often lack the “hardcore” experience described above. They operate at a higher level, cleaner and simpler. Note Java is cleaner than c++. In fact I feel weaker as java programmer compared to a c++ programmer.

Q1: I have successfully mastered a few sizable codebases in C++, java, c#. So how many more successful experiences do I need to feel competent?
A: ….?

Virtually every codebase feels too big at some time during the first 1-2 years, often when I am in a low mood, despite the fact that in my experience, I was competent with many of these large codebases.
I think Ashish, Andrew Yap etc were able to operate well with limited understanding.
I now see the whole experience as a grueling marathon. Tough for every runner, but I tend to start the race assuming I’m the weakest — impostor syndrome.
Everyone has to rely on log and primitive code browsing tools. Any special tools are usually marginal value. With java, live debugger is the most promising tool but still limited pain-relief. Virtually all of my fellow developers face exactly the same challenges so we all have to guess. I mean Yang, Piroz, Sundip, Shubin, … virtually all of them, even the original authors of the codebase. Even after spending 10Y with a codebase, we could face opaque issues. However, these peers are more confident against ambiguity.

new languages lose limelight soon

New languages come into and go out of fashion, like hairstyle and cars.

  • Java has lost the lime light for 10 years but is still fairly popular due to job market and high-profile and “cool” companies like google.
  • C and C++ are among the top 5 longest-living languages. They are unpopular among the young, but they remain important.
  • c# is a curious case. I think the really important windows applications are still written in c++.
  • Many high-level languages go out of fashion, including C#, D, ..
  • Most scripting or dynamic languages go out of fashion

 

c++toolchain complexity imt new languages #%%advantage

The modern languages all feature dramatically simplified tool chain. In contrast, c++ tool chain feels much bigger to me, including profilers, static analyzers, binary file dumpers, linkers ..

This is one of the real obstacles to new entrants, young or old. This is also my (slowly growing) competitive advantage. I feel some people (like Kevin of Macq) know more, but most developers have a cursory working knowledge in this field.

I was frustrated for years by the complex and messy build tools in c++. Same for the other new entrants — Rahul spent a month setting up Eclipse CDT…

This learning curve, entry barrier … is a direct consequence to the c++ “sweet spot” as Stroustrup described — inherently complex codebase close to hardware.

I wrote dozens of blogposts about c++ build issues. For example, on windows, my strawberryPerl + git_bash + notepad++ setup is unknown to many. These fellow developers struggle with MSVS or Eclipse !

Due to the bigger ecosystem needed to support c++, new features are added at a slower pace than languages having a central organization.

new lang2challenge c++on efficiency@@

I asked Stroustrup — efficiency is the traditional strength of C and C++,  both memory efficiency and speed.. Is that still true? He immediately said yes.

I think it was clear in his mind that c/c++ were still the most efficient languages around. He did say Fortran is optimized for scientific computing.

I later asked him — any new language he watches out for. He said none, without real hesitation, no ifs or buts.

Recalling that conversation, I feel new languages are usually more high-level and easier to use. They are more likely to use heap to provide a “consistent interface” and avoid the complexities of a low-level language.

If I’m right, then these languages can’t and won’t optimize for efficiency as a top priority. Efficiency is possibly a 2nd priority.

##longevity rating: java is outlier ] language_war

Among languages, java is way ahead of the pack. We need to treat java as exceptional, outlier. With that, c++ looks rather solid and decent. Shall we merge this table into my pastTechBet.xlsx sheet? Partially merged, but no need to update the sheet.

longevity rating #bias tech skill mkt share prominence domains
80 % java robust 2000’s
40 % py rising 2000’s
50 % c/c++ fell 1980’s HFT, gaming,
telco,embedded, AI/ML
20 % c#/dotnet fell 2010’s
30% php ? 2000’s
10% perl FELL 2000’s automation
40% javascript rising 2000’s
30 % RDBMS/sql fell 1990’s
70 % socket robust 1990’s probably
90% TCP/IP dominant 1970’s
20 % MOM robust
90 % http stack dominant 2000’s
90 % unix “tradition” dominant beyond memory

language war: 4 criteria

  1. criteria-E: efficiency, performance including platform-specific optimization.
    • Ling of MS felt threading support is crucial
    • many interactive or batch applications don’t care much about latency
  2. criteria-P1: proven, widely used in “my” community (me = tech lead), with mature ecosystem.
  3. criteria-F: killer features, including OO
  4. criteria : RAD, simplicity, ease-of-use

It is instructive to study the “dethrone” stories.

  • Case: On server-side, java dethroned c++ due to P1, RAD, E
    • In contrast, windows GUI seldom use java. They use 1)c++ 2)c# due to P1, F and E
  • Case: Python dethroned perl due to RAD
  • Case: shell scripting is very old but survived, due to F
  • Case: php survived due to F (designed for web server side), RAD

20Y long term trend – demand for high-level language skillset is rising (unsteadily) relative to java/c++. The reasons are numerous and include RAD, F, P1 Q: which factor is the biggest threat to java? A: F, RAD, not E ! I guess the RAD productivity gap between java and python isn’t so big. For large modularized projects, java probably has a productivity advantage.

Q: will c++ (or even java) be relegated to assembly’s status? No but I wonder why.
 

##[19] cited strengths@java

In this post we compare c++, python, javascript, c#

  • [G3] Scalability and performance [1] – James Governor has a saying: “When web companies grow up, they become Java shops”.Java is built for scalability in mind, which is why it is so popular among enterprises and scaling startups. Twitter moved from Ruby to Java for scaling purposes.
  • [G9] community support [1] such as stackoverflow —
  • [G9] portability [1] — on Linux and Android.
  • [G9] versatile — For web, batch jobs, and server-side. MS is a java shop, using java for trading. but DDAM is not a typical MS app. DDAM has many batch jobs but the UI is all in web java.
    • python and c++ are also versatile
  • [G5] Java has high correlation with fashionable technologies — hadoop; cloud; big data; microservices… Python and javascript are also in the league.
  • [G3] proven —
    • web apps are the biggest market segment. Some (js/php/ruby) of the top 10 most popular languages are used exclusive for web. Java is more proven than c#, python, c++.
    • enterprise apps (complex business logic + DB) are my primary focus. java is more proven than python, javascript, php, c#
  • [G3=a top-3 strength]

[1] https://stackify.com/popular-programming-languages-2018/ explains Java’s popularity

 

%%geek profile cf 200x era, thanks2tsn

Until my early 30’s I was determined to stick to perl, php, javascript, mysql, http [2] … the lighter, more modern technologies and avoided [1] the traditional enterprise technologies like java/c++/c#/SQL/MOM/Corba . As a result, my rating in the “body-building contest” was rather low.

Like assembly programming, I thought the “hard” (hardware-friendly) languages were giving way to easier, “productivity” languages in the Internet era. Who would care about a few microsec? Wrong…. The harder languages still dominate high-end jobs.

Analogy?

* An electronics engineering graduate stuck in a small, unsuccessful wafer fab
* An uneducated pretty girl unable to speak well, dress well.

Today (2017) my resume features java/c++/py + algo trading, quant, latency … and I have some accumulated insight on core c++/c#, SQL, sockets, connectivity, ..

[1] See also fear@large codebase
[2] To my surprise, some of these lighter technologies became enterprise —

  1. linux
  2. python
  3. javascript GUI
  4. http intranet apps

##coreJava QQ topics: stand-out from crowd

Q: what core java QQ topics (not necessarily skills) to make you stand out from the crowd?

  1. (Best example) low-level core threading features — higher market value than ecosystem features like concurrency utilities, which are so numerous that most candidates are not expected to know in-depth
  2. jGC
  3. latency tuning — including JIT
  4. collections internals

— second tier QQ topics

  • reflection, AOP, bytecode engineering? never popular
  • advanced generics? out of fashion
  • class loader internals? out of fashion
  • JNI? never popular
  • jvm internals? never popular and never needed on any project

my”native”language==C : feeling good till 70

See also post on CivilEngineers.

Context: speaking to interviewers, colleagues, I like to say my native programming language is … C. C is the first language I studied in-depth on my own, in 1994. C was also the first professional programming language in my very first job. I’m proud of my association with C because :

  • My dad is a specialist on 墨子. C is like 孔子. C never completely fell out of fashion for system programmers.
  • C is the most important language to Unix system programming (kernel, sockets, standard library…). As of 2019, system programing knowledge is growing progressively more important to my job interviews.
  • threading and data structure are among the top 5 most important and evergreen interview topics, both “born” in C.
    • Most thread implementations are related to system libraries.
    • all important data structures are low level and implemented in C more efficiently than other languages
  • In terms of depth — I think C, c++, java, c# have the most depth. I am slowly building my grasp of this depth. I think the accumulation is good.
  • In terms of of churn and accu — C is among the best. See [17] j^c++^c# churn/stability…
  • In terms of it’s relation to other languages — C is the #1 most important, as Confucius is in Chinese culture. Java shows barely visible heritage from C. In contrast, C#, python, perl etc show a visible heritage from C. I feel most popular languages today inherits from C or are created in C.
  • In terms of longevity — C is #1, the grand-daddy in the short history of programming languages. (C++ might come 2nd.) In contrast, all the popular languages will probably come and go — java, python, c#, javascript
  • Mark of Quoin seem to suggest that my low-level experience is less valuable than experience using c++ libraries, but I think most people would agree that the high-level experience is superficial, lower accumulation, high churn, and offers no insight.

array=#1 important data structure

C supports only array (horizontal) and struct (vertical). I feel most standard libraries across languages are designed based on the same. Array + graph are about the only data structures in those libraries and in CIV.

For cross-language coding drill, we should probably keep our focus on arrays.

For comp science algorithm research, there’s more energy focused on array than any other data structure.

performance domain is low-level. Array (among various data structures) is the real focus of micro tuning and hardware optimizations.

growing popularity@scripting languages #threat2c++

See also sg19 %% attitude on java, relative2c++ and ..

By 2019, I have noticed growing popularity of high-level languages like GO, node.js and python … all dynamic, scripting languages, in contrast to compiled languages. However, java is still fairly relevant. I won’t say too much about the possible reasons — ease of learning, ease of finding and training new hires…

i feel this trend is a continuation + evolution of a longer trend — from c/c++ to java/c#.

One direct impact of this trend — demand for c++ skill is shrinking[3], again. The remaining c++ jobs tend to pay lower than these “easier” languages or require deep expertise (like HFT shops) .. very bad deals for the c++ job seekers.

[3] It’s important to remember that demand for c# and perl developers is also shrinking.

come-n-go /Flash-in-the-pan — In my younger days (early 2000’s), there were also a growing choice of scripting languages but only one became general purpose — Perl. I feel GO, node.js may lose popularity over the coming years. These languages are popular partly because they are new. I described elsewhere that among the scripting languages, only 1 or 2 will dominate. Perl was dethroned by python.

See y re-enter c++if jobs fewer — I still believe c++ is harder and “builds character“, unless you spend 90% of your c++ career using wrappers that other people wrote and don’t bother to look under the hood.

c++^java..how relevant ] 20Y@@

See [17] j^c++^c# churn/stability…

C++ has survived more than one wave of technology churn. It has lost market share time and time again, but hasn’t /bowed out/. I feel SQL, Unix and shell-scripting are similar survivors.

C++ is by far the most difficult languages to use and learn. (You can learn it in 6 months but likely very superficial.) Yet many companies still pick it instead of java, python, ruby — sign of strength.

C is low-level. C++ usage can be equally low-level, but c++ is more complicated than C.

[17] j^c++^c# churn/stability…

This comparison has a bias towards java. My observation is finance-centric.

Q: why do I feel c# as a t-investment is not as long-living as java and C (the longest-living)?
A: Java is not tied to any OS. Java is used on windows + many unix derivatives including linux and android, whereas c# and objective-C are tied to particular platforms.

However look at Perl. It is cross-platform but was displaced by vbscript on windows and python.

 notes worst – best factor
Microsoft doesn’t care about this c# jav c++ protection of Your investment
c# jav c++ churn
c++ has no single owner making those decisions #worst to best c# jav c++ stability of features
#younger to older c# jav c++ longevity(new tech tends to die young
or changing too much)
JVM is huge help c++ c# jav ease of troubleshooting
c++ c# jav maintainability of app
c# can be back-end but less proven;
c++ can crash easily — no exception to catch
c# c++ jav underlying stability as
long-running server
windows is murky. Java: type erasure c++ c# jav  dark corners undocumented
#high to low c++ c# jav syntax complexity
#worst to cleanest c++ c# jav clean language
#low to high jav c++ c# expressiveness
c++ much higher #low to high c++ entry barrier at senior level
c# can be ez to learn, but java is even cleaner  #easy to hard java c# c++ initial learning
only Microsoft provides #low to high c# c++ jav std+3rdParty libraries
c# — only on windows c++ c# jav popularity
c# — limited to Windows and GUI/web c# c++ jav wide appeal, general-purpose
each lang has x% high-end jobs but c++
percentage (30%) is highest
c# salary
c++ c# java job market depth

which programming languages changed toooo much

No elaboration please. Pointers will do.

perl 6?
python 3?  but the excessive changes are not taken up!
php5 was better received.
C# ?
MFC -> winforms -> WPF ? Different technologies. More like delphi/powerBuilder/coldFusion declining than a language evolving.
java 8? not much change. Lambda is the biggest
c++11? not much change. Rval ref is the biggest

what c++ topics are valuable in the long run on WallSt

When you first pick up a language (like c++) from 11 textbooks, you form a view of what topics (know-how/insights) are important, and worth spending time on.

However, once in the field you will get a different view. Over the years, I realized
++ remote debugger is critical

++ lots of C techniques are widely used on Wall St.

++ MSVS is a dominant tool, so is GNU toolkit.

++ multiple inheritance is still relevant

— most whiteboard coding will use C constructs + vector/string, not fancy c++ features

— STL containers are widely adopted, but not everything in STL is equally important.
— portability isn’t a issue.

— template techniques are powerful but seldom used on Wall St
— boost shared_ptr is widely used, but not other boost libraries

— strings — many teams have their in-house development, so the std string is less imp

— I feel OO techniques are not as popular/easy as in java/c#

So over 50 years (if I were to work that long), i’m going to discover all the important topics. Now question is how to quickly discover them, so as to focus on the most valuable.

Answer — I feel a beginner is better off changing project frequently. The more diverse projects you take on, the faster you discover those important “topics”.

The rolling stone gathers no moss? Well, i feel this is not the case for the beginner dynamic traveling consultant:)

stack to heap trend in languages

  • I guess you seldom see malloc() and free() in business applications. I have seen a number of c applications but didn’t notice them. However, how does c pointers deal with automatic destruction of the objects they reference? I guess all such objects are declared and used in the same scope, never declared in an inner func and used in an outer func?
  • C++ biz apps use lots of new(), but by default, if you don’t call new() or malloc(), C++ uses the stack and global area. In terms of language support and encouragement, stack is still easier to use than heap in C++. I consider C++ as a transition between C and newer languages.
  • STL seems to be mostly heap, to support expansion
  • Java uses the heap more than the stack. Every object is on the heap. Memory leak becomes too likely so java had to provide garbage collector.
Conclusion: c -> c++ -> STL -> java, I see less stack and more heap usage. However, Stroustup pointed out the performance advantage of stack over heap.