your brain power^brain power invested ] localSys

Background – “how fast you figure things out relative to your peers”.

For each team member AA, the struggle is the same — AA’s brain power vs the cumulative brain power that has gone into the local system which measures the complexity. If the local system complexity is too high then AA would struggle and take a long time (before he gives up).

The “local system” could include firmwide frameworks, or something open-source.

I prefer a local system created by low site-specific brain power, like one with standard SQL/stored-procs, standard noSQL, standard data encoding (FIX, Json..), standard java/c++ libraries, including OSS.

  • RTS and OC – relatively small amount of site-specific brain power in the system.
  • PWM comm – actually small amount of local system complexity but time given is too short
  • Barc – brand new codebase .. low site-specific brain power.
  • Quartz — the worst
Advertisements

iterator invalidated{erase: opaque

See also

I had multiple encounters with STL iterator invalidation, esp. after erase().

  • sometimes I get correct result
  • sometimes I get segfault
  • sometimes I get incorrect result

Opaque i.e. no obvious clues, so no keyword to google

Luckily my code was 20-lines

Luckily i could reproduce it once a while.

This is another example of tough bugs to hunt down.

%%GTD xp: 2types@technical impasse#难题

See also post on alpha geeks…
See also post on how fast you figure things out relative to peers
See also ##a few projects technically too challeng` 4me
See also https://bintanvictor.wordpress.com/2017/03/26/google-searchable-softwares/
see also https://bintanvictor.wordpress.com/2017/05/29/transparentsemi-transparentopaque-languages/

tuning? never experienced this challenge in my projects.
NPE? Never really difficult in my perience.

#1 complexity/opacity/lack of google help

eg: understanding a hugely complex system like the Quartz dag and layers
eg: replaying raw data, why multicast works consistently but tcp fails consistently
eg: adding ssl to Guardian. Followed the standard steps but didn’t work. Debugger was not able to reveal anything.
Eg: Quartz dag layers
Eg: Quartz cancelled trade

#2 Intermittent, hard to reproduce
eg: Memory leak is one example, in theory but not in my experience

eg: crashes in GMDS? Not really my problem.

eg: Quartz preferences screen frequently but intermittently fails to remember the setting. Unable to debug into it i.e. opaque.

q[nm] instrumentation #learning notes

When you want to reduce the opacity of the c++ compiled artifacts, q(nm) is instrumental. It is related to other instrumentation tools like

c++filt
objdump
q(strings -a)

Subset of noteworthy features:
–print-file-name
–print-armap? Tested with my *.a file. The filename printed is different from the above
–line-numbers? Tested
–no-sort
–demangle? Similar to c++filt
–dynamic? for “certain” types of shared libraries
–extern-only

My default command line is


nm --print-armap --print-file-name --line-numbers --demangle
nm --demangle ./obj/debug/ETSMinervaBust/src.C/ReparentSource.o //worked better

In May 2018, I ran nm on a bunch of *.so files (not *.a) to locate missing symbol definitions. Once I found a needed symbol is exported by libabc.so, I had to add -labc to my g+ command line.

##transparent^semi-transparent^opaque languages

With a transparent language, I am very likely (high correlation) to have higher GTD/productivity/KPI.

Stress — correlates with opacity. Would I take a job in php, SQL, javascript, perl?

Confidence to take on a gig — The more transparent, the higher my confidence — like py, javascript

Bootstrap — with a transparent language, I’m confident to download an open source project and hack it (Moodle …). With an opaque language like C++, I can download, make and run it fine, but to make changes I often face the opacity challenge. Other developers are often more competent at this juncture.

Learning — The opaque parts of a language requires longer and more “tough” learning, but sometimes low market value or low market depth.

Competitiveness — I usually score higher percentiles in IV, and lower percentiles in GTD. The “percentile spread” is wider and worse with opaque languages. Therefore, I feel 滥竽充数 more often

In this context, transparency is defined as the extent to which you can use __instrumentation__ (like debugger or print) to understand what’s going on.

  • The larger systems tend to use the advanced language features, which are less transparent.
  • The more low-level, the less transparent.

–Most of the items below are “languages” capable of expressing some complexity:

  • [T] SQL, perl, php, javascript, ajax,
  • [T] stored proc unless complex ones, which are uncommon
  • [T] java threading is transparent to me, but not other developers
  • [S] java reflection-based big systems
  • [T] regular c++, c# and java apps
  • [O]… but consider java dynamic proxy, which is used in virtually every non-trivial package
  • [T] most python scripts
  • [S] … but look at python import and other hacks. Import is required in large python systems.
  • [O] quartz
  • [S] Spring underlying code base. I initially thought it was transparent. Transparent to Piroz
  • [O] Swing visual components
  • [O] C# WCF, remoting
  • [T] VBA in Excel
  • —-below are not “languages” even in the generalized sense
  • [S] git .. semi-transparency became stressor cos KPI!
  • [O] java GC … but NO STRESS cos so far this is not a KPI
  • [O] MOM low level interface
  • [S] memory leak detectors in java, c#, c++
  • [O] protobuf. I think the java binding uses proxies
  • [T] XML + simple SAX/DOM
  • [S =semi-transparent]
  • [O =opaque]

 

python: routine^complex tasks

XR,

Further to our discussion, I used perl for many years. 95% of my perl tasks are routine tasks. With py, I would say “majority” of my tasks are routine tasks i.e. solutions are easy to find on-line.

  • routine tasks include automated testing, shell-script replacement, text file processing, query XML, query various data stores, query via http post/get, small-scale code generation, simple tcp client/server.
  • For “Complex tasks” , at least some part of it is tricky and not easily solved by Googling. Routine reflection / concurrency / c++Integration / importation … are documented widely, with sample code, but these techniques can be pushed to the limit.
    •  Even if we just use these techniques as documented, but we combine them in unusual ways, then Google search will not be enough.
    • Beware — decorators , meta-programming, polymorphism, on-the-fly code-generation, serialization, remote procedure call … all rely on reflection.

When you say py is not as easy as xxx and takes several years to learn, I think you referred to complex tasks.

I can see a few reasons why managers choose py over java for certain tasks. I heard there are a few jvm-based scripting languages (scala, groovy, clojure, jython …) but I guess python beats them on several fronts including more packages (i.e. wheels) and more mature, more complete and proven solutions, familiarity, reliability + wider user base.

One common argument to prefer any scripting language over any compiled language is faster development. True for routine tasks. For complex tasks, “your mileage may vary”. As I said, if the software system requirement is inherently complex, then implementation in any language will be complex. When the task is complex, I actually prefer more verbose source code — possibly more transparent and less “opaque”.

Quartz is one example of a big opaque system for a complex task. If you want, I can describe some of the complex tasks (in py) I have come across though I don’t have the level of insight that some colleagues have.

When you said the python debugger was less useful to you than java debugger, it’s a sign of java’s transparency. My “favorite” opaque parts of py are module import and reflection.

If python (or any language) has excellent performance/stability + good on-line resources [1] + reasonable library of components comparable to the mature languages like Java/c++, then I feel sooner or later it will catch on. I feel python doesn’t have exactly the performance.

[1] documentation is nice-to-have but not sufficient. Many programmers don’t have time to read documentation in-depth.

sys instrumentation: win32 murkier than linux #c++?

See also

I recently browsed [[inside windows debugging]]. In the past I also browsed a few *nix system programming books. I’m more afraid of windows than *nix. Windows is more dark and murky; linux is more open and consistent. As a beginner, I feel in linux the challenge is interpreting tool output but on windows it’s getting past the installation and security restrictions.

  • complication: GUI — As explained in this blog (http://bigblog.tanbin.com/2013/04/for-administration-prefer-command-line.html), GUI tends to hide a lot of features in layers and compartments. Console tools provide the same amount of power features …
    • if there’s a security or registry or other issues that reduce the provided feature set to below 100%, a console tool is more likely to report it to the user but GUI tools often “intelligently” handle it as if it’s normal.
    • MSVS is the primary windows instrumentation tool – very complicated.
  • complication: tool installation – Linux tools are not easy to install, but Windows tools tend to be even harder to install.
    • ** registry.
    • ** On any platform GUI tools are, on average, harder to install. Linux tools are less likely GUI.
  • complication: security restrictions – for a novice, 1/3 of the problems with windows tools are related to security/permission. Even under an administrator account, there are many security restrictions we may need to painstakingly disable! I don’t have any example now. I feel windows was criticized as vulnerable so Microsoft added a lot of security features.
    • ** some security features seem to be controlled by the domain administrators.
  • complication: environment factors – If you run the same tool in 2 machines with different results it’s often due to environment factors. In Linux, environment variables and some utilities are the only environment factors I know. In windows there seem to be many more places to check.
    • ** registry
    • ** certificates
    • ** control panel items
    • ** secpol.msc
  • complication: tool availability – many windows tools are commercial. On windows, I generally find the open-source tools better-quality, but for some tasks there’s only one tool – shipped by Microsoft. If I feel it is often crippled. On linux the best tools are invariably open-source.
  • complication: undocumented features — Even if you follow the all documented procedures, still you may hit something unexpected, due to undocumented features. This happens more often on windows then linux.

Note undocumented features affect not only instrumentation tools, but other components of the platform, which affect our debugging.

untold horror stories in Microsoft dev environment

Other horror stories:

* http://weblogs.asp.net/psteele/archive/2007/01/31/windows-service-stops-updating-event-log.aspx
* http://blogs.msdn.com/b/saurabh_singh/archive/2009/01/30/getting-fatal-error-lnk1181-cannot-open-input-file-kernel32-lib.aspx
* In my projects, logs folder can become inaccessible even though no one but my own services were touching it.
———
http://mariusbancila.ro/blog/tag/regasm/ looks like a typical horror story about Microsoft development platform.

– too many binary files involved. In unix environments ascii is the format for most files developers need to deal with. I feel from the beginning unix favors ascii. That tradition has endured the test of time and is going strong. I feel ascii is Clean and efficient. If a platform must use so many binary files, it’s a sign of weakness and non-ideal design. Simplicity is not easy to achieve or maintain. Look at copy-paste.

– outcome depends too much on the idiosyncrasies of the IDE. In unix-like environments, all the standard dev tools are command line tools, usually more transparent than a GUI tool.

– environment dependency and esp. the Mysterious Monstrous Murky registry — the worst problem IMO. It’s a local environmental config that’s pretty invisible. That means the same source code compiled on another machine may not behave the same. Even if you copy everything to another machine it may behave inconsistently.

– The platform is not robust enough. You may say it’s all developer’s fault, being inexperienced with the platform and tools. Well, I have substantially fewer such problems on other dev platforms. I’d venture to say a unix veteran hits fewer such problems than a windows veteran. If you follow the standard procedure and hit this kind of roadblock, it’s disappointing and stressful. This platform tends to throw up too many such roadblocks. Other platforms are more consistent and robust.

Look, every dev platform has complexity under the hood. As a higher-layer app developer we shouldn’t need to know a lot under the hood. The platform should be robust and just work. When we are forced to look under the hood, usually it’s a premature, unprofessional dev platform. As a 3-year old eclipse user, I  feel eclipse is such a premature tool, but other tools in java land are simpler and very robust.

This story also reminds me of my very first programming project in Microsoft dev environment. It’s a VBA project. Some basic FTP functionality was rather hard to implement. The MS-supplied infrastructure was not fully documented. Took several false starts. I soon left the MS platform for greener pastures — LAMP, javascript, unix, java, … and never looked back. Notably, Mysql had the best documentation and no surprise.

A recent project experience is again similar. We just couldn’t get auto update to work on a SL-OOB application. Auto-update once worked but now stopped working. What changed? We checked all the known issues… mystery, opaque, hidden under the hood, undocumented.