opaque challenge:(intermediate)data browser #push the limit

Opacity — is a top 3 (possibly biggest) fear and burden in terms of figure-things-out-relative-to-cowokers on a localSys. The most effective and direct solution is some form of instrumentation tool for intermediate data. If you develop or master an effective browsing tool, however primitive, it would likely become a competitive advantage in terms of figure-out speed, and consequently earn you some respect.

LocalSys — I feel most of the effective data browser knowledge is localSys knowledge.

If you are serious about your figure-out speed weakness, if you are seriously affected by the opaque issues, then consider investing more time in these browsing tools.

Hard work, but worthwhile.

  • eg: Piroz built a Gemfire data browser and it became crucial in the Neoreo project
  • #1 eg: in my GS projects, the intermediate data was often written into RDBMS. Also important — input and output data are also written into RDBMS tables. Crucial in everyday trouble-shooting. I rank this as #1 in terms of complexity and value. Also this is my personal first-hand experience
  • #2 eg: RTS rebus — during development, I captured lots of output CTF messages as intermediate data… extremely valuable
    • Once deployed, QA team relied on some web tools. I didn’t need to debug production issues.
    • I remember the static data team save their static data to RDBMS, so they relied on the RDBMS query tool on a daily basis.

Now some negative experiences

  • eg: I’m not too sure, but during the Stirt QZ dev projects I didn’t develop enough investigation skill esp. in terms of checking intermediate data.
  • eg: in Mvea, we rely on net-admin commands to query order state, flow-element state and specific fills… Not as convenient as a data store. I never became proficient.
    • I would say the FIX messages are logged consistently and serve as input and output data browser.

Many projects in my recent past have no such data store. I don’t know if there’s some effective solution to the opacity, but everyone else face the same issue.

Advertisements

opaque c++trouble-shooting: bustFE streamIn

This is a good illustration of fairly common opaque c++ problems, the most dreadful/terrifying species of developer nightmares.

The error seems to be somewhat consistent but not quite.

Reproducing it in dev enviroment was a first milestone. Adding debug prints proved helpful in this case, but sometimes it would take too long.

In the end, I needed a good hypothesis, before we could set out to verify it.

     81     bool SwapBustAction::streamInImpl(ETSFlowArchive& ar)
     82     { // non-virtual
     83       if (exchSliceId.empty())
     84       {
     85         ar >> exchSliceId;
     86       }
    104     }
    105     void SwapBustAction::streamOutImpl(ETSFlowArchive& ar) const
    106     { // non-virtual
    107       if (exchSliceId.size())
    108       {
    109         ar << exchSliceId;
    110       }

When we save the flow element to file, we write out the exchSliceId field conditionally as on Line 107, but when we restore the same flow element from file, the function looks for this exchSliceId field unconditionally as on Line 85. When the function can’t find this field in the file, it hits BufferUnderflow and aborts the restore of entire flow chain.

The serialization file uses field delimiters between the exchSliceId field and the next field which could be a map. When the exchSliceId field is missing, and the map is present, the runtime would notice an unusable data item. It throws a runtime exception in the form of assertion errors.

The “unconditional” restore of exchSliceId is the bug. We need to check the exchSliceId field is present in the file, before reading it.

In my testing, I only had a test case where exchSliceId was present. Insufficient testing.

camp_out]ff: %%coping strategy/strength

Exec summary: If I find myself engaged, I would want to spend extra quiet hours including weekends working in office and make progress on localSys. I would be able to apply myself and match my brain power against the accumulated brainpower baked into the local code base.

By “Camp-out” I mean all forms of extra time spent working in office. Camp-out is a concrete form of engagement. Usually the engaging period is a honeymoon but hopefully it can become a more sustained focus on localSys. If it does happen (I won’t push myself too hard…) I would hit better GTD, productivity, and earn some respect from some colleagues (not necessarily the boss).

More importantly, I would feel proud of myself and free of regret or guilt. See warning below.

Capture the mood. Eg: You get an engaging project that give you insight into portable GTD, tools, or localSys.

  • Xp: Qz — I did capture the mood a few (like 3+) times. I felt really good even though the camp-out didn’t “save me” in the end. Like U.S. presidential election, the “end” result depends on many many things so it’s unwise to use it as a big paint brush and wipe out all the effort.
  • Xp: Macq — I made big efforts to camp out to work on c++ build system and python. No regrets. The camp-out didn’t really build my foundation in localsys as the expectation was too high. No regrets!
  • xp: OC — I did a few times, mostly to learn c#. No regrets.

— Warning: camp-out may not earn me respect, GTD, enough localSys mileage. Ashish feels I would survive.

I feel I may still get PIP, but after such an all-out effort, I would be decisive to leave. Consider Qz and Macq.

— I see this as my competitive strength and an effective coping strategy.

I used to feel I’m kind of inadequate if I have to camp out and sacrifice family time just to keep my head above water. Wrong! Most of my peers simply don’t have the absorbency, and productive solitude to do this. Many can’t even imagine it.

I said my son may have more or less math abilities, but his effort (chiefly absorbency) is probably lower than his classmates. My daughter’s abilities is unknown, but her absorbency is better than her brother. Similarly, my absorbency is higher than my peers as demonstrated by camp-out.

— Initially, May need 6M-9M to get over the initial hump. I can even rent a temp home for 6M if I feel engaged. See kids only on weekends.

— As contractor I feel the motivation but zero pressure to camp out; As employee I feel pressure to camp out. Clearly I feel better as contractor.

— Q: In my experience, part of the camp-out hours are often spent on blogging (flight control) and morale-boosting workout. OK?

— Q: However, if code is too opaque, I won’t be able to figure things out 😦 What can I do?

— Q: What if I receive a PIP.. will I be able to persuade myself to camp out?
A: Yes. I did in GS, Qz and Macq. No honeymoon, but I have been able to overcome the huge negative energy, convert it partially to motivation and get myself to camp out and focus on GTD.

I like this imagery — I felt like a boxer on my back-foot, not hopeful but not pessimistic either.

— Q: If in the mood for localSys but need to sleep? Shall i go home?
A: don’t go home to sleep. Sleep in office. Go home to eat then come back to office.

 

c++complexity≅30% above java #c#=in_between

Numbers are just gut feelings, not based on any measurement. I often feel “300% more complexity” but it’s nicer to say 30% 🙂

  • in terms of interview questions, I have already addressed in numerous blog posts.
  • see also mkt value@deep-insight: java imt c++
  • — tool chain complexity in compiler+optimizer+linker… The c++ compiler is 200% to 400% (not merely 30%) more complex than java… see my blogpost on buildQiurks. Here are some examples:
  • undefined behaviors … see my blogposts on iterator invalidation
  • RVO — top example of optimizer frustrating anyone hoping to verify basic move-semantics.
  • See my blogpost on gdb stepping through optimized code
  • See my blogpost on on implicit
  • — syntax — c++ >> c# > java
  • java is very very clean yet powerful 😦
  • C++ has too many variations, about 100% more than c# and 300% more than java
  • — core language details required for GTD:
  • my personal experience shows me c++ errors are more low-level.
  • Java runtime problems tend to be related to the (complex) packages you adopt from the ecosystem. They often use reflection.
  • JVM offers many runtime instrumentation tools, because JVM is an abstract, simplified machine.
  • — opacity — c++ > c# > java
  • dotnet IL bytecode is very readable. Many authors reference it.
  • java is even cleaner than c#. Very few surprises.
  • — more low-level — c++ > c# > java.
  • JVM is an excellent abstraction, probably the best in the world. C# CLR is not as good as JVM. A thin layer above the windows OS.

your brain power^brain power baked into localSys

Background – “how fast you figure things out relative to your peers”.

For each team member AA, the struggle is the same — AA’s brain power vs the cumulative brain power that has gone into the local system which measures the complexity. If the local system complexity is too high then AA would struggle and take a long time (before he gives up).

The “local system” could include firmwide frameworks, or something open-source.

I prefer a local system created by low site-specific brain power, like one with standard SQL/stored-procs, standard noSQL, standard data encoding (FIX, Json..), standard java/c++ libraries, including OSS.

  • RTS and OC – relatively small amount of site-specific brain power in the system.
  • PWM comm – actually small amount of local system complexity but time given is too short
  • Barc – brand new codebase .. low site-specific brain power.
  • Quartz — the worst

iterator invalidated{erase: opaque

See also

I had multiple encounters with STL iterator invalidation, esp. after erase().

  • sometimes I get correct result
  • sometimes I get segfault
  • sometimes I get incorrect result

Opaque i.e. no obvious clues, so no keyword to google

Luckily my code was 20-lines

Luckily i could reproduce it once a while.

This is another example of tough bugs to hunt down.

%%c++keep crash` I keep grow`as hacker #zbs#Ashish

Note these are fairly portable zbs, more than local GTD know-how !

My current c++ project has high data volume, some business logic, some socket programming challenges, … and frequent crashes.

The truly enriching part are the crashes. Three months ago I was afraid of c++, largely because I was afraid of any crash.

Going back to 2015, I was also afraid of c++ build errors in VisualStudio and Makefiles, esp. those related to linkers and One-Definition-Rule, but I overcame most of that fear in 2015-2016. In contrast, crashes are harder to fix because 70% of the crashes come with no usable clue. If there’s a core file I may not be able to locate it. If I locate it, it may not have symbols. If it has symbols the crash site is usually in some classes unrelated to any classes that I wrote. I have since learned many lessons how to handle these crashes:

  • I have a mental list like “10 common crash patterns” in my log
  • I have learned to focus on the 20% of my codebase that are most convoluted, most important, most tricky and contribute most to debugging difficulties. I then invest my time strategically to rewrite (parts of) that 20% and dramatically simplify them. I managed to get familiar and confident with that 20%.
    • If the code belongs to someone else including 3rd party, I try to rewrite it locally for my dev
  • I have learned to pick the most useful things to log, so they show a *pattern*. The crashes usually deviate from the patterns and are now easier to spot.
  • I have developed my binary data dumper to show me the raw market data received, which often “cause” crashes.
  • I have learned to use more assertions and a hell lot of other validations to confirm my program is not in some unexpected *state*. I might even overdo this and /leave no stoned unturned/.
  • I figured out memset(), memcpy(), raw arrays are the most crash-prone constructs so I try to avoid them or at least build assertions around them.
  • I also figured signed integers can become negative and don’t make sense in my case so I now use unsigned int exclusively. In hind sight not sure if this is best practice, but it removed some surprises and confusions.
  • I also gained quite a bit of debugger (gdb) hands-on experience

Most of these lessons I picked up in debugging program crashes, so these crashes are the most enriching experience. I believe other c++ programs (including my previous jobs) don’t crash so often. I used to (and still do) curse the fragile framework I’m using, but now I also recognize these crashes are accelerating my growth as a c++ developer.

%%GTD xp: 2types@technical impasse#难题

See also post on alpha geeks…
See also post on how fast you figure things out relative to peers
See also ##a few projects technically too challeng` 4me
See also https://bintanvictor.wordpress.com/2017/03/26/google-searchable-softwares/
see also https://bintanvictor.wordpress.com/2017/05/29/transparentsemi-transparentopaque-languages/

tuning? never experienced this challenge in my projects.
NPE? Never really difficult in my perience.

#1 complexity/opacity/lack of google help

eg: understanding a hugely complex system like the Quartz dag and layers
eg: replaying raw data, why multicast works consistently but tcp fails consistently
eg: adding ssl to Guardian. Followed the standard steps but didn’t work. Debugger was not able to reveal anything.
Eg: Quartz dag layers
Eg: Quartz cancelled trade

#2 Intermittent, hard to reproduce
eg: Memory leak is one example, in theory but not in my experience

eg: crashes in GMDS? Not really my problem.

eg: Quartz preferences screen frequently but intermittently fails to remember the setting. Unable to debug into it i.e. opaque.

tech strength: Depth [def]

Some top geeks I know are fast at reading code + logs. Few rely on documentation. I’m OK not the fastest.

Some top geeks in “sister” teams of my team are experts with instrumentation tools and techniques. I guess other top geeks seldom need a lot of instrumentation. I feel they lack the experience but make up for it in other skills.

Some top geeks keep gaining depth if they focus on one complex system. I might have potential here. “Deep” is slightly different from “complex”. Deep means you need to slow down, /quiet down/, absorb, go through thick->thin cycles, get a laser focus, look again, and think deeper.

Perhaps related to my Depth capacity, I’m not that fast with timed online coding tests.

  • — some well-defined, often named technical subjects with depth, often opaque to the uninitiated
  • rvr, RVO
  • RAII, swap
  • [b] TMP including SFINAE, CRTP
  • [b] java generics
  • [b] JNI, python extension module — complexities lurking by the /dimly lit path/ due to low-level interfacing
  • java/c# reflection techniques in practice
  • ? python introspection
  • [B] concurrency in general and java threading in particular
  • [B] algorithms and data structures
  • clever SQL single join to solve tricky problems.
  • [B] query/sproc tuning — depth created by the large number of tables/indices and variations in queries
  • ? serialization over network
  • ? ADL techniques
  • any math and statistics subject
  • [b/B=books have been written on this topic]

q[nm] instrumentation #learning notes

When you want to reduce the opacity of the c++ compiled artifacts, q(nm) is instrumental. It is related to other instrumentation tools like

c++filt
objdump
q(strings -a)

Subset of noteworthy features:
–print-file-name
–print-armap? Tested with my *.a file. The filename printed is different from the above
–line-numbers? Tested
–no-sort
–demangle? Similar to c++filt but c++filt is more versatile
–dynamic? for “certain” types of shared libraries
–extern-only

My default command line is


nm --print-armap --print-file-name --line-numbers --demangle
nm --demangle ./obj/debug/ETSMinervaBust/src.C/ReparentSource.o //worked better

In May 2018, I ran nm on a bunch of *.so files (not *.a) to locate missing symbol definitions. Once I found a needed symbol is exported by libabc.so, I had to add -labc to my g+ command line.

##transparent^semi-transparent^opaque languages

With a transparent language, I am very likely (high correlation) to have higher GTD/productivity/KPI.

Stress — correlates with opacity. Would I take a job in php, SQL, javascript, perl?

Confidence to take on a gig — The more transparent, the higher my confidence — like py, javascript

Bootstrap — with a transparent language, I’m confident to download an open source project and hack it (Moodle …). With an opaque language like C++, I can download, make and run it fine, but to make changes I often face the opacity challenge. Other developers are often more competent at this juncture.

Learning — The opaque parts of a language requires longer and more “tough” learning, but sometimes low market value or low market depth.

Competitiveness — I usually score higher percentiles in IV, and lower percentiles in GTD. The “percentile spread” is wider and worse with opaque languages. Therefore, I feel 滥竽充数 more often

In this context, transparency is defined as the extent to which you can use __instrumentation__ (like debugger or print) to understand what’s going on.

  • The larger systems tend to use the advanced language features, which are less transparent.
  • The more low-level, the less transparent.

–Most of the items below are “languages” capable of expressing some complexity:

  • [T] SQL, perl, php, javascript, ajax,
  • [T] stored proc unless complex ones, which are uncommon
  • [T] java threading is transparent to me, but not other developers
  • [S] java reflection-based big systems
  • [T] regular c++, c# and java apps
  • [O]… but consider java dynamic proxy, which is used in virtually every non-trivial package
  • [T] most python scripts
  • [S] … but look at python import and other hacks. Import is required in large python systems.
  • [O] quartz
  • [S] Spring underlying code base. I initially thought it was transparent. Transparent to Piroz
  • [O] Swing visual components
  • [O] C# WCF, remoting
  • [T] VBA in Excel
  • —-below are not “languages” even in the generalized sense
  • [S] git .. semi-transparency became stressor cos KPI!
  • [O] java GC … but NO STRESS cos so far this is not a KPI
  • [O] MOM low level interface
  • [S] memory leak detectors in java, c#, c++
  • [O] protobuf. I think the java binding uses proxies
  • [T] XML + simple SAX/DOM
  • [S =semi-transparent]
  • [O =opaque]

 

[16]python: routine^complex tasks #XR

XR,

Further to our discussion, I used perl for many years. 95% of my perl tasks are routine tasks. With py, I would say “majority” of my tasks are routine tasks i.e. solutions are easy to find on-line.

  • routine tasks include automated testing, shell-script replacement, text file processing, query XML, query various data stores, query via http post/get, small-scale code generation, simple tcp client/server.
  • For “Complex tasks” , at least some part of it is tricky and not easily solved by Googling. Routine reflection / concurrency / c++Integration / importation … are documented widely, with sample code, but these techniques can be pushed to the limit.
    •  Even if we just use these techniques as documented, but we combine them in unusual ways, then Google search will not be enough.
    • Beware — decorators , meta-programming, polymorphism, on-the-fly code-generation, serialization, remote procedure call … all rely on reflection.

When you say py is not as easy as xxx and takes several years to learn, I think you referred to complex tasks.

It’s quite impressive to see that some powerful and useful functionalities can be easily implemented by following online tutorials. By definition these are routine tasks. One example that jump out at me is reflection in any language. Python reflection can be powerful like a power drill to break a stone wall. Without such a power drill the technical challenge can look daunting. I guess metaclass is one such power drill. Decorator is a power drill I used in %%logging decorator with optional args

I can see a few reasons why managers choose py over java for certain tasks. I heard there are a few jvm-based scripting languages (scala, groovy, clojure, jython …) but I guess python beats them on several fronts including more packages (i.e. wheels) and more mature, more complete and proven solutions, familiarity, reliability + wider user base.

One common argument to prefer any scripting language over any compiled language is faster development. True for routine tasks. For complex tasks, “your mileage may vary”. As I said, if the software system requirement is inherently complex, then implementation in any language will be complex. When the task is complex, I actually prefer more verbose source code — possibly more transparent and less “opaque”.

Quartz is one example of a big opaque system for a complex task. If you want, I can describe some of the complex tasks (in py) I have come across though I don’t have the level of insight that some colleagues have.

When you said the python debugger was less useful to you than java debugger, it’s a sign of java’s transparency. My “favorite” opaque parts of py are module import and reflection.

If any language has excellent performance/stability + good on-line resources [1] + reasonable library of components comparable to the mature languages like Java/c++, then I feel sooner or later it will catch on. I feel python doesn’t have exactly the performance. In contrast, I think php and javascript can achieve very high performance in their respective usage domains.

[1] documentation is nice-to-have but not sufficient. Many programmers don’t have time to read documentation in-depth.

sys instrumentation: win32 murkier than linux #c++@@

See also

I recently browsed [[inside windows debugging]]. In the past I also browsed a few *nix system programming books. I’m more afraid of windows than *nix. Windows is more dark and murky; linux is more open and consistent. As a beginner, I feel in linux the challenge is interpreting tool output but on windows it’s getting past the installation and security restrictions.

  • complication: GUI — As explained in this blog (http://bigblog.tanbin.com/2013/04/for-administration-prefer-command-line.html), GUI tends to hide a lot of features in layers and compartments. Console tools provide the same amount of power features …
    • if there’s a security or registry or other issues that reduce the provided feature set to below 100%, a console tool is more likely to report it to the user but GUI tools often “intelligently” handle it as if it’s normal.
    • MSVS is the primary windows instrumentation tool – very complicated.
  • complication: tool installation – Linux tools are not easy to install, but Windows tools tend to be even harder to install.
    • ** registry.
    • ** On any platform GUI tools are, on average, harder to install. Linux tools are less likely GUI.
  • complication: security restrictions – for a novice, 1/3 of the problems with windows tools are related to security/permission. Even under an administrator account, there are many security restrictions we may need to painstakingly disable! I don’t have any example now. I feel windows was criticized as vulnerable so Microsoft added a lot of security features.
    • ** some security features seem to be controlled by the domain administrators.
  • complication: environment factors – If you run the same tool in 2 machines with different results it’s often due to environment factors. In Linux, environment variables and some utilities are the only environment factors I know. In windows there seem to be many more places to check.
    • ** registry
    • ** certificates
    • ** control panel items
    • ** secpol.msc
  • complication: tool availability – many windows tools are commercial. On windows, I generally find the open-source tools better-quality, but for some tasks there’s only one tool – shipped by Microsoft. If I feel it is often crippled. On linux the best tools are invariably open-source.
  • complication: undocumented features — Even if you follow the all documented procedures, still you may hit something unexpected, due to undocumented features. This happens more often on windows then linux.

Note undocumented features affect not only instrumentation tools, but other components of the platform, which affect our debugging.

untold horror stories in Microsoft dev environment

Other horror stories:

* http://weblogs.asp.net/psteele/archive/2007/01/31/windows-service-stops-updating-event-log.aspx
* http://blogs.msdn.com/b/saurabh_singh/archive/2009/01/30/getting-fatal-error-lnk1181-cannot-open-input-file-kernel32-lib.aspx
* In my projects, logs folder can become inaccessible even though no one but my own services were touching it.
———
http://mariusbancila.ro/blog/tag/regasm/ looks like a typical horror story about Microsoft development platform.

– too many binary files involved. In unix environments ascii is the format for most files developers need to deal with. I feel from the beginning unix favors ascii. That tradition has endured the test of time and is going strong. I feel ascii is Clean and efficient. If a platform must use so many binary files, it’s a sign of weakness and non-ideal design. Simplicity is not easy to achieve or maintain. Look at copy-paste.

– outcome depends too much on the idiosyncrasies of the IDE. In unix-like environments, all the standard dev tools are command line tools, usually more transparent than a GUI tool.

– environment dependency and esp. the Mysterious Monstrous Murky registry — the worst problem IMO. It’s a local environmental config that’s pretty invisible. That means the same source code compiled on another machine may not behave the same. Even if you copy everything to another machine it may behave inconsistently.

– The platform is not robust enough. You may say it’s all developer’s fault, being inexperienced with the platform and tools. Well, I have substantially fewer such problems on other dev platforms. I’d venture to say a unix veteran hits fewer such problems than a windows veteran. If you follow the standard procedure and hit this kind of roadblock, it’s disappointing and stressful. This platform tends to throw up too many such roadblocks. Other platforms are more consistent and robust.

Look, every dev platform has complexity under the hood. As a higher-layer app developer we shouldn’t need to know a lot under the hood. The platform should be robust and just work. When we are forced to look under the hood, usually it’s a premature, unprofessional dev platform. As a 3-year old eclipse user, I  feel eclipse is such a premature tool, but other tools in java land are simpler and very robust.

This story also reminds me of my very first programming project in Microsoft dev environment. It’s a VBA project. Some basic FTP functionality was rather hard to implement. The MS-supplied infrastructure was not fully documented. Took several false starts. I soon left the MS platform for greener pastures — LAMP, javascript, unix, java, … and never looked back. Notably, Mysql had the best documentation and no surprise.

A recent project experience is again similar. We just couldn’t get auto update to work on a SL-OOB application. Auto-update once worked but now stopped working. What changed? We checked all the known issues… mystery, opaque, hidden under the hood, undocumented.