python to dump binary data in hex digits

Note hex() is a built-in, but I find it inconvenient. I need to print in two-digits with leading 0.

def Hex(data): # a generator function
  i=0
  for code in map(ord,data):
    yield "%02x " % code
    i += 1
    if i%8==0: yield ' '

print ''.join(Hex("\x0a\x00")); exit(0)
Advertisements

q(nm) instrumentation #learning notes

When you want to reduce the opacity of the c++ compiled artifacts, q(nm) is instrumental. It is related to other instrumentation tools like

c++filt
objdump
q(strings -a)

Subset of noteworthy features:
–print-file-name
–print-armap? Tested with my *.a file. The filename printed is different from the above
–line-numbers? Tested
–no-sort
–demangle? Similar to c++filt
–dynamic? for “certain” types of shared libraries
–extern-only

My default command line is

nm –print-armap –print-file-name –line-numbers –demangle

q(g++ -g -O) together

https://linux.die.net/man/1/g++ has a section specifically on debugging. It says

GCC allows you to use -g with -O

I think -g adds additional debug info into the binary to help debuggers; -O turns on complier optimization.

By default, our binaries are compiled with “-g3 -O2”. When I debug these binaries, I can see variables but lines are rearranged in source code, causing minor problems. See my blog posts on gdb.

gdb q(next) over if/else +function calls #optimized

I used an optimized binary. Based on limited testing, un-optimized doesn’t suffer from these complexities.

Conventional wisdom: q(next) differs from q(step) and should not go into a function

Rule (simple case): When you are on a line of if-statement in a source code, q(next) would evaluate this condition. If the condition doesn’t involve any function call, then debugger would evaluate it and move to the “presumed next line”, hopefully another simple statement.

Rule 1: suppose your “presumed next line” involves a function call, debugger would often show the first line in the function as the actual “pending”. This may look like step-into!

Eg: In the example below. Previous pending is showing L432 (See Rule 2b to interpret it). The presumed line is L434, but L434 involves a function call, so debugger actually shows L69 as the “pending” i.e. the first line in the function

Rule 2 (more tricky): suppose presumed line is an if-statement involving a function call. Debugger would show first line in the function as the pending.

Eg: In the example below, Previous pending was showing L424. Presumed line is L432, but we hit Rule 2, so actual pending is L176, i.e. first line in the function.

Rule 2b: when debugger shows such an if-statement as the “pending”, then probably the function call completed and debugger is going to evaluate the if-condition.

424 if (isSendingLevel1){
425 //……
426 //……….
427 //……..
428 //……….
429 } // end of if
430 } // end of an outer block
431
432 if (record->generateTopOfBook()
433 && depthDb->isTopOfTheBook(depthDataRecord)) {
434 record->addTopOfBookMarker(outMsg);
435 }

#1 challenge if u rely@gdb to figure things out: optimizer

Background: https://bintanvictor.wordpress.com/2015/12/31/wall-st-survial-how-fast-you-figure-things-out-relative-to-team-peers/ explains why “figure things out quickly” is such a make-or-break factor.

In my recent experience, I feel compiler optimization is the #1 challenge. It can mess up GDB step-through. For a big project using automated build, it is often tricky to disable every optimization flag like “-O2”.

More fundamentally, it’s often impossible to tell if the compiled binary in front of you was compiled as optimized or not. Rarely the binary shows it.

Still, compared to other challenges in figuring things out, this one is tractable.

gdb skill level@Wall St

I notice that, absolutely None of my c++  veteran colleagues (I asked only 3) [2] is a gdb expert as there are concurrency experts, algo experts [1], …

Most of my c++ colleagues don’t prefer (reluctance?) console debugger. Many are more familiar with GUI debuggers such as eclipse and MSVS. All agree that prints are often a sufficient debugging tool.

[1] Actually, these other domains are more theoretical and produces “experts”.

[2] maybe I didn’t meet enough true c++ alpha geeks. I bet many of them may have very good gdb skills.

I would /go out on a limb/ to say that gdb is a powerful tool and can save lots of time. It’s similar to adding a meaningful toString() or operator<< to your custom class.

Crucially, it could help you figure things out faster than your team peers. I first saw this potential when learning remote JVM debugging in GS.

— My view on prints —
In perl and python, I use prints exclusively and never needed interactive debuggers. However, in java/c++/c# I heavily relied on debuggers. Why the stark contrast? No good answer.

Q: when are prints not effective?
A: when the edit-compile-test cycle is too long, not automated but too frequent (like 40 times in 2 hours) and when there is real delivery pressure. Note the test part could involve many steps and many files and other systems.
A: when you can’t edit the file at all. I have not seen it.

A less discussed fact — prints are simple and reliable. GUI or console debuggers are often poorly understood. Look at step-through. Optimization, threads, and exceptions often have unexpected impacts. Or look at program state inspection. Many variables are hard to “open up” in console debuggers. You can print var.func1()

 

gdb stop@simple assignments #compiler optimize

Toggle between -O2 and -O0, which is the default non-optimized compilation.

In my definition, A “simple assignment” is one without using functions. It can get value from another variable or a literal. Simple assignments are optimized away under -O2, so gdb cannot stop on these lines. This applies to break point or step-through.

In particular, if you breakpoint on a simple assignment then “info breakpoint” will show a growing hit count on this breakpoint, but under -O2 gdb would never stop there. -O0 works as expected.

As another illustration, if an if-block contains nothing but simple assignment, then gdb has nowhere to stop inside it and will only stop after the if-block. You won’t know whether you entered it. -O0 works as expected.

%%GTD xp: 2types@technical impasse#难题

See also post on alpha geeks…
See also post on how fast you figure things out relative to peers
See also ##a few projects technically too challeng` 4me
See also https://bintanvictor.wordpress.com/2017/03/26/google-searchable-softwares/
see also https://bintanvictor.wordpress.com/2017/05/29/transparentsemi-transparentopaque-languages/

tuning? never experienced this challenge in my projects.
NPE? Never really difficult in my perience.

#1 complexity/opacity/lack of google help

eg: understanding a hugely complex system like the Quartz dag and layers
eg: replaying raw data, why multicast works consistently but tcp fails consistently
eg: adding ssl to Guardian. Followed the standard steps but didn’t work. Debugger was not able to reveal anything.
Eg: Quartz dag layers
Eg: Quartz cancelled trade

#2 Intermittent, hard to reproduce
eg: Memory leak is one example, in theory but not in my experience

eg: crashes in GMDS? Not really my problem.

eg: Quartz preferences screen frequently but intermittently fails to remember the setting. Unable to debug into it i.e. opaque.