q(nm) instrumentation #learning notes

When you want to reduce the opacity of the c++ compiled artifacts, q(nm) is instrumental. It is related to other instrumentation tools like

q(strings -a)

Subset of noteworthy features:
–print-armap? Tested with my *.a file. The filename printed is different from the above
–line-numbers? Tested
–demangle? Similar to c++filt
–dynamic? for “certain” types of shared libraries

My default command line is

nm –print-armap –print-file-name –line-numbers –demangle

q(g++ -g -O) together

https://linux.die.net/man/1/g++ has a section specifically on debugging. It says

GCC allows you to use -g with -O

I think -g adds additional debug info into the binary to help debuggers; -O turns on complier optimization.

By default, our binaries are compiled with “-g3 -O2”. When I debug these binaries, I can see variables but lines are rearranged in source code, causing minor problems. See my blog posts on gdb.

gdb q(next) over if/else +function calls #optimized

I used an optimized binary. Based on limited testing, un-optimized doesn’t suffer from these complexities.

Conventional wisdom: q(next) differs from q(step) and should not go into a function

Rule (simple case): When you are on a line of if-statement in a source code, q(next) would evaluate this condition. If the condition doesn’t involve any function call, then debugger would evaluate it and move to the “presumed next line”, hopefully another simple statement.

Rule 1: suppose your “presumed next line” involves a function call, debugger would often show the first line in the function as the actual “pending”. This may look like step-into!

Eg: In the example below. Previous pending is showing L432 (See Rule 2b to interpret it). The presumed line is L434, but L434 involves a function call, so debugger actually shows L69 as the “pending” i.e. the first line in the function

Rule 2 (more tricky): suppose presumed line is an if-statement involving a function call. Debugger would show first line in the function as the pending.

Eg: In the example below, Previous pending was showing L424. Presumed line is L432, but we hit Rule 2, so actual pending is L176, i.e. first line in the function.

Rule 2b: when debugger shows such an if-statement as the “pending”, then probably the function call completed and debugger is going to evaluate the if-condition.

424 if (isSendingLevel1){
425 //……
426 //……….
427 //……..
428 //……….
429 } // end of if
430 } // end of an outer block
432 if (record->generateTopOfBook()
433 && depthDb->isTopOfTheBook(depthDataRecord)) {
434 record->addTopOfBookMarker(outMsg);
435 }

#1 challenge if u rely@gdb to figure things out: optimizer

Background: https://bintanvictor.wordpress.com/2015/12/31/wall-st-survial-how-fast-you-figure-things-out-relative-to-team-peers/ explains why “figure things out quickly” is such a make-or-break factor.

In my recent experience, I feel compiler optimization is the #1 challenge. It can mess up GDB step-through. For a big project using automated build, it is often tricky to disable every optimization flag like “-O2”.

More fundamentally, it’s often impossible to tell if the compiled binary in front of you was compiled as optimized or not. Rarely the binary shows it.

Still, compared to other challenges in figuring things out, this one is tractable.

gdb skill level@Wall St

I notice that, absolutely None of my c++  veteran colleagues (I asked only 3) [2] is a gdb expert as there are concurrency experts, algo experts [1], …

Most of my c++ colleagues don’t prefer (reluctance?) console debugger. Many are more familiar with GUI debuggers such as eclipse and MSVS. All agree that prints are often a sufficient debugging tool.

[1] Actually, these other domains are more theoretical and produces “experts”.

[2] maybe I didn’t meet enough true c++ alpha geeks. I bet many of them may have very good gdb skills.

I would /go out on a limb/ to say that gdb is a powerful tool and can save lots of time. It’s similar to adding a meaningful toString() or operator<< to your custom class.

Crucially, it could help you figure things out faster than your team peers. I first saw this potential when learning remote JVM debugging in GS.

— My view on prints —
In perl and python, I use prints exclusively and never needed interactive debuggers. However, in java/c++/c# I heavily relied on debuggers. Why the stark contrast? No good answer.

Q: when are prints not effective?
A: when the edit-compile-test cycle is too long, not automated but too frequent (like 40 times in 2 hours) and when there is real delivery pressure. Note the test part could involve many steps and many files and other systems.
A: when you can’t edit the file at all. I have not seen it.

A less discussed fact — prints are simple and reliable. GUI or console debuggers are often poorly understood. Look at step-through. Optimization, threads, and exceptions often have unexpected impacts. Or look at program state inspection. Many variables are hard to “open up” in console debuggers. You can print var.func1()


gdb stop@simple assignments #compiler optimize

Toggle between -O2 and -O0, which is the default non-optimized compilation.

In my definition, A “simple assignment” is one without using functions. It can get value from another variable or a literal. Simple assignments are optimized away under -O2, so gdb cannot stop on these lines. This applies to break point or step-through.

In particular, if you breakpoint on a simple assignment then “info breakpoint” will show a growing hit count on this breakpoint, but under -O2 gdb would never stop there. -O0 works as expected.

As another illustration, if an if-block contains nothing but simple assignment, then gdb has nowhere to stop inside it and will only stop after the if-block. You won’t know whether you entered it. -O0 works as expected.

%%GTD xp: 2types@technical impasse#难题

See also post on alpha geeks…
See also post on how fast you figure things out relative to peers
See also https://bintanvictor.wordpress.com/2017/03/26/google-searchable-softwares/
see also https://bintanvictor.wordpress.com/2017/05/29/transparentsemi-transparentopaque-languages/

tuning? never experienced this challenge in my projects.
NPE? Never really difficult in my perience.

#1 complexity/opacity/lack of google help

eg: understanding a hugely complex system like the Quartz dag and layers
eg: replaying raw data, why multicast works consistently but tcp fails consistently
eg: adding ssl to Guardian. Followed the standard steps but didn’t work. Debugger was not able to reveal anything.
Eg: Quartz dag layers
Eg: Quartz cancelled trade

#2 Intermittent, hard to reproduce
eg: Memory leak is one example, in theory but not in my experience

eg: crashes in GMDS? Not really my problem.

eg: Quartz preferences screen frequently but intermittently fails to remember the setting. Unable to debug into it i.e. opaque.

Locate msg]binary feed #multiple issues solved

Hi guys, thanks to all your help, I managed to locate the very first trading session message in the raw data file.

We hit and overcame multiple obstacles in this long “needle search in a haystack”.

  • · Big Obstacle 1: endian-ness. It turned out the raw data is little-endian. For my “needle”, the symbol integer id 15852(in decimal) or 3dec(in hex) is printed swapped as “ec3d” when I finally found it.

Solution: read the exchange spec. It should be mentioned.

  • · Big Obstacle 2: my hex viewers (like “xxd”) adds line breaks to the output, so my needle can be missed during my search. (Thanks to Vishal for pointing this out.)

Solution 1: xxd -c 999999 raw/feed/file > tmp.txt; grep $needle tmp.txt

The default xxd column size is 16 so every 16 bytes output will get a line break — unwanted! So I set a very large column size of 999999.

Solution 2: in vi editor after “%!xxd -p” if you see line breaks, then you can still search for “ec\_s*3d”. Basically you need to insert “\_s*” between adjacent bytes.

Here’s a 4-byte string I was able to find. It span across lines: 15\_s*00\_s*21\_s*00

  • · Obstacle 3: identify the data file among 20 files. Thanks to this one obstacle, I spent most of my time searching in the wrong files 😉

Solution: remove each file successively, starting from the later hours, and retest, until the needle stops showing. The last removed file must contain our needle. That file is a much smaller haystack.

o one misleading info is the “9.30 am” mentioned in the spec. Actually the message came much earlier.

o Another misleading info is the timestamp passed to my parser function. Not sure where it comes from, but it says 08:00:00.1 am, so I thought the needle must be in the 8am file, but actually, it is in the 4am file. In this feed, the only reliable timestamp I have found is the one in packet header, one level above the messages.

  • · Obstacle 4: my “needle” was too short so there are too many useless matches.

Solution: find a longer and more unique needle, such as the SourceTime field, which is a 32-bit integer. When I convert it to hex digits I get 8 hex digits. Then I flip it due to endian-ness. Then I get a more unique needle “008e0959”. I was then able to search across all 14 data files:

for f in arca*0; do

xxd -c999999 -p $f > $f.hex

grep -ioH 008e0959 $f.hex && echo found in $f


  • · Obstacle 5: I have to find and print the needle using my c++ parser. It’s easy to print out wrong hex representation using C/C++, so for most of this exercise I wasn’t sure if I was looking at correct hex dump in my c++ log.

o If you convert a long byte array to hex and print without whitespace, you could see 15002100ffffe87600,but when I added a space after each byte, it looks like 15 00 21 00 ffffe876 00, so the 3rd byte was overflowing without warning!

o If you forget padding, then you can see a lot of single “0” when you should get “00”. Again, if you don’t include white space you won’t notice.

Solution: I have worked out some simplified code that works. I have a c++ solution and c solution. You can ask me if you need it.

  • · Obstacle 6: In some cases, sequence number is not in the raw feed. In this case the sequence number is in the feed, so Nick’s suggestion was valid, but I was blocked by other obstacles.

Tip: If sequence number is in the feed, you would probably spot a pattern of incrementing hex numbers periodically in the hex viewer.