debugger stepping into library

I often need my debugger to step into library source code.

Easy in java:

c++ is harder. I need to find more details.

  • in EclipseCDT, STL source code is available to IDE ( probably because class templates are usually in the form of header files), and debugger is able to step through it, but not so well.

Overall, I feel debugger support is significantly better in VM-based languages than c++, even though debugger was invented before these new languages.

I guess the VM or the “interpreter” can serve as an “interceptor” between debugger and target application. The interceptor can receive debugger commands and suspend execution of the target application.

error stack trace: j^c++

Without stack trace, silent crashes are practically intractable.

In my java and python /career/ (I think c# as well) , exceptions always generate a console stack trace. The only time it didn’t happen was a Barclays JNI library written in c++..

In c++, getting the stack trace is harder.

  • when I used the ETSFlowAsert() construct I get a barely usable stack trace, with function names, without line numbers
  • [[safeC++]] described a technique to generate a simple stack trace with some line numbers but I have never tried it
  • the standard assert() macro doesn’t generate stack trace
  • In RTS and mvea, memory access errors lead to seg-fault and core dump. In these contexts we are lucky because the runtime environment (host OS, standard library, seg-fault signal handler ..) cooperate to dump some raw data into the core file, but it’s not as reliable as the JVM runtime … Here are some of the obstacles:
    • core files may be suppressed
    • To actually make out this information, we need gdb + a debug build of the binary with debug symbols.
    • it can take half an hour to load the debug symbols
    • My blogpost %%first core dump gdb session describes how much trouble and time is involved to see the function names in the stack trace.


gdb: dump STL container %%experience

First let’s talk about custom containers. GDB would show the field names of an object, but frequently not the values. I guess integers values might show up but more than half the fields are pointers ( actually char-array field would be easy to print.)

If I call a function on the object, I have to be very lucky and very careful. q(->) has never worked for me so far, so I need to use q(*) to de-reference every pointer before calling a method on the pointee, and pray it works. works on std::map …

A simple experiment using

  • g++ -g theFile.cpp && gdb -iex ‘add-auto-load-safe-path .’ ./a.out
  • (gdb) print *(li._M_impl._M_start+1) # can print 2nd element if it’s std::string or double
    • Note before vector initialization, gdb already shows the addresses inside the vector, but some addresses are not populated. Just retry after the initialization.
  • std::unordered_map is doable:
    • (gdb) print **(tm._M_buckets) # prints first pair in a hash table bucket
    • (gdb) print *((**(tm._M_buckets))._M_next) # next pair in the same bucket
  • std::map content is harder
    • (gdb) print *(int*)(tm._M_t._M_impl._M_header._M_left+1) # prints one key
    • (gdb) print *(int*)(tm._M_t._M_impl._M_header._M_right+1) # prints another key in the pair
    • (gdb) print *(int*)((void*)(tm._M_t._M_impl._M_header._M_right+1)+sizeof(int)) #prints the value in the pair.
      • the (void*) is needed before we add sizeof(value_type). Without the cast, the pointer arithmetic would be different.
      • from the key field to value field, we move by 4 bytes (i.e. sizeof value_type) from  0x6050e0 to 0x6050e4. It’s actually easy to manually type .. print *0x6050e4
      • I suspect the _M_right pointer is seated at the “color” field. Increment to the key field?

loading .gdbinit

My experiments show that $HOME/.gdbinit is discovered. I actually changed the $HOME env variable:)

However, I hit

  warning: not using untrusted file "/v/global/user/b/bi/bint/.gdbinit"

, even though I added q(gdb -iex ‘set auto-load safe-path …’). I guess the warning come up before the -iex option takes effect

  gdb -ix '/path/to/.gdbinit' # also failed due to untrusted file
  g++ -std=c++0x -g dump.cpp && gdb -iex 'add-auto-load-safe-path .' ./a.out # working

gdb symbol-loading too time-consuming

After I attach gdb, it immediately starts a prolonged symbol loading process. It’s better to skip the loading, and selectively load some symbols. describes how to use ~/.gdbinit, but I had no permission to write to the ~/

gdb -iex ‘set auto-solib-add off’ …. # worked

–loading a particular *.so file

I got “No loaded shared libraries match the pattern” and fixed it by

shar #instead of /full/path/to/

gdb to show c++thread wait`for mutex/condVar shows my experiment using gdb supplied by StrawberryPerl.

On this g++/gdb set-up, “info threads” shows thread id number 1 for main thread, “2” for the thread whose pthread_self() == 2 … matching 🙂

The same “info-threads” output also shows

  • one of the worker threads is executing sleep() while holding lock (by design)
  • the other worker threads are all waiting for the lock.
  • At the same time, the main thread is waiting in a conditional variable, so info-threads shows it executing a different function.

gdb cheatsheet

#1 tip:
gdb –tui # starts an upper screen showing a moving marker against the current line. If you started gdb without “–tui”, you can still use Ctrl-X Ctrl-A.

#2 tip:
After hitting a breakpoint, you can use q(n) i.e. q(next) and just Enter to repeat previous q(n).

I found it easier to use q(break filename:lineno) # Filename doesn’t need full path J

Other essential commands

  • b myFunc
  • b myClass::myFunc
  • info break # or info b to list all breakpoints
  • · bt # or backtrace
  • · frame 0
  • · list
  • · p for print
  • · info variables
  • · info locals
  • · info args
  • · n # for step-over
  • · s # for step-in
  • · fin # for step-out
  • ctrl-L toclear gdb command screen

gdb –tui #split screen

You can also start gdb normally, then switch to split screen, with “ctrl-x  ctrl-a” (thanks to Gregory).

Upper screen shows the source code with a moving marker

–Here’s my full gdb command showing

  • –args to run the target executable with arguments
  • redirect stderr to a file so my gdb screen isn’t messed up — not always effective

gdb –tui –args $base/shared/tp_xtap/bin/xtap -D 9 -c $base/etc/test_replay.cfg 2 >

gdb q[next] over if/else +function calls #optimized

I used an optimized binary. Based on limited testing, un-optimized doesn’t suffer from these complexities.

Conventional wisdom: q(next) differs from q(step) and should not go into a function

Rule (simple case): When you are on a line of if-statement in a source code, q(next) would evaluate this condition. If the condition doesn’t involve any function call, then debugger would evaluate it and move to the “presumed next line”, hopefully another simple statement.

Rule 1: suppose your “presumed next line” involves a function call, debugger would often show the first line in the function as the actual “pending”. This may look like step-into!

Eg: In the example below. Previous pending is showing L432 (See Rule 2b to interpret it). The presumed line is L434, but L434 involves a function call, so debugger actually shows L69 as the “pending” i.e. the first line in the function

Rule 2 (more tricky): suppose presumed line is an if-statement involving a function call. Debugger would show first line in the function as the pending.

Eg: In the example below, Previous pending was showing L424. Presumed line is L432, but we hit Rule 2, so actual pending is L176, i.e. first line in the function.

Rule 2b: when debugger shows such an if-statement as the “pending”, then probably the function call completed and debugger is going to evaluate the if-condition.

424 if (isSendingLevel1){
425 //……
426 //……….
427 //……..
428 //……….
429 } // end of if
430 } // end of an outer block
432 if (record->generateTopOfBook()
433 && depthDb->isTopOfTheBook(depthDataRecord)) {
434 record->addTopOfBookMarker(outMsg);
435 }

#1 challenge if u rely@gdb to figure things out: optimizer

Background: explains why “figure things out quickly” is such a make-or-break factor.

In my recent experience, I feel compiler optimization is the #1 challenge. It can mess up GDB step-through. For a big project using automated build, it is often tricky to disable every optimization flag like “-O2”.

More fundamentally, it’s often impossible to tell if the compiled binary in front of you was compiled as optimized or not. Rarely the binary shows it.

Still, compared to other challenges in figuring things out, this one is tractable.

gdb stop@simple assignments #compiler optimize

Toggle between -O2 and -O0, which is the default non-optimized compilation.

In my definition, A “simple assignment” is one without using functions. It can get value from another variable or a literal. Simple assignments are optimized away under -O2, so gdb cannot stop on these lines. This applies to break point or step-through.

In particular, if you breakpoint on a simple assignment then “info breakpoint” will show a growing hit count on this breakpoint, but under -O2 gdb would never stop there. -O0 works as expected.

As another illustration, if an if-block contains nothing but simple assignment, then gdb has nowhere to stop inside it and will only stop after the if-block. You won’t know whether you entered it. -O0 works as expected.

%%first core dump gdb session

gdb $base/shared/tp_xtap/bin/xtap   ~/nx_parser/core

### 1st arg is the executable responsible for the core dump. If this executable is correct, then you should not get “No symbol table is loaded. Use the file command.” If you do, then try

(gdb) file /path/to/xtap

I think “symbol” means the variable/function names. I think gdb sees only … address not names. To translate them, presumably you need *that* file.

In my case the “xtap” executable is a debug-build, verified with “file” and “objdump” commands, according to

Anyway, we need to load the symbols. For a large executable (like minerva) with many *.so libraries, 30 minutes may be needed. Once the symbols are loaded, you should run “bt” to load the back trace

(gdb) bt

Now choose one of the frames such as the 2nd most recent function i.e. Frame #1

(gdb) frame 1

Now gdb shows the exact line number and the line of source code. You can see before/after lines with “list”. Those lines may belong to other functions collocating in the same source file.

(gdb) list

remote debugging MSVS@winxp; app@win2008 or 2003 is the best guide I have found. Some comments —

Firewall – I didn’t need to change anything

— I ran the msvsmon.exe from a share. Just double click to run the exe
** msvsmon.exe need not run-as admin
** msvsmon.exe.config needs no change
** MUST add my name under msvsmon->tool->permission. Repeat in every msvsmon.exe session — Not saved.

— Attach to process
** transport = default
** qualifier = eqdpr@uwsamapos01, where eqdpr is the msvsmon.exe user on the remote
** refresh

Should see every process — Connected:) Once connected, before attach, msvsmon window will show who connected when.

You can start the remote app before or after Attach.

3 requirements to debug into c++from another language

The driver app (java, c#, MS-excel …) had better use dynamic linked library to access C++ code. (Not sure if a static library can suffice.) Before the app terminates, we attach our debugger (gdb, MSVC etc) to that process.,6419.0.html has a popular write-up about MSVC debugger. i guess if you use g++ to compile, then you may need gdb; if you used MSVC to compile, then you may need MSVC debugger.

Obviously We need source code used to compile the library. Of course, the library must be compiled with -G.

Annoying trivial exceptions when debugging Eclipse/MSVS

My eclipse debugger always stops at URLClassPath$JarLoader. Solution – Try turning off Window>Preferences>Java>Debug>Suspend execution on uncaught exceptions?

For MSVS, I find it extrememly useful to turn on “break on exception” but some exceptions like System.TypeLoadException are useless. To disable them, Debug->Exceptions->Find and uncheck the specific exceptions

local variables unavailable in eclipse debugger

If you can’t inspect a method argument (or local variable) object using Expressions/Watch/Inspect … Be creative and try Eclipse debug view -> Variables. It’s possible that local vars are not visible, but class fields are. Many fields of “this” are objects, so their fields are visible. Using a combination of fields you can often deduce the values of those local vars. Simple example — if a local var is set to someStaticMap.get( this.age%10 ), you can deduce that value.

Here’s the original context of this blog — During remote debugging one day, it turned out all these variables are non-viewable because we had built the project without including “debug information”. If your build-master allows, then the best solution is —

  javac -g:vars # Local variable debugging information added to byte code in generated *.class files

Other eclipse users had the same issue —