If you ever printed stack with gdb(1), you may noticed it's slow. It's OK while debugging, but surely not suitable for some-kind-of-realtime. That's because gdb performs extraction of every symbol of all files to which observing executable been linked.
Here is nice replacement for this - pstack(1), but only for x86 binaries. Here is attempt to do this for x86_64 too. It uses libunwind to unroll stack frames, and then Perl-script (omfg) to extract symbols and debug-info and to make pretty output.
Here maybe a nice use-case on servers: to automatically print out backtrace of monitored processes which is starved, right before killing then. In most cases it's enough to dig the problem, even w/o debug symbols. Or ... just to see how to perform unrolling remote stack with libunwind since I didn't find any example :-)
ExampleAnyway, here is example of it's output:
$./pstack64 20794 20794:./a.out #0 0x00007fb66ee42020 in /lib/x86_64-linux-gnu/libc-2.15.so: nanosleep@@GLIBC_2.2.5 #1 0x00007fb66ee41edc in /lib/x86_64-linux-gnu/libc-2.15.so: __sleep (/build/buildd/eglibc-2.15/posix/../sysdeps/unix/sysv/linux/sleep.c:138) #2 0x0000000000400561 in /tmp/a.out: fn #3 0x0000000000400571 in /tmp/a.out: a #4 0x0000000000400581 in /tmp/a.out: main #5 0x00007fb66eda576d in /lib/x86_64-linux-gnu/libc-2.15.so: __libc_start_main (/build/buildd/eglibc-2.15/csu/libc-start.c:258) #6 0x0000000000400489 in /tmp/a.out: _start
This program don't use any dynamic libraries but libc. Here is example of another program written in C++ and using Qt:
./pstack64 9190 9190:keepassx #0 0x00007f9c71a3eb03 in /lib/x86_64-linux-gnu/libc-2.15.so: __GI___poll (/build/buildd/eglibc-2.15/io/../sysdeps/unix/sysv/linux/poll.c:87) #1 0x00007f9c70c2a036 in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.3: - #2 0x00007f9c70c2a164 in /lib/x86_64-linux-gnu/libglib-2.0.so.0.3200.3: g_main_context_iteration #3 0x00007f9c726cf3bf in /usr/lib/x86_64-linux-gnu/libQtCore.so.4.8.1: QEventDispatcherGlib::processEvents(QFlags) #4 0x00007f9c72c6ad5e in /usr/lib/x86_64-linux-gnu/libQtGui.so.4.8.1: - #5 0x00007f9c7269ec82 in /usr/lib/x86_64-linux-gnu/libQtCore.so.4.8.1: QEventLoop::processEvents(QFlags) #6 0x00007f9c7269eed7 in /usr/lib/x86_64-linux-gnu/libQtCore.so.4.8.1: QEventLoop::exec(QFlags) #7 0x00007f9c726a3f67 in /usr/lib/x86_64-linux-gnu/libQtCore.so.4.8.1: QCoreApplication::exec() #8 0x000000000041be9e in /usr/bin/keepassx: - #9 0x00007f9c7197976d in /lib/x86_64-linux-gnu/libc-2.15.so: __libc_start_main (/build/buildd/eglibc-2.15/csu/libc-start.c:258) #10 0x000000000041caa1 in /usr/bin/keepassx: -Keepassx binary is stripped, so we can't see it's procedures, dashes printed instead. As you can see C++ names are demangled. BTW, I wondered it's straightforward with c++filt coming with GNU binutils. To make a short story longer ( :-) ), I'll put a piece of README here:
BenefitsIt's easy to read :-) It shows symbols much faster than `gdb -batch` since it performs a lazy lookup. Works well with executables and shared objects. Falling back to dynamic symbols lookup if none of them found in (debug) table.
DrawbacksIt's strongly depends on GNU binutils and therefore it's Linux-only It doesn't support threads (even if you pick up right LWP)
Permissions to traceSince unwind uses ptrace(2), it's worth to note what in latest Linux-distro it's forbidden to trace "foreign" processes by default. For example, see /etc/sysctl.d/10-ptrace.conf in *Ubuntu, or simply run pstack64 with sudo.
It's worth to say what many distributions of Linux provide so-called "separated debug-info": dynamic libraries or even executables containing DWARF records. Since debug info in DWARF doesn't affect other sections and do not require any transformation of executable code, it might be easily excluded (stripped) from object file. But since the size is critical, after being compiled shared libraries usually stripped and /usr/lib/ contains nothing but symbols for dynamic loader (likewise extracted by pstack64 too, anyway). But the original one may be installed too.
For example, here is a libc6-dbg in Ubuntu which provides /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.15.so. The thing is, it can be easily used instead of runtime libc since all virtual addresses (or section offsets) are valid for debug-version too.
BTW, it's interesting to glance on this short introduction to DWARF.