Wednesday, July 25, 2012

pstack for amd64

If you ever printed stack with gdb(1), you may noticed it's slow. It's OK while debugging, but surely not suitable for some-kind-of-realtime. That's because gdb performs extraction of every symbol of all files to which observing executable been linked.

Here is nice replacement for this - pstack(1), but only for x86 binaries. Here is attempt to do this for x86_64 too. It uses libunwind to unroll stack frames, and then Perl-script (omfg) to extract symbols and debug-info and to make pretty output.

Here maybe a nice use-case on servers: to automatically print out backtrace of monitored processes which is starved, right before killing then. In most cases it's enough to dig the problem, even w/o debug symbols. Or ... just to see how to perform unrolling remote stack with libunwind since I didn't find any example :-)


Anyway, here is example of it's output:
$./pstack64 20794

#0 0x00007fb66ee42020 in /lib/x86_64-linux-gnu/ nanosleep@@GLIBC_2.2.5
#1 0x00007fb66ee41edc in /lib/x86_64-linux-gnu/ __sleep (/build/buildd/eglibc-2.15/posix/../sysdeps/unix/sysv/linux/sleep.c:138)
#2 0x0000000000400561 in /tmp/a.out: fn
#3 0x0000000000400571 in /tmp/a.out: a
#4 0x0000000000400581 in /tmp/a.out: main
#5 0x00007fb66eda576d in /lib/x86_64-linux-gnu/ __libc_start_main (/build/buildd/eglibc-2.15/csu/libc-start.c:258)
#6 0x0000000000400489 in /tmp/a.out: _start

This program don't use any dynamic libraries but libc. Here is example of another program written in C++ and using Qt:

./pstack64 9190
#0 0x00007f9c71a3eb03 in /lib/x86_64-linux-gnu/ __GI___poll (/build/buildd/eglibc-2.15/io/../sysdeps/unix/sysv/linux/poll.c:87)
#1 0x00007f9c70c2a036 in /lib/x86_64-linux-gnu/ -
#2 0x00007f9c70c2a164 in /lib/x86_64-linux-gnu/ g_main_context_iteration
#3 0x00007f9c726cf3bf in /usr/lib/x86_64-linux-gnu/ QEventDispatcherGlib::processEvents(QFlags)
#4 0x00007f9c72c6ad5e in /usr/lib/x86_64-linux-gnu/ -
#5 0x00007f9c7269ec82 in /usr/lib/x86_64-linux-gnu/ QEventLoop::processEvents(QFlags)
#6 0x00007f9c7269eed7 in /usr/lib/x86_64-linux-gnu/ QEventLoop::exec(QFlags)
#7 0x00007f9c726a3f67 in /usr/lib/x86_64-linux-gnu/ QCoreApplication::exec()
#8 0x000000000041be9e in /usr/bin/keepassx: -
#9 0x00007f9c7197976d in /lib/x86_64-linux-gnu/ __libc_start_main (/build/buildd/eglibc-2.15/csu/libc-start.c:258)
#10 0x000000000041caa1 in /usr/bin/keepassx: -
Keepassx binary is stripped, so we can't see it's procedures, dashes printed instead. As you can see C++ names are demangled. BTW, I wondered it's straightforward with c++filt coming with GNU binutils. To make a short story longer ( :-) ), I'll put a piece of README here:


It's easy to read :-) It shows symbols much faster than `gdb -batch` since it performs a lazy lookup. Works well with executables and shared objects. Falling back to dynamic symbols lookup if none of them found in (debug) table.


It's strongly depends on GNU binutils and therefore it's Linux-only It doesn't support threads (even if you pick up right LWP)

Permissions to trace

Since unwind uses ptrace(2), it's worth to note what in latest Linux-distro it's forbidden to trace "foreign" processes by default. For example, see /etc/sysctl.d/10-ptrace.conf in *Ubuntu, or simply run pstack64 with sudo.

Separated debug-info

It's worth to say what many distributions of Linux provide so-called "separated debug-info": dynamic libraries or even executables containing DWARF records. Since debug info in DWARF doesn't affect other sections and do not require any transformation of executable code, it might be easily excluded (stripped) from object file. But since the size is critical, after being compiled shared libraries usually stripped and /usr/lib/ contains nothing but symbols for dynamic loader (likewise extracted by pstack64 too, anyway). But the original one may be installed too.

For example, here is a libc6-dbg in Ubuntu which provides /usr/lib/debug/lib/x86_64-linux-gnu/ The thing is, it can be easily used instead of runtime libc since all virtual addresses (or section offsets) are valid for debug-version too.

BTW, it's interesting to glance on this short introduction to DWARF.

Sunday, July 1, 2012

memcached: dump to disk


Memcached is well-known, excellent memory storage. But what if you need to dump it's content to disk? This may be need, for example, in the following case: you have memcached with around 50% hitrate. Your service' average load is about 70% in rush hour. So, if your cache-server will reboot, you'll lose your cache, and requests will suddenly double, your users will suffer for this time due to timeouts.
Sounds realistic for you? Then, try this fork of memcached: memcached-dd.
As said in README, usage is straightforward: just add `-F file' option to command-line. Memcached will read this `file' at start and write to file.tmp when SIGUSR2 received. Then (after successfull write and sync), it will rename file.tmp -> file. So, `file' should be never truncated. For example:
$ memcached -F /tmp/memcache.dump -m 64 -p 11211 -l

Some notes to be clear

  • Dump performs in separate thread and doesn't block memcached itself
  • If you using TTL for your data, being restored the data will have the same TTL as in the time of dump.
  • All expired and flushed (flush_all command) content left behind
  • There is no any schedule-like maintaining for dumps, it's better to do with crontab and/or your own scripts


I'll show the usage with Perl script. Assume, you have downloaded and built memcached-dd; see INSTALLATION section in README if in doubt. This Perl-scenario will load fake data into memcached:
use Cache::Memcached;

$memd = new Cache::Memcached {
    'servers' => [ "" ]

$| = 1;

for (my $i = 0; $i <= 10000; $i++) {
    $memd->set( "key_$i", "x"x100 . " [$i]" );
    # my $val = $memd->get( "xkey_$i");

    if ($i % 1000 == 0) {
        print "\r$i...";
Having this script, launch the memcached-dd:
$ ./memcached -P /tmp/ -F /tmp/memcached.dump -m 128 -p 11211 -l
# now load the data into memcached:
$ perl ./
# and now, assuming memcached has this data, dump it:
$ kill -USR2 `cat /tmp/`
1Mb dumped: 10001 items (0 expired during dump, 0 nuked by flush)
Moving temprorary /tmp/memcached.dump.tmp -> /tmp/memcached.dump

OK, now you have file /tmp/memcached.dump with all 10000 records dumped. You may reload it anytime launching memcached-dd with the same -F (assuming you killed memcached):

$ ./memcached -P /tmp/ -F /tmp/memcached.dump -m 128 -p 11211 -l
Now check this keys with netcat:
$ echo get key_1 | nc 11211
VALUE key_1 0 104
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx [1]

As you can see, content successfully restored from dump. Hope this will help for your particular usecase. If you have some problems with memcached-dd, feel yourself free to post this issue