At a customer site, we had a C++ program (renamed for the purpose of this blog to myprogram) which was failing after running for some time. It’s a program processing messages from an external system and seemed to fail only when the processing of the message triggered some given operations. The error message was:

lookup error: /home/xxx/bin/xxx/myprogram: undefined symbol: _Z22CxxxPxxxExxxPxxxR6CDBManRKSsRSt6vectorISsSaISsEE

The first thought was to use ldd to check whether everything was fine:

# ldd myprogram
        linux-gate.so.1 =>  (0xffffe000)
        libxxx230.so => not found
        libmc3adv.so => not found
        libsybdb.so => not found
        libxxxutl.so => not found
        libxxxdb.so => not found
        libxxxcustom.so => not found
        libxerces-c.so.23 => not found
        libdl.so.2 => /lib/libdl.so.2 (0x4001e000)
        libstdc++.so.5 => /usr/lib/libstdc++.so.5 (0x40022000)
        libm.so.6 => /lib/tls/libm.so.6 (0x400df000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x40101000)
        libc.so.6 => /lib/tls/libc.so.6 (0x40109000)
        /lib/ld-linux.so.2 (0x40000000)

So many not found ! Ok I must have forgotten something… Of course, LD_LIBRARY_PATH is probably not set in my shell:

# pidof myprogram
10132
# export `cat /proc/10132/environ | strings | grep LD_LIBRARY_PATH`

• pidof return the id of the process running the specified program
• /proc/10132/environ contains the whole environment of the specified process
• strings is required to convert the 0x00 separated strings of /proc/xxx/environ
• The rest just gets the line LD_LIBRARY_PATH=… and uses export to set it in my shell as well

Now I have the same LD_LIBRARY_PATH as the process running myprogram and can rerun ldd:

ldd myprogram
        linux-gate.so.1 =>  (0xffffe000)
        libxxx230.so => /home/xxx/libs/libxxx230.so (0x40018000)
        libmc3adv.so => /home/xxx/libs/libmc3adv.so (0x40051000)
        libsybdb.so => /opt/sybase-12.5/OCS-12_5/lib/libsybdb.so (0x403a5000)
        libxxxutl.so => /home/xxx/libs/libxxxutl.so (0x404e1000)
        libxxxdb.so => /home/xxx/libs/libxxxdb.so (0x405c5000)
        libxxxcustom.so => /home/xxx/libs/libxxxcustom.so (0x4068d000)
        libxerces-c.so.23 => /home/xxx/libs/libxerces-c.so.23 (0x40695000)
        libdl.so.2 => /lib/libdl.so.2 (0x40985000)
        libstdc++.so.5 => /usr/lib/libstdc++.so.5 (0x40988000)
        libm.so.6 => /lib/tls/libm.so.6 (0x40a46000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x40a68000)
        libc.so.6 => /lib/tls/libc.so.6 (0x40a70000)
        libstdc++-libc6.1-1.so.2 => /usr/lib/libstdc++-libc6.1-1.so.2 (0x40b8a000)
        libpthread.so.0 => /lib/tls/libpthread.so.0 (0x40bd2000)
        /lib/ld-linux.so.2 (0x40000000)

Everything looks OK. So it’s not that a library is missing. Also the paths to the libraries are OK.

Executing ldd -r -d myprogram basically just shows me the same plus an error with the missing symbol.

Finally, comparing the size of the library files on this system and at another customer site with the exact same version of the software gave us the answer: somehow a different version of one of our library landed there and didn’t contain the missing symbol.

So the search ended there. Otherwise we’d have had to use the following:

Use c++filt to find out the method behind the mangled symbol name:

# c++filt _Z22CxxxPxxxExxxPxxxR6CDBManRKSsRSt6vectorISsSaISsEE
CxxxPxxxExxxPxxx(CDBMan&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&)

Use nm to list the symbols in the libraries shown by ldd:

# nm /home/xxx/libs/libxxxdb.so

This will show the mangled symbol names. For demangled symbols, use the -C option:

 # nm -C /home/xxx/libs/libxxxdb.so