Unless otherwise noted, articles © 2005-2008 Doug Spencer, SecurityBulletins.com. Linking to articles is welcomed. Articles on this site are general information and are NOT GUARANTEED to work for your specific needs. I offer paid professional consulting services and will be happy to develop custom solutions for your specific needs. View the consulting page for more information.


Debugging Applications on Linux systems

From SecurityBulletins.com

Jump to: navigation, search

Debugging Core Files

Written October 2006 by Doug Spencer

If you have a program that has a segmentation fault, the first thing to do is to get a core dump. Many Linux distributions don't automatically save core files. To set saving of core files, run the following command from the BASH shell:

ulimit -c unlimited

Start the prgram that is generating segmentation faults from that shell and it will inherit the core file settings. Once it dumps a core file, you can begin tracking down the source of the problem. Tracking down the source of the problem is usually done with the GNU Debugger, gdb as follows:

$ gdb vmware-serverd
GNU gdb Red Hat Linux (6.3.0.0-1.132.EL4rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...(no debugging symbols found)
Using host libthread_db library "/lib64/tls/libthread_db.so.1".

(gdb) core core.4762
Reading symbols from shared object read from target memory...(no debugging symbols found)...done.
Loaded system supplied DSO at 0xffffe000
Core was generated by `/usr/sbin/vmware-serverd'.
Program terminated with signal 6, Aborted.
warning: svr4_current_sos: Can't read pathname for load map: Input/output error

Reading symbols from /lib/tls/libpthread.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/tls/libpthread.so.0
Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/tls/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/tls/libc.so.6
Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /lib/libnsl.so.1...
(no debugging symbols found)...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/tls/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/tls/libm.so.6
Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /usr/lib/vmware/lib/libcrypto.so.0.9.7/libcrypto.so.0.9.7...
(no debugging symbols found)...done.
Loaded symbols for /usr/lib/vmware/lib/libcrypto.so.0.9.7/libcrypto.so.0.9.7
Reading symbols from /usr/lib/vmware/lib/libssl.so.0.9.7/libssl.so.0.9.7...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/vmware/lib/libssl.so.0.9.7/libssl.so.0.9.7
Reading symbols from /usr/lib/vmware/perl5/site_perl/5.005/auto/IO/IO.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/vmware/perl5/site_perl/5.005/auto/IO/IO.so
Reading symbols from /usr/lib/vmware/perl5/site_perl/5.005/i386-linux/auto/XML/Parser/Expat/Expat.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/vmware/perl5/site_perl/5.005/i386-linux/auto/XML/Parser/Expat/Expat.so
Reading symbols from /usr/lib/vmware/perl5/site_perl/5.005/i386-linux/auto/VMware/VmPerl/VmPerl.so...done.
Loaded symbols for /usr/lib/vmware/perl5/site_perl/5.005/i386-linux/auto/VMware/VmPerl/VmPerl.so
Reading symbols from /usr/lib/vmware/perl5/site_perl/5.005/i386-linux/auto/VMware/HConfig/HConfig.so...done.
Loaded symbols for /usr/lib/vmware/perl5/site_perl/5.005/i386-linux/auto/VMware/HConfig/HConfig.so
Reading symbols from /usr/lib/vmware/perl5/site_perl/5.005/i386-linux/auto/MIME/Base64/Base64.so...done.
Loaded symbols for /usr/lib/vmware/perl5/site_perl/5.005/i386-linux/auto/MIME/Base64/Base64.so
Reading symbols from /usr/lib/vmware/perl5/site_perl/5.005/auto/POSIX/POSIX.so...done.
Loaded symbols for /usr/lib/vmware/perl5/site_perl/5.005/auto/POSIX/POSIX.so

#0  0x080de29a in Util_GetCurrentThreadId ()
(gdb) bt
#0  0x080de29a in Util_GetCurrentThreadId ()
#1  0x08210bd4 in SyncRecMutex_Lock ()
#2  0x0812b6c3 in VmdbDbLock ()
#3  0x08123c34 in Vmdb_IsSet ()
#4  0x081342c4 in VmdbCnxSafeRelease ()
#5  0x08122eba in Vmdb_ProcessCallbacks ()
#6  0x0813a819 in WQPool_GetPoll ()
#7  0x080e3227 in Poll_Callback ()
#8  0x080e352e in Poll_Callback ()
#9  0x080e28e8 in Poll_LoopTimeout ()
#10 0x080e29b8 in Poll_Loop ()
#11 0x08079314 in VMServerdRegisterListener ()
#12 0x08078595 in main ()

From the information shown above, you can see the core file was created when the program was in the Util_GetCurrentThreadId subroutine. Doing a backtrace by typing bt gives you a list of all the subroutines that were called when getting to that point. In the example shown, you see the Util_GetCurrentThreadId routine was called by the SyncRecMutex_Lock, which was called by VmdbDbLock and so on.

At this point, if you have the source code, you can look in the routine that is responsible for the error. The SIGABRT described above was a result of some missing packages in that particular Centos 4/RHEL 4 Linux installation.

Generally, a segmentation fault (SIGSEGV) is the result of using strcpy instead of strncpy and specifying a maximum number of characters to copy or a similar error. Doing a strcpy will try to copy a string until it gets a NULL character that signifies the end of the text. If that NULL has been stripped or was never added to the string, the copy will try to exceed its allocated memory, which usually results in a segmentation fault, commonly shown as "Segmentation fault (core dumped)" or a similar message. Echoing $? from the BASH shell, Korn Shell, or Bourne Shell immediately after a program crashes will print the error that caused its exit. If you run "kill -l" it will show a listing of all the errorlevels as follows:

$ kill -l
 1) SIGHUP       2) SIGINT       3) SIGQUIT      4) SIGILL
 5) SIGTRAP      6) SIGABRT      7) SIGBUS       8) SIGFPE
 9) SIGKILL     10) SIGUSR1     11) SIGSEGV     12) SIGUSR2
13) SIGPIPE     14) SIGALRM     15) SIGTERM     16) SIGSTKFLT
17) SIGCHLD     18) SIGCONT     19) SIGSTOP     20) SIGTSTP
21) SIGTTIN     22) SIGTTOU     23) SIGURG      24) SIGXCPU
25) SIGXFSZ     26) SIGVTALRM   27) SIGPROF     28) SIGWINCH
29) SIGIO       30) SIGPWR      31) SIGSYS      34) SIGRTMIN
35) SIGRTMIN+1  36) SIGRTMIN+2  37) SIGRTMIN+3  38) SIGRTMIN+4
39) SIGRTMIN+5  40) SIGRTMIN+6  41) SIGRTMIN+7  42) SIGRTMIN+8
43) SIGRTMIN+9  44) SIGRTMIN+10 45) SIGRTMIN+11 46) SIGRTMIN+12
47) SIGRTMIN+13 48) SIGRTMIN+14 49) SIGRTMIN+15 50) SIGRTMAX-14
51) SIGRTMAX-13 52) SIGRTMAX-12 53) SIGRTMAX-11 54) SIGRTMAX-10
55) SIGRTMAX-9  56) SIGRTMAX-8  57) SIGRTMAX-7  58) SIGRTMAX-6
59) SIGRTMAX-5  60) SIGRTMAX-4  61) SIGRTMAX-3  62) SIGRTMAX-2
63) SIGRTMAX-1  64) SIGRTMAX   
Personal tools