Unless otherwise noted, articles © 2005-2008 Doug Spencer, SecurityBulletins.com. Linking to articles is welcomed. Articles on this site are general information and are NOT GUARANTEED to work for your specific needs. I offer paid professional consulting services and will be happy to develop custom solutions for your specific needs. View the consulting page for more information.
Debugging Applications on Linux systems
From SecurityBulletins.com
Debugging Core Files
Written October 2006 by Doug Spencer
If you have a program that has a segmentation fault, the first thing to do is to get a core dump. Many Linux distributions don't automatically save core files. To set saving of core files, run the following command from the BASH shell:
ulimit -c unlimited
Start the prgram that is generating segmentation faults from that shell and it will inherit the core file settings. Once it dumps a core file, you can begin tracking down the source of the problem. Tracking down the source of the problem is usually done with the GNU Debugger, gdb as follows:
$ gdb vmware-serverd GNU gdb Red Hat Linux (6.3.0.0-1.132.EL4rh) Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu"...(no debugging symbols found) Using host libthread_db library "/lib64/tls/libthread_db.so.1". (gdb) core core.4762 Reading symbols from shared object read from target memory...(no debugging symbols found)...done. Loaded system supplied DSO at 0xffffe000 Core was generated by `/usr/sbin/vmware-serverd'. Program terminated with signal 6, Aborted. warning: svr4_current_sos: Can't read pathname for load map: Input/output error Reading symbols from /lib/tls/libpthread.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/tls/libpthread.so.0 Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libdl.so.2 Reading symbols from /lib/tls/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/tls/libc.so.6 Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/ld-linux.so.2 Reading symbols from /lib/libnsl.so.1... (no debugging symbols found)...done. Loaded symbols for /lib/libnsl.so.1 Reading symbols from /lib/tls/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/tls/libm.so.6 Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libcrypt.so.1 Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libnss_files.so.2 Reading symbols from /usr/lib/vmware/lib/libcrypto.so.0.9.7/libcrypto.so.0.9.7... (no debugging symbols found)...done. Loaded symbols for /usr/lib/vmware/lib/libcrypto.so.0.9.7/libcrypto.so.0.9.7 Reading symbols from /usr/lib/vmware/lib/libssl.so.0.9.7/libssl.so.0.9.7...(no debugging symbols found)...done. Loaded symbols for /usr/lib/vmware/lib/libssl.so.0.9.7/libssl.so.0.9.7 Reading symbols from /usr/lib/vmware/perl5/site_perl/5.005/auto/IO/IO.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/vmware/perl5/site_perl/5.005/auto/IO/IO.so Reading symbols from /usr/lib/vmware/perl5/site_perl/5.005/i386-linux/auto/XML/Parser/Expat/Expat.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/vmware/perl5/site_perl/5.005/i386-linux/auto/XML/Parser/Expat/Expat.so Reading symbols from /usr/lib/vmware/perl5/site_perl/5.005/i386-linux/auto/VMware/VmPerl/VmPerl.so...done. Loaded symbols for /usr/lib/vmware/perl5/site_perl/5.005/i386-linux/auto/VMware/VmPerl/VmPerl.so Reading symbols from /usr/lib/vmware/perl5/site_perl/5.005/i386-linux/auto/VMware/HConfig/HConfig.so...done. Loaded symbols for /usr/lib/vmware/perl5/site_perl/5.005/i386-linux/auto/VMware/HConfig/HConfig.so Reading symbols from /usr/lib/vmware/perl5/site_perl/5.005/i386-linux/auto/MIME/Base64/Base64.so...done. Loaded symbols for /usr/lib/vmware/perl5/site_perl/5.005/i386-linux/auto/MIME/Base64/Base64.so Reading symbols from /usr/lib/vmware/perl5/site_perl/5.005/auto/POSIX/POSIX.so...done. Loaded symbols for /usr/lib/vmware/perl5/site_perl/5.005/auto/POSIX/POSIX.so #0 0x080de29a in Util_GetCurrentThreadId () (gdb) bt #0 0x080de29a in Util_GetCurrentThreadId () #1 0x08210bd4 in SyncRecMutex_Lock () #2 0x0812b6c3 in VmdbDbLock () #3 0x08123c34 in Vmdb_IsSet () #4 0x081342c4 in VmdbCnxSafeRelease () #5 0x08122eba in Vmdb_ProcessCallbacks () #6 0x0813a819 in WQPool_GetPoll () #7 0x080e3227 in Poll_Callback () #8 0x080e352e in Poll_Callback () #9 0x080e28e8 in Poll_LoopTimeout () #10 0x080e29b8 in Poll_Loop () #11 0x08079314 in VMServerdRegisterListener () #12 0x08078595 in main ()
From the information shown above, you can see the core file was created when the program was in the Util_GetCurrentThreadId subroutine. Doing a backtrace by typing bt gives you a list of all the subroutines that were called when getting to that point. In the example shown, you see the Util_GetCurrentThreadId routine was called by the SyncRecMutex_Lock, which was called by VmdbDbLock and so on.
At this point, if you have the source code, you can look in the routine that is responsible for the error. The SIGABRT described above was a result of some missing packages in that particular Centos 4/RHEL 4 Linux installation.
Generally, a segmentation fault (SIGSEGV) is the result of using strcpy instead of strncpy and specifying a maximum number of characters to copy or a similar error. Doing a strcpy will try to copy a string until it gets a NULL character that signifies the end of the text. If that NULL has been stripped or was never added to the string, the copy will try to exceed its allocated memory, which usually results in a segmentation fault, commonly shown as "Segmentation fault (core dumped)" or a similar message. Echoing $? from the BASH shell, Korn Shell, or Bourne Shell immediately after a program crashes will print the error that caused its exit. If you run "kill -l" it will show a listing of all the errorlevels as follows:
$ kill -l 1) SIGHUP 2) SIGINT 3) SIGQUIT 4) SIGILL 5) SIGTRAP 6) SIGABRT 7) SIGBUS 8) SIGFPE 9) SIGKILL 10) SIGUSR1 11) SIGSEGV 12) SIGUSR2 13) SIGPIPE 14) SIGALRM 15) SIGTERM 16) SIGSTKFLT 17) SIGCHLD 18) SIGCONT 19) SIGSTOP 20) SIGTSTP 21) SIGTTIN 22) SIGTTOU 23) SIGURG 24) SIGXCPU 25) SIGXFSZ 26) SIGVTALRM 27) SIGPROF 28) SIGWINCH 29) SIGIO 30) SIGPWR 31) SIGSYS 34) SIGRTMIN 35) SIGRTMIN+1 36) SIGRTMIN+2 37) SIGRTMIN+3 38) SIGRTMIN+4 39) SIGRTMIN+5 40) SIGRTMIN+6 41) SIGRTMIN+7 42) SIGRTMIN+8 43) SIGRTMIN+9 44) SIGRTMIN+10 45) SIGRTMIN+11 46) SIGRTMIN+12 47) SIGRTMIN+13 48) SIGRTMIN+14 49) SIGRTMIN+15 50) SIGRTMAX-14 51) SIGRTMAX-13 52) SIGRTMAX-12 53) SIGRTMAX-11 54) SIGRTMAX-10 55) SIGRTMAX-9 56) SIGRTMAX-8 57) SIGRTMAX-7 58) SIGRTMAX-6 59) SIGRTMAX-5 60) SIGRTMAX-4 61) SIGRTMAX-3 62) SIGRTMAX-2 63) SIGRTMAX-1 64) SIGRTMAX
