Thursday 12 September 2013

Enable Core File Dumps for Application Crashes or Segmentation Faults

Core dumps are really useful debugging tool for system administrators especially when applications like Apache or MySQL crashes. When an application is poorly written or has a bug on, it may tries to access memory location that it is not supposed to and segmentation fault occurs.

Redhat and it's clones core file creation is disabled by default. You can verify this by

# ulimit -c
0

Where zero is the size of the core dump file. Do the following steps

Step 1: Set limit for core dump file 

You can restrict your core dump file to 100MB (102400KB), you should say that in kilobytes.

vim /etc/security/limits.conf

Replace  #*               soft    core            0  with  *               soft    core            102400

Step 2: Set limit for all the users

Now you need to change this in /etc/profile file. If you are using RHEL you can find

# vim /etc/profile
ulimit -S -c 0 > /dev/null 2>1

Replace the above ulimit command in /etc/profile with the following

ulimit -S -c 102400 > /dev/null 2>1

If you don't find it in any of your Redhat clones don't worry just add this to /etc/profile file.

Step 3: Enable debugging for all the applications

To enable debugging for all the applications, edit  /etc/sysconfig/init file and add the following
# vim /etc/sysconfig/init

DAEMON_COREFILE_LIMIT=102400

If you want to enable core dumping for a particular application you can say that in it's corresponding file in /etc/sysconfig/ directory.

Now edit /etc/sysctl.conf file and add the following

# vim /etc/sysctl.conf

kernel.core_pattern = /tmp/core-%e-%s-%u-%g-%p-%t
fs.suid_dumpable = 2

When an application crashes or killed by a signal, a core dump file is created inside the directory /tmp named core, but you can define the core dump file name with the following template which can contain % specifiers which are substituted by the following values when a core file is created:

%% - A single % character
%p - PID of dumped process
%u - real UID of dumped process
%g - real GID of dumped process
%s - number of the signal causing dump
%t - time of dump (seconds since 0:00h, 1 Jan 1970)
%h - hostname
%e - application file name

Reload the settings in /etc/sysctl.conf by running the following command
# sysctl -p

Step 4: Test to see core dump file is creating on crash

Now I'm going to kill Apache using the signal SIGQUIT, which dumps core on termination. Don't try this in production environment.

# kill -s SIGQUIT `cat /var/run/httpd/httpd.pid`

Now you can see a core dump file is created in the /tmp dictionary.

# ls -l /tmp
-rw-------  1 root  root  5431296 Jan 12 21:07 core-httpd-3-0-0-4277-1379034432

Where httpd is the name of application which was killed by the signal SIGQUIT, 3 is the signal number, two  zeros means GID and UID of root, 4277 is the PID and 1379034432 is the time of crash.

Step 5: Analyse the core dump file

You need gdb (GNU Debugger) to analyse the core dump file. You may not have gdb on your system, install gdb.

# yum install gdb

You need use the gdb command as follows
# gdb <path of the application> <path of the core dump file>

# which httpd
/usr/sbin/httpd

Now you may run gdb

# gdb /usr/sbin/httpd /tmp/core-httpd-3-0-0-24631-1374966080
Core was generated by `/usr/sbin/httpd'.
Program terminated with signal 3, Quit.

You may get error message like “Missing separate debuginfos, use: debuginfo-install httpd-2.2.15-28.el6.centos.i686", you need to have yum-utils packages installed on your system, yum-utils packages provides the command debuginfo-install. Check if you have installed yum-utils package on your system.

# rpm -qa | grep yum-utils
yum-utils-1.1.30-14.el6.noarch

If not install it and run. If you still getting error message like “Could not find debuginfo for main pkg: httpd-2.2.15-28.el6.centos.i686”, edit  the file /etc/yum.repos.d/CentOS-Debuginfo.repo and set enabled=1. If you use some other Redhat clone you need to use the appropriate Debuginfo repo file.

No comments:

Post a Comment