Segmentation faults and core dumps on Fedora Linux 28

Introduction

This is a brief article about segmentation faults and how to handle core dumps on Fedora Linux 28. The intended audience is C programmers and people not familiar with Fedora Linux 28. For educational purposes we will create a tiny flawed C program that will crash. We then get its core dump and analyze it to see what went wrong. In this case the bug is extremely simple, but this is just to demonstrate the basic concepts. I am not sure whether this article is completely suitable for novices, but the C code is certainly very simple and accessible even to beginners. However, some of the concepts presented here require a bit deeper understanding of computers' and Linux's workings.

What is a core dump?

According to a Wikipedia article:

When a program misbehaves on Linux, it is sometimes possible to get the process' core dump and use it to analyze why the program crashed.

Creating a flawed C program

Type this C program and save it as coredumptest.c or download the source code using this link.

#include <stdio.h>

void foo(const char *s);
void bar(const char *s);

void bar(const char *s)
{
	printf("%s\n", s);
}

void foo(const char *s)
{
	bar(s);
}

int main(int argc, char *argv[])
{
	foo(argv[1]);
	return 0;
}

If you do not have it yet, install GNU Compiler Collection gcc with:

Then compile our test program with:

You should get a warning about unused parameter argc. GCC was indeed quite helpful and normally we would pay attention to all the warnings, but ignore the warning for now.

It was very important that we used -g command line option so that GCC was instructed to include debug symbols in our resulting coredumptest executable.

According to the Wikipedia article:

We want to have the debug symbols inserted in, because we will later use GNU debugger (gdb) to analyze our program and the related core dump.

If you do not have it yet, install GNU Debugger gdb with:

Running the flawed executable

Now run the flawed executable with:

You should get the error message:

This message comes from the Linux command shell bash. It was bash that executed our coredumptest and monitored how the coredumptest child process exited using wait Linux system call. If you are interested in wait specification, you can see its manual page by typing:

It describes what this Linux system call does. In particular, we now know that bash must have called the following C macros after wait returned:

In this case, both WIFSIGNALED and WCOREDUMP macros returned true, so bash emitted the message we saw earlier: "Segmentation fault (core dumped)". Do not worry if you did not understand all of this, these are just the technical details that are not needed in order to understand the big picture.

UPDATE June 21st, 2018: I just checked and Bash does not call WCOREDUMP macro at all. Instead it uses macro WIFCORED. It is conditionally defined in Bash's own source bash/include/posixwait.h depending on whether POSIX or non-POSIX system is used.

The message from bash actually has its roots in the Fedora Linux 28 operating system's Linux kernel. "Segmentation fault", or more technically SIGSEGV, is a signal that almost always kills a user space process. In our case, Linux kernel detected invalid memory access and delivered the SIGSEGV signal to the violating user space process. It was our buggy coredumptest that was guilty.

It is possible for user space processes to catch many signals, including SIGSEGV. But our program has not installed a signal handler function for SIGSEGV, so Linux kernel resorts to SIGSEGV signal's default action: kernel kills the misbehaving user space process and looks to its own kernel configuration to see whether it should generate a core dump. If core dumps are enabled and process' ulimit core dump size is not exceeded, then Linux kernel will create a core dump. You probably guessed that the message "core dumped" means that core dump was indeed created.

What is ulimit? It is bash command shell's builtin command for setting and querying process limits. Internally, it uses Linux system calls setrlimit and getrlimit to set and query process limits. To verify that bash has unlimited size for core dumps, issue command:

You should see the reply:

To see all limits, type:

What does Fedora Linux 28 do with core dumps?

On Fedora Linux 28, core dumps are enabled by default. To verify this, issue command:

You should see the following result:

Back in the old days, things were pretty messy. Unix and Linux operating systems dumped core files all over the place, depending on where the killed user space process' working directory happened to be. It was not a clean solution. Luckily core dumping has improved.

On Fedora Linux 28 and other modern Linux distributions, kernel now sends core dumps to an external user space program /usr/lib/systemd/systemd-coredump for saving and pre-processing. Much of the work is thus delegated to systemd-coredump.

How does kernel send the core dump to systemd-coredump? Very elegantly: systemd-coredump reads the data from its stdin, standard input. So it is familiar Unix/Linux piping, but this time the data comes from Linux kernel and goes to the user space! The receiving program systemd-coredump need not be aware of the fact that its input data comes from the kernel, it just reads stdin like usual.

To see information about core dumps, type:

It explains the meaning of %P %u %g %s %t %c %h %e parameters:

To see which RPM package the program comes from, try command:

You should see reply like the following:

So like the name suggests, systemd-coredump is part of systemd RPM-package.

Fedora Linux 28 saves core dumps in directory /var/lib/systemd/coredump, but we need not access that directory directly.

Analyzing core dumps

Like systemd-coredump, coredumpctl command belongs to the systemd RPM. If you want to verify that, just type:

and you should see something like:

Handling core dumps is now easy and convenient. To see all core dumps available we can simply type:

without any parameters. But because we know the name of our flawed program, we could also issue a more specific command:

We should get a result like this:

TIME                            PID   UID   GID SIG COREFILE  EXE
Wed 2018-06-20 19:17:13 EEST  16559  1000  1000  11 present   /home/kalevi/c/coredumptest

Since this program has crashed just once, we can identify it with by name and run GNU Debugger with:

If there were many core dumps for coredumptest, we could use Process Identifier, i.e. PID, to select the correct dump:

Or if you want to be really specific, you can use COREDUMP_* predicates like this:

That command forms logical AND of all the predicates, so in pseudo program code the selection above means:

Anyway you choose the core dump, the output will be pretty long and it will resemble this:

           PID: 16559 (coredumptest)
           UID: 1000 (kalevi)
           GID: 1000 (kalevi)
        Signal: 11 (SEGV)
     Timestamp: Wed 2018-06-20 19:17:13 EEST (39min ago)
  Command Line: ./coredumptest
    Executable: /home/kalevi/c/coredumptest
 Control Group: /user.slice/user-1000.slice/session-2.scope
          Unit: session-2.scope
         Slice: user-1000.slice
       Session: 2
     Owner UID: 1000 (kalevi)
       Boot ID: 632a5ac708f641169decad624740b442
    Machine ID: 3571feec2e5c442fa1085a3524a7cf29
      Hostname: localhost.localdomain
       Storage: /var/lib/systemd/coredump/[kalevi: shortened for HTML]
       Message: Process 16559 (coredumptest) of user 1000 dumped core.
                
                Stack trace of thread 16559:
                #0  0x00007f886e20b9e1 __strlen_avx2 (libc.so.6)
                #1  0x00007f886e120c82 _IO_puts (libc.so.6)
                #2  0x00000000004004fe n/a (/home/kalevi/c/coredumptest)
                #3  0x0000000000400519 n/a (/home/kalevi/c/coredumptest)
                #4  0x000000000040053e n/a (/home/kalevi/c/coredumptest)
                #5  0x00007f886e0d218b __libc_start_main (libc.so.6)
                #6  0x000000000040042a n/a (/home/kalevi/c/coredumptest)

GNU gdb (GDB) Fedora 8.1-15.fc28
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/kalevi/c/coredumptest...done.
[New LWP 16559]
Core was generated by `./coredumptest'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f886e20b9e1 in __strlen_avx2 () from /lib64/libc.so.6
Missing separate debuginfos, use: dnf debuginfo-install glibc-2.27-15.fc28.x86_64
(gdb) 

We have now entered gdb. It is a symbolic debugger, and it has read debug symbols from /home/kalevi/c/coredumptest and we are now ready to analyze the core dump. Type:

to get a backtrace like this:

(gdb) bt
#0  0x00007f886e20b9e1 in __strlen_avx2 () from /lib64/libc.so.6
#1  0x00007f886e120c82 in puts () from /lib64/libc.so.6
#2  0x00000000004004fe in bar (s=0x0) at coredumptest.c:8
#3  0x0000000000400519 in foo (s=0x0) at coredumptest.c:13
#4  0x000000000040053e in main (argc=1, argv=0x7ffc80a62788) at coredumptest.c:18

So how do we make sense of that? First of all, we need to read the backtrace backwards, starting from #4 and progressing towards #0. Our C program starts executing in function main.

1. main calls foo
2. foo calls bar
3. bar calls puts
4. puts calls __strlen_avx2

This seems very strange. Remember that in our coredumptest.c code we had function bar defined like this:

void bar(const char *s)
{
	printf("%s\n", s);
}

Why on earth was puts called instead of printf? Is the backtrace reliable at all? What is going on here?

Well, this was all some kind of magic done by gcc when it compiled our coredumptest.c. It has absolutely nothing to do with this demonstration, it is only a potential source of unnecessary confusion! Do not worry about this. The GNU C compiler was able to deduce that it can replace our call to printf with an equivalent, but more efficient call to puts, since all we are doing is printing a single constant character string. So do not let this fact confuse you. It was a speed optimization done by GCC, but normally the backtrace would reflect our code more closely and you could expect to see printf instead of puts.

Where is the bug then and what good is this backtrace, you might ask. Relax and look closely at the line:

#3  0x0000000000400519 in foo (s=0x0) at coredumptest.c:13

It shows that our function foo was called with parameter s containing value 0x0. That is the bug right there! 0x0 means the NULL pointer and that is not what we want. We expected that argv[1] would contain a pointer to valid memory containing start of a character string, but instead we got a NULL pointer. Why? Because we ran our program with:

We should have done something like:

Then argv[1] would have contained a pointer to valid memory. So to fix our coredumptest.c, we should modify our main to have a check:

int main(int argc, char *argv[])
{
    if (argc != 2) {
        fprintf(stderr, "usage: %s [message]\n", argv[0]);
        exit(EXIT_FAILURE);
    }   

    foo(argv[1]);
    return 0;
}

For getting exit function prototype, we would also need to add:

#include <stdlib.h>

in the beginning.

But if the NULL pointer was the cause of the bug, why did the crash happen? Look at our backtrace again:

#0  0x00007f886e20b9e1 in __strlen_avx2 () from /lib64/libc.so.6
#1  0x00007f886e120c82 in puts () from /lib64/libc.so.6
#2  0x00000000004004fe in bar (s=0x0) at coredumptest.c:8
#3  0x0000000000400519 in foo (s=0x0) at coredumptest.c:13
#4  0x000000000040053e in main (argc=1, argv=0x7ffc80a62788) at coredumptest.c:18

We see that foo called bar with 0x0 too. That is quite clear. So it just passed the NULL pointer on. After that we miss some information, because puts is an external GNU C library function. Our Fedora Linux 28 did not contain debug symbols for the C library, so gdb is unable to show exact information. But we do know that puts has called __strlen_avx2 and we can be 99,99999999% sure that __strlen_avx2 has tried to calculate the length of the character string. In other words, it has tried to dereference the NULL pointer, causing Linux kernel to detect the memory violation, to generate signal SIGSEGV and to kill the offending coredumptest process.

Just for fun, let's install debug symbols for GNU C library glibc with command:

Then we can run:

again to see whether we will get more information this time. Giving command bt we get:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  __strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:62
62		VPCMPEQ (%rdi), %ymm0, %ymm1
(gdb) bt
#0  __strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:62
#1  0x00007fcc48826c82 in puts () at ioputs.c:35
#2  0x00000000004004fe in bar (s=0x0) at coredumptest.c:8
#3  0x0000000000400519 in foo (s=0x0) at coredumptest.c:13
#4  0x000000000040053e in main (argc=1, argv=0x7ffd92d2a778) at coredumptest.c:18

All right, we now know a bit more. We can see that the crash occurred inside a function written in x86_64 assembly! You see, GNU C library contains parts that are written in assembly and those parts are naturally machine dependent. AVX2 stands for "Advanced Vector Extensions" and it is a CPU feature according to this Wikipedia article. Because the C library functions are called so frequently, it is worth optimizing some of them using assembly. Obviously, strlen-avx2.S is one of those, because strlen is such a commonly called C function. It has to be as fast as possible and it makes sense to fine-tune it on every machine architecture.

The file name is ../sysdeps/x86_64/multiarch/strlen-avx2.S and the line number is 62. Assembly code is VPCMPEQ (%rdi), %ymm0, %ymm1. I do not know AMD64 assembly, so that does not help me and in any case, we would have to read the whole function in order to understand this. But as I said earlier, with this tiny sample program coredumptest.c, we can be sure that NULL pointer dereference is causing our program to crash. So there is no need to inspect this problem any further.

Conclusion

I hope this article provided you with some information about segmentation faults and how the resulting core dumps could be useful to programmers and system administrators.