When programs crash in certain specific ways, the kernel can generate a core dump file. A core dump is a file containing an image of the process’s memory at the moment when it was terminated.

A core dump can later be used with tools such as gdb to generate a backtrace of the program and debug the root cause of the crash.

Relevant runtime parameters

[permalink]

The RLIMIT_CORE resource limit defines a maximum size for core dump. Like most resource limits, it has a soft limit (which can be adjusted by an unprivileged user) and a hard limit (the maximum value to which an unprivileged user can adjust the soft limit). These can be queried via ulimit -Hc and ulimit -c respectively. ulimit is a shell built-in, consult your shell’s documentation for more details¹. The default value is zero, which means that no core dumps are generated.

The kernel.core_pattern kernel attribute defines where core dumps will be stored. If the first character of this pattern is |, then the rest of the attribute is interpreted as the path to an program which will be executed with the core dump piped into its stdin. The default value is core, so core dumps would be generated into a file named core in the process’s working directory.

If the value of kernel.core_pattern does not start with a pipe, the core dump is only saved if the terminated process had permission to write to the resulting path. However, if the pattern does start with a pipe, the new process started by the kernel runs with elevated privileges (e.g.: as root).

Saving core dumps

[permalink]

I take the simplest approach, saving all core dumps into a common directory. I want any user to be able to write dumps, but only members of the wheel group can read dumps, and only the ones that they own.

mkdir /var/coredump/
chown root:wheel /var/coredump
chmod 1775 /var/coredump

The 1 in the above permissions is the sticky bit: only the owner of a file within the directory can delete or rename it. This setup allows any local user to fill the disk completely by forking and crashing continuously.

Next, I’ll set the kernel.core_pattern attribute (this can be made permanent via a configuration file in /etc/sysctl.d):

sysctl -w kernel.core_pattern='/var/coredump/%t_%P_%u_%s_%f'

From the above pattern:

• %t: UNIX time of dump
• %P: Global PID
• %u: Global UID
• %s: Signal number
• %f: Executable filename

I initially tried %E (executable path) instead of %f first. It saves the full path to the executable replacing / with !. I didn’t find the full path of any use, so opted for the simpler variation.

Testing the setup

[permalink]

First, I’ll create a quick helper that segfaults right away. C makes this easier than any other language!

int main(int argc, char *argv[]) {
	int *i = 0;
	*i = 42;
}

I’ll then raise the core size resource limit:

ulimit -c 256

And then run my test program that segfaults:

cc segfault.c -o segfault
./segfault
doas ./segfault

I now see both core dumps in /var/coredump²:

.rw------- 201k hugo hugo 2024-06-24 13:07 1719227265_11806_1000_11_segfault
.rw------- 201k root root 2024-06-24 13:07 1719227268_11866_0_11_segfault

• 1719227265 is the timestamp.
• 11806 is the PID.
• 1000 is the UID.
• 11 is the signal number
• segfault is the name of my test program.

I also confirm that permissions are as expected:

> file 1719227265_11806_1000_11_segfault 1719227268_11866_0_11_segfault
1719227265_11806_1000_11_segfault: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from './segfault'
1719227268_11866_0_11_segfault:    regular file, no read permission

I can also debug this dump by pointing gdb to the original binary and the coredump itself:

gdb ./segfault /var/coredump/1719227677_18833_1000_11_segfault

Portability

[permalink]

• RLIMIT_CORE is standard and exists in the BSDs as well.
• kernel.core_pattern seems to be Linux-specific.

Piping to a dedicated handler

[permalink]

A more sophisticated setup could pipe to a dedicated handler. This handler could extract additional process information, drop privileges, and save the dump along with this extra information into a location where it has exclusive write permissions.

For saving the metadata, a .info file seems like the obvious choice, but one could also use extended file attributes³. Extended attributes would ensure that if a coredump is deleted, no orphan metadata is left behind.

For my use-case, I don’t really need any more information. Even the filename patterns described above already have more data than I need.