‹ back home

Notes on saving coredumps

2024-06-24 #experiment #notes

When programs crash in certain specific ways, the kernel can generate a core dump file. A core dump is a file containing an image of the process’s memory at the moment when it was terminated.

A core dump can later be used with tools such as gdb to generate a backtrace of the program and debug the root cause of the crash.

Relevant runtime parameters

The RLIMIT_CORE resource limit defines a maximum size for core dump. Like most resource limits, it has a soft limit (which can be adjusted by an unprivileged user) and a hard limit (the maximum value to which an unprivileged user can adjust the soft limit). These can be queried via ulimit -Hc and ulimit -c respectively. ulimit is a shell built-in, consult your shell’s documentation for more details1. The default value is zero, which means that no core dumps are generated.

The kernel.core_pattern kernel attribute defines where core dumps will be stored. If the first character of this pattern is |, then the rest of the attribute is interpreted as the path to an program which will be executed with the core dump piped into its stdin. The default value is core, so core dumps would be generated into a file named core in the process’s working directory.

If the value of kernel.core_pattern does not start with a pipe, the core dump is only saved if the terminated process had permission to write to the resulting path. However, if the pattern does start with a pipe, the new process started by the kernel runs with elevated privileges (e.g.: as root).

Saving core dumps

I take the simplest approach, saving all core dumps into a common directory. I want any user to be able to write dumps, but only members of the wheel group can read dumps, and only the ones that they own.

mkdir /var/coredump/
chown root:wheel /var/coredump
chmod 1775 /var/coredump

The 1 in the above permissions is the sticky bit: only the owner of a file within the directory can delete or rename it. This setup allows any local user to fill the disk completely by forking and crashing continuously.

Next, I’ll set the kernel.core_pattern attribute (this can be made permanent via a configuration file in /etc/sysctl.d):

sysctl -w kernel.core_pattern='/var/coredump/%t_%P_%u_%s_%f'

From the above pattern:

I initially tried %E (executable path) instead of %f first. It saves the full path to the executable replacing / with !. I didn’t find the full path of any use, so opted for the simpler variation.

Testing the setup

First, I’ll create a quick helper that segfaults right away. C makes this easier than any other language!

int main(int argc, char *argv[]) {
	int *i = 0;
	*i = 42;

I’ll then raise the core size resource limit:

ulimit -c 256

And then run my test program that segfaults:

cc segfault.c -o segfault
doas ./segfault

I now see both core dumps in /var/coredump2:

.rw------- 201k hugo hugo 2024-06-24 13:07 1719227265_11806_1000_11_segfault
.rw------- 201k root root 2024-06-24 13:07 1719227268_11866_0_11_segfault

I also confirm that permissions are as expected:

> file 1719227265_11806_1000_11_segfault 1719227268_11866_0_11_segfault
1719227265_11806_1000_11_segfault: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from './segfault'
1719227268_11866_0_11_segfault:    regular file, no read permission

I can also debug this dump by pointing gdb to the original binary and the coredump itself:

gdb ./segfault /var/coredump/1719227677_18833_1000_11_segfault


Piping to a dedicated handler

A more sophisticated setup could pipe to a dedicated handler. This handler could extract additional process information, drop privileges, and save the dump along with this extra information into a location where it has exclusive write permissions.

For saving the metadata, a .info file seems like the obvious choice, but one could also use extended file attributes3. Extended attributes would ensure that if a coredump is deleted, no orphan metadata is left behind.

For my use-case, I don’t really need any more information. Even the filename patterns described above already have more data than I need.

  1. POSIX specifies that this value is defined in 512byte blocks, but behaviour diverges between popular shells. ↩︎

  2. You might have noticed that I set a limit of 128KB, but the dumps are 201KB. This seems to be a bug in zsh↩︎

  3. I’ve been reading a lot on Haiku’s design lately. It uses extended attributes in several interesting ways. Storing metadata for non-portable files like this seems like a good usage of extended attributes, although also not strictly necessary. ↩︎

Have comments or want to discuss this topic?
Send an email to ~whynothugo/public-inbox@lists.sr.ht (mailing list etiquette)

— § —