After the system goes into sleep and wakes up again, ImapGoose would just hang for like 15–20 minutes before finally detecting the disconnection and reconnect. This was super annoying, and understanding the root cause was not trivial at all.
ImapGoose sends a NOOP to the server every three minutes. This NOOP is basically like a ping to which the server must reply. The timeout it set to thirty seconds, so why did it take around 20 minutes to detect the disconnection?
Part 1
While the system is asleep, the server sends a RST indicating that the connection is terminated. This message is never delivered because our host is no longer on the network. The system wakes up again and has no impression that the connection is dead. The Linux kernel puts the NOOP message send by ImapGoose in its internal outbound buffer. While there’s data pending to be written in the outbound buffer, the underlying TCP keepalive is paused. Sending this message fails on a dead connection, so Linux keeps retrying, with an increasing back-off… for slightly over 18 minutes.
Having a hypothesis, I had to prove it. I tried using a VM, but failed.
Apparently, slirp, the default networking backend used by QEMU, will inject
the RST into the VM even if it was asleep. I later learnt that there’s a
different backend, passt, which likely would have worked, but I’d trashed all
the code for producing and VM and its reproduction environment months ago.
In the end, I used a Linux network namespace running ImapGoose inside of it, and a nftables rule drops all incoming traffic after the initial start-up. This simulates scenario where the connection was closed by the other side and messages got dropped. I can then measure the time before the disconnection is detected, and see if the number align with my hypothesis.
I couldn’t be bothered with writing all the one-shot scaffold to prove the hypothesis and used an LLM which produces a surprisingly clear and readable script (this isn’t part of ImapGoose’s codebase). The numbers proved it right:
- Applying the DROP netfilter rule right after receiving a NOOP response, the disconnection was detected within the usual TCP keepalive timeout (~154s).
- Appyling the DROP netfilter rule right before sending a NOOP, the disconnection was detected after the TCP retransmission timeout (~949s).
Knowing the root cause of the issue, I wrote a small change to ImapGoose: when sending a NOOP, also set an in-application timer for 30 seconds. If the timer expires and we got no reply, assume the connection is dead and reconnect.
Part 2
The above worked, and ImapGoose now would reconnect after wake-ups in at most 3:30m. This doesn’t sound like very long, but it really is an issue: I open my laptop, it wakes-up, I check my mail, data is stale, I see nothing new in 15 seconds, I close it back again.
Having to wait three minutes in these cases is a terrible experience.
I need to detect when the system wakes up from sleep immediately, and Linux has an interface which lets me infer this, with relatively good reliability:
Timers have a TFD_TIMER_CANCEL_ON_SET flag, which cancels the timer if the
clock’s time has changed. When the system wakes up after sleeping, the clock
monotonic clock is delayed and immediately gets adjusted. This adjustment
cancels my timer immediately. No waiting up to three minutes, I can now check
the connection immediately, and reconnect if it’s dead.
The NOOP still needs 30 seconds (thanks to the 30 second timeout from part 1). Still not perfect, but definitely an improvement.
TFD_TIMER_CANCEL_ON_SET is a Linux-specific flag, so this second fix only
applies to Linux.
Closing words
This is a huge improvement when using ImapGoose for systems which go to sleep often, and was the biggest bug pending.
ImapGoose still won’t detect when the network disconnects or when switching to a
new one. There are no portable APIs for this. I could use a PF_ROUTE socket on
BSD and a AF_NETLINK socket on Linux, which lets me know “some interface
disconnected”, but not whether it’s the interface I’m using.
I’m still considering gluing SIGUSR1 to “reconnect now”, so users with
different userspace can glue ImapGoose to automatically reconnect on conditions
they deem appropriate.
I’m annoyed that there’s no portable API to let the kernel inform an application
when it wakes up from sleep. OpenBSD exposes APM_NORMAL_RESUME events through
kqueue, but as far as I can tell, Linux exposes nothing.
Unix is old, from a time where putting the system to sleep like this wasn’t even a dream. The common, portable interfaces haven’t kept up.