After the acceptance of my grant request to work further on pimsync, I’ve been focusing on it greatly this past month.
When pimsync downloads a WebCal calendar, it needs to split all the events into
individual files before they can be written to a CalDAV or filesystem (vdir)
storage. Each separate file needs to have the timezone metadata for the event it
contains. My initial implementation was a naive one: copy all the timezones into
all the files. This works, but makes files much bigger than they need to be. If
a WebCal calendar has a hundred entries with a dozen different timezones, each
of the hundred resulting files ends up having those dozens of timezones
repeated. Every tool which later interacts with them needs to pointlessly parse
lots of unused data. This is no longer the case, and pimsync now only copies
over relevant timezones. Along with this change, I’ve done several other
performance improvements to the whole WebCal splitting logic as well as the
underlying vparser library.
The configuration file now also supports specifying TLS fingerprints with colons as separators, a common convention which vdirsyncer also used.
I’ve implemented I/O timeouts: both when connecting and for reads and writes. This requires careful implementation: when fetching a large item, we don’t want to time out if the entire operation takes too long, but only if the server takes too long. To implement this kind of timeouts, they need to be at the TCP-stream layer, rather than on top of the HTTP client. The implementation is quite clean, but the underlying hyper library is in a state of flux, where its HTTP client is “legacy” but a non-legacy one doesn’t exist yet…
I’d like to upstream my timeout logic, but that will likely have to wait until the situation upstream stabilises further. This “legacy” HTTP client also has some bugs, like swallowing TCP errors in some cases. While the HTTP client abstraction is considered legacy, the underlying hyper library is well maintained, but there’s just a gap in usability in the stable portions of it. Additionally, this legacy client has some issues (like sometimes swallowing errors). I’ll try patching around these issues first, but might need to implement my own minimal client (relying on the existing HTTP library itself, of course, I have no desire to reinvent all the underlying layers).
The timeout tests also led me to find some issues with timers on OpenBSD. After much debugging, I found that the kernel runs times on a 100Hz ticker, so timers end up being rounded to 10ms intervals. For short intervals (e.g.: 20ms), this is a huge margin of error. I worked around the issue by making the tests 50ms slower, but the underlying situation really bothers me.
I’ve cleaned up some stale database columns from pimsync’s status database. These were leftovers from earlier designs, and removing them requires updating all sites which still referenced them and implementing data migrations so that existing installations can continue working seamlessly.
Finally, I’ve finished the refactoring of the socket implementation which I
mentioned before. Now pimsync can be pointed to a unix domain socket (e.g.: for
a locally running CalDAV server or one tunnelled in some way). The same
underlying logic is used for supporting the de-facto standard http_proxy
environment variable, which I expect to have finalised in the coming couple of
weeks.