‹ back home

vdirsyncer status update 2023-03

2023-03-28 #status-update #vdirsyncer

This is a long overdue update on the status on vdirsyncer’s rewrite. I keep thinking “I’ll just finish this one bit before publishing it”, but that always seems to be a day or two in the future, so here goes.


I’m very happy to announce that the NLnet foundation has agreed to fund my work on vdirsyncer. Thanks to this, I’ve been able to put a lot more focus hours into this project, which would have not been possible without this support. I’m very thankful for their support and the opportunity it provides.

The storages module

My first focus was the storage implementations themselves. Storages are basically filesystem (e.g.: vdir), webcal (i.e.: an icalendar file over http), CalDav, etc. As I mentioned before, my intent is to have a standalone vstorage crate which has all the storage implementations. The goal is to abstract away all the storage-specific quirks and details, so that when working on vdirsyncer any storage provides the same API, regardless of how it works under the hood.

I want all storage operations to be asynchronous: there will be a lot of networking going on all the time, and blocking every time is just going to be slow. Using worker threads is an option, but that doesn’t really scale well.

My first approach was to define the Storage traits using Rust’s experimental async trait feature. However these are not object safe, so application code wouldn’t be able to operate on storages with the right implementation defined at runtime. This is a no-go for my use-case, since storage types (and therefore, their implementation) are determined at runtime based on vdirsyncer’s configuration. That aside, async trait is only available in Rust nightly, and, while it’s been so for a few years, there’s no indicator of this feature becoming stable anytime soon.

So I moved onto the #[async_trait] approach. This doesn’t rely on any nightly features and simply wraps the code to return a Future under the hood. In order to have storages be interchangeable at runtime, I also had to Box all the storage’s return types. This adds a bit of indirection and is a bit less efficient. I tried to work these performance issues for days, but then realised that I was doing the “too much premature optimisation” thing we developers sometimes do: the current approach is likely to yield results more efficient than what any interpreted language could achieve anyway, and I’m sure we’ll find other bottlenecks worth improving further down the line.

With that out of the way, I finished defining a first version of the traits that all implementations will use as well as two of the implementations themselves: filesystem and webcal. There’s even a little example which fetches a remote webcal and saves it into a local vdir. Not very exciting, but it’s written mostly to prove that the API and implementations work.

The next item on the agenda was the CalDav storage implementation, and storages are paused at the moment until the next item is done…

A CalDav implementation

In order to implement the storage type for CalDav itself I need a CalDav client. I’ve been working on this the past few weeks. A big time sink was implementing auto-discovery of servers, context path, etc. This was actually not on my radar ahead of time, but it seemed like a natural place to start.

CalDav auto-discovery

Auto-discovery allows configuring a CalDav client with just a username, password and a bare domain such as example.com. Auto-discovery will use DNS to resolve the real server and port for the CalDav server for example.com (which can be something like https://caldav.example.com:8443/dav/.

This helps mostly with hosted services; people who are self-hosting usually know the under-the-hood details. I’ve tested this with fastmail.com and posteo.de – providing just the bare domain as input results in the client communicating to the right server/port combination. Note that I’m not making anything up here; I’m just following the specification for Locating Services for Calendaring Extensions to WebDAV (CalDAV) and vCard Extensions to WebDAV (CardDAV).

The fun of Parsing XML

CalDav (and WebDav) uses a lot of XML under the hood for representing requests and responses. Pretty much anything that’s not calendar data itself is modelled in XML. For my initial approach I used quick_xml with its serde capabilities, which (de)serialised everything for me into concrete Rust types, all nice and simple.

Regrettably, quick_xml’s serde implementation doesn’t support namespaces, and ignoring namespaces broke under a few scenarios. It’s not even clear how it could be updated to support namespaces, so that turned out to be a dead end.

I had to switch to quick_xml’s lower-level API for parsing XML manually. The design of the lower-level parser API is very similar to a SAX parser, but with inversion of control. Basically, the parser reads the underlying bytes and provides a stream of Event instances indicating what it has found (new node, text, etc). This allows modelling the higher-level parser as a sort of state machine, which is what I’ve done.

I thought I had all the XML parsing out of the day and then discovered a scenario which forced me to refactor the whole thing. Each query gets a multistatus object, which has multiple responses. Typically these responses look like this:

    <ns0:status>HTTP/1.1 200 OK</ns0:status>

But when no prop objects are included the propstat element is omitted, and the status is one level higher:

    <ns0:status>HTTP/1.1 404 Not Found</ns0:status>

Upon re-reading the specification I don’t know why I overlooked this the first time, but after about a whole day, I refactored everything to work with both cases, which seems to have been the last of all the XML parsing fun.

Testing with real servers

After writing most of the primitive operations for my CalDav client (e.g.: list/create/update/get/delete resources/collections), I started out some basic testing.

I wrote a few integration tests which are relatively simple, for example:

This is very simple and straightforward, but running the same test with different servers has helped iron out some bugs due to slight differences in how they behave. I really appreciate having heterogeneous servers to test with since it helps make my result a lot more resilient.

Having tested these primitives individually at this level of abstraction also gives a lot more confidence in using this functionality in higher level code.

I’ve also found a few bugs in servers running these tests.

The first bug was on Fastmail.com. They provide a free test account for vdirsyncer’s integration testing. Fastmail uses (and apparently contribute a lot to) Cyrus IMAP. Cyrus IMAP, despite the name, is also a CalDav server. I triggered the bug and reported it the next day. It turns out that a fix for the issue had been pushed earlier that same day.

The second bug is untriaged yet, but the reproducer looks pretty solid. It mostly affects discovery: when asking the server “what features do you support”, it doesn’t mention CalDav. Everything else works, but it just hurts discovery because vdirsyncer can’t provide solid confirmation that it’s reached the correct server/port/path combination.

A helper/testing command line interface

The CalDav client library is designed to be re-usable by other projects, and I intend to include a small cli for interactive usage. This cli will just include basic functionality like printing discovery data, list and fetch individual items, and other primitives like that. The main use-case is for it to be used as a debugging/development tool. In particular, I’d like to ask others to test discovery via the cli in order to find if there are any well-configured services where discovery fails.

libdav v0.1.0

I’ve published a version 0.1.0 of libdav. In reality, this is mostly a milestone in the whole rewrite’s development, but if you’re in need of one, feel free to give it a try. Do keep in mind that this is very early in development so, (a) it likely still has bugs, and (b) the API might change over time. Feedback, as always, is welcome.

Aside from CalDav, this has it own tiny WebDav client (given that CalDav is just an extension to WebDav) with mostly the basics implemented for this use case. I’ve also laid out the groupwork for the CardDav client on top of it for syncing contacts. That one is missing a few bits and some intense integration testing.

Next item: synchronisation

The next item on the agenda is synchronisation itself. I have some clear ideas on how it’s going to work (greatly inspired by the existing implementation, of course).

Note on maintenance status of vdirsyncer

I’ve been putting less time into maintenance and bugfixes of the existing codebase. Basically, every hour that I put into maintaining legacy code is an hour that’s not spent on the new codebase. As a single developer, I have to choose one or finish neither.

— § —