This is a long overdue update on the status on vdirsyncer’s rewrite. I keep thinking “I’ll just finish this one bit before publishing it”, but that always seems to be a day or two in the future, so here goes.
Funding
I’m very happy to announce that the NLnet foundation has agreed to fund my work on vdirsyncer. Thanks to this, I’ve been able to put a lot more focus hours into this project, which would have not been possible without this support. I’m very thankful for their support and the opportunity it provides.
The storages module
My first focus was the storage implementations themselves. Storages are
basically filesystem (e.g.: vdir), webcal (i.e.: an icalendar file over http),
CalDav, etc. As I mentioned before, my intent is to have a standalone
vstorage crate which has all the storage implementations. The goal is to
abstract away all the storage-specific quirks and details, so that when working
on vdirsyncer any storage provides the same API, regardless of how it works
under the hood.
I want all storage operations to be asynchronous: there will be a lot of networking going on all the time, and blocking every time is just going to be slow. Using worker threads is an option, but that doesn’t really scale well.
My first approach was to define the Storage traits using Rust’s experimental
async trait feature. However these are not object
safe, so application code wouldn’t be able to operate on storages with the
right implementation defined at runtime. This is a no-go for my use-case, since
storage types (and therefore, their implementation) are determined at runtime
based on vdirsyncer’s configuration. That aside, async trait is only
available in Rust nightly, and, while it’s been so for a few years, there’s no
indicator of this feature becoming stable anytime soon.
So I moved onto the #[async_trait] approach. This doesn’t rely
on any nightly features and simply wraps the code to return a Future under
the hood. In order to have storages be interchangeable at runtime, I also had
to Box all the storage’s return types. This adds a bit of indirection and is
a bit less efficient. I tried to work these performance issues for days, but
then realised that I was doing the “too much premature optimisation” thing we
developers sometimes do: the current approach is likely to yield results more
efficient than what any interpreted language could achieve anyway, and I’m sure
we’ll find other bottlenecks worth improving further down the line.
With that out of the way, I finished defining a first version of the traits
that all implementations will use as well as two of the implementations
themselves: filesystem and webcal. There’s even a little example which fetches
a remote webcal and saves it into a local vdir. Not very exciting, but it’s
written mostly to prove that the API and implementations work.
The next item on the agenda was the CalDav storage implementation, and storages are paused at the moment until the next item is done…
A CalDav implementation
In order to implement the storage type for CalDav itself I need a CalDav client. I’ve been working on this the past few weeks. A big time sink was implementing auto-discovery of servers, context path, etc. This was actually not on my radar ahead of time, but it seemed like a natural place to start.
CalDav auto-discovery
Auto-discovery allows configuring a CalDav client with just a username,
password and a bare domain such as example.com. Auto-discovery will use DNS
to resolve the real server and port for the CalDav server for example.com
(which can be something like https://caldav.example.com:8443/dav/.
This helps mostly with hosted services; people who are self-hosting usually know the under-the-hood details. I’ve tested this with fastmail.com and posteo.de – providing just the bare domain as input results in the client communicating to the right server/port combination. Note that I’m not making anything up here; I’m just following the specification for Locating Services for Calendaring Extensions to WebDAV (CalDAV) and vCard Extensions to WebDAV (CardDAV).
The fun of Parsing XML
CalDav (and WebDav) uses a lot of XML under the hood for representing requests
and responses. Pretty much anything that’s not calendar data itself is modelled
in XML. For my initial approach I used quick_xml with its serde
capabilities, which (de)serialised everything for me into concrete Rust types,
all nice and simple.
Regrettably, quick_xml’s serde implementation doesn’t support
namespaces, and ignoring namespaces broke under a few
scenarios. It’s not even clear how it could be updated to support namespaces,
so that turned out to be a dead end.
I had to switch to quick_xml’s lower-level API for parsing XML manually. The
design of the lower-level parser API is very similar to a SAX parser, but with
inversion of control. Basically, the parser reads the underlying bytes and
provides a stream of Event instances indicating what it has found (new node,
text, etc). This allows modelling the higher-level parser as a sort of state
machine, which is what I’ve done.
I thought I had all the XML parsing out of the day and then discovered a scenario which forced me to refactor the whole thing. Each query gets a multistatus object, which has multiple responses. Typically these responses look like this:
<ns0:response>
<ns0:href>/user/calendars/Q208cKvMGjAdJFUw/qJJ9Li5DPJYr.ics</ns0:href>
<ns0:propstat>
<ns0:status>HTTP/1.1 200 OK</ns0:status>
<ns0:prop>
<ns0:getetag>"adb2da8d3cb1280a932ed8f8a2e8b4ecf66d6a02"</ns0:getetag>
<ns1:calendar-data>calendar-data-goes-in-here</ns1:calendar-data>
</ns0:prop>
</ns0:propstat>
</ns0:response>
But when no prop objects are included the propstat element is omitted, and
the status is one level higher:
<ns0:response>
<ns0:href>/user/calendars/Q208cKvMGjAdJFUw/rKbu4uUn.ics</ns0:href>
<ns0:status>HTTP/1.1 404 Not Found</ns0:status>
</ns0:response>
Upon re-reading the specification I don’t know why I overlooked this the first time, but after about a whole day, I refactored everything to work with both cases, which seems to have been the last of all the XML parsing fun.
Testing with real servers
After writing most of the primitive operations for my CalDav client (e.g.: list/create/update/get/delete resources/collections), I started out some basic testing.
I wrote a few integration tests which are relatively simple, for example:
- Create a collection
- Create a couple of resources
- Check that listing the collection returns two resources with the right names
- Fetch one and check the data is what we saved
- Delete both resources
- List items in the collection and confirm that it is now empty
This is very simple and straightforward, but running the same test with different servers has helped iron out some bugs due to slight differences in how they behave. I really appreciate having heterogeneous servers to test with since it helps make my result a lot more resilient.
Having tested these primitives individually at this level of abstraction also gives a lot more confidence in using this functionality in higher level code.
I’ve also found a few bugs in servers running these tests.
The first bug was on Fastmail.com. They provide a free test account for vdirsyncer’s integration testing. Fastmail uses (and apparently contribute a lot to) Cyrus IMAP. Cyrus IMAP, despite the name, is also a CalDav server. I triggered the bug and reported it the next day. It turns out that a fix for the issue had been pushed earlier that same day.
The second bug is untriaged yet, but the reproducer looks pretty solid. It mostly affects discovery: when asking the server “what features do you support”, it doesn’t mention CalDav. Everything else works, but it just hurts discovery because vdirsyncer can’t provide solid confirmation that it’s reached the correct server/port/path combination.
A helper/testing command line interface
The CalDav client library is designed to be re-usable by other projects, and I intend to include a small cli for interactive usage. This cli will just include basic functionality like printing discovery data, list and fetch individual items, and other primitives like that. The main use-case is for it to be used as a debugging/development tool. In particular, I’d like to ask others to test discovery via the cli in order to find if there are any well-configured services where discovery fails.
libdav v0.1.0
I’ve published a version 0.1.0 of libdav. In reality, this is
mostly a milestone in the whole rewrite’s development, but if you’re in need of
one, feel free to give it a try. Do keep in mind that this is very early in
development so, (a) it likely still has bugs, and (b) the API might change over
time. Feedback, as always, is welcome.
Aside from CalDav, this has it own tiny WebDav client (given that CalDav is just an extension to WebDav) with mostly the basics implemented for this use case. I’ve also laid out the groupwork for the CardDav client on top of it for syncing contacts. That one is missing a few bits and some intense integration testing.
Next item: synchronisation
The next item on the agenda is synchronisation itself. I have some clear ideas on how it’s going to work (greatly inspired by the existing implementation, of course).
Note on maintenance status of vdirsyncer
I’ve been putting less time into maintenance and bugfixes of the existing codebase. Basically, every hour that I put into maintaining legacy code is an hour that’s not spent on the new codebase. As a single developer, I have to choose one or finish neither.