Responses API in libdav
libdav exposed functionality via
methods where all parameters must be specified. For example:
caldav.create_collection(
"/calendars/tasks/",
&[&names::CALENDAR],
&[&prop] // These are properties in XML form.
).await?;
The problem with this approach is adding new functionality. For example, I want
to allow specifying a display name, colour or other attributes of a calendar
when creating it. Adding extra arguments doesn’t scale well: all call sites need
to be updated each time a new argument is added, and all call sites need to
specify several None values for properties which they’re not using. It becomes
verbose and unreadable.
I’ve finally refactored libdav to use a new “Requests API”, where each
functionality is exposed via a custom Request type implementing a DavRequest
trait. This trait has two methods: prepare_request and parse_response. As
you’ve hopefully guessed, these methods prepare a new request and parse the
response respectively. The WebDavClient has a new request() method which
takes an instance of a DavRequest and executes it.
These separate prepare_request and parse_response methods result in a
cleaner layout and make testing much simpler by virtue of being able to test
both of these without any network I/O. On this topic, I strongly recommend
reading Network protocols, sans I/O.
DavRequest implementations like CreateCalendar have methods to specify
additional fields using a builder pattern. Here’s an example using all the
fields for this request type:
let create_calendar = CreateCalendar::new("/calendars/work/")
// All of these calls are optional.
.with_display_name("Work Calendar")
.with_colour("#FF0000")
.with_order(10)
.with_components(&[CalendarComponent::VEvent, CalendarComponent::VTodo]);
webdav.request(create_calendar);
The resulting code is a greater total number of lines of code, but each functionality is nicely split into its own file, making code easier to read, follow and maintain. I recently wrote about the virtues of having a greater amount of simpler code.
I’m pleased with the result of the implementation, and I will add additional features (which were already planned) without having to add new arguments to functions for each new feature introduced.
Streaming API for vstorage
I’ve refactored the synchronisation portion of vstorage/pimsync to operate on a
“stream of operations”. The general principle is that a plan with the actions to
perform during synchronisation is a futures_util::stream::Stream of
Operation objects, which each encode all the information required for their
execution.
The executor now also takes a Stream of operations too, executing them one at a
time. The main goal of this refactor was to improve how the daemon mode
operates: after the initial “full” synchronisation, the daemon will monitor for
individual changes, and send only operations for those individual changes to be
executed — instead of the current approach which just performs another full
sync. This Stream-based design enables doing exactly this, and also leaves room
for a lightweight implementation of DAV:sync-token from RFC 6578.
DAV:sync-token allows querying the server for exactly which changes have
happened since the last sync without listing all items from all collections for
local comparison over and over.
The new Stream-based executor also allows improved concurrency control. At
present, up to eight concurrent operations execute at once, but I’ll fine tune
this further in future. At present, the underlying hyper library needs some
improvements for fine-grained control of connection pool.
The new implementation probably does a few more memory indirections or CPU cycles, but nothing which is practically noticeable. Due to the streaming nature of the new design, memory usage should remain a bit lower, since a single collection is enumerated at once now, rather than all of them beforehand.
The sync command, which performs a single synchronisation allowed viewing the
whole plan before its execution. This functionality remains: the whole plan is
simply collected into a huge array for display purposes, and then executed. This
usage is an edge case, so I don’t consider the extra cost has any meaningful
impact.
Pre-fetched items during planning
I also refactored the sync algorithm to simplify its logic. Previously, it would pre-fetch all items that were new or had changed, then enter a planning phase where it would again identify new or changed items by comparing their ETags and hashes. I realise that if an item wasn’t pre-fetched, that already implies it hasn’t changed; only pre-fetched items can differ from their previous versions (even if some may end up unchanged because their edits cancel out).
At the same time, items which have changed are guaranteed to be pre-fetched. As a result, the old code path that performed a late, lazy fetch was both unnecessary and effectively unreachable — but still needed to be implemented because of the earlier structure. With the refactor, that branch is gone, which makes reading the code more straightforward.
Auto-conflict resolution
I’ve implemented automatic conflict resolution. Pimsync will automatically resolve conflicts by keeping the item from whichever side is specified in the configuration file, if any (there is no default value for this and the functionality is disabled unless explicitly configured). This feature was present in vdirsyncer and is now available in Pimsync too.
ImapGoose
Since my last status update, I’ve also been working a lot of ImapGoose. I’ve written about it on separate posts (intro, update) over the last month. It’s currently close to a v1.0.0 release, with two issues blocking the release:
ImapGoose focuses on a single use case: synchronising an IMAP server with a local Maildir. Its narrower focus has made development much faster and much smoother. Pimsync on the other hand, tries to cover all use cases in the same domain, with a variety of storage backends. In the case of IMAP, there are plenty of other tools for legacy servers and different scenarios. In the case of Calendar and Contacts, we have a shortage of tools, so I’ll continue trying to cover all the usual scenarios.