I’ve been planning a full rewrite of vdirsyncer for quite some time now. So far work on vdirsyncer (both on the original version and on this rewrite) has happened during free time and is mostly unpaid. My intention is to seek sponsors for this rewrite, so that I can dedicate more hours to this project and deliver a well polished product to the open source ecosystem.
By becoming a sponsor, you will allow me to focus on this endeavour and spend much more time on it than I ever could on “another side project”.
This is a detailed description of everything that’s been planned so far. Given the considerable length of the project, progress will be delivered through several intermediate milestones. This also ensures a more modular design with multiple reusable parts.
What is vdirsyncer
vdirsyncer is a command line tool to synchronise two calendars and contacts
between a variety of supported storages (including CalDav and local
filesystem). A very popular usage scenario is synchronizing a server with a
local directory and using other applications to to view or edit the local
entries. Vdirsyncer can also syncronise between two remote storages, or two
local storages.
The current implementation
There’re a few hard-to-solve problems with the current design.
Much of the current design was written while learning all the quirks of involved standards, and without fully knowing the final shape vdirsyncer would have. It evolved to where it is, and maintenance has become pretty hard and painful: different internal bits are way too intertwined and it’s not possible to alter one part without affecting the others. In other words, there’s a strong lack of modularity.
On the distribution side, shipping python applications to end users has also become a pain point. The python version on each distribution varies, as does the version of many individual dependencies, making supporting all of them a dependency hell. Additionally, due to the fact that distributions ship individual libraries as separate packages, packaging vdirsyncer also implies packaging all its dependencies, which is often a burden for distribution package maintainers. There’s also no easy way to ship a single binary to end-users, which could be very useful for special debug builds, or builds with special changes to individual users trying to inspect an unusual scenario.
The current vdirsyncer was designed first as a command line tool, which makes it hard to be integrated into larger systems or to be used programmatically as part of larger applications. There has been much interest in reusing it in many context, both as a whole or some of its parts, but this is generally not possible.
Goals
The goal is to ship a solid version of vdirsyncer well designed from the start, and to implement most of its functionality as a library. I’ll still ship a command line tool, though it will actually just be a thin wrapper delegating everything to the library itself, keeping most of the underlying implementation reusable.
Rust has been chosen for this implementation, since its expressive type system allows implementing this in very safe ways. Rust is well-known for making it easier to write code that can account for all possible scenarios, rather than having to use exception-handling as a flow-control mechanism. Using Rust libraries from other languages (e.g.: from Python) is also a solved problem, which implies that developers who are not using Rust can still leverage this work.
The first milestone will be a storages library, which will allow interacting with different calendar/contacts storages without having to worry about internals or protocol implementations. It will expose raw entries without parsing their contents.
The second milestone will leverage the above and implement synchronisation between any two storages (including one-way and other features currently supported).
The third milestone will be the vdirsyncer cli itself, which will be a single binary that mostly wraps around the above two libraries.
Finally, a fourth milestone will be to parse CalDav/CardDav items from storages and expose typed entries. The goal of this is to make the storages libraries well usable by programs (web, desktop, mobile) that need to interact with these storages. These can be used by all sorts of calendar or contacts applications with little to no overhead.
I intend to also make this an educational endeavour, documenting progress, and potentially doing live streams if there is interest in it. These items aren’t set in stone; I’d like to hear more from [potential] sponsors. Monthly status updates (as are very common for funded open source work) will also happen.
Writing a GUI for vdirsyncer is not in my initial goals - although I do intend to provide all the foundations necessary to build one.
Technical details
The following is a general idea of what libraries would look like. I’ve been refining these for months now, and while the final results may be subtly different, they should not stray too much from this design.
The storage module
The first key item is the storage module with multiple Storage
implementations. The intent is to have one type per supported storage type
(e.g.: Vdir<Ics>, Vdir<Vcf>, SingleFileCalendar, CalDav, CardDav,
GoogleCalDav, EteSync<Ics>, etc), all of them having the same API (by
implementing a common trait).
Each one is initialised purely with a URL describing it (e.g.:
caldav://user:password@host:port/path or file+ical:///path/to/calendar.ics)
and a URL fragment indicating whether it is read only or not (e.g.:
#read_only=true). Applications using the library can avoid exposing the URL
to users if they prefer, but for an internal representation, it is the simplest
thing to handle.
Each Storage implementation handles all the reading/writing/communicating
with the actual storage itself, both for data and metadata. In-memory will be
implemented internally at this layer too.
While vdirsyncer will be the obvious user of this module, other tools can use
it for reading storages too. For example, a calendar app can read a local
Vdir<Ics> or even a CalDav storage. Providing a base for other developers
to write calendar and contacts apps is an important goa.
The SyncPair type
A SyncPair represents synchronisation settings between two storages. It
contains the set of synchronisation rules like direction (e.g.: a->b,
bi-directional, etc) and conflict resolution rules (if any).
A SyncPair reads both storages and can determine which operations to execute
to sync both ways.
A SyncPair also needs a KnownState to operate, which is the previous state of
both collections (this allows understanding which one has changed and which one
hasn’t when we find discrepancies between the two).
The KnownState will be passed to the SyncPair on creation, and can be read
from different forms of storage. Our cli tool will save to disk as we currently
do, but potential library users should be able to serialise it into a database
or however they deem fit.
Library
An important aspect of doing a rewrite is avoiding raising exceptions as a
flow-control mechanism – when something fails it will return an appropriate
error, allowing the calling layer to handle this properly. That is, no
panic!() or exposing any unsafe behaviour.
This means, amongst other things, that if a single collection has a fatal issue, vdirsyncer will continue working on everything else. This also enables writing things like long-lived processes that use vdirsyncer as a library. Having a daemon that does automatic syncing has also been a highly demanded feature.
Note that while this would also have technically been possible in Python, the language doesn’t really have clear constructs for returning errors that MUST be handled.
Cli tool (e.g.: vdirsyncer 2)
So far everything described is libraries, and the final piece is the small command line tool that reads a configuration file and calls this library with the right parameters.
Audience
This development seeks to enrich the open source ecosystem by providing not just a tool, but also libraries that can boost other developments by getting the complicated bits out of the way. Therefore, the target audience of this development shall include:
- Current users of vdirsyncer who want a lighter version that has less dependency issues.
- Developers of servers who need to handle calendar/contacts collections.
- Developers of contacts or calendar applications for desktop or mobile.
- Engineers working on hosted solutions involving synchronisation calendars or contacts.
Note that developers don´t just imply Rust developers, since libraries wrapping Rust code can be created in many languages, including Python and the commonly used mobile development languages.
Contributing
If you would like to sponsor this project, see my page on liberapay. Don’t hesitate to reach out if you have additional inquiries or would like to collaborate in some other way.