I’ve implemented support for DAV:sync-collection (from RFC 6578) in pimsync.
With this extension, requesting for a list of items (events, todos, contacts,
etc), returns a unique token. The next time that the list of items is requested,
the client (e.g.: pimsync) includes this token, and the server only returns new
items or those which have been changed, substantially reducing the amount of
data transferred, and slightly simplifying the total work done locally.
A large portion of this feature was implemented with when implementing JMAP support, since JMAP also supports this unconditionally. The work done recently focused mostly on the WebDAV-specific portions which were still missing, and on actual testing with real servers.
I’m quite pleased with the result, and was positively surprised at how smooth this went — I expected this task to hit a lot more blockers and issues. I’m also glad that the generic “incremental sync” work done for JMAP fit right in for this. I did design it with WebDAV in mind, but I still expected that more refinement would be required. I was not.
Incremental sync will be used automatically, falling back to regular sync if it is unavailable. In fact, incremental sync is tried first, and if that fails, the server is queried on whether it supports incremental sync or not (since we can’t distinguish some unrelated failures from “unsupported”). This means that usages using incremental sync perform less queries, whereas scenarios without it will have additional queries. If incremental sync is unsupported, this fact is cached, so it the additional query is only one per storage per process lifetime.
The work on incremental sync was sponsored by NLNet. Since I’d over-estimated how much this task would take, some of the leftover funds (destined for development hours) will be used to implement the upcoming Web Push support for WebDAV.
I also performed lots of smaller performance improvements all round too. One big pending performance improvement is concurrency: each pair usually does only one operation at a time. Massive concurrency would be easy to implement, but we don’t want massive concurrency: we need to limit the maximum amount of open files and open connections to a given server to something reasonable. I.e., bound concurrency. This isn’t hard, it just requires care.
Another new features is being able to disable syncing properties via the
properties ignore directive. This had been requested before, and is
particularly useful when synchronising from WebCal, which do not supports
properties. If a name was given to the calendar on the other side, a one-way
sync form a WebCal storage would quickly override it.