Live tests
A belated status update for July, since the final task has been “99% ready” for over fifteen days, and trickled way into August.
This last task was generating reports for live tests. The idea behind live tests is to check that libdav is fully compatible with various expectations out there. A colleague recently asked whether this was really necessary: standards mean you just target the standard and know your code works on all servers.
In practice, this is far from true. These live tests have immense value:
- I’ve found bugs in my implementation which only reproduce with one server, while others are lenient and accept technically-invalid requests.
- I’ve found bugs in servers, many of which I’ve reported upstream, several having been fixed already.
- I’ve found bugs in my reasoning, which led to bogus tests where both my library and the server were correct, but I learnt something new.
Test results yield a clear summary of what’s expected to work and what isn’t. A few servers have quirks handling escaped symbols. A few newer servers are prone to races when deleting or have broken calendar creation behaviour.
And developers of new CalDAV/CardDAV servers can use this suite to test themselves — a result of me reporting the bugs I find.
The tricky thing about these tests is that they’re not unit tests, and they even have dependencies. For example, if creating a calendar fails, then testing whether we can delete one is skipped, rather than a failure. The tests which are expected to fail or need to be skipped therefore vary per server. This was so far done with hack macros which skipped compiling tests for a target server, so the test suite needed to be recompiled for each target server.
The new implementation is a “regular” Rust application, which runs a multi-root tree of tests, many of which depend on others. The server details are read from a simple toml file, as are known failures.
It took me a lot of iterating to get to the final shape, because this was a task where I knew what I didn’t want, but I didn’t have a fully clear picture in mind before starting — I needed to explore the way until I found it.
I’m relatively happy with the result, but this is now great foundation with relatively simple tests — I need to add plenty more, which will happen gradually as I find bugs or quirks in different servers. Or (hopefully not), as bugs are found in libdav itself.
The latest version of the report is here. These are generated automatically, and I have yet to settle on a concrete place where I’ll publish the latest version.
Of note, these tests now also run against Davis and Rustical, two relatively newer CalDAV servers.
Encoding issues
These tests also led me to find some encoding issues: the characters #, [
and ] are illegal in path components in URLs, and were not being encoded
properly (I mixed up two relatively similar character sets, hence why almost
every other one was handled correctly).
I must have read through RFC 3986 a dozen times already, and I fear this won’t be the last.
Per-collection protections
It is now possible to enable protections on a per-collection basis. If I delete an important calendar on one host by accident, I don’t want this deletion synchronising to others. On the other hand, I have some (programmatically-generated) calendars which are sometimes empty, and this should sync fine.
I do admit I have mixed feelings about the feature. It seems a bit too specific and hand-holding to avoid accidental deletions, but it is also legitimately useful.
jmap.io
Pimsync is now listed under clients in https://jmap.io/software/ since it’s very much a JMAP client, even if not optimised for this protocol yet.