One of my goals for re-writing vdirsyncer was to write “live tests”. These tests are series of operations against a real CalDav server, failing if the server doesn’t behave as expected (or if it doesn’t support a specific feature).
I’ve had this working for a couple of weeks now, and while it always feels like “it could be better” (the code quality isn’t great, but it’s also not my worst), the reality is that it fulfils its goal and can be improved later as needed. This is, after all, a test tool, and not something end-user facing.
Profiles
Profiles are tiny files which contain all the information run these integration tests with a real server. Here’s the one for baikal as an example:
host = "http://localhost:8002"
username = "baikal"
password = "baikal"
[xfail]
CreateAndDeleteCollection = "https://github.com/sabre-io/Baikal/issues/1182"
The xfail second lists tests that are known to fail with a given server. This
helps detect any new regressions, since a couple of tests are known to fail
due to servers not implementing some required features.
Results
I’ve tried to make the results as easy to read as possible, since I like to run these tests again when I change something internally to make sure it doesn’t break with any servers.
These are the current results:
🗓️ Running tests for: live_tests/xandikos.profile
- CreateAndDeleteCollection: ✅ passed
- CreateAndForceDeleteCollection: ✅ passed
- SetAndGetDisplayName: ✅ passed
- SetAndGetColour: ✅ passed
- CreateAndDeleteResource: ✅ passed
- CreateAndFetchResource: ✅ passed
- FetchMissingResource: ✅ passed
- CheckAdvertisesSupport: ✅ passed
✅ Tests 8 completed.
🗓️ Running tests for: live_tests/baikal.profile
- CreateAndDeleteCollection: ⚠️ expected failure: https://github.com/sabre-io/Baikal/issues/1182
- CreateAndForceDeleteCollection: ✅ passed
- SetAndGetDisplayName: ✅ passed
- SetAndGetColour: ✅ passed
- CreateAndDeleteResource: ✅ passed
- CreateAndFetchResource: ✅ passed
- FetchMissingResource: ✅ passed
- CheckAdvertisesSupport: ✅ passed
✅ Tests 8 completed.
🗓️ Running tests for: live_tests/radicale.profile
- CreateAndDeleteCollection: ✅ passed
- CreateAndForceDeleteCollection: ✅ passed
- SetAndGetDisplayName: ✅ passed
- SetAndGetColour: ✅ passed
- CreateAndDeleteResource: ✅ passed
- CreateAndFetchResource: ✅ passed
- FetchMissingResource: ✅ passed
- CheckAdvertisesSupport: ✅ passed
✅ Tests 8 completed.
🗓️ Running tests for: live_tests/fastmail.profile
- CreateAndDeleteCollection: ⚠️ expected failure: precondition failed
- CreateAndForceDeleteCollection: ✅ passed
- SetAndGetDisplayName: ✅ passed
- SetAndGetColour: ⚠️ expected failure: https://github.com/cyrusimap/cyrus-imapd/issues/4489
- CreateAndDeleteResource: ✅ passed
- CreateAndFetchResource: ✅ passed
- FetchMissingResource: ✅ passed
- CheckAdvertisesSupport: ⚠️ expected failure: server does not adviertise caldav support (unreported)
✅ Tests 8 completed.
🗓️ Running tests for: live_tests/nextcloud.profile
- CreateAndDeleteCollection: ⚠️ expected failure: server does not return etags (unreported)
- CreateAndForceDeleteCollection: ✅ passed
- SetAndGetDisplayName: ✅ passed
- SetAndGetColour: ✅ passed
- CreateAndDeleteResource: ✅ passed
- CreateAndFetchResource: ✅ passed
- FetchMissingResource: ✅ passed
- CheckAdvertisesSupport: ⚠️ expected failure: https://github.com/nextcloud/server/issues/37374
✅ Tests 8 completed.
As you can see, these tests cover all the “primitive” operations. Everything else is built on top of them, so as long as they work, more complex flows should work too. I’ll of course refine as needed if I find any bugs along the way.
The source is available in the vdirsyncer-rs repository.