It’s been a busy couple of months. I had to take some unexpected time off for personal reasons in January, followed by FOSDEM and the postmarketOS hackathon which kept me plenty busy.
pimsync
I’ve implemented an “interactive sync” (via pimsync sync -i), which shows a
list of all operations to be performed and waits for confirmation before
executing. This is especially useful when storages are in some unusual state and
one wants to be certain that sync will perform the expected operations.
While releases of pimsync and related libraries always depend on stable versions
of dependencies, during development these often require the latest main commit
of some dependency. I’ve switched to using submodules to track these, which
should greatly simplify compiling pimsync for anyone who wants to try out the
latest development version.
Pimsync now also ships with zsh completion scripts. Writing these was actually more fun than I expected it to be. I’ve kept some tidy notes and will publish a short guide on how to write zsh completions in future. Completion scripts for other shells are welcome.
Conflict resolution for properties is now implemented: when the name or description of a collection has diverged on both storages, this conflict can now be resolved. Pimsync displays a prompt with options to keep A, keep B, or manually provide a new value.
The biggest change has been one-way sync. It is now possible to configure
pimsync to synchronise two storages one way, effectively making B a clone of A.
This required several internal refactors: I had to split out the “analyse the
state of this pair” logic from “determine the new set of actions for this pair”
logic. The latter is implemented via a vstorage::sync::Mode trait, so
any library consumer could potentially implement their own custom logic (e.g.:
sync only deletions one way, but only new items the other way).
I’m happy with how this last change turned out, but somehow feel that the code can be simplified further at this point. Complexity is a necessary ingredient to progress: sometimes we need to add some complexity to see the path forward and then simplify. At this stage, I’m looking to refine and simplify the general implementation a bit.
I’ve started some work on sync-token (RFC 6578) support. This allows pimsync
to only query a server for items which have changed since the last
synchronisation process (instead of having to fetch a whole list and determine
this manually). This dramatically reduces network usage (and local processing)
for each synchronisation. Server support for this feature varies, so the current
behaviour will remain as fallback. Several of the foundational pieces are there,
but the full feature is not yet done.
davcli
davcli now uses the latest version of
libdav. While this is mostly a debugging/inspection tool, it also serves as an
example of using the library.
Matridge
The postmarketOS team relies on Matrix a lot for communication and coordination. While I’m not a fan of the latest VC-funded hype, it’s true that Matrix is widely used in the FLOSS ecosystem, and better than the usual variety of proprietary silos. Even so, I didn’t want to deal with yet-another messaging client, so I set up a local XMPP->Matrix gateway using matridge with the intent of using the Matrix network from an existing client.
The gateway had a lot of missing essential features, so I had no choice but to set out and implement them, and fix a variety of issues. Patches were quickly merged upstream, and we even had an opportunity to sync up with the lead developer at FOSDEM.
At this point, the gateway is fully usable with a new Matrix account without having to bootstrap it on some other client. It now properly handles invites, joining rooms and even implements commands like changing the account password. All this is doable from your favourite XMPP client!
Both the XMPP and Matrix ecosystems are richer thanks to this.
hare-dbus
hare-dbus was already in a pretty solid working state since last year, but lacked support for non-blocking use. I’ve now implemented support for non-blocking mode and integration with hare-ev. The current non-blocking implementation works, but serialising and parsing messages is still a bit tedious since it requires manually using the mid-level message API.
The pending work here is to adapt to the code generator: it can currently generate stubs for using hare-dbus’s blocking API. The message types, parsers and serialises which it currently generates are fine to use with the non-blocking API, but the functions used to send the messages and callbacks are tied to the blocking API. These two need to be split (making the blocking stubs optional), and I need to implement stubs for the non-blocking API.
This has been shaping up really nicely. Working with hare is really nice, it’s simple, lean and doesn’t add extra abstraction layers between application code and the native OS APIs.
hare-lsp
While I do enjoy writing code in Hare, I really missed the extremely useful IDE integration which Go offers via gopls. I wrote hare-lsp last year to fill in that gap, but while working on hare-dbus, I found lots of scenarios where go-to-definition didn’t work. I kept around samples and one-line explanations of each scenario, and later circled back to implement tests for them and finally fix hare-lsp. It should now provide a much more reliable experience, although a lot of scenarios can’t be fixed until the hare stdlib implements the type inference used by the compiler.
ImapGoose
kqueue/inotify re-implementation
ImapGoose’s kqueue implementation was essentially unusable. This was mostly
due to a limitation in the general-purpose library used to track files:
fsnotify.
When monitoring a directory for file creations or deletions, kqueue needs one file descriptor per directory. When monitoring for file changes, kqueue needs one file descriptor per file in said directories. Due to the nature of Maildir, files are only created or deleted (or moved), but never changed in-place. So ImapGoose doesn’t need to track file changes, and only needs one file descriptor per directory (or two per mailbox).
fsnotify doesn’t have an API to detect only file creation or deletion. This
also seems undesirable upstream and didn’t fit its general-purpose use case.
I dropped this dependency, re-implemented the inotify backend (which is now
optimised for this specific use case), and implemented a new kqueue backend
which is also optimised for ImapGoose. fsnotify’s kqueue backend also kept a
list of files in-memory to determine which files changed, but this duplicates
the logic already implemented by ImapGoose using a status repository (which
persists when the process exits too).
Pending work on ImapGoose
The main issue in ImapGoose which is blocking a v1.0.0 release is [recovering from sleep]: when a system goes to sleep and wakes up again, ImapGoose isn’t detecting a timeout. Sometimes it takes tens of minutes to detect this, even though it has a timer every 3 minutes. I still can’t fathom what’s going on, and have been unable to reproduce the issue in a controlled test environment.