For my personal use case, pimsync works fine, and it’s been a great “vdirsyncer version 2”. In particular, I find the daemon mode of enormous value. In case a server rejects a single calendar event or contact, pimsync will also continue and sync everything else. These are the two main features which have made a difference for me. I can mostly run it in the background and forget about it

For other use cases, some features are still missing, and I’ll be working on addressing these in the coming months.

By the end of this project, all users should be able to move to pimsync. Of course, nobody’s force to migrate, and users happy with vdirsyncer can continue using it, although support will wind down substantially. That said, I hope the improvements in pimsync are attractive for most use cases.

With the Python ecosystem being in a perpetual state of change and backwards incompatibility, I suspect that in a few years vdirsyncer will gradually become harder to run: functionality or libraries on which it relies will slowly stop working on newer versions of Python. To be frank, I find this quite disappointing, even if I’m sun-setting the project.

The man page pimsync-migration(7) (also available online) contains a list of all missing features. If you’re using any functionality which is missing in pimsync and also missing from this list, please reach out by opening an issue or send an email to the list so I can bring it into the roadmap.

Beyond porting vdirsyncer features, I don’t have any major plans for pimsync, and I expect it to enter a long period of stability, with minor fixes. I do have several other related projects (with good synergy but low coupling) in mind.

Pimsync 0.5.7 is out, with mostly minor improvements and bug fixes.

The pimsync sync command (which synchronises once and exits) now shows a human-readable summary of operations being performed. pimsync sync -i allows reviewing operations before they happen. This is especially useful during initial setup (to ensure that everything is working as intended) and for debugging unusual scenarios.

The sync command now exits with a non-zero status on error. An obvious behaviour, but I overlooked this previously since my main focus was the daemon command, which never exits.

I’ve made good progress with http_proxy support and rewriting the buggy Unix Domain Socket support to be much smoother. The previous implementation used a library which rewrote URL domains to include the socket path. This is quirky, and broke under conditions that are rather common. In Pimsync’s case, the socket path is declared in the configuration file and does not change. The new implementation uses a newly written code for this, since no existing library fit this (rather common) use case. I’d like to upstream this Unix Domain Socket support into hyper-util, but upstream is in an odd state of flux: the old HTTP client which we’re using is a legacy one, and the new one isn’t ready yet. Upstreaming support for legacy interfaces might not be the best time investment.

Sway

[permalink]

I implemented support for the ext-workspace-v1 protocol in sway, which will be released as part of sway 1.12. This will enable using a standard Wayland protocol to control workspaces, and should substantially improve interoperability of status bars and similar shell components.

I also fixed a small bug where fullscreen windows which didn’t draw a surface large enough didn’t cover the whole screen. This is a niche edge case which I only encountered once with an emulator, but the issue had been open for a long time and it was actually a pretty straightforward fix.

Khal

[permalink]

I’ve tagged a release of khal 0.14.0, with many fixes and improvements mostly made by contributors — I mostly reviewed, merged and tagged the release. Thanks to everyone who contributed for this release.

ImapGoose

[permalink]

I tackled a bug report in which the listener connection was not properly re-established after a network error. This led me to iron out the entire error handling for the listener connection. After applying the fixes, some paths to improving the surrounding code became obvious, and the code for handling disconnections is now much tidier, in essentially a single mini event loop which handles all possible failure modes for the listener.

ImapGoose now also handles a server resetting its MODSEQ, a niche scenario which isn’t even mentioned in the specification (and I guess, technically allowed). I did not expect this to actually happen, but the fix is the result of this actually happening to someone and them having to work around it.

Detection of non-existent mailboxes was not always accurate. Sometimes ImapGoose assumed an unrelated error, when in reality, the error was a missing mailbox. ImapGoose now unconditionally tries to create a mailbox, which succeeds if the mailbox does not exist, and will simply fail again if the error was something else. Regrettably, error detection in IMAP is not very clean, but this has no practical downsides.

That’s all for this last month. Better late than never!

The full documentation for it is in the XDG base directory specification.

Prior art

[permalink]

• dumb_runtime_dir: PAM module with less than 100 lines of C code.
• pam_rundir: PAM module, functionally equivalent to the above, over 400 lines of code.
• pam_systemd: Tied to systemd. Relies on PAM too.
• pam_elogind: Tied to elogind, an unmaintained fork of systemd-logind.
• mkdir /tmp/run-$UID: simple, but insecure (see below).
• mkrundir: Small binary written by myself, less than 40 lines, but requires setuid (and I’d like to address this).

Requirements

[permalink]

Concurrent sessions for the same user should share the same runtime directory. For example, when logging in via a physical seat on a host, and then starting an SSH session, they both need to use the same runtime directory path.

A native approach might use /tmp/run-$USER, but this is vulnerable to another user account creating a directory in that location, inadvertently controlling ownership of sensitive files. Likewise, another user simply creating a file there, blocking creating of the runtime dir. Using a deterministic location in a world-writeable location is insecure.

Approaches

[permalink]

All possible approaches have their caveats and compromises. We can only pick which compromise we prefer.

• PAM model: requires installing and relying on PAM, a huge amount of complexity and attack surface for something so trivial. They also wouldn’t work when simply using loging on a tty.
• setuid/setgid binary: probably the simplest approach, but usage of setuid/setgid is frowned upon from a security point of view.
• Using a deterministic location in /tmp is vulnerable to races/hijacking.
• Somewhere in $HOME: survives reboots. This lacks any proper clean-up mechanism, but can also result in stale files and break application expectations.
• Dedicated daemon: expose a socket to a specific group, and have the daemon create the directory on request. Works, but having an additional daemon for this seems like an overkill.
• Randomised location in /tmp: requires coordination across concurrent sessions.

Additionally, PAM, setuid or a dedicated daemon all require root involvement in configuring or initialising a user session after user login. This kind of initialisation should be handled inside the user session: the root user should only be involved if strictly necessary.

Randomised location in /tmp

[permalink]

Of all the above approaches, I opted for using a non-deterministic (i.e., unpredictable) location in /tmp.

A “name file” somewhere in $HOME keeps the location of the current runtime directory. Upon login, a helper called via ~/.zprofile checks this file. If it contains the location of a directory owned by the current user with the right permission, it is used as the current $XDG_RUNTIME_DIR. Otherwise, a lock is taken, a new directory is created, and the name file is replaced. The lock ensures that two concurrent executions don’t overstep each other when replacing the name file.

This is simple enough, and can be implemented in a small, simple binary.

• Install a package: pacman -S $PKG_NAME. ¹
• View dependencies of a package: pacman -Qi $PKG_NAME.
• Which package owns a file: pacman -Qo /path/to/file.

The list goes on. If you’ve never used this package manager, these commands are as obscure as it gets. You’d never guess the right flags, and you’d never guess what they do just by reading them. But if you’re proficient with pacman, those commands feel obvious, and it’s easy to think that anyone with doubts should just read the documentation.

During the many years I used Arch and pacman, I never questioned this. It felt normal and natural, just the way things are.

I’ve now been using Alpine for a couple of years, and apk is quite a bit more obvious:

• Install a package: apk add $PKG_NAME.
• View dependencies of a package: apk info --depends $PKG_NAME.
• Which package owns a file: apk info --who-owns /path/to/file.

You might not guess the exact names of flags to use based on what you want to do, but you’ll definitely guess what these commands do just by reading them. Looking back at my years using pacman, I ask myself how that ever felt natural or intuitive.

It just takes stepping outside of our daily habits to notice how cryptic some of them are. We often use tools for years until they become second nature, and never question whether they make sense, if they truly are “easy to use” or we’re just used to them. Every once in a while, we need to step out of our bubble and use a different tool for our daily tasks, so we can have a clearer, less biased picture of the tools we rely upon.

I’ve come to think of this as the Pacman syndrome: familiarity makes an unintuitive interface feel natural.

Disclaimer

[permalink]

I use pacman as an example because it’s the program that made me realise this so clearly. I look back and think “yeah, this thing is weird and non-obvious, I can understand why folks using something else see it the way they do”.

pacman is great. It’s lightweight and fast. And compared to most package managers, it’s still among the better ones.

pimsync

[permalink]

I’ve implemented an “interactive sync” (via pimsync sync -i), which shows a list of all operations to be performed and waits for confirmation before executing. This is especially useful when storages are in some unusual state and one wants to be certain that sync will perform the expected operations.

While releases of pimsync and related libraries always depend on stable versions of dependencies, during development these often require the latest main commit of some dependency. I’ve switched to using submodules to track these, which should greatly simplify compiling pimsync for anyone who wants to try out the latest development version.

Pimsync now also ships with zsh completion scripts. Writing these was actually more fun than I expected it to be. I’ve kept some tidy notes and will publish a short guide on how to write zsh completions in future. Completion scripts for other shells are welcome.

Conflict resolution for properties is now implemented: when the name or description of a collection has diverged on both storages, this conflict can now be resolved. Pimsync displays a prompt with options to keep A, keep B, or manually provide a new value.

The biggest change has been one-way sync. It is now possible to configure pimsync to synchronise two storages one way, effectively making B a clone of A. This required several internal refactors: I had to split out the “analyse the state of this pair” logic from “determine the new set of actions for this pair” logic. The latter is implemented via a vstorage::sync::Mode trait, so any library consumer could potentially implement their own custom logic (e.g.: sync only deletions one way, but only new items the other way).

I’m happy with how this last change turned out, but somehow feel that the code can be simplified further at this point. Complexity is a necessary ingredient to progress: sometimes we need to add some complexity to see the path forward and then simplify. At this stage, I’m looking to refine and simplify the general implementation a bit.

I’ve started some work on sync-token (RFC 6578) support. This allows pimsync to only query a server for items which have changed since the last synchronisation process (instead of having to fetch a whole list and determine this manually). This dramatically reduces network usage (and local processing) for each synchronisation. Server support for this feature varies, so the current behaviour will remain as fallback. Several of the foundational pieces are there, but the full feature is not yet done.

davcli

[permalink]

davcli now uses the latest version of libdav. While this is mostly a debugging/inspection tool, it also serves as an example of using the library.

Matridge

[permalink]

The postmarketOS team relies on Matrix a lot for communication and coordination. While I’m not a fan of the latest VC-funded hype, it’s true that Matrix is widely used in the FLOSS ecosystem, and better than the usual variety of proprietary silos. Even so, I didn’t want to deal with yet-another messaging client, so I set up a local XMPP->Matrix gateway using matridge with the intent of using the Matrix network from an existing client.

The gateway had a lot of missing essential features, so I had no choice but to set out and implement them, and fix a variety of issues. Patches were quickly merged upstream, and we even had an opportunity to sync up with the lead developer at FOSDEM.

At this point, the gateway is fully usable with a new Matrix account without having to bootstrap it on some other client. It now properly handles invites, joining rooms and even implements commands like changing the account password. All this is doable from your favourite XMPP client!

Both the XMPP and Matrix ecosystems are richer thanks to this.

hare-dbus

[permalink]

hare-dbus was already in a pretty solid working state since last year, but lacked support for non-blocking use. I’ve now implemented support for non-blocking mode and integration with hare-ev. The current non-blocking implementation works, but serialising and parsing messages is still a bit tedious since it requires manually using the mid-level message API.

The pending work here is to adapt to the code generator: it can currently generate stubs for using hare-dbus’s blocking API. The message types, parsers and serialises which it currently generates are fine to use with the non-blocking API, but the functions used to send the messages and callbacks are tied to the blocking API. These two need to be split (making the blocking stubs optional), and I need to implement stubs for the non-blocking API.

This has been shaping up really nicely. Working with hare is really nice, it’s simple, lean and doesn’t add extra abstraction layers between application code and the native OS APIs.

hare-lsp

[permalink]

While I do enjoy writing code in Hare, I really missed the extremely useful IDE integration which Go offers via gopls. I wrote hare-lsp last year to fill in that gap, but while working on hare-dbus, I found lots of scenarios where go-to-definition didn’t work. I kept around samples and one-line explanations of each scenario, and later circled back to implement tests for them and finally fix hare-lsp. It should now provide a much more reliable experience, although a lot of scenarios can’t be fixed until the hare stdlib implements the type inference used by the compiler.

ImapGoose

[permalink]

kqueue/inotify re-implementation

[permalink]

ImapGoose’s kqueue implementation was essentially unusable. This was mostly due to a limitation in the general-purpose library used to track files: fsnotify.

When monitoring a directory for file creations or deletions, kqueue needs one file descriptor per directory. When monitoring for file changes, kqueue needs one file descriptor per file in said directories. Due to the nature of Maildir, files are only created or deleted (or moved), but never changed in-place. So ImapGoose doesn’t need to track file changes, and only needs one file descriptor per directory (or two per mailbox).

fsnotify doesn’t have an API to detect only file creation or deletion. This also seems undesirable upstream and didn’t fit its general-purpose use case.

I dropped this dependency, re-implemented the inotify backend (which is now optimised for this specific use case), and implemented a new kqueue backend which is also optimised for ImapGoose. fsnotify’s kqueue backend also kept a list of files in-memory to determine which files changed, but this duplicates the logic already implemented by ImapGoose using a status repository (which persists when the process exits too).

Pending work on ImapGoose

[permalink]

The main issue in ImapGoose which is blocking a v1.0.0 release is [recovering from sleep]: when a system goes to sleep and wakes up again, ImapGoose isn’t detecting a timeout. Sometimes it takes tens of minutes to detect this, even though it has a timer every 3 minutes. I still can’t fathom what’s going on, and have been unable to reproduce the issue in a controlled test environment.

iDEAL has a few notable advantages: it’s a purely local system (avoiding any foreign intermediaries or fees), payments flow directly through the banking system, and I don’t have to share sensitive card information with merchants. I instruct my bank to send money: I don’t give a merchant permission to extract it.

Sadly, iDEAL is being phased away by Wero. Wero is marketed as a “truly European” payment method, touted as an improvement. In reality, it is quite the opposite: it introduces strong dependency on Google and Apple, two foreign companies with terrible track records on user rights.

Wero supports only apps that run on Google’s and Android proprietary locked-down operating systems. It does not support using a regular browser on a regular computer. These facts are confirmed in their FAQ.

The choice is clear: to continue using online payments, one must use a mobile device entirely under the control of Google or Apple and agree to their contractual terms, or stop participating altogether. Wero is legally European, but in practice it locks users into foreign ecosystems.

What really drives me crazy is the shameless propaganda around the initiative. Wero is presented a European, independent, future-proof payment system. But it’s an American lock-in disguised as progress. The system relies entirely on foreign corporations subject to laws and rulings outside Europe — jurisdictions currently campaigning against European data sovereignty.

Person 1: I went to see my dentist today.
Person 2: What did they say?

I’ve heard variants of the above since the nineties, but this is not at all a modern trend. The following is example from William Shakespeare (1594):

There’s not a man I meet but doth salute me
As if I were their well-acquainted friend.

Note the usage of “their”, referring to the singular noun “a man”.

The following example is from Jane Austen (1813):

To be sure you knew no actual good of me —
but nobody thinks of that when they fall in love.

The key is that “nobody” is grammatically singular: we say “nobody is here”, not “nobody are here”.

The following Biblical example (James 2:15–16) uses “them” in singular:

Suppose a brother or a sister is without clothes and daily food.
If one of you says to them, “Go in peace; keep warm and well fed,” but does nothing about their physical needs, what good is it?

Examples of this usage go back at least as far back as the 14th century.

Let’s first understand the base mechanism which everything else builds upon: how Vim discovers and loads plugins…

The runtimepath list

[permalink]

Vim defines a runtimepath variable, a set of directories inside which it will search for plugins. For my Neovim installation, runtimepath includes the following:

• $HOME/.config/nvim
• /etc/xdg/nvim
• $HOME/.local/share/nvim/site
• $HOME/.local/share/nvim/site/pack/*/start/*
• /usr/local/share/nvim/site
• /usr/share/nvim/site
• /usr/share/nvim/runtime
• /usr/share/nvim/runtime/pack/dist/opt/netrw
• /usr/share/nvim/runtime/pack/dist/opt/matchit
• /usr/lib/nvim
• $HOME/.local/share/nvim/site/pack/*/start/*/after
• /usr/share/nvim/site/after
• /usr/local/share/nvim/site/after
• $HOME/.local/share/nvim/site/after
• /etc/xdg/nvim/after
• $HOME/.config/nvim/after

The above list is, in part, defined as subdirectories of XDG_DATA_HOME and XDG_DATA_DIRS. Use :echo &runtimepath to view the exact values on your setup, or run :h runtimepath to see the documentation.

Inside any of these directories, Vim gives special treatment to a few special subdirectories:

• plugin/: is loaded automatically.
• after/plugin/: is loaded automatically and last, overriding any defaults.
• ftplugin/: is lazy-loaded and executed when a file of a matching filetype is opened.
• autoload/: is lazy-loaded on demand. More on this below.
• lua/: its contents are loaded via require('modulename'). More on this below.

Vim’s autoload mechanism

[permalink]

When Vim tries to execute a Vimscript function called autoload#filename#functionname(), Vim will search for a function called functionname in the file autoload/filename.vim. Vim will search for this file in any of the directories defined by runtimepath.

Essentially, placing files in the correct path in any of these directories will result in Vim lazy-loading the file when the function is first called, so placing code here is enough to lazy load it.

These are only loaded once, so if you have custom code that needs to be executed for HTML and XML files, you can include an ftplugin/html.vim and ftplugin/xml.vim which both autoload the same common file.

Neovim’s lua search paths

[permalink]

When using require('mymodule') in lua code, Neovim will search for the file mymodule in the lua directory inside any of the directories defined in runtimepath. Note that Lua support is entirely Neovim-specific.

The packadd command

[permalink]

Files in $HOME/.local/share/nvim/site/pack/*/start/* are loaded automatically. On the other hand, files in $HOME/.local/share/nvim/site/pack/*/opt/* are loaded manually via packadd. Generally, this is only needed for plugins which initialise pre-emptively (e.g.: by using plugin rather than ftplugin or autoload), but you only want to initialise them in specific scenarios.

Installing a plugin

[permalink]

Given the mechanisms described above, installing a plugin is just a matter of placing its source files into the correct directory, and plugins already include files in the correct layout.

Sometimes you want specific plugins to only initialise if you open a file of a given type. In the case of plugins which require explicit initialisation (e.g.: require('myplugin').setup()), the ideal place to do this is in ftplugin/python.lua (replacing python with whichever filetype you care about).

Vim plugins can be configured and lazy loaded with greater ease: they typically take configuration via global variables (e.g.: vim.g.myplugin_use_monochrome = 1 or let g:myplugin_use_monochrome = 1), and then lazy-load, for example, on a matching filetype. Such plugins only load when a matching file has been opened, lazy initialise, but allow configuration to be trivially defined in init.vim or init.lua. Neovim Lua plugins typically use a .setup() function to initialise, requiring additional scaffold if you want to delay loading them until necessary.

Plugin managers

[permalink]

Plugin managers basically fetch plugins (e.g.: git clone into the correct location). Some will place plugins into opt instead of start. Some plugin managers implement lazy-loading of plugins on demand, but this should not be necessary for plugins which properly implement lazy initialisation (e.g.: a Go plugin should ensure that it is auto-loaded by having its main entry point in ftplugin/go.vim or ftplugin/go.lua).

Plugin managers do provide certain conveniences, and there’s nothing wrong with using them, but it’s important to understand that they’re not a strict technical requirement.

In particular, when dealing with plugins which do not contemplate lazy-loading, plugin managers can help work around their initialisation and lazy-load them even if they’re not designed with this in mind. This can be achieved by using packadd, but requires additional manual setup.

Modern plugin managers have grown into orchestration frameworks. Most of that functionality compensates for plugin design, not editor limitations.

There are a few large alternatives to using plugin managers…

Distribution package manager

[permalink]

The same distribution channel which shipped Neovim likely has some plugins which can be installed the same way. Availability of plugins varies per distribution. The AUR and Nixpkgs in particular are known to include a large variety of plugins.

Tracking plugins with dotfiles

[permalink]

I track my dotfiles (including all my Neovim configuration) via git, so tracking plugins as git submodules has some natural synergy. Git itself can fetch and track plugins, and also gives me a clear overview if any plugins have local modifications which I may have forgotten about. Plugins are pinned to an exact version, and I can upgrade a plugin and its configuration in the same commit. Initial setup is also trivial.

The only manual step required is running :TSUpdate each time that the tree-sitter plugin is updated. Ideally, this shall change in future as the ecosystem starts to stabilise and we can rely on tree-sitter binaries installed through the usual channels.

In my dotfiles repository, I use the following command to add new plugins as submodules, installing them into the start directory so they are loaded automatically:

git submodule add \
    https://github.com/ibhagwan/fzf-lua \
    home/.local/share/nvim/site/pack/plugins/start/fzf-lua
git submodule add \
    https://github.com/lewis6991/gitsigns.nvim/ \
    home/.local/share/nvim/site/pack/plugins/start/gitsigns.nvim

This is the core of what a plugin manager ultimately does: place plugin sources into runtimepath. Having understood this, using a plugin manager becomes a matter of preference rather than necessity.

Each time I made a small tweak on an existing change, I had to switch back to the terminal and git add -p, skip irrelevant changes, and stage the one I want.

At some point I thought to myself “you can do better”. In recent months, I added two sets of keyboard mappings which have dramatically changed how I work.

Jump to changes

[permalink]

The set of mappings is:

nnoremap <expr> ]c &diff ? ']c' : '<Cmd>Gitsigns next_hunk<CR>'
nnoremap <expr> [c &diff ? '[c' : '<Cmd>Gitsigns prev_hunk<CR>'

These two are symmetrical: ]c jumps to the next change in the current file and [c jumps to the previous change in the current file (using gitsigns.nvim). The mapping is a bit long/complex because it intentionally avoids overriding the default behaviour when in diff mode.

This makes reviewing and potentially operating on changes in a file fast and straightforward.¹

Stage changes

[permalink]

The second set of mappings is:

nnoremap <buffer> <Leader>ga <Cmd>Gitsigns stage_hunk<CR>
xnoremap <buffer> <Leader>ga :Gitsigns stage_hunk<CR>

The first of these mappings stages the current hunk (contiguous range of changed lines) to be committed. The second mapping applies only in visual mode (e.g.: when text is selected) and only stages the selection.

This lets me easily stage changes to be committed from the editor, without having to jump to the terminal. The main annoyance of jumping to the terminal and using git add -p was that I have to search for these changes again, while I already have my cursor on them in the editor!

I also still use git add -p when staging large amounts of changes in bulk. I use whichever would be faster, given the state of changes.

Other operations

[permalink]

I only really use those two mappings for git operations inside the code editor. It also shows lines which have been added/changed/removed on the left next to sign numbers. Everything else I still do via the git command line tool.

For my own usage, I don’t see much value in doing things like git checkout, git commit, git fetch, git rebase, etc in the editor. I’ve seen plenty of full-blown git integration in code editors, and I do not find these more convenient than my current approach. In addition, they typically lack support for a lot of the non-basic features that I’ve come to rely upon over the years.

Perhaps there are more specific git operations which would be convenient inside the code editor. At this time, I don’t know of any and can’t think of any. A few more years of use might tell.

I run Unbound as a DNS resolve, and I will configure it to delegate the .local to a avahi2dns. avahi2dns uses D-Bus to talk to Avahi for the actual resolution. There are several layers and processes involved due to the lack of a dedicated solution: an mDNS proxy which just exposes a local DNS interface.

Setup

[permalink]

Configure unbound to resolve .local domains using mDNS via avahi.

apk add avahi avahi2dns
rc-update add avahi-daemon
rc-update add avahi2dns
service avahi-daemon start
service avahi2dns start

Add the following to /etc/unbound/unbound.conf.d/avahi-local.conf:

forward-zone:
	name: "local"
	forward-addr: 127.0.0.1@5354
server:
	do-not-query-localhost: no
	domain-insecure: "local"

Finally, reload unbound:

service unbound reload

I’ve written before on setting up quick and simple VMs with qemu. That article covered manually setting up VMs, which then need to be further customised via serial console or SSH.

Automating provisioning of a VM via serial console or SSH is non-trivial. I’d need a mechanism on the host to determine when the VM is ready to receive commands, but there’s no obvious way to do so, other than parsing the output of the serial port, waiting for a login: prompt, and then feeding the command. This approach is brittle and error-prone.

tiny-cloud

[permalink]

I opted for using images with tiny-cloud. tiny-cloud is a tool for auto-configuring VMs, and it’s intended for hosting providers. These VM images start up, and automatically seeks a configuration file on how to provision itself. There’s essentially two approaches to providing a configuration mechanism: via a well-known endpoint (a static URL at [fd00:ec2::254] or 169.254.169.254) or a CD image with label cidata.

Surprisingly, the CD image approach is much easier for a development/testing setup with static configuration. I use the official upstream images, the nocloud variant being the one that properly auto-detects a CD. I opted for the variant with properties Release=3.22.2, Arch=x86_64, Firmware=UEFI, Bootstrap=“Tiny Cloud” and Machine=Virtual.

Preparing metadata

[permalink]

Preparing the metadata is quite straightforward.

First I prepared a meta-data file with a machine identification profile:

instance-id: dev-vm-001
hostname: alpine-dev.whynothugo.nl

And a user-data file with user configuration data (for a VPS provider, this is data that varies depending on the tenant’s requested settings):

#cloud-config
users:
  - name: hugo
    ssh_authorized_keys:
      - ssh-ed25519
        AAAAC3NzaC1lZDI1NTE5AAAAIJiHg+cQwtqA7Ay+Fsimh9O5QnALlt6561TEu/Z67mU1
        hugo@hyperion.whynothugo.nl
    groups: wheel
    shell: /bin/sh

packages:
  - git

runcmd:
  - echo "Setup complete"

Most of this should be quite self-explanatory, and runcmd can be used for further programmatic configuration of the guest.

Creating the VM

[permalink]

I prepared a script which creates the VM itself:

# Fetch the image once, manually.
# curl -LO https://dl-cdn.alpinelinux.org/alpine/v3.22/releases/cloud/nocloud_alpine-3.22.2-x86_64-uefi-tiny-r0.qcow2

# Keep the original image clean.
cp nocloud_alpine-3.22.2-x86_64-uefi-tiny-r0.qcow2 alpine.qcow2

cloud-localds seed.iso user-data meta-data

# tiny-cloud auto-grows the partition, but the disk itself needs to be larger.
qemu-img resize alpine.qcow2 +5G
qemu-system-x86_64 \
	-drive file=alpine.qcow2,format=qcow2 \
	-drive file=seed.iso,format=raw \
	-drive if=pflash,format=raw,readonly=on,file=/usr/share/OVMF/OVMF_CODE.fd \
	-drive if=pflash,format=raw,readonly=on,file=/usr/share/OVMF/OVMF_VARS.fd \
	-nic user,hostfwd=tcp:127.0.0.1:2222-:22 \
	-m 2G \
	-smp cores=2 \
	-monitor unix:monitor.sock,server,nowait \
	-nographic

A brief explanation of the above:

• I copy the original image since the bootstrap process mutates it; using a copy allows me to quickly start over.
• cloud-localds prepares the previously mentioned ISO image with filename seed.iso.
• I resize (grow) the image because the default is full. I’ll need extra space to install my own packages.
• -drive if=pflash arguments configure the UEFI firmware; the VM won’t start without those.
• -nic user sets up an interface using user-space networking (this allows running the VM without needing to elevate privileges for any network setup.
  • hostfwd=tcp:127.0.0.1:2222-:22 forwards port localhost:2222 from the host to port 22 on the VM. This allows me to SSH into the VM. When running other services, I’ll forward the appropriate port as necessary.
• -monitor unix:monitor.sock,server,nowait sets up a monitor socket at ./monitor.sock. I can use this one to send commands to qemu, instructing it to shut down the VM, put it to sleep, resume, etc.

First run

[permalink]

During start-up, the VM logs:

 * Tiny Cloud - boot phase ...
   ++ expand_root: starting
GPT PMBR size mismatch (278527 != 10764287) will be corrected by write.
The backup GPT table is not on the end of the device. This problem will be corrected by write.
Re-reading the partition table failed.: Resource busy
resize2fs 1.47.2 (1-Jan-2025)
Filesystem at /dev/sda2 is mounted on /; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 21
The filesystem on /dev/sda2 is now 5381100 (1k) blocks long.

   ++ expand_root: done
   ++ set_ephemeral_network: starting
   ++ set_ephemeral_network: done
   ++ set_network_interfaces: starting
   ++ set_network_interfaces: done
   ++ enable_sshd: starting

It also auto-configures the network interface (via RA and DHCP).

Near the end of the start-up sequence, it logs:

 * Tiny Cloud - early phase ...
   ++ save_userdata: starting
   ++ save_userdata: done
 [ ok ]
 * Tiny Cloud - main phase ...
   ++ userdata_user: starting
   ++ userdata_user: done
   ++ create_default_user: starting
create_default_user: already exists
   ++ create_default_user: done
   ++ set_hostname: starting
   ++ set_hostname: done
   ++ userdata_bootcmd: starting
   ++ userdata_bootcmd: done
   ++ userdata_groups: starting
   ++ userdata_groups: done
   ++ userdata_users: starting
chpasswd: password for 'hugo' changed
   ++ userdata_users: done
   ++ userdata_write_files: starting
   ++ userdata_write_files: done
   ++ userdata_ntp: starting
   ++ userdata_ntp: done
   ++ userdata_package_update: starting
   ++ userdata_package_update: done
   ++ userdata_package_upgrade: starting
   ++ userdata_package_upgrade: done
   ++ userdata_packages: starting
fetch https://dl-cdn.alpinelinux.org/alpine/v3.22/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.22/community/x86_64/APKINDEX.tar.gz
(1/9) Installing brotli-libs (1.1.0-r2)
(2/9) Installing c-ares (1.34.5-r0)
(3/9) Installing nghttp2-libs (1.65.0-r0)
(4/9) Installing libpsl (0.21.5-r3)
(5/9) Installing libcurl (8.14.1-r2)
(6/9) Installing libexpat (2.7.3-r0)
(7/9) Installing pcre2 (10.46-r0)
(8/9) Installing git (2.49.1-r0)
(9/9) Installing git-init-template (2.49.1-r0)
Executing busybox-1.37.0-r19.trigger
OK: 99 MiB in 95 packages
   ++ userdata_packages: done
   ++ set_ssh_keys: starting
set_ssh_keys: no ssh key found
   ++ set_ssh_keys: done
   ++ ssh_authorized_keys: starting
   ++ ssh_authorized_keys: done
 [ ok ]
ssh-keygen: generating new host keys: RSA ECDSA ED25519
 * Starting sshd ... [ ok ]
 * Tiny Cloud - final phase ...
   ++ userdata_runcmd: starting
Setup complete
   ++ userdata_runcmd: done
   ++ bootstrap_complete: starting
   ++ bootstrap_complete: done
 [ ok ]

I can now log into the VM via ssh -p 2222 hugo@localhost.

Controlling the VM

[permalink]

Commands can be sent over monitor.sock using echo "$COMMAND_HERE" | socat - UNIX-CONNECT:monitor.sock. An “interactive shell” can be obtained with socat stdio UNIX-CONNECT:monitor.sock. Some example commands:

• info status: show a brief summary of the VM status
• system_powerdown: send ACPI shutdown signal (graceful shutdown)
• quit: immediately stop the VM
• stop: pause/freeze VM execution
• cont: continue VM execution
• system_wakeup: wake up from sleep
• help: show all available commands

Further steps

[permalink]

This is the basis for programmatically providing VMs using an upstream image as a base. With further customisation of user-data, I can configure VMs for specific purposes, and use them in scripted environments.

libdav exposed functionality via methods where all parameters must be specified. For example:

caldav.create_collection(
    "/calendars/tasks/",
    &[&names::CALENDAR],
    &[&prop] // These are properties in XML form.
).await?;

The problem with this approach is adding new functionality. For example, I want to allow specifying a display name, colour or other attributes of a calendar when creating it. Adding extra arguments doesn’t scale well: all call sites need to be updated each time a new argument is added, and all call sites need to specify several None values for properties which they’re not using. It becomes verbose and unreadable.

I’ve finally refactored libdav to use a new “Requests API”, where each functionality is exposed via a custom Request type implementing a DavRequest trait. This trait has two methods: prepare_request and parse_response. As you’ve hopefully guessed, these methods prepare a new request and parse the response respectively. The WebDavClient has a new request() method which takes an instance of a DavRequest and executes it.

These separate prepare_request and parse_response methods result in a cleaner layout and make testing much simpler by virtue of being able to test both of these without any network I/O. On this topic, I strongly recommend reading Network protocols, sans I/O.

DavRequest implementations like CreateCalendar have methods to specify additional fields using a builder pattern. Here’s an example using all the fields for this request type:

let create_calendar = CreateCalendar::new("/calendars/work/")
    // All of these calls are optional.
    .with_display_name("Work Calendar")
    .with_colour("#FF0000")
    .with_order(10)
    .with_components(&[CalendarComponent::VEvent, CalendarComponent::VTodo]);
webdav.request(create_calendar);

The resulting code is a greater total number of lines of code, but each functionality is nicely split into its own file, making code easier to read, follow and maintain. I recently wrote about the virtues of having a greater amount of simpler code.

I’m pleased with the result of the implementation, and I will add additional features (which were already planned) without having to add new arguments to functions for each new feature introduced.

Streaming API for vstorage

[permalink]

I’ve refactored the synchronisation portion of vstorage/pimsync to operate on a “stream of operations”. The general principle is that a plan with the actions to perform during synchronisation is a futures_util::stream::Stream of Operation objects, which each encode all the information required for their execution.

The executor now also takes a Stream of operations too, executing them one at a time. The main goal of this refactor was to improve how the daemon mode operates: after the initial “full” synchronisation, the daemon will monitor for individual changes, and send only operations for those individual changes to be executed — instead of the current approach which just performs another full sync. This Stream-based design enables doing exactly this, and also leaves room for a lightweight implementation of DAV:sync-token from RFC 6578. DAV:sync-token allows querying the server for exactly which changes have happened since the last sync without listing all items from all collections for local comparison over and over.

The new Stream-based executor also allows improved concurrency control. At present, up to eight concurrent operations execute at once, but I’ll fine tune this further in future. At present, the underlying hyper library needs some improvements for fine-grained control of connection pool.

The new implementation probably does a few more memory indirections or CPU cycles, but nothing which is practically noticeable. Due to the streaming nature of the new design, memory usage should remain a bit lower, since a single collection is enumerated at once now, rather than all of them beforehand.

The sync command, which performs a single synchronisation allowed viewing the whole plan before its execution. This functionality remains: the whole plan is simply collected into a huge array for display purposes, and then executed. This usage is an edge case, so I don’t consider the extra cost has any meaningful impact.

Pre-fetched items during planning

[permalink]

I also refactored the sync algorithm to simplify its logic. Previously, it would pre-fetch all items that were new or had changed, then enter a planning phase where it would again identify new or changed items by comparing their ETags and hashes. I realise that if an item wasn’t pre-fetched, that already implies it hasn’t changed; only pre-fetched items can differ from their previous versions (even if some may end up unchanged because their edits cancel out).

At the same time, items which have changed are guaranteed to be pre-fetched. As a result, the old code path that performed a late, lazy fetch was both unnecessary and effectively unreachable — but still needed to be implemented because of the earlier structure. With the refactor, that branch is gone, which makes reading the code more straightforward.

Auto-conflict resolution

[permalink]

I’ve implemented automatic conflict resolution. Pimsync will automatically resolve conflicts by keeping the item from whichever side is specified in the configuration file, if any (there is no default value for this and the functionality is disabled unless explicitly configured). This feature was present in vdirsyncer and is now available in Pimsync too.

ImapGoose

[permalink]

Since my last status update, I’ve also been working a lot of ImapGoose. I’ve written about it on separate posts (intro, update) over the last month. It’s currently close to a v1.0.0 release, with two issues blocking the release:

1. Slow reconnection after sleep/resume
2. Initial import does not pre-process corrupt newlines

ImapGoose focuses on a single use case: synchronising an IMAP server with a local Maildir. Its narrower focus has made development much faster and much smoother. Pimsync on the other hand, tries to cover all use cases in the same domain, with a variety of storage backends. In the case of IMAP, there are plenty of other tools for legacy servers and different scenarios. In the case of Calendar and Contacts, we have a shortage of tools, so I’ll continue trying to cover all the usual scenarios.

Old programming wisdom tells us that to debug code, you need to be smarter than the person who wrote it. If I use all my smarts to write code then I am, by definition, not smart enough to debug it.

During the past months, I’ve been changing how I write code a bit. I’ve identified that I focus too much on keeping the total amount of lines low. Keeping code concise is great, but obsessing too much with total lines of code is impairing my ability to write better software.

I’ve recently been writing more code, but shorter functions; a larger amount of simple abstractions. A function with 90 lines is pretty hard for a human to follow. Splitting it into multiple functions with clearly defined scopes is typically easier to understand and maintain. These multiple functions might be 30% more code in total, but I only ever need to keep in mind a single one of them. Ideally, what each function does is explained in a short brief sentence. When reading the code for any one of these functions, I only need to remember what the others do, but don’t need to worry about their internal implementation details (i.e. I don’t need to remember how they do it).

Too much code is not good, but too little code is also not good. Try to find the sweet spot. Don’t be scared of adding more code, especially if it removes code from the already-complicated places.

Most of what I’m saying today is obvious, and some of it are lessons that we learn during the first couple of years of programming. Yet it’s still wisdom that we should all remember to reflect upon, even after a few decades of coding.

The goal is to find an ideal model to use on my own devices for translating this language pair.

Overview

[permalink]

I ran this in batches, with different sets of models each. I supplied all models in each batch a set of texts. A native Chinese speaker proficient in English evaluated the translations and ranked them from best to worst.

First batch of tests

[permalink]

My first set of subjects were LibreTranslate (with its current default model), Opus-MT and NLLB-600M. All of these run perfectly fine on consumer hardware.

Below is the verdict of the first batch of tests (I didn’t keep more granular results for the first batch).

English to Chinese:

[permalink]

• LibreTranslate: good quality, unnaturally spoken, but clear.
• Opus-MT: more naturally spoken, but results not necessarily as accurate.
• NLLB: worst of the three.

Based on these results, the recommendations are:

• For communicating with a Chinese speaker, use Opus-MT.
• For language learning, use LibreTranslate (more literal).

Chinese to English

[permalink]

• LibreTranslate: not as good as the others, tense also wrong, misses a word.
• Opus-MT: mixes up tense, but otherwise good.
• NLLB: is very literal (not in a bad way).

Based on these results, the recommendations are:

• For interpreting text in Chinese, use Opus-MT.
• For communicating with an English speaker, use NLLB.

Second batch of tests

[permalink]

For my second batch of comparison, I used some newer (and also heavier) models:

• Opus-MT: same as before.
• NLLB-600M: same as before.
• NLLB-3.3B: heavier (and slower) than NLLB-600M.
• MADLAD-400: substantially heavier, and trickier to run on consumer hardware.
• Tower: Technically an LLM trained specially for translations, much heavier.

For this second batch of tests, I translated from Chinese to English, and kept more detailed notes of the results.

Test 2.1

[permalink]

• Input: 但说真的，一想到总有一天我也会把这个故事忘诸脑后，也是有点感伤呢。
• Opus-MT (6.70s): But really, it’s kind of sad to think that someday I’m going to forget about it.
• NLLB (7.93s): But honestly, it’s a bit painful to think that one day I’ll forget this story too.
• NLLB-3.3B (16.82s): But to be honest, it’s a little sad to think that one day I’ll forget this story too.
• MADLAD-400 (30.37s): But, honestly, it’s a little sad to think that one day I’ll forget this story.
• Tower (331.99s): But to be honest, when I think about the fact that someday I will forget this story, I feel a little sad.

Opus-MT omits the subject in the second part, which really washes out the meaning of the sentence. MADLAD omits “too”, which was considered as changing the original meaning too much. Tower was considered the best since it retained the original order of the statements, which in turn maintains its melancholic tone.

The ranking for these is (from best to worst):

• Good results: Tower, NLLB-3.3B, NLLB.
• Failed results: MADLAD-400, Opus-MT.

Test 2.2

[permalink]

The translation by deekseek.com is included here for reference.

• Input: 也就是一刹那的事，一种说不清道不明的疲倦的伤怀爬上他的心头。天边的乌云正在散去，几缕阳光挟着风的轻巧和雨的温柔钻进窗棂，吻了吻桌前那盆已开始枯萎的雏菊，春末的气息就这样被若无其事地搅乱了。
• Opus-MT (8.06s): That is, at a moment, an inexplicable tired wound that climbs over his heart. The clouds of the sky are dissipating, with the light of the wind and the tenderness of the rain rushing into the window, kissing the moist daisies that have begun to fade in front of the table, and the breath of the end of the spring is disturbed by nothing.
• NLLB (10.22s): And it came to pass that a sore wound of fatigue, unknown to the way, came upon his head. And the clouds of the sky were scattered, and the light wind and the gentle rain of the sun drove into the curtain, and kissed the basin before the table, which was beginning to wither, and the breath of the end of spring was disturbed.
• NLLB-3.3B (28.12s): It was a moment, a weary sorrow, which he could not explain, that climbed up his heart. The clouds of the sky were dissipating, and the light of the sun, with the gentle wind and the gentle rain, penetrated into the curtains, kissing the dew that had begun to wither in the pot before the table, and the breath of late spring was thus disturbed without cause.
• MADLAD-400 (58.45s): At that moment, a sort of unspeakable fatigue and sadness climbed up to his heart. The dark clouds were dissipating, and a few rays of sunlight, caught between the lightness of the wind and the gentleness of the rain, penetrated the window, kissed the daisies in the pot before the kiss table, which had begun to wither, and the air of the end of spring was thus disturbed as if nothing had happened.
• Tower (621.65s): It was over in a flash, a kind of fatigue that was hard to describe crept into his heart. The clouds on the horizon were dispersing, and a few rays of sunlight, carried by the wind and the rain, crept through the window and kissed the dying chrysanthemum
• Deepseek.com: In the blink of an eye, an indescribable, weary melancholy crept into his heart. The dark clouds on the horizon were dispersing, and a few strands of sunlight, carrying the lightness of the wind and the tenderness of the rain, slipped through the window lattice. They kissed the pot of withering daisies on the desk, quietly unsettling the late spring atmosphere.

Tower’s results here are truncated, and I later discovered that this is due to how I configured my test service. This model is much less straightforward to run than the others, and requires a lot of tuning and understanding of the domain space which I lack.

• Ranking for passing results: MADLAD-400, NLLB-3.3B.
• Ranking for failing results: Opus-MT, NLLB.

Test 2.3

[permalink]

• Input: 澄澈而晴朗的苍穹万里无云，暖得有些过分的阳光仿佛给远方的风景草木都蒙上一层微微颤动的空气薄纱。这是故乡难得的好天气。周围很是安静，除了夏风轻轻掠过树木的声音和虫鸟的鸣叫，什么也听不见。他的马低头靠了过来，在他脸上嗅了嗅，喷出一个响鼻。
• Opus-MT (7.79s): The clear and clear sky is cloudless, and some of the excessive sun warms up as if it were covered with a small tremor of air in distant landscapes. This is a fine weather for home. The surroundings are very quiet, and nothing can be heard except the sound of the summer breezes over the trees and the sound of the bugbirds. His horse leans down, sniffs in his face and snorts a loud nose.
• NLLB (11.98s): The clear and clear cloudless, warm and overly sunny grassland was covered with a thin layer of air that gave the distant landscape a slightly creeping tinge. It was a strange weather in his hometown. It was very quiet, and nothing could be heard except the sound of the summer breeze gently sweeping through the trees and the cries of insect birds. His horse’s lower head leaned over, sniffed in his face, and sprayed a whistling nose.
• NLLB-3.3B (30.70s): The clear and sunny sky was cloudless, the sun was warm and a little too much, as if the air were thin and slightly shimmering over the distant grass. This was the rare good weather in my hometown. It was very quiet, nothing was heard around except the sound of the summer wind gently sweeping through the trees and the chirping of insect birds. His horse leaned over, sniffed on his face, and spat out a rattling nose.
• MADLAD-400 (93.79s): The sky was clear and clear, and the sun was warm enough to cover the distant landscape with a slightly fluttering veil of air. It was a rarely good weather in his homeland. It was quiet, and nothing was heard except the sound of the summer wind blowing through the trees and the chirping of insects and birds. His horse leaned over, sniffed his face, and gave a sniff.
• Tower (737.99s): The sky was clear and bright, with not a cloud in the vast expanse of blue. The warm sunshine seemed to cast a light veil over the distant scenery and vegetation. This was a rare good day in the homeland. The surroundings were quiet, except for the sound of the

Again, Tower’s result is truncated, due to the configuration I used for its output. Its results would have been considered the best of all five had it not been truncated.

Ranking for passing results: MADLAD-400, NLLB-3.3B. Failed: NLLB, Opus-MT.

Test 2.4

[permalink]

• Input: 这大概是一条颠扑不破的真理，一段平淡的旧事总有戏剧化的开端。
• Opus-MT (6.45s): This is probably an intransigent truth, and there is always a dramatic beginning to a flat old story.
• NLLB (7.93s): This is probably an unflinching truth, a plain old thing that always begins dramatically.
• NLLB-3.3B (17.41s): It’s probably an unbreakable truth that a plain old thing always has a dramatic beginning.
• MADLAD-400 (26.26s): It is probably an incontrovertible truth that a dull old story always has a dramatic beginning.
• Deepseek.com: This is likely an unshakable truth: a mundane old tale always possesses a dramatic beginning.

At this point I removed Tower from the list. It’s clear that it consistently provides the best results, but requires an excessive amount of RAM (~25GB), and over 10 minutes to process these samples. Ideally, Tower needs to be run on GPU rather than CPU, and much more fine tuning than I was interested in doing for this experiment. It can technically run on consumer hardware, but not on most of my devices.

Ranking:

• Ranking for passing results: MADLAD-400
• Ranking for failing results: NLLB-3.3B, Opus-MT, NLLB.

Test 2.5

[permalink]

• Input: 旅游签证的发放对象是以观光旅游为目的的申请人。因此，除观光旅游外的短期访问目的都不属于此范围（除了部分签证）。
• Opus-MT (1.23s): Tourism visas are granted to applicants for tourist purposes.
• Opus-MT (6.92s): Tourism visas are granted to applicants for tourist purposes. Thus, short-term visits, with the exception of tourist tours, do not fall within this scope (with the exception of partial visas).
• NLLB (8.59s): The object of the tourist visa is the applicant for tourist purposes. Therefore, no short-term visit purposes other than tourist tourism fall within this scope (except for some visas).
• NLLB-3.3B (33.25s): Tourist visas are issued to applicants for the purpose of sightseeing tourism. Therefore, short-term visits for purposes other than sightseeing tourism are not covered (except for some visas).
• MADLAD-400 (32.56s): Tourist visas are issued to applicants for the purpose of tourism. Therefore, short-term visits for purposes other than tourism are not covered (except for some visas).

During the initial run Opus-MT only translated the first sentence. It turns out that this model only supports translating a single sentence at a time. I worked around this by updating my test script to split sentences, translate them individually, and then join them again.

All models passed this test, from best to worst: NLLB-3.3B, MADLAD-400, NLLB, Opus-MT.

Test 2.6

[permalink]

• Input: 对于已注册用户，可以点击页面右上角的“登录”按钮，并输入自己的账号和密码进行登录。登录成功后，用户即可享受网页版提供的各项功能，如浏览商品、下单购买、查看订单等。
• Opus-MT (7.08s): For registered users, you can click on the " Login " button at the top right corner of the page and enter your own account number and password to log in. After a successful login, users will be able to enjoy the various functions offered on the web page, such as browsing commodities, purchase orders, viewing orders, etc.
• NLLB (8.93s): For registered users, you can click the login button in the top right corner of the page and enter your account and password to log in. After successful log-in, you can enjoy the features of the web version, such as browsing goods, ordering, ordering, etc.
• NLLB-3.3B (22.84s): For registered users, you can click on the login button in the upper right corner of the page and enter your account and password to log in. After successful login, you can enjoy the features offered by the web page, such as browsing products, ordering purchases, viewing orders, etc.
• MADLAD-400 (47.52s): For registered users, you can click the “Login” button in the upper right corner of the page and enter your account number and password to log in. After successful login, users can enjoy the various functions offered by the web version, such as browsing products, placing purchase orders, viewing orders, etc.

All models passed this test, from best to worst: MADLAD-400, Opus-MT/NLLB-3.3B (draw), NLLB.

Test 2.7

[permalink]

• Input: 身后有余忘缩手，眼前无路想回头。
• Opus-MT (6.84s): Behind them, there is no way to look back.
• NLLB (7.65s): I have a shrinking hand behind me, and I can’t look back.
• NLLB-3.3B (16.79s): The people who are left behind are those who have forgotten and have no way of looking back.
• MADLAD-400 (35.74s): There is no way to turn back.
• Human translation: Forget to withdraw while there is still space behind, and only think of turning back when no path is seen in front.

All models failed this test.

Third batch of tests

[permalink]

The third and final batch of tests focuses on the same models as the second batch, but translating English to Chinese.

Test 3.1

[permalink]

• Input: The database was built from scratch. It did not have this many functions. And guess what? People are now asking for more! There is no end to their greed.
• Opus-MT (1.67s): 数据库是从头建立起来的。 它没有这么多职能。 你猜怎么着? 现在人们在要求更多! 他们的贪婪是没有尽头的。
• NLLB (8.32s): 据说数据库从头开始,它没有这么多功能. 猜猜什么?人们现在要求更多!他们的贪无止境.
• NLLB-3.3B (19.62s): 数据库是从头开始建立的. 它没有这么多功能. 猜猜吧? 人们现在要求更多! 他们的贪没有尽头.
• MADLAD-400 (35.68s): 数据库是从零开始构建的,它没有这么多的功能。 猜猜看? 人们现在要求更多! 他们的贪婪没有止境。

Ranking (all pass): Opus-MT, MADLAD, 3.3B, NLLB

Test 3.2

[permalink]

Input: This man is completely out of his sense. He speaks madness. Nonsense!

• Opus-MT (7.30s): 这个男人完全疯了。 他讲疯话。 胡说八道!。
• NLLB (8.00s): 这个人完全失去了头脑,他说话是疯狂的.
• NLLB-3.3B (16.22s): 这个人完全失去了理智.他说疯狂.胡说八道!
• MADLAD-400 (24.70s): 这个人完全失去理智了 他说的疯了 胡说八道

Ranking:

• Passing: Opus-MT
• Failed: MADLAD / NLLB-3.3B (draw), NLLB

Test 3.3

[permalink]

Input: It is hard to tell which one he is quietly and deeply gazing at, the bloody hue of the sinking sun at the horizon, or that pale azure moon hanging in the early summer sky. Translation: en → zh

• Opus-MT (6.72s): 很难分辨他静静地、深深地注视着谁, 地平线下沉的太阳的血腥光芒, 或夏日初的苍蓝的月亮。
• NLLB (8.61s): 很难说他静静地深地看着哪个,即向平线下沉的阳光的血色,还是在夏天早些时候的蓝色月亮.
• NLLB-3.3B (22.13s): 很难说他静静地深深地凝视着哪一个, 血的阴影的日落在地平线上, 或那个白的蓝色的月亮悬挂在夏天的天空.
• MADLAD-400 (35.45s): 很难分辨他是静静地、深深地凝视着哪一个,是地平线上落下的血色太阳,还是初夏天空中那颗苍白的蓝月亮。

Ranking:

• Passing: MADLAD, Opus-MT.
• Failing: NLLB-3.3B, NLLB.

Conclusions

[permalink]

Tower consistently seems to provide the best translations, but is also in another category of resource requirements and speed. I find it unsuitable for modern consumer hardware.

Of the remaining models, all failed at least two tests. There is also a clear asymmetry depending on the direction of the translation. When translating simple or technical text (e.g.: not metaphors or poetry), all the models provide useful results.

Opus-MT stands out positively when translating English to Chinese, and does a generally okay job translating Chinese to English. It is by far the lightest and fastest of the models, and quite suitable for average consumer devices.

MADLAD-400 stands out in the opposite direction, when translating Chinese to English, but can also yield imperfect results. This model is noticeably slower than Opus-MT, and uses a huge amount of RAM (between 30 and 40GB).

NLLB-3.3B does a pretty good job translating Chinese to English, but did rather poorly in all tests in the opposite direction. NLLB-3.3B uses around 13–16GB, while NLLB-600M uses around 2.5–3GB. This makes the former somewhat unsuitable for most devices (especially phones), while the latter can run on much more diverse hardware.

Scanning and task queue

[permalink]

An important bug quickly came to light: the local maildirs are never fully re-scanned. If any changes happened while ImapGoose was not running, those changes would not be detected the next time it ran— or ever!

My first approach at fixing this introduced substantial complexity into the task dispatching mechanism. There were no obvious faults with it, but it was too complex for any obvious faults to be visible. I redesigned the task queue keeping in mind all possible scenarios.

The resulting design is also quite simple: the IMAP listeners emits events indicating which mailbox has changed (this is all the information we get). The filesystem watcher emits events indicating which mailbox and file changed. In case of the initials start-up event, no file is specified, and this implies that the entire directory needs to be scanned for new files, deleted files, or changed flags.

The dispatcher accumulates events received and executes sync tasks based on this. Sync tasks are basically three:

• Scan a single message in filesystem (and sync change to IMAP).
• Scan entire filesystem (and sync changes to IMAP).
• Scan IMAP (and sync changes to local filesystem).

There is no longer a distinction between “Full Sync” and “Incremental Sync” when deciding what to do. When performing an IMAP scan, the workers use CONDSTORE/QRESYNC (which returns a list of messages that have changed since last time), or fetch all UIDs depending on whether the mailbox has been seen before or not. This is a decision on how to scan the remote, and not a decision on what to do, so it’s one layer down.

A noticeable change of the refactor is that scanning the remote mailbox for changes is no longer implied in all sync events: if only a local file (or files) changed, we can determine what to do based on that file (or those files) and the status repository. Since we’ve been monitoring the remote mailbox, we know its latest state already and can decide what to do without any network I/O.

All these changes brought around another simplification. Previously, during the start-up sequence, both the listener and watcher would trigger full sync events for all known mailboxes. This implied duplicate sync events for mailboxes on both sides. I had implemented some special treatment for these to be properly deduplicated. Now queued sync events are directional, so while both the listener and watcher queue events, they’re complementary events: one syncs local changes to the IMAP server and the other syncs remote changes to the local filesystem. We don’t need special de-duplication for them, and this simplifies the start-up sequence of the dispatcher.

Nested mailboxes

[permalink]

Some issues surfaced relating to nested mailboxes. My initial design mapped the hierarchy separation to a period in Maildir names, inspired by the same behaviour in offlineimap. The mailbox lists/mine was stored on disk as lists.mine, but the mailbox lists.mine would also be stored on disk with the same name— a collision.

Eventually, I settled on fully supporting nested mailboxes. notmuch supports these just fine, and there’s realistically no problem with this. It’s likely what users of nested mailboxes expect too. The only limitation is that the mailbox names cur, new and tmp are prohibited, since these names have special meanings in Maildir structures.

Because directory mapping changed, installations running a version prior to v0.2.0 need to migrate. The new -m command executes a migration, moving directories and updating the status database accordingly.

Single sqlite connection

[permalink]

Some users reported an occasional “database is locked” error from sqlite. This was caused by Go using multiple connections for sqlite by default. While one connection was writing, the other tried to write the same row and failed due to the row being locked. Fixing this was as simple as adding a call to db.SetMaxOpenConns(1). For unknown reasons, I was never able to personally reproduce this issue, despite having synchronised hundreds of thousands of messages.

Placing new messages in new/

[permalink]

Some MUAs make a distinction between the new/ and cur/ directories inside a maildir. ImapGoose now places new unseen messages in new/ and other messages in cur/. This should also help writing scripts to notify of new recent unseen messages.

Concurrent edits on flags

[permalink]

It’s possible for a message to have conflicting flag edits, and this is now handled appropriately. For example: a message is synchronised initially with no flags. The IMAP version is externally edited to add the Replied flag and the local message is edited to add the Seen flag. When synchronising after these two events, we need to detect which flags changed exactly. When ImapGoose reads the IMAP message with only the Replied flag, it will compare it with the status. The message had no flag last time, which implies that the Replied flag is new. This is then added to the local copy, but this operation needs to only append a flag without overwriting others (in this case, the new Seen flag which was added locally).

Additionally, when saving the result into the status, we don’t record the Seen flag, because that one is not in sync. This particular operation synced remote-to-local flags only, and the status now records that the last time items were in sync, they all had the Replied flag.

In this particular scenario, we’ll always have another pending task to sync local-to-remote, which will detect the Seen flag that was added locally, and replicate that to the remote server, eventually having both sides with the same view.

It took me several iterations to get the algorithm that was just right for this. My initial approach was to do a two-way sync of flags, but this introduced some added complexity and was somewhat non-obvious to follow. The final design is solid and simple to reason about.

Ignoring flags in filename matches

[permalink]

When looking for a file with an expected name, it’s possible that the file does not exist, but the message hasn’t been deleted— it’s been renamed. This is because every time that flags change in a Maildir message, the message has to be renamed (the flags are reflected in the filename). This caused some issues in cases where ImapGoose tries to read a file and its flags have changed (and the change hasn’t been processed yet). To work around this, when looking for a file, ImapGoose needs to find any other file in the same directory with the same prefix and different flags. This is a computationally expensive operation, since it requires reading all the directory entries.

The extra cost doesn’t seem to have a noticeable impact on performance. This operation cannot be skipped, because otherwise when a message’s flags change, ImapGoose would delete the remote message, and then re-upload it with the new flags.

User service files

[permalink]

v0.3.0 includes service files for running ImapGoose as a user service under OpenRC and systemd (thanks Clayton).

Initial index import

[permalink]

I added support for converging an existing setup. That is, an IMAP account and a set of local Maildirs which are already in-sync. An additional condition is required for this to work: messages in the Maildir must have a U=71931 portion in their filename, where 71931 is the UID of its counterpart on the IMAP server. OfflineIMAP encodes filenames this way, and I’ve heard that mbsync does too.

This initial import is implemented as imapgoose -i, which scans all local messages and compares them to the remote message with the same UID. If they are the same, ImapGoose stores into its status database that these two messages are the same and in-sync. No changes are made during this operation. Its output is as follows:

# Repeated for every mailbox:
Mailbox indexed mailbox=INBOX identical=433 mismatched=1 missing=6
# And finally:
Account indexed account=personal identical=83084 mismatched=4 missing=44

mismatched indicates that a file has a U= flag, but doesn’t match the other copy byte-to-byte or that its flags have diverged. The latter case is the usual one. missing indicates a file has a U=, but no message exists on the server with that UID.

This initial import feature allows quickly migrating to ImapGoose from other tools “importing” the existing configuration and syncing only newer changes in future. Such migrations are one-way. After ImapGoose has imported a mailbox like this, running it again without -i will start synchronising messages. Switching back to the previous tool at this point might be problematic, since its own internal status is ignorant of the changes done by ImapGoose.

imapgoose -i is safe to run at a later time, and is essentially idempotent. It should never be used while another instance is already running.

Simplicity through constraints

[permalink]

Much of ImapGoose’s simplicity comes from supporting only one specific use case.

I’ll use vdirsyncer as a counter-example: it can sync filesystem to caldav, or caldav to caldav, or webdav to filesystem, etc. Due to this permutation of choices, all these “backends” are implemented using a common API. This somewhat constrains the design in some ways. We can’t have one storage implement a feature that others do not. Or we need others to provide fallback, or to make the feature “optional”, and have call sites work around a feature’s potential non-availability.

On the other hand, in ImapGoose, the IMAP listener and Mailbox watcher return entirely different event types:

// Listener sends: "remote changed"
type RemoteEvent struct {
    Mailbox string // Just mailbox name
}

// Watcher sends: "local changed"
type LocalEvent struct {
	Mailbox string
	AbsPath string // Empty = full mailbox, non-empty = specific file
}

When scanning an IMAP remote, we can ask it for a list including only items which have changed, when we scan a filesystem, we need to read all file entries ourselves. This is why ImapGoose can’t sync one IMAP server to another: it’s designed around this asymmetry.

Service discovery

[permalink]

The server configuration parameter is now optional. If no server is specified, it is determined by using DNS-based service discovery. As a reminder, your local DNS server MUST be a validating DNSSEEC server in order to avoid MITM attacks.

Performance improvements

[permalink]

Large synchronisation operations sometimes took quite long. It turns out that I had lots of N+1 queries. I’m aware of the usual wisdom against these, but I wrote them thinking about sqlite’s article titled Many Small Queries Are Efficient In SQLite. Ultimately, one query with all results is still dramatically faster.

With that particular issue fixed, performance is good enough at this point that I’m not particularly inclined to tinker with it further. I am aware of places where there is room for theoretical improvements, but ImapGoose can max out my network uplink, so it’s pointless.

CRLF normalisation

[permalink]

Messages transmitted over the network are always transmitted with \r\n (carriage return, newline) at the end of each line. When we download messages, they’re saved with these line endings. When we upload messages, they are expected to have the same line ending. In theory, any tool that writes to a Maildir SHOULD respect this convention. In practice, they don’t. notmuch even removes a single \r from a single line when a message is moved across mailboxes.

In theory, changing any lines of a message might invalidate any signature which it may contain. In practice, if the message was missing CR, it couldn’t have been transmitted anywhere without being changed anyway. Since messages can’t be uploaded without fixing a missing CR, they’re now automatically fixed (and a warning is logged).

Messages which we download are guaranteed to end in CRLF (because of how the IMAP protocol encodes them), so messages which were downloaded and moved around are not being tampered in any way.

Keeping connections alive

[permalink]

Ideally, the connection with the NOTIFY listener remains permanently open. Keeping the connection open is relatively cheap. Through it, we can receive updates immediately, react instantly and fetch any updates that happened remotely.

If the connection is dropped, ImapGoose reconnects and sets up a new NOTIFY listener, but also needs to ask for the new status of all mailboxes. This adds some network overhead which is not ideal (but necessary) during each reconnection.

Many servers close idle connections after a while, so ImapGoose sends a NOOP (basically a ping) to the server every 15 minutes to keep the connection alive. Initial development showed that servers commonly close connections after 30 minutes, so this worked well to keep those connections alive. However, many servers use a much shorter timeout (as short as 5 minutes). When connected to these servers, ImapGoose would need to reconnect every 5 minutes, adding unnecessary overhead.

Making the interval configurable isn’t practical since people don’t have a convenient way to determine the right value. I changed the interval to 3 minutes, which works for servers with short timeouts while adding minimal burden for those with longer timeouts.

I’m still considering implementing auto-detection of how long a server takes to disconnect an idle connection. It’s possible to distinguish between a clean disconnect and a network glitch. ImapGoose could record the interval after which the server closes the connection, store this timeout, and only send pings every half a period.

Development reflections

[permalink]

Purity of git history

[permalink]

I’ve been iterating on how I code with these new projects.

I did a lot of short micro-commits, then reviewing, keeping notes of bugs, and iterating with the model. When I’m done implementing a feature, I have a large amount of commits, many of which have changes known to be faulty in some way, and some introducing hundreds of lines of code which are removed in the following commit. All these commits are steps in the right direction, but are a huge amount of noise. They all get squashed into one commit.

This results in larger commits, some a bit harder to follow since they mix a new feature with a refactor of another function which I noticed was necessary along the way. These types of commits are harder to review, and not ideal when sending somebody else patches. But when working on a new project where nobody else is looking at changes, these details don’t really matter.

By removing this burden of “keep small tidy commits” I’ve also moved faster, since I can focus on fixing the task without focusing on keeping clean commits. When sending patches to someone else, I sometimes need to split a large change into multiple commits, sometimes in a shuffled order. This kind of rebasing takes time, and is only meaningful if someone else will be reviewing commits, in order, and understanding the full context.

Multiple flags in Go

[permalink]

Go likes to be special and different in some unique aspects. One of these is command line arguments and flags. Go doesn’t support the classic single-letter flags (like ls -lt). It only supports single-dash full-word flags like -list -time.

ImapGoose has -m and -v, but using -mv didn’t work: Go interprets that as a single mv flag rather than two separate flags.

I could have written a paragraph in the documentation warning about this, but it seems cleaner to just ignore Go’s flag parsing abilities and parse these flags on my own. It’s honestly less than 50 lines of code anyway.

Current status

[permalink]

The latest releases contain mostly minor fixes and optimisations. Work on ImapGoose has slowed down, with it being stable at this point. I’ll gather feedback on the current version for some days, and then tag a v1.0.0.

ImapGoose is highly optimised to reduce the amount of network traffic and tasks performed. To do so, it relies on a few modern IMAP extensions and only supports modern email servers. “Modern servers” in the context of email means servers which support extensions which were standardised between 2005 and 2009.

ImapGoose uses the CONDSTORE extension (standardised in 2006), which basically allows it to tell the server “I last saw this mailbox when it was in state XYZ, please tell me what’s new”. This avoids the need to download an entire message list (which can be tens of thousands of emails), making incremental syncs much more efficient. It also uses the QRESYNC extension (standardised in 2008) so that the server includes a list of deleted messages too (i.e. VANISHED). Finally, ImapGoose uses the NOTIFY extension (standardised in 2009), which allows an IMAP client to tell the server “please let me know when there are changes to these mailboxes”, and then leave a connection open. NOTIFY has two nice consequences: (1) the client doesn’t need to ask the server if there have been any changes at regular intervals, and (2) the client is informed of any changes immediately, so they can be processed without delay. Unlike the older IDLE extension (from 1996), NOTIFY (from 2009) allows monitoring multiple mailboxes per connection, rather than just one.

In this article, I’ll cover some of the general design details, inner workings and other development details.

General mode of operation

[permalink]

First off, ImapGoose keeps a small status database with some minor metadata about the last-seen status of both the server and local Maildirs. This includes the mapping between server UIDs and filesystem filenames. Its general design is strongly inspired by how OfflineIMAP works.

At start-up, ImapGoose lists all mailboxes in the server and in the local filesystem. It then starts monitoring them (the server via NOTIFY, the client via inotify/kqueue), so we receive notifications of any changes that may happen after our initial listing. This ensures that, for example, if we receive a new email while performing the initial sync, we get a notification for it.

Once monitoring is set up, ImapGoose queues a task to perform a full sync of each mailbox. Initially, we determine if this is the first time we see this mailbox by its absence in the status database. If this mailbox has not been seen before, then we request all messages. The server returns all of these along with a HIGHESTMODSEQ, which we store in the status database. This HIGHESTMODSEQ is a numeric property of each mailbox and increases every time a change occurs inside that mailbox. If a mailbox has been seen before, then we can ask the server for changes since that HIGHESTMODSEQ, which delivers only the minimal amount of data which we need, and nothing else about all the other thousands of unchanged messages.

When a message is present in the server and absent in the filesystem (or vice versa), we need to determine whether it is a new message, or if it is a message that was previously present in both and deleted from the local filesystem. To determine this, we use the status database and apply the exact same algorithm as offlineimap. It’s simple and well tested.

At times, ImapGoose may disconnect from the server (for example, due to a laptop disconnecting from Wi-Fi, or going into sleep mode). It will try to re-connect automatically using an exponential back-off: after 1 second, then after 2 seconds, 4 seconds, 8 seconds, 16 seconds, 32 seconds,… all the way up to 17 minutes. Then it will continue retrying every 17 minutes. This means users don’t really have to worry about ImapGoose’s current state, whether it’s still working, etc. It knows how to back-off when there’s no network and how to get back to work when it is feasible again.

As mentioned above, ImapGoose “queues” sync tasks. Internally, it uses a task queue; when changes are detected on the server, a task to sync that entire mailbox is queued. A worker picks this up from the queue, asks for changes in that mailbox, and synchronises them. When changes are detected in the filesystem, a task to sync that particular message is queued. It may happen that multiple messages arrive in quick succession for the same mailbox. In this case, we don’t want to trigger multiple syncs of the same mailbox, and we especially don’t want two workers to sync the same mailbox concurrently: this would quickly lead to duplicate emails.

To work around concurrent syncs and redundant mailbox updates, ImapGoose uses a “dispatcher”, which hands off sync tasks to workers. When a task to sync a specific mailbox is handed to a worker, that mailbox is marked as “busy”, and we don’t process other tasks for that queue until that worker notifies that it has finished its work on that mailbox. While a worker is synchronising a mailbox, we may receive several notifications that changes have happened to that mailbox. These changes could be the result of the changes made by the worker, or they could be new emails being delivered, so we have to queue another task to sync that mailbox. These tasks are kept in queue until the worker frees up the mailbox, and the dispatcher additionally de-duplicates them: synchronising a mailbox just once after the last change notification is enough to synchronise the changes in all the notifications.

When a message changes in the filesystem, ImapGoose receives an inotify event. This doesn’t trigger a sync of the full mailbox, but instead a “targeted” sync, which focuses only on that email message. We know that a single message has changed, so there’s no point in re-scanning the thousands of messages in the mailbox. These targeted syncs are taken into account in deduplication; they only get de-duplicated if the path for them is the same.

While the connection which is listening for changes from the server is kept alive by sending periodic NOOP commands, the connections for workers are allowed to time out. If no activity is happening, these connections simply time out, but a connection is re-established once a worker needs it again. Great care has been taken to avoid unnecessary churn in all possible aspects.

Prior art

[permalink]

Before developing ImapGoose, I studied prior art in the field. In particular, offlineimap does a great job at synchronising mailboxes. However, it doesn’t “keep in sync” in the same way; offlineimap needs to execute periodic syncs, doesn’t rely on modern extensions, and tends to “hang” when there are network time-outs. ImapGoose is new and has no existing users, so it can just require modern extensions or declare other scenarios as unsupported. Existing tools have to maintain compatibility for existing users, which might rely on some legacy email server. If I couldn’t rely on NOTIFY, implementing ImapGoose in such a clean efficient way would not have been possible. If I couldn’t rely on CONDSTORE and QRESYNC, I would have had to download lists of thousands of emails each time even a single one changes. Thanks to UIDPLUS, the server returns the UID of a newly uploaded message, and we don’t need any ugly workarounds to retrieve it.

If someone needs to sync data from legacy servers, plenty of tools are still out there, providing the best experience which those servers can offer.

Development

[permalink]

When working on ImapGoose, I focused exactly on my needs for my particular use case: keep my local mailboxes in sync with an IMAP server. There’s no other supported scenario, there’s no fallback for legacy servers, and there’s no support for alternative email backends. All these constraints allowed me to focus on making a tool that’s great for a single use case: it does one thing and does it well.

I strongly believe that my keeping tight constraints (e.g.: focusing on just one use case, ignoring support for legacy servers, keeping things as simple as possible) helped develop this much faster and with much cleaner results.

I started with a very clear picture of how the whole thing would work. I was also familiar with go-imap, and knew it to be a well designed and well implemented IMAP library. My immense appreciation goes to emersion and the contributors who’ve worked on it. I didn’t need to worry about the inner details of talking to an IMAP server, parsing responses, tracking connection state, etc. go-imap provides a simple idiomatic Go interface for IMAP commands and their responses.

go-imap was lacking two features which I needed: support for the NOTIFY command and for VANISHED (rfc5162). While still standing on the shoulders of giants, I implemented both of these and sent patches for both of them (NOTIFY, VANISHED). Until those are merged, ImapGoose is built using my own (temporary) fork which has those two patches applied.

Configuration

[permalink]

For configuration, I opted for the very simple and straightforward scfg configuration format. The configuration file looks something like:

account example {
    server imap.example.com:993
    username hugo@example.com
    password-cmd pass show email/example
    local-path ~/mail/example
}

Naming

[permalink]

I wanted something easy to remember, easy to pronounce and that won’t yield thousands of unrelated search engine results. There’s also room for an obvious mascot/logo: a goose wearing a postman’s hat carrying an envelope, using the colour palette from the Go ecosystem. Please reach out if you are an illustrator willing to contribute with artwork.

Open source

[permalink]

ImapGoose is open source and distributed under the terms of the ISC licence. The source code is available via git. Feedback is welcome, including bug reports.

I’ve finalised support for JMAP in pimsync, and tagged a release with it. It’s still experimental. While preliminary testing shows it works fine, it still hasn’t had extensive long-term testing. Feedback for it is most welcome, but please keep frequent back-ups of your data while testing it. This is still in an early state.

On my previous status update I mentioned that I’d be switching to the calcard library for converting iCalendar to and from JSCalendar. calcard uses serde for serialisation and deserialisation, and that resulted in a mismatch with my own libjmap, which (at the time) used the json library. I therefore ported libjmap to use serde instead. The resulting code is much clearer. After a few iterations, it is also much more reliable.

With libjmap and calcard using the same serialisation representations, they fit together much nicer. I then moved to port the JmapStorage implementation to use this.

Etag and State

[permalink]

The most complex part of the storage implementation was handling the mismatch between Etag and State. In CalDAV, each item has an Etag property, a version identifier which changes each time an item changes: if an item has changed, its Etag has changed. Pimsync stores an Etag value on a per-item basis, and tracks it each time an item is updated. This happens internally in the synchronisation logic, quite detached from individual storage implementations. Whenever we update an item, we tell the server to only update it if its Etag matches the one last seen. If it doesn’t match, then the item has been modified by some other client, and a conflict needs to be resolved.

For the filesystem storage, we use a combination of inode number + mtime as an Etag, and this matches the same conditions just fine.

JMAP doesn’t have an equivalent to Etag. It has a State value, but that changes whenever any item changes. We still use State as an Etag internally, since we don’t have anything else on which we can rely, but this brings about several complications.

As soon as we update a single item, we’ve invalidated the Etag/State for all other items in that storage. My early prototypes simply omitted the usage of these, but this could easily result in conflicts being ignored and data being overwritten. I.e.: losing changes made by some other client. This is unacceptable for anything beyond an early prototype.

In order to use the State value while keeping within the constraints of both pimsync and JMAP, I had to change the process of updating an item to the following sequence:

• Try to update the item passing the last known State.
• If the process completes, then all is good.
• If the process fails due to a State mismatch, then:
  • Fetch the current State.
  • Request all changes since our item’s previously recorded State and the current State.
  • Check whether this particular item has changed between those two States.
  • If the item has not been modified, retry updating it, passing this new State as conditional.
  • If the item has been modified, then we have a real conflict — someone else has modified it since we last saw it.

This works, but has a really ugly side: uploading a single item can end up causing four network round trips. We can do better.

Tracking state locally

[permalink]

In order to avoid continuously querying the server for state changes, I moved to caching states and their changes inside the JmapStorage implementation.

Essentially, it keeps a mapping of states which show up, and items which were modified between that state and the previous one. Each time that an item is uploaded (or deleted), we pass an “expected state” to the server and it returns a new state. We record in the cache that this new state transition only modifies the items which we’ve modified.

When we later update (or delete) some other item, we’ll have a State for the last time we’ve seen that item from the server. We can check our local cache, and if there’s a full path of transitions between that state and the current one, we can determine whether any of the intermediate states modified this item. If none of them did, then we can send the request telling the server to update the item assuming that it has not changed since the most recently seen state.

This implementation has mixed results. When continuously running (e.g.: pimsync daemon) there are plenty of cache hits and it saves a lot of network queries. When running ad-hoc (e.g.: pimsync sync) the cache usually doesn’t have enough updates to make any meaningful difference.

All of this is, essentially, a band aid. Execution of pimsync’s synchronisation algorithm currently expects to update items one at a time, but JMAP allows us to send a single request updating any amount of items. Using this API, we could dramatically reduce the amount of queries required. This requires large changes to the executor, which could also benefit the singlefile storage (which saves N events into a single iCalendar file). This is somewhat of an invasive change, which I’d like to address at some point in future, but there are higher priorities at the moment.

Concurrency

[permalink]

Due to how state changes occur, JMAP is incapable of dealing with concurrent writes. If two clients send a write operation, they’ll instruct the server to only apply the changes if no other change has occurred. But because the state tracking is for all items, if any other item changes, then the operation fails. Because of this, the JmapStorage implementation only sends a single write query at a time. Any more than that would, by definition, always fail. Again, this will be ironed out once we simply write all changes in a single query, but it’s also an issue for concurrent clients trying to operate on entirely different items.

Final bits

[permalink]

The final bits of the implementation were integrating the JmapStorage into pimsync. Mostly, allowing configuration blocks for storages with type jmap. At this time, these storages need to specify jmap/icalendar or jmap/vcard. This is somewhat of an annoyance, since the same storage can actually handle both. In future, I’ll move the declaration of item type into pair blocks, so a single storage declaration can be re-used for both.

Collection IDs and URL segments

[permalink]

Unrelated from JMAP: it is now possible to configure which URL segment is used for collection identifiers. A few months ago, I explained why this is needed on some unusual configurations. This includes Google’s CalDAV implementation.

The new collection_id_segment configuration directive allows using the second-to-last segment as a collection id. See the manual page for reference documentation.

While native OAuth support is still on the road map, this change allows using a proxy such as google-dav-proxy¹ with pimsync. It requires a bit more setup, but should be feasible to use today. If you do try it out, please let me know the results.

As part of my work to implement JMAP support for pimsync, I need to convert JSCalendar entries to and from iCalendar (and likewise, JSContact entries to and from vCard).

Fortunately, there were draft RFCs documenting how to do all these conversions¹. Essentially, all fields in iCalendar format have an equivalent field in JSCalendar format and there’s special syntax to keep around unsupported fields in a lossless way so that they can be recovered when converting back.

While reading the RFC I paid special attention to recurrence rules, which I feared might be a great issue. Fortunately, recurrence rules are represented in more or less the same format, so I don’t need to parse them. The same applies for other recurrence-related fields.

iCalendar events can have either (1) a start datetime² and an end datetime, or (2) a start datetime and a duration. JSCalendar only supports start datetime and duration, so anything in the first format needs to be converted. I didn’t think much of this in the two times that I read the entire RFC. This was a grave

mistake.

My general design

[permalink]

The general approach to my design was as follows:

When reading an iCalendar/vCard file, read each event/todo/contact and once we reach the end of it, return a JSON object with its equivalent on the other type.

When reading a JSCalendar/JSContact file, there’s a function to convert each JSON object into its counterpart, and these functions call each other, forming a sort of tree in the call stack.

vCard <> JSContact

[permalink]

My simple design worked wonders for vCard <> JSContact, and I managed to implement a rough prototype for pimsync³.

I also had unit tests converting JSContact into vCard and back, and a few JSCalendar items into iCalendar and back. Everythiing worked wonders (for the fields that were implemented, a few were just left as “pending”).

At this point, my design seemed solid, so I started building more around it, and slowly working on the iCalendar<>JSCalendar aspect of it. Again, I noticed that converting ’end datetime’ into durations requires a bit more thinking and left it for later, without giving it a second thought.

Stable UIDs

[permalink]

Potentially, vCard files may lack a UID, and conversion needs to add one. When converting the same input multiple times, the process SHOULD add the same UID. This is something which didn’t entirely fit well with my design, but pimsync should take care of this beforehand anyway: all its synchronisation process requires that all entries have a UID anyway.

Timezones are nightmares

[permalink]

This isn’t the first time that timezones sprung up to haunt me in what seemed like a perfectly good design for something that was going to work fine.

It turns out that if we have the start datetime for an event and an end datetime, calculating the duration isn’t that trivial. To do so, we need to take the timezone into account. Does an event start on the day before a daylight transition and end the following day at the same time? Then its duration is 23 or 25 hours, but not 24 hours!

Given how my entire converter converted and serialised each component in isolation, I didn’t have the timezone in scope when converting an event. In fact, the timezone might be present further down in the same file, so it hasn’t even been parsed yet.

Now, it’s far from impossible to re-implement things in a different way to take this into account, but it did imply that I had to scrap most of my code and start over, with a design that keeps some form of “partially converted” representation in memory until all necessary information is found further down in the same file.

Motivation

[permalink]

I admit that being hit by this situation strongly reduced my motivation. Work on JMAP support along with JSCalendar and JSContact has been taking far more than anticipated, and at this stage I mostly want to get it over with and move on to other more pressing features.

As I was dealing with having to re-start the same work with an entirely new approach, the folks from Stalwart Labs posted an email to the JMAP mailing list announcing calcard: a project to convert iCalendar/JSCalendar and vCard/JSContact.

I peeked at the code and was impressed. It’s has mostly minimal dependencies, and takes into account all possible fields. They first convert items into an intermediate representation, and then into the other type. I wasn’t a fan of this approach at first, but then realised it’s the only thing that really works out if you need to work around the timezone issue that blocked me.

calcard bundles rust-rrule, a library to parse recurrence rules which I have used and contributed to in the past. I’m happy that they’ve reused an existing implementation that works, and I also have some fixes for rrule which can hopefully trickle down too.

Moving on

[permalink]

At this point, it makes more sense to adopt calcard for pimsync’s JMAP backend. I’ll contribute fixes or improvements upstream as needed, which should be a better use of time and result in a more robust solution.

This is a strange status update: usually I share something that works, or at least an experiment that demonstrates that an idea does not. In this case, I failed to implement a converter due to a flawed design. There is no greater lesson to be learnt from it either.

The lady at the desk was friendly and polite. She asked me for the amount, gave me a quote (which, honestly, was pretty bad, but it’s not like there’s any better choice in the Netherlands). She then inquired about the funds, requested my ID, address and a few other bits of personal information. She then started a phone call, asked me to wait for a bit. I waited for twenty-something minutes while she herself was on hold. Eventually she told me that she could not proceed with the operation. She explained that the phone call is part of the regular procedure, where she calls their offices in the UK, sends all my personal information, and the folks from the UK then approve or reject the operation. In this case, they’d told her that she was not authorised to offer me any exchange or any transaction. They refused me service.

The next people in line were there for the exact same transaction as I was, and completed their operation swiftly. I approached the desk again and asked the same lady more details about my situation. She was not given a reason, even after asking further, and was simply the messenger here: this faceless corporation had decided I was on a “do not serve” list and I was not welcome as a client there. As I inquired further, she indicated that I should not come back, since they would refuse to provide me any sort of service, for any amount, in future.

Considering that this is the only currency exchange office in the city (and in this province), I’m appalled and unsettled that they would simply refuse to serve me, without explanation of recourse. Why? The employee explain that she had no way of knowing, and she did not believe that I could determine why either.

We can only guess what might be the reason. Are some of my friends of the wrong ethnicity? Maybe those close to me are voting for the wrong parties? Your guess is as good as mine. There’s no accountability here, and financial institutions can simply decide that any of us are not welcome and that’s the end of it.

This isn’t my first encounter with such arbitrary systems…

Dangerous ethnicities

[permalink]

Back in 2019 I met this girl at a friend’s birthday party. She worked at a bank in the software development sector, and told me about the project on which she’d been working. Their system collected data from various sources on people applying for financial services (e.g.: loans) and would indicate if someone was eligible, or raise a red flag. In the latter case they would have to deal with substantial additional bureaucracy, and often times would not be able to access these services anyway.

She seemed quite proud of her work, and told me her team had demoed it the previous week in front of the whole office. They’d shown the report that the system generated for each person in the team. In her case, the system flagged her as “dangerous”, and she was not eligible for a loan. Her grandmother was from Iran, and because there’s frequently cases of money laundering or other irregularities in Iran, she’s immediately flagged too. Despite her being a Dutch citizen, born and raised in the Netherlands, and working in this Dutch financial institution, her bloodline was “too risky” for her employer to lend her money.¹

I was confused, and honestly, heartbroken. This person was telling me how proud they were of their work, building a system that discriminates them based on their bloodline. I questioned what she thought about it, and she explained “well, the rules are there…” —she paused a few times— “the rules are there…”. I remained silent and she eventually finished her sentence after repeating it a few times “the rules are there to protect us”. Even though this was six years ago, every time I remember this situation it brings me great sadness. Here was a person who’d worked hard to build a system which would discriminate against them, and yet stood proudly defending their work.

I don’t think this situation is a complete outlier either. In this corporatocracy that is the modern world, a large proportion of the population works hard to sustain and enforce the same system that oppresses them. Generally, because there’s no other choice. Either work doing that, or starve. Frequently, it’s not even obvious that this is the case, given the layers of indirection.

It’s hard not to think of The Matrix (1999). Anyone in this society is unknowingly an agent which enforces its rules, despite it being the system that keeps them trapped. When an outlier tries to change the system, the men in suits will come after them — men who don’t have to follow the rules themselves

I have few requirements for a watch. It should: (1) keep the time, (2) show the time in a clear, readable fashion, and (3) track steps taken each day, (4) have a battery duration of no less than a week. Heart rate monitor is a nice plus, and the Pinetime includes one.

Aside from meeting these requirements, the Pinetime has the upsides of being (really) cheap, running open source software, and using standard 20mm straps. These straps are the same as plenty of other regular watches, there’s immense variety and they’re available in countless shops.

Keeping track of time

[permalink]

This sounds like an obvious requirement for a watch. But my last two Fitbit smartwatches didn’t meet this requirement. After having my Fitbit watches for over a year, the time would start being off. It sounds like a silly issue and one that would be easy to fix. That was far from being the truth. The watches didn’t allow manually adjusting the time — instead they sync the time automatically with the iOS app. Syncing with the app worked fine, and it would reflect the amount of steps tracked and other measurements. But the time remained wrong. It didn’t sync. Their support staff always said “maybe the timezone is wrong on your profile, check that”. The timezone was fine — if it had been off, the time would have been off by an exact amount of hours, not by twenty-something or thirty-something minutes.

When this happened to the first Fitbit, I thought it was a fluke. When this happened to the second one, I started to see a pattern. Some time later, a friend had the same issue and stopped wearing hers too.

So while “it keeps the time” is an obvious requirement for a watch, be aware that not all smartwatches make it this far.

Showing the time clearly

[permalink]

The Pinetime has only a few watch faces. One of them shows the time in clear large numbers, with good contrast on a backlit display and easily readable in the dark. It also shows the date in smaller font underneath. There’s no need to convert angles into hours and minutes, no obscuring the time with implicit ante meridiem and post meridiem, just obvious numbers. Sure it’s easy to convert post meridiem times to 24hs time, but I hate waking up, seeing 07:00 on a clock, and having to wonder if it’s seven in the morning or seven in the evening. I see no reason for which my watch should show anything other than the exact time, clear and straight. Infinitime can do exactly this.

The Pinetime has a few other watch faces. They look cool, but are not of my interest. It’s also possible to develop your own, although it seems that you need to actually know C++ to write a watch face. That’s definitely a downside, but not one that affects me.

Tracking steps

[permalink]

InfiniTime tracks how many steps I take each day quite accurately. It’s interesting trivia which I care about, since it gives me a vague reference of whether I’ve been too sedentary or not.

Sadly, it only tracks steps taken today. There’s a patch to also record steps taken the previous day. I mostly only care about this when I’m up past midnight. I don’t have any interest in keeping historical records of steps. I understand others do want this. If you do, you might need a companion app on a phone or laptop which syncs the data off the device. I don’t use any such thing, so can’t comment on them.

Sometimes if I take a single step, the step doesn’t register immediately. Instead, when I take three more, the total of four steps register at once. I suspect that the device needs several steps worth of readings to identify false positives, or might do processing in tiny batches. To be frank, I haven’t cared enough to dig into it, but a fascinating aspect of this device is that development was and is done entirely in the open. Not only is the source code available, but discussion of development is also public, so if you’re really curious you can peek at why things work this way and how the algorithms work.

Battery duration

[permalink]

Battery lasts about 30 days.

There’s an “always on” mode for the display which reduces that to about three. The display is not OLED, nor optimised for this. I don’t use this “always on” mode, but do wish that I could configure the device to automatically enable this mode while it’s charging.

Leaving Bluetooth on also dramatically reduces battery. I think to around a week? I only enable Bluetooth when I want to upgrade the software, or sync the time. The time only needs to be synced after I let the battery get down to zero percent.

The heart rate monitor also reduces battery duration substantially. I seldom use it, so I’m not sure how long the battery lasts when using it continuously.

It’s cheap

[permalink]

USD 26.99

Accidental taps

[permalink]

An annoying issue is accidental taps. Any skin touching the watch screen triggers a tap (or swipe). Some activities (including sleep) can trigger the occasional accidental tap. A few times I’ve woken up to find that some random menu had been opened and some toggle changed, or the watch face was changed. This only happens a couple of times per month, and typically it’s just a menu having been opened without any changes made. But it’s still annoying.

There’s a patch which only unlocks the screen when the button is pressed and not when tapping the screen. In theory an algorithm could detect all false taps, and that would be the ideal solution. The patch works in the meantime. I’m not sure we’ll ever have completely perfect detection for false taps.

Community maintained software

[permalink]

InfiniTime is open source and community maintained software. There’s a lot of people working on improving it, and I really like the spirit behind it.

Both the code itself and discussions on feature implementations are public, and some issues have really interesting discussion on how the algorithms were implemented. This includes non-obvious algorithms like counting steps, optimising the heart rate monitor, and reducing the detection rate of false taps (this last one still not being quite there yet).

If you’re interested in data analysis in resource constrained environments, you’re going to love this project. Personally, that’s a bit out of my area of expertise, so I can only provide feedback on whether things are an improvement or not.

The open source nature of the project also helps me test patches made by others. Some of these patches, like the ones pointed out above, fix issues which affect me directly. My Pinetime actually runs a build of InfiniTime which includes both patches. Building my own image is actually really easy using a docker container which ships all the required build tools.

Unique charger

[permalink]

The charging cradle is its own unique thing. You can’t re-use the charging cable for anything else but a Pinetime. I kinda get why they did it — a USB-C port would accumulate dirt or liquid easily and be a pain to maintain. Qi charging would likely have made the design much more complex, but would still be a great improvement, and I’d love to see this in a second iteration of this device. The design is open source though, so a capable enough hacker could design a variation with Qi charging.

The unique charging cradle isn’t an issue because of the great battery duration. I only really need to bring the charger with me if I’m travelling somewhere more than four weeks, which isn’t often at all. For anything shorter, I can just charge the watch before I leave.

Build quality

[permalink]

The build quality is great. The watch is no larger than a standard watch and the casing is solid metal. It’s sturdy, durable and water-resistant (rated at IP67).

So I try different models and tools, and it’s all incredibly underwhelming. It’s honestly hard to believe that people get work done using these tools, because I can spend a few hours on them (without getting even close to finishing the task at hand) and realise that I could have done it myself in 25 minutes.

I tell myself “learning to use Vim took a long time, but then it paid off” eventually. But I could (slowly) write text with Vim the first day. I can spend an entire day with a GPT and produce nothing of value.

GPTs work great for finding the exact word to complete a sentence. They’re surprisingly good at finding the exact type annotation for a Python function. They can find nuanced bugs in a single function which I copy-paste into the GPT. But anything beyond writing a simple function always leads to useless junk. Often times, they solve big problems by just importing a library that does not exist, and calling a function which does the bulk of the logic. ChatGPT told me the other day “if you don’t want any dependencies, you’re going to have to implement it yourself”. But couldn’t actually implement the necessary code. Large portions of code have lots of hidden logic bugs, and when they fix one they introduce another.

And then I see another post on Hacker News, about somebody using GPTs and how they achieved great results on this and that. Part of me wants to think that those articles are fake to generate hype, but the reality is that several of them are written by well-known developers who’ve been around for over a decade. Some of the results are actually publicly available online.

I’m in a state where I can’t reconcile my own results with other people’s results. I hear people saying “this hammer is indestructible”, but when I pick it up, it’s just origami: made of paper, intricate, delicate, very cool-looking but I can’t even hammer a tomato with it.

Early this year I signed up to finish my university degree, a licentiate in computer science. I finished all the coursework years ago, and only my dissertation was missing. That absorbed a lot of time during May and June, but it’s now finally complete.

My research (PDF in Spanish) went into mounting fuse filesystems without any form of privilege escalation (some platforms use setuid, others require root). At a high level, a service runs with elevated privileges, and allows authorised users to mount filesystems via the kernel’s fuse interface. The bulk of the work was implementing the proof of concept.

Clients wishing to mount a new filesystem connect to the service via a socket, then specify a mount path, and finally exchange the file descriptor used for communicating fuse operations with the kernel.

The proof of concept allows not only mounting filesystems without setuid binaries, but it also allows handing over the file descriptor to a process inside a sandbox (even inside a docker container), so that a fuse service can run in highly confined environments—a notable departure from requiring privileges to run setuid binaries.

The source is an ugly cannibalisation of the libfuse project (and therefore under the LGPLv2/GPLv2 licences), and I hope to be able to release it at some point, even if only as a prototype. In theory, the general design should be usable in BSDs as well.

pimsync + davmail

[permalink]

Pimsync has now been confirmed to work with DavMail. DavMail exposes a pretty standard CalDAV server, but a couple of early adopters found a bug in pimsync which prevented it from completing the discovery process.

Fixing the bug was straightforward, and pimsync v0.4.4 has now been reported to work fine with DavMail.

JMAP and libjmap

[permalink]

My other point of focus these past weeks has been JMAP support for pimsync.

Status quo

[permalink]

Existing libraries for interacting with JMAP focus on email, and have no support for Calendars or Contacts (although both of these JMAP extensions are experimental).

The main contender was jmap-client. Adding support for Calendars and Contacts is possible, but the general design doesn’t seem to be a good fit for pimsync. The main blocker is that this library brings its own I/O (see: sans I/O) and (high level) HTTP client. This makes it a poor fit for pimsync, which already has its own I/O and HTTP client implementations. Pimsync also needs a lower-level HTTP client due to the configuration options exposed.

Cyrus as a JMAP server

[permalink]

In order to work on a JMAP client, I needed a JMAP server. The only implementation which currently supports calendars and contacts is Cyrus IMAP.

I tried to build Cyrus from source in an isolated environment, but came across some build issues on Alpine. I reported these upstream. The developers provided fixes for each issue along the way. Cyrus IMAP now builds and works fine on Alpine, and likely on other musl-based distributions too.

While iterating on the above, I used cyrus-docker-test-server to continue on my own work. Running a server is as simple as:

docker build -t cyrus-testserver .
docker run -it \
  -p 8080:8080 -p 8143:8143 -p 8110:8110 -p 8024:8024 -p 8001:8001 \
  cyrus-testserver

Hacking on a JMAP client

[permalink]

My first prototype code worked, but was quite ugly, since I learnt quite a few quirks about JMAP while writing it. It doesn’t matter how many times you read the RFCs, you’ll still run into surprises when writing the client. The current code is a second prototype.

Some of the design choices in jmap-client make a bit of sense to me now, but I’m still not entirely convinced by others. I’ve reached out asking if they’d consider moving to a sans I/O approach. If so, I believe it’s best to collaborate with them and attempt to mould their client library to fit my requirements (even though I’d have to use a fork in the short term in order to properly support Calendars and Contacts until they are fully standardised).

After having written the current JMAP implementation, I believe that requests would be best designed with a builder pattern. It would make writing JMAP-based applications a lot more pleasant for downstream consumers. Such an implementation would be quite time consuming, and it’s hard to justify it for pimsync’s more basic needs.

libjmap

[permalink]

I’ve extracted my JMAP client implementation into a new library, libjmap. It exposes the few basic functions required by pimsync:

impl<C> JmapClient<C>
where
    C: Service<http::Request<String>, Response = Response<Incoming>> + Sync + Send + 'static,
    <C as Service<http::Request<String>>>::Error: std::error::Error + Send + Sync,
{
    pub async fn discover_session_url(
        mut client: C,
        scheme: Scheme,
        domain: String,
    ) -> Result<Uri, DiscoverError>;
    pub async fn create_collection<T: Collection>(&self, name: &str) -> Result<Record>;
    pub async fn create_record<T: Collection>(
        &self,
        calendar_id: &str,
        record: &JsonValue,
    ) -> Result<Record>;
    pub async fn delete_collection<T: Collection>(&self, collection_id: &str) -> Result<String>;
    pub async fn delete_record<T: Collection>(&self, record_id: &str) -> Result<String>;
    pub async fn get_collections<T>(&self) -> Result<Vec<T>>;
    pub async fn get_records<T: Collection>(&self, calendar_id: &str) -> Result<Vec<Record>>;
    // …
}

The get_collections function also fetches properties (display name, colour, etc), but a function to update properties is still missing.

I’m still missing the support for the state parameter. Each time that the server returns any data, it returns a state, an opaque value which changes any time that data on the server changes. This allows sending requests specifying operations which are only executed if the data has not changed on the server since we last saw it. This mainly avoids race conditions. It’s closely related to HTTP’s Etag header, with the biggest difference being that the state changes any time any item changes, while an Etag applies only to individual items.

As such, the current (incomplete) JMAP backend in pimsync is subject to race conditions if multiple clients operate on the same server at the same time.

pimsync JMAP support

[permalink]

JMAP support in pimsync is still incomplete, and unreleased.

I need to convert data in JSCalendar format into iCalendar and vice versa, for which I’ve been following the draft RFC. The code at this point works, but mostly handles the basic cases. There is no public release of this yet.

Finally, there is a bit of a mismatch between how JMAP handles a global state and pimsync handles a per-item ETag. I have a plan on how to work around this, although this unfortunately adds a bit of extra network overhead.

I believe that some refactors can be done to pimsync which would improve its integration with JMAP (and also with the missing singlefile storage), but I have no plans to execute on these any time soon.

Upcoming work

[permalink]

In the coming weeks I’ll continue with the JMAP work. Once that is finished, I’ll move onto OAuth and Google support. After that, I intend to implement any missing features which are present in vdirsyncer, so that pimsync can fully replace it for all use cases.

Some folks who dislike an organisation still buy from it anyway. They argue that their individual purchase is just another grain of sand in the desert—too small to make a difference. While this is true, it’s only half of the picture. When we decide to boycott one organisation, we also buy the same goods or services from another organisation.

When we decide to boycott one unethical organisation, it’s not just about reducing their profits, but also about funding their competition, however small they may be. For small, honest businesses, every Euro counts. Every sale makes a difference. Every customer helps them stay alive.

Here’s a summary of the steps.

Connecting to the device

[permalink]

First, on the iOS device, open the Bluetooth settings. The device is discoverable only while this screen is visible. Then, on the Alpine Linux host, open bluetuith (or any other program used to connect to Bluetooth devices), scan for devices, and connect to the iPhone. A PIN will be shown on both devices. Ensure that it matches, and press OK.

Enable the hotspot

[permalink]

On the iOS device, go to Settings > Personal Hotspot, and enable the toggle. Something equivalent is likely possible on devices with other operating systems.

Connecting to the Bluetooth PAN

[permalink]

Install bluez-tools (I submitted a package recipe to the aports repo as part of this work).

Use bt-device -l to list all devices. Find the MAC address of the iOS device, and then run bt-device -i XX:XX:XX:XX:XX:XX to ensure that the iOS device shows NAP. If it doesn’t, then the personal hotspot is off.

Finally, run bt-network -c XX:XX:XX:XX:XX:XX nap, using the MAC address of the device found above.

The first time both devices reported that the connection was established, but no traffic flowed. After disconnecting and reconnecting twice, everything worked.

Not only was the IPv4 issue resolved, but the hotspot works perfectly, better than the Wi-Fi hotspot has ever worked on an iOS device.

Usability

[permalink]

When using the Wi-Fi hotspot, after the last device disconnects, the iOS device stops broadcasting the access point. Reconnecting to it requires unlocking the device and opening the Personal Hotspot screen again. It’s often necessary to toggle it off and on again.

When using the Bluetooth hotspot, my laptop can connect to the iOS device via Bluetooth at any time, and the bt-network… command requests access to the network. I don’t need to reach for the iPhone every time.

Bluetooth is likely less power-hungry than Wi-Fi (at least for my typical use case). I admit I haven’t actually measured this. If you do measure this, please send me your data.

This patchset follows on a similar previous patchset which implemented renaming of issue trackers from a few of months back.

Both of these were pretty basic features which were missing. Implementing them was not that tricky: the codebase for sourcehut is well-organised and quite legible (despite there still being a few quirks lingering here and there). I’m happy to be using a forge where the community is empowered to contribute with such ease.

I’m in the process of rewriting the pimsync manual pages from man(7) into mdoc(7). I’ve so far rewritten pimsync(1) and pimsync.conf(5). I’m not a huge fan of having the raw format in the repository (it’s not the easiest for newcomers to edit), but I’m quite pleased with the end result. The renders produced by a man pager are much tidier and lots of little quirks have been fixed. The same is true of the html pages, which now also include proper links, including links to other sections on the same page.

I’ve implemented support for a --socket-path flag in way-secure and tagged a new 0.2.0 version. This new flag should make way-secure much easier to use in shell scripts, and for simple experimentation.

I’ve tagged new releases of khal. v0.11.4 includes a large accumulation of fixes and smaller changes, but mostly retains backwards compatibility. 0.13.0 supports the icalendar==6.0.0, which had produced some major breakage, but won’t work with older dependencies or older, unmaintained versions of Python. To be clear, I’ve mostly tagged these new releases and published the new version: most of the work was done by other contributors (thanks for all the patches!).

There hasn’t been any major progress on the pimsync side. I’ve been writing prototype code to interact with JMAP servers, but this needs a few more iterations before it’s in a presentable state. Once this starts shaping up, I’ll start working on JMAP support for pimsync.

Frequent criticisms of man pages are “they don’t have links to each other” and “they don’t re-flow if I make the window narrower”. These are perfectly valid complains, but the reality is that man page do support these features: it’s the programs that we use to read man pages that don’t implement them.

The format

[permalink]

Man pages are stored in the mdoc(7) format and in the legacy man(7) format¹. The actual format for these files isn’t the easiest to read in the world, but it’s definitely not worse than XML or JSON. mdoc has purely semantic markup. A portion of an mdoc man page looks something like the following²:

.Sh NAME
.Nm openrc
.Nd stops and starts services for the specified runlevel
.Sh SYNOPSIS

In this case .Sh, .Nm and .Nd are macros which mark the following text as “section header”, “document name” and “document description” respectively. If you try to manually read or edit these files you’ll need the to consult the macro overview.

References (“links”)

[permalink]

Two macros are of particular interest:

• .Xr: “Cross reference”: link to another manual page.
• .Sx: Reference to another section in the same manual page.

Both of these can be rendered as links. In fact, when mdoc(7) pages are converted into HTML, both of these macros are converted into actual links. Both of the above linked manual pages include examples of this. The .Sh (section header) macro renders as an anchor, so links can point to it directly.

Ironically, the documentation for mdoc(7) renders links when displayed as HTML on a browser, but not when displayed in a terminal via the man(1) command.

Conclusion

[permalink]

We need better man page readers. Ones that let us follow references as links.

Currently man(1) formats a man pages and pipes it to less(1). We can’t easily add supports for links here: we need a pager which understands man pages natively.

While we’re at it, we could also re-flow text when the terminal window is made narrower.

If adjusting brightness is not possible, then the next obvious thing to do was to turn off HDR mode¹. I set out to find the toggle for this in settings, and right above it was what I really needed: an “SDR content in HDR mode” slider.

The slider worked exactly as expected. Great, no need to disable HDR.

But the experience hadn’t been great. This slider probably shouldn’t be hidden down at the end of the Display settings. In fact, the typical “brightness” slider should be repurposed to perform exactly this function. This slider won’t control brightness for game which support HDR; those games have an in-game slider. So the OS’s brightness slider, once adapted to control “SDR brightness in HDR mode” would need a little clarification underneath: “Does not apply to HDR games; see in-game menu”.

The first game we played did support HDR, and the first screen which is showed was one to properly adjust brightness for it. HDR is a really cool feature, but if brightness control is not a first-class feature, then the value of HDR quickly drops to a negative one.

Addendum

[permalink]

Huge thanks for everyone involved in getting HDR working on this software stack; I’m really pleased to see the improvements to the graphics stack that have been happening on Linux in recent times.

I’ve implemented a full documentation website. It provides guides on setting up pimsync, common scenarios, the security model as well as HTML renders of the manual pages. The man pages continue to be the canonical reference documentation, whereas the website includes more usage guides and explanations. It also includes instructions for building and hints for packages. If you’re interested in packaging pimsync for your distributions, please check out the packaging instructions.

I tagged a v0.4.1 release of pimsync. The changelog is also available in the documentation website.

If you find some gap in the documentation, or some detail is not clear, please reach out. At this stage, feedback is needed in order to continue refining it.

The vdir storage format

[permalink]

The vdir specification is now published on a page of its own. Previously this format was documented in vdirsyncer’s documentation, but the format itself was designed so that different tools can operate on the same set of files avoiding conflicts or races as much as possible.

The vdir storage format is a simply convention on how to store calendar and address book data locally, as stand-alone files. It is easy to implement and allows interoperability across various tools. It also puts people in full control of their data, since all of it is stored in standardised formats easily accessible in the local filesystem.

Publishing the specification as a stand-alone document should better reflect it’s intended purpose: being a specification which other tools can follow in order to improve interoperability.

Tools such as pimsync, khal, todoman, tdx, ab-bday, ab-tidy, vdirsyncer, and others already rely on this format, and I hope that more tools will join in future.

Repairing items

[permalink]

pimsync implements a repair command, which repairs invalid items in storages. Typically, these are items with a missing UID or with duplicate UIDs.

If additional common scenarios of invalid items surface, I’ll extend this command to handle them as well (assuming that there is a clear / unambiguous path to fixing them).

content-line-writer

[permalink]

I’ve published a new release of the content_line_writer crate. This library implements escaping of content lines for iCalendar and vCard files, and is usable in applications that need to generate either of these and don’t want to deal with the nuances of folding lines.

It’s not terribly interesting yet, but it’s one of the building block to start working on JSCalendar and JMAP support.

Critics might argue this would collapse productivity or disincentivise work. But if we already produce enough, why must we keep labouring like we don’t? The real disincentive is forcing unemployment on some while overworking others in a system that, by its own efficiency, has made so many jobs unnecessary.

The solution isn’t to invent useless busywork or cling to outdated 40-hour weeks—it’s to distribute necessary labor fairly. A shorter work week wouldn’t mean less gets done; it would mean what must get done is made by the work of all, freeing time for life beyond work. The obstacle isn’t economics, but our insistence that survival must be earned through toil, even when toil is no longer needed.

I mostly needed this because of how many crappy websites around the web automatically play audio when opened, even in background. American news and media sites are a common offender (but definitely not the only ones): try to read an article and they’ll start automatically playing some unrelated video¹.

Firefox has blocked auto-playing media for many releases now, and the feature has grown reliable enough that I no longer need mutetab. It’s definitely nice for a browser to provide this as a first-class feature, but I still wish we didn’t need to work around crap like this in the first place.

I used my laptop (which hadn’t been updated) as a reference to compare what’s changed and why a different font is being used. Not new fontconfig rules were installed, so fontconfig should behave the same.

GTK’s default font is configured via dconf, and is stored under the key org.gnome.desktop.interface.font-name (this is true even if my desktop isn’t GNOME; GTK configuration keys often start with org.gnome…). Checking this configuration value revealed that the default font had changed.

This value can be changed interactively using the dconf-editor GUI, or via the command line. The command line incantation to restore the previous default is:

dconf write /org/gnome/desktop/interface/font-name '"Cantarell 11"'

The double quotes are required because dconf expects a value surrounded by quotes, and the terminal will “swallow” the outer quote and treat their inner content as a single string.

Source of the change

[permalink]

The default values for dconf are stored in some XML files in /usr/share/glib-2.0/schemas. In this case, the value is in org.gnome.desktop.interface.gschema.xml, which is provided by the gsettings-desktop-schemas. The upstream changelog for this project lists the change in default font under Major changes in 48.beta. I’m not running beta; I’m running the final release, but it seems that this is one of those projects where stable releases don’t mention all applicable major changes; end users are expected to read all changelog details for release candidates and betas too.

There are two major differences that tools like pimsync and vdirsyncer need to keep in mind.

Calendar paths

[permalink]

The first is not a big deal, and relates to how Google defines the paths for each calendar. Most servers host all of a users’s calendars under a single path, such as:

/users/hugo@whynothugo.nl/personal/
/users/hugo@whynothugo.nl/work/
/users/hugo@whynothugo.nl/travel/

Both vdirsyncer and pimsync follow a convention based on this: the last component in a calendar’s path is treated as its name, and is also used as a name for a local directory which mirrors that calendar.

Google uses something that looks vaguely like so:

/user/some-user/calendars/aluMp7OOf2/events/
/user/some-user/calendars/GCmYjRTK1t/events/
/user/some-user/calendars/LZ81HRKLRv/events/

Note that each calendar is a collection named “events” and is itself contained in a collection with a unique name.

Vdirsyncer implements Google support as a special type of storage, and uses the second-to-last component as the unique name for a calendar.

For pimsync, I intend to do something similar; I’ll implement a behaviour to use the second-to-last component as a calendar name, but leave this behind an independent configuration directive. This setting will automatically default to True if you’re syncing with https://www.googleapis.com, but shall also be usable in other settings where a similar behaviour is required.

OAuth instead of Basic Auth

[permalink]

Google only offers OAuth for authentication to their CalDAV services. The flow in this authentication mechanism is quite different from Basic Auth (and Digest Auth), which is what most other providers and implementation use.¹

When using Basic authentication, a user simply provides a username and password, and pimsync uses that to authenticate itself.

When using Google’s flavour of CalDAV, the steps are a lot more:

1. Obtain a client_id and client_secret via Google’s Web interface. For web-based services, the operator of the service provisions these credentials and users don’t need to deal with this, but for software that runs on a local host, end users need to do this themselves. This only needs to be done once, but can be quite off-putting and confusing for new users.²
2. Configure pimsync with this client_id and client_secret and initiate the authentication flow. This opens a web-browser, where the user logs in through Google’s website.
3. After logging in, the user is redirected to a URL they configured in step 1. Vdirsyncer runs an HTTP service in a local port to handle this redirection, but it’s always a source of issues when users are configuring a remote host which doesn’t expose arbitrary ports. The only flow usable by desktop applications was deprecated by Google some years ago without a usable substitute.
4. Finally, pimsync can handle the tokens returned by Google to perform authentication. There are two tokens: one is used to authenticate requests and expires quite often. The other token is used to renew the first. Both of these need to be periodically persistent, which is what makes the whole mechanism stateful.

Personal thoughts on the situation

[permalink]

Personally, I’d suggest to folks to just stay away from Google and use some other hosting provider that isn’t contiguously campaigning against user’s digital rights and digital autonomy. But the reality is that a lot of folks simply don’t have a choice. The most typical scenario is one’s employer using Google Workspaces to coordinate internally. People who are in such situation simply need to access calendars on Google’s servers and don’t have any choice on their service provider.

And then, if tools like pimsync don’t support Google’s CalDAV services, individuals and organisations would have a much harder time migrating away onto another platform.

Proposed solutions

[permalink]

I have been pondering for a long time two potential solutions…

An OAuth proxy

[permalink]

The first approach is to implement an OAuth proxy. This proxy is configured once by an operator, and then usable by themselves or by their family and community. The proxy is configured once with a pair of client_id and client_secret, negating the need for everyone else to provision their own. When users create their account on the proxy, they also log in with their Google accounts, and the access tokens are stored by the proxy, encrypted with their password. Finally, users simply configure pimsync to talk to this server, which decrypts their tokens with their password and relays requests to Google’s services.

I like the approach in this solution: each of the two components involved do one thing (i.e.: good separation of concerns), it doesn’t further increase complexity in pimsync, and the proxy is also usable by any CalDAV or CardDAV application which supports Basic Auth. It doesn’t just enable pimsync to use Google’s CalDAV services, but any other CalDAV application, with the application requiring any custom support.

Alas, for the typical sole user configuring one or two of their own computers, this doesn’t make life a lot easier — it merely makes things harder by having to configure two separate services and glue then together.

I still think that the proxy is a useful tool to be implemented in future, but it won’t be the primary solution for pimsync. I have a much longer design specification this idea, so please reach out if you’d be interested in implementing it.

Native OAuth support

[permalink]

The second potential solution is to implement OAuth support in pimsync itself. Most feedback I’ve received has also hinted that this is the most user-friendly approach, and this is what I plan to implement in future.

As I mentioned a few months ago, libdav itself uses a middleware approach for extension functionality to its CalDAV and CardDAV clients. Functionality like implementing authentication and setting custom headers is already feasible with this approach, so this library doesn’t need any custom support for OAuth. This is great news, because it allows keeping this library compact and well-focused.

In order to make this feature a reality, the following functionality needs to be implemented:

1. Support configuring OAuth configuration flows. The OAuth parameters themselves shall be provided separately, to avoid having to repeat them for multiple storages (the example below re-uses the same configuration for CalDAV and CardDAV). The client_id and client_secret parameters can be specified as an external command with the same syntax as passwords. A full example:

auth google_foo {
    type oauth
    auth_url https://accounts.google.com/o/oauth2/v2/auth
    client_id 123
    client_secret 123
    token_file ~/.local/share/pimsync/foo.tokens
    scope https://www.googleapis.com/auth/calendar
    scope https://www.googleapis.com/auth/carddav
}

storage calendars_foo {
    type caldav
    url https://apidata.googleusercontent.com/caldav/v2/
    auth google_foo
}

storage contacts_foo {
    type caldav
    url https://www.googleapis.com/.well-known/carddav
    auth google_foo
}

2. A middleware that implements an OAuth flow on top of the tower service trait. Existing Rust libraries which implement OAuth all seem to implement too much, including their own internal IO (i.e.: the opposite of sans-io). They are not composible with other libraries, including libdav. I’ll implement a minimal library for OAuth clients, and a lightweight middleware to fit it into the tower stack.

3. Include the library itself as a dependency of pimsync, and write the final glue code to use the middleware in cases of a matching configuration.

4. Finally, if the URL matches any of the two URLs above, use the alternative logic for calendar names that I mentioned above in Calendar Paths.

Note that all OAuth related URLs can be provided in the configuration, so the implementation is usable with any server which wishes to implement OAuth in future, and not just the one that exists right now.

It’s a simple tool, yet I find it immensely useful. Given my usage of existing standards and conventions, it integrates well with existing tools. It is usable with pimsync (or vdirsyncer) to synchronise this birthday calendar event onto a server and onto other devices.

If all your calendar applications on all your devices integrate into your address book and show birthdays, then this won’t be of much use. But if you have any device which doesn’t, this solution works anywhere without any special support from your calendar.

This initial release implements all the basic features. For future versions I have a few useful addition in mind:

• -a to create recurring events for anniversaries.
• -m to monitor the input directory and update calendars immediately as soon as a contact is modified.
• Include the age in each recurring instance of an event.
• Localisation. E.g.: Support for event descriptions in other languages.

Let me know if this tool is of use to you or if you find any bugs!

Pimsync synchronises a few metadata attributes for calendars (displayname, colour, description, order). I had an intent to add owner to this list. This turns out to be impractical.

A WebDAV server exposes a calendar-user-address-set property, which is the address (typically an email) of a user. This property would represent the owner for a calendar. I planned to synchronise this property for every calendar, so that applications inspecting a calendars on a local directory can understand the owner of each one. If one of my calendars belongs to hugo@whynothugo.nl, and another to hugo@example.com, applications would be able to automatically detect this information and use it for sending out invites (via iTIP) with the correct Organiser or for sending RSVPs with the correct Attendant.

I overlooked one important detail. The calendar-user-address-set property is not defined for each calendar; it’s defined for the “user principal”, a collection which contains multiple calendars, typically all the calendars belonging to a single user. Due to this detail there are too many cases where the results of synchronising it would be unexpected, if not outright invalid.

Example 1: I share my work calendar with a colleague, and they synchronise it into their server. The “owner” property is not per-calendar, but at one level up. Synchronising the owner would change the owner for all their personal calendars, which would trickle into all sorts of issues. The only solution is to ignore the owner in this case.

Example 2: I synchronise collections from two different CalDAV sources into a single local vdir. The two CalDAV sources have a different owner. There is no way to synchronise the two conflicting values into the local storage.

Finally, pimsync’s approach to storages and collections is that collections (e.g.: calendars or address books) have properties, not the whole storage. The minimal unit which we synchronise is a collection, and having properties at a higher level simply doesn’t fit the model.

I can’t think of an approach for this feature which doesn’t produce undesirable results in surprising ways. So I’ve decided to omit this feature, the alternative would be to surprise users in how their data is overwritten. By the way, changing the calendar-user-address-set on the server would also impact how the server processes replies, and I can imagine someone going crazy trying to understand the source of the bug until the pinpoint it back to pimsync.

Other components of the project do implement related functionality. libdav provides a function to obtain the current calendar address set, and davcli exposes this information when inspecting a server. Both of these features shall remain in place.

Tracking issue: https://todo.sr.ht/~whynothugo/pimsync/8

Pimsync “shell” mechanism

[permalink]

vdirsyncer allowed specifying a short in-line shell script for retrieving credentials. I’ve implemented this same mechanism in pimsync. It’s really just a shortcut, but it makes lives easier for people, and the implementation cost is minimal.

You can now specify that credentials are to be retrieved via a command:

# This is safe to use with any special character.
password {
    cmd hiq -dFpassword proto=carddavs username=…
}

Or via a short sh snippet:

# Shell rules apply for special characters here, but you can pipe much easier
password {
    shell pass show communication/migadu.com | head -1
}

In reality, the latter is an alias for the following, which is a bit less comprehensible for non-hardcore shell users:

password {
    cmd sh -c "pass show communication/migadu.com | head -1"
}

Tracking issue: https://todo.sr.ht/~whynothugo/pimsync/136

Pimsync v0.3.0

[permalink]

I’ve tagged a new release with the above features a few other bugfixes. See the changelog for details.

Roadmap

[permalink]

I’ve tagged prioritised tasks in the issue tracker as todo. This is mostly for my own visibility, and these fall under the current grant from NLNet.

I had a list of tasks that fell under the scope of the grant, but each time that I needed to prioritise tasks, I needed to cross-check that list with the issue tracker, which ended up in a lot of pointless overhear. Tagging all issues is the obvious solution in hindsight.

Facts

[permalink]

When rendering fonts, especially at small sizes or on high-resolution screens, the edges of the characters are often anti-aliased to make them look smooth. Anti-aliasing works by blending the font colour with the background colour at the edges.

Human vision is non-linear: our eyes are much more sensitive to changes in dark tones than in lighter tones. E.g.: The difference between 10% and 20% brightness is much more noticeable to us than the difference between 80% and 90% brightness.¹

To account for our non-linear vision, RGB values are stored using a non-linear scale called gamma space. Gamma space allocates a larger range to dark tones, making images look more natural to our eyes.

We also represent colours in non-linear space due to technical reason related to ancient CRTs for which we use non-linear space, but that’s mostly historical baggage.

Gamma space

[permalink]

When storing or displaying an image, pixel values are often raised to a power (1/2.2) to account for the non-linear response of displays.

To perform accurate calculations on colours we first need to convert them from a non-linear space into a linear space. To do this, pixel values are raised to the inverse power (2.2).

This value (2.2) is called gamma. I’ll continue using 2.2 below to keep things simple.

Problem

[permalink]

Colour values are represented in non-linear scale, so if we perform calculations on them without first converting them into a linear scale, we get inaccurate results.

This results in incorrect blending of colours, where edges of text might be too dark, appear less sharp, or be inconsistent across different background. Contrast might also not be as good as expected.

Example

[permalink]

Let’s say we’re blending black and white with 50% coverage. Our colours are values between 0 and 1. Often these are represented by integers between 0 and 255, but the mathematics are easier to explain with values between 0 and 1.

Example operating in non-linear space

[permalink]

If we operate with non-linear colour values, we blend by applying the following operation:

Keep in mind that we’re using a non-linear scale, so 0.5 is not the “halfway” value between 0 and 1.

Example operating in linear space

[permalink]

If we operate by first transforming the colours into linear space, the result is entirely different. This process is called “gamma correction”.

First convert the values from non-linear space into linear space. For these particular values (0 and 1), they are the same, but this is not true for any other value.

Second, perform the blending operation, in linear space:

Finally, transform the result back into the original non-linear space:

The difference is far from trivial. For a visual reference, the following two rectangles have their background colour set to 0.5 and 0.73 respectively.

The first of the above rectangles is much close to black, and you might even notice that text on it has less contrast than ideal.

If you want to see some visual examples of blending text with proper gamma correction, see the sample screenshot in the PR which implements this feature for foot. The text is noticeably sharper.

Performance

[permalink]

There’s a lot of maths behind applying gamma correction. The above example does this for a single channel, but real usages have three channels (red, green, blue).

Fortunately, GPUs are designed for exactly this, and applying gamma correction using the GPU has an insignificant overhead.

Disabling web fonts is a known accessibility feature for users with dyslexia, low vision, or other cognitive impairments. I also consider my usage of this toggle an accessibility tweak, despite not belonging to any of the above groups.

My experience was quite positive although not without its caveats. The main thing that breaks is when sites use web fonts for icons. Don’t use web fonts for icons. They are an unnecessary hack and break when enabling this accessibility feature.

The good

[permalink]

As intended, most sites became easier to read, especially those that use exotic fonts with poor readability.

As an unexpected side effect, pages loaded noticeably faster. The improvement was immediately obvious and the explanation is simple: instead of having to wait for an additional font resource to load before rendering, the browser can render text on websites much sooner. This was a very pleasant side effect.

The bad

[permalink]

Instead of using SVG or PNG for icons and small images many websites encode these into fonts, and display them as if they were text. These icons no longer render when disabling web fonts. Instead, they just render a placeholder for a character, or sometimes random letters and symbols.

This ugly hack originally existed to support Internet Explorer, which lacked inline SVG support and had other limitations making, web fonts the easiest workaround. Despite IE being long dead in 2025, web developers continue using font icons out of habit. It doesn’t help that a lot of online documentation still promotes this practice.

Many popular website themes and icon libraries still rely on font icons. Some have migrated, but many haven’t. This process takes time and effort, and outdated documentation doesn’t help here.

I tried to fix some of the more popular tools and themes used for documentation websites in the open source ecosystem. But the sad reality is that most developers don’t seem to care about accessibility.

A few specific sites break quite badly without web fonts. One of the worst offenders is Google’s documentation websites. Individual icons are replaced with a whole word or a short sentence, but these overlap with the text around them, making the whole thing unreadable. I have no idea what kinds of horrible hacks they’re using, but it’s honestly amazing how much complexity has been piled onto simply rendering some text, hyperlinks and a few icons.

Conclusion

[permalink]

Don’t use web fonts for icons. Use the formats that were invented for this purpose instead. IE, the only reason to use these, is long dead.

When reviewing the accessibility of your website, check how it behaves with web fonts disabled. Keep this as an item to your accessibility checklist.

Related

[permalink]

As a reminder, font icons seldom align properly with text.

I find great value in tools and libraries with minimal dependencies. Simple code is a sign of good design. Too many layers of abstraction overcomplicate software, leading to maintenance headaches and security risks. It’s called over-engineering for a reason.

Armin Ronacher captures this frustration perfectly in Build It Yourself and I share his sentiment, especially when it comes to Rust. I highly recommend giving his article a read. It’s a well-written piece on how we can do better.

Optimised discovery process

[permalink]

When libdav performs DNS-based service discovery for a given base_url it now first checks whether the input base_url is a valid CalDAV/CardDav context. If it is, then discovery can be skipped entirely and that input URL is used instead. This skips unnecessary work, and allows using libdav in situations where discovery is misconfigured on a server.

Pimsync uses libdav for discovery, so this improvement affects pimsync as well.

pimsync storage monitoring

[permalink]

I’ve finalised a design for storage monitoring in pimsync. It can now monitor a storage for changes, and react immediately when they happen. This allows synchronising changes immediately, rather than on an interval every N minutes. Currently, monitoring in only implemented for the filesystem storage on Linux (using inotify). An implementation for BSD using kqueue is pending. CalDAV and CardDAV have no mechanism to monitor for events, although there is discussion ongoing to integrate with Web Push. Once this new protocol feature is stabilised, I’ll dive into implementing it for pimsync.

A caveat of how monitoring is integrated is that a full synchronisation is executed. I.e.: when a single item changes, pimsync still checks all items to see if they have changes. This adds a bit of extra work (although it’s still minimal CPU use). I intend to iterate upon this further in future, executing a partial synchronisation (which only sync the item that has changes and doesn’t read or write anything else), but I have other priorities in the immediate future.

pimsync faster start-up

[permalink]

pimsync now resolves parameters specified via a cmd early during start-up. This ensures that all prompts for credentials happen quickly, and network IO happens after these prompts, so you don’t have to wait around a second or two while pimsync starts. I’ve also tweaked how locking happens internally: previously pimsync lock storages so that only one operation happened on each storage at a time. Now, pimsync only lock individual files, so that we only avoid concurrent writes to a same file, but multiple operations can run concurrently on each storage.

Various libdav improvements

[permalink]

I’ve made substantial changes to handling of escaped characters and reserved characters in paths to DAV resources. Mishandling of escape sequences is a common issue for many corner cases in vdirsyncer. I am confident that this set of changes properly handles all scenarios for libdav and pimsync.

I’ve backfilled the changelog with changes in recent versions of libdav as well, so that consumers can properly track what’s changed and how to upgrade. I’ve also made sure that all public interfaces are documented. If anything is still unclear, please reach out.

Context

[permalink]

todoman optionally depends on py3-repl. If the latter is installed, then todoman gains a new repl command which starts an interactive console.

One way of indicating that todoman optionally depends on py3-repl is to add it as an optional dependency. This requires support for optional dependencies in the package manager and all related tools.

A pitfall of this approach is that there is later no clear indicator that py3-repl is installed as an optional dependency for todoman. If it is marked as explicitly installed, then there’s no “hint” for me to later remember that I installed it as an optional dependency. I’ll likely delete it thinking that I don’t use it directly and nothing depends on it. If it is marked as a installed as a dependency, then it will removed automatically, since nothing explicitly depends on it.

A package manager might have a different behaviour: when cleaning up, it can keep around any package that is an optional dependency for another. This would result in many stale unwanted packages being kept around.

Solution

[permalink]

Create a new meta-package: todoman-repl. This depends on todoman and py3-repl. If I want todoman with the optional functionality, I simply install todoman-repl. The package manager won’t remove py3-repl, but it also won’t show up as installed explicitly. In fact, the list of desired packages reflects my exact intent: “I want todoman with the repl functionality”.

Advantages

[permalink]

This does not require any additional complexity or dedicated features in the package manager or packaging toolchain.

When the package todoman-repl is installed, it clearly reflects the intent of the user to install todoman with its repl functionality. The repl dependency is a directly dependency of an explicitly installed package.

And what I’ve described is precisely the problem with it. At any idle moment, I can open the app to pass a single moment and end up being sucked into it for tens of minutes or even hours. It’s very hard to just use it for five to ten minutes a day. It’s designed to be addictive and is addictive.

The only solution for me is to simply not have TikTok installed on my phone. Like many addictive things in life, the solution is zero consumption, because consuming a small amount is much harder than abstaining completely.

Public perception around the addictiveness of TikTok is mixed. Some say it’s no addictive at all, while others claim that it’s the best thing in the world and they can quit whenever they what — they just don’t feel like quitting.

Jan 9 14:52:37 anchor soju[71321]: 2025/01/09 14:52:37 listener 0.0.0.0:6697: accept error (retrying in 1s): accept tcp 0.0.0.0:6697: accept4: too many open files

Sounds like soju needs more file descriptors that the current limit. I need to increase the maximum allowed file descriptors.

The fix

[permalink]

First, give the _soju user a dedicated class:

-_soju:*:895:895::0:0:soju user:/var/soju:/sbin/nologin
+_soju:*:895:895:soju:0:0:soju user:/var/soju:/sbin/nologin

Then raise the limit for this class by creating /etc/login.conf.d/soju:

soju:\
        :openfiles-max=4096:\
        :openfiles-cur=128:\
        :tc=default:

Finally, rebuild the capability database and restart soju:

cap_mkdb /etc/login.conf
rcctl restart soju

Sources

[permalink]

• passwd(5)
• login.conf

In the last couple of days, I’ve learnt that my approach was unsound.

I will use the path /path/to/theitem%2Fwithslash.ics as an example. This path refers to a resource named theitem%2Fwithslash.ics inside the collection /path/to/. However, decoding this path, it would return /path/to/theitem/withslash.ics, which points to resource withslash.ics in the /path/to/theitem/ collection. Clearly these are not the same.

URLs describe resources which don’t share the name naming limitations as regular file systems. Resources names may contain any byte sequence, including / (which needs to be escaped as %2F).

This behaviour is explicitly called out in the RFC3986, section 2.2:

reserved    = gen-delims / sub-delims

gen-delims  = ":" / "/" / "?" / "#" / "[" / "]" / "@"

sub-delims  = "!" / "$" / "&" / "'" / "(" / ")"
            / "*" / "+" / "," / ";" / "="

[…] URIs that differ in the replacement of a reserved character with its corresponding percent-encoded octet are not equivalent. Percent-encoding a reserved character, or decoding a percent-encoded octet that corresponds to a reserved character, will change how the URI is interpreted by most applications. Thus, characters in the reserved set are protected from normalization and are therefore safe to be used by scheme-specific and producer-specific algorithms for delimiting data subcomponents within a URI.

The next section makes a clarification for unreserved characters:

URIs that differ in the replacement of an unreserved character with its corresponding percent-encoded US-ASCII octet are equivalent: they identify the same resource.

When implementing client, it’s probably safest to treat the path component in URLs as an opaque string and not to alter it in any way. However, comparing whether two components are the same requires normalising them, by percent-decoding non-reserved characters, while maintaining percent-encoded reserved characters in their original form.

Double-encoding in WebDAV

[permalink]

In WebDAV, resources are listed inside an XML response. In this case, some characters need to be encoded as XML entities. I.e.: ", ', <, > and & need to be encoded as ", "', <, > and & respectively. These will be decoded when parsing the XML, which happens in a separate layer, and is invisible to HTTP.

Copying photos out of an iPhone another device is not trivial. The approach suggested by Apple seems to be to copy your photos onto their servers, but actually downloading them is a pain. You can either download them one at a time, or request request a takeout, which takes about a week. A ridiculous approach when you want to move files from one device to another that’s 30cm away.

In my experience, the simplest approach (which is not at all simple) to copying photos seems to be using usbmuxd with ifuse. These set of tools allow mounting the phone onto the local file system, and moving the regular photo files out.

When using usbmuxd, please keep in mind that all users on your target host have complete access to the iOS device. Even on a single-user system, this means non-interactive accounts like those used by background system services have access, including the user nobody.

The latest release of usbmuxd happened in 2020, and doesn’t work with recent devices or recent versions of iOS. You’ll need to rebuild from source. All build dependencies can be installed with:

doas apk add -t .usbmuxd-dev \
    automake \
    autoconf \
    libtool \
    libusb-dev \
    libplist-dev \
    libimobiledevice-dev \
    libimobiledevice-glue-dev

Then, clone the source repository and build it as indicated in the instructions:

git clone https://github.com/libimobiledevice/usbmuxd.git
cd usbmuxd
./autogen.sh
make
doas make install

usbmuxd has some magic mechanism to automatically run when a device is plugged in, but it is quick hacky and unreliable. Simply run the daemon manually:

doas usbmuxd --foreground

Now plug the iOS device with a USB cable, and mount the iPhone’s filesystem:

mkdir -p ~/mount/iphone
ifuse ~/mount/iphone

Note that ifuse forks to the background immediately, so while it seems that it may have exited, it’s still running.

Photos and videos will be available under ~/mount/iphone/DCIM. They can be moved around as regular files.

When moving photos out of the phone, the Photos app will still show them as present. This is because the Photos app keeps its own database of photos. A workaround for this issue is to delete the Photo app’s database. It will then re-create a new empty database, and show no photos.

To delete the Photo app’s database, use:

cd ~/mount/iphone/PhotoData
rm Photos.sqlite Photos.sqlite-shm Photos.sqlite-wal

Note that Albums and favourites are generally lost during this process. This information is also lost when moving the photos out, so I don’t bother with organising photos on-device at all.

Finally, I use this quick and dirty script to sort photos into directories by date taken. Do not run this script in a directory with existing subdirectories; it may overwrite data inside of them.

Note that screenshots don’t have date and time metadata, so this also filters them out rather quickly (I personally don’t store screenshots with my “real” photos).

Packages from other distributions are welcome to start publishing packages. Please report any issues, including lack of documentation. The README now explains requirements, compilation and installation, something obvious that somehow slipped through the cracks until this point.

I have updated libdav to rely on the hyper/tower middleware stack, which is currently the standard in the Rust ecosystem. This allows consumers of this library to alter requests and responses in any way that they need, including custom authentication, setting a User-Agent header (or any other header), domain-specific logging, etc. Version 0.6.1 includes further documentation and examples on this.

Pimsync itself leverages the above feature, and now properly sets a User-Agent header. This can also be customised in the configuration file. The pimsync.conf(5) man page explains this in detail.

As you might notice from the above link, manual pages are now published in HTML form, making them convenient to read online. I’ve created a small and simple landing website for pimsync too, mostly as an index of existing material.

Another small change is that pimsync will now embed the proper version, regardless of whether it is build from tarballs or a git checkout. I wrote a separate article with details on this process.

The storage for WebCal calendars is now named webcal, which is less ambiguous than http (CalDAV also uses HTTP). The documentation and source code was inconsistent everywhere. The term webcal is now used consistently in all places.

Current blockers for a v1.0.0 release

[permalink]

• Publish man pages as HTML online
• Publish a landing index.html page
• Implement protections if collections have been reconfigured.
• Flush stale data for collections that are no longer synced.
• Move the interval directive inside individual storage definitions.
• Implement conflict resolution from a and from b.

The recipe presented here ensures that source tarballs include the right version string (even tarballs auto-generated by code forges), and builds done from a full git checkout use the output of git-describe.

The examples below work for a Rust project, but the general solution is applicable to any language.

Including the version into tarballs

[permalink]

Most code forges provide tarballs for tags and individual commits. These are generated using git-archive. Git provides an export-subst feature which allows us to replace a string with the output of git-describe, effectively injecting the corresponding version into the tarball.

First of all, we’ll create a .gitattributes file which includes a file where the replacement will be executed, and the export-subst attribute. E.g.:

/build.rs export-subst

This will replace the string $Format:%(describe)$ in build.rs with the corresponding version.

Injecting the version during builds

[permalink]

The above merely replaces the string into a file when generating tarballs, but we still need to consider builds done using a full git checkout of the repository (which will have the un-replaced string).

When building from a git checkout, we can determine the right version string using git describe --tags.

The following example uses a replaced version, if it has been replaced, and otherwise falls back to git describe:

use std::process::Command;

fn main() {
    if std::env::var("PIMSYNC_VERSION").is_err() {
        let version = "$Format:%(describe)$"; // Will be replaced by git-archive.
        let version = if version.starts_with('$') {
            match Command::new("git").args(["describe", "--tags"]).output() {
                Ok(o) if o.status.success() => String::from_utf8_lossy(&o.stdout).trim().to_owned(),
                Ok(o) => panic!("git-describe exited non-zero: {}", o.status),
                Err(err) => panic!("failed to execute git-describe: {err}"),
            }
        } else {
            String::from(version)
        };
        println!("cargo:rustc-env=PIMSYNC_VERSION={version}");
    }
}

As you’ll notice, I basically inject the version into an PIMSYNC_VERSION variable. This is only done if PIMSYNC_VERSION is unset, allowing anyone compiling to override the version by setting the PIMSYNC_VERSION variable.

Using the environment variable

[permalink]

Finally, the application needs to use this variable as a version. This is done by simply loading the environment variable into a &str:

pub const VERSION: &str = env!("PIMSYNC_VERSION");

The env! macro evaluates an environment variable at compile time. This would fail if the variable is unset, but the variable is set unconditionally. If it is unset, it is because someone has been tinkering with the build system and broke it.

Caveats

[permalink]

There are none. This relies on code-forge auto-generated tarballs, using functionality in git itself. You don’t need to manually produce tarballs, and builds done from any commit or tag will reflect their source properly.

The idea of a different name originally came to mind due to difficulty pronouncing the original name, and having to spell it out every time I spoke about it in person.

But this wasn’t the deciding factor for a new name. The main issue with naming it vdirsyncer is that is makes references everywhere confusing. Documentation, issues, tutorials and blog posts that refer to vdirsyncer become ambiguous, since it’s not clear which version they’re talking about.

With the new edition having a different feature set and different configuration, it just makes sense to treat it as a different project – even if a spiritual successor. The initial release of pimsync will be missing some features in vdirsyncer, so some folks will also continue to use the previous version for the time being. Distributions should likewise be able to ship both packages during a transition period, and end users will be able to co-install both programs until they’ve migrated to the new one.

The documentation for both will remain online. This separate names, it will be clear which documentation refers to which project.

Despite having a new name, the main goal and general implementation design remains the same. Thanks @untitaker for writing the original implementation!

The name pimsync comes from the PIM acronym, which stands for Personal information manager. This term describes tools that manage personal information, including address books and calendars. It is easier to pronounce at loud and to explain verbally.

Documentation

[permalink]

Writing the documentation was a lot more pleasant and fulfilling that I expected. I admit that I overestimated the effort required for this.

At the moment, the following manual pages are available:

• pimsync(1) - general command usage
• pimsync.conf(5) - pimsync configuration file
• pimsync-migration(7) - migration guide from vdirsyncer

All reference documentation shall be published as manual pages. These same pages are also rendered as HTML pages and published into a proper website. For the moment, the HTML pages are published at a temporary mirror.

Tutorials and how-tos will likely exist in the pimutils website, and contributions are most welcome.

Explanations on why things are designed the way they are will continue being published in these status updates. There’s also a lot of internal documentation in the library documentations for the different components of pimsync.

The documentation is by no means complete. There’s likely plenty of room for improvement, but it should be a good starting point to understand how to configure and use pimsync. I expect to continue improving it based on user feedback and frequent questions. If anything is not clear, that is an issue that needs to be fixed.

Configuration overhaul

[permalink]

I discussed configuration formats a bit last year. At the time, I went with TOML, mostly given that it seemed the one with least difference from the previous implementation.

While TOML is a good serialisation format, it is not a very good configuration format for humans to write. To make matters worse, the code that parsed the TOML configuration was bloated and pointlessly complex. I was happy to drop all of it.

I don’t want to ship a stable release on pimsync with TOML and later regret that and impose yet another change on end users. I intend to avoid breaking changes as much as possible after the stable release has been done.

The new configuration format is much simpler and friendlier for humans to understand and write. The syntax is technically the scfg format, but documentation doesn’t expect familiarity or understanding of the underlying format. Instead, the documentation focuses on explaining how to write the configuration file itself.

Unix socket support

[permalink]

The implementation for using CalDav and CardDav over a Unix socket has been completed. See Unix domain socket support for vdirsyncer for more background on this.

Versioning discontinuity

[permalink]

Previous beta releases were in the order of v2.0.0-beta1, given that this was the second iteration of vdirsyncer. With the rename, this doesn’t make sense: the next stable release will be v1.0.0 of pimsync.

Given that no distribution has packaged any of the beta versions, this should not be an issue.

Each task is an individual process which can be managed with the usual process management tools. This means that it is also possible to run an instance of snooze as an individual service via one’s service manager under the usual service supervision tree.

The man page describes its usage quite well, and this post describes its mode of operation and usage in greater detail. The snooze command is currently packaged by a few distributions, and its source is public domain.

A minimal usage example is the following:

snooze -H9 -M40 echo "it is now 09:40"

Note that the snooze command exits after running the command. For recurring tasks, the service manager should re-start it once it exits, preparing for the next execution. This design also ensures that there are never any concurrent executions.

As a bonus use case, snooze can be used easily schedule a one-shot which inherits the current execution environment.

Notes on cron

[permalink]

The snooze command’s use case overlaps with cron, but the former doesn’t depend on a service that needs to be executed by root. It kind of bothers me that cron service that needs to run as root and requires privilege escalation for a user to operate.

cronie, dcron, fcron bcron and anacron all have the same design as cron. It’s perfectly fine on a server, but on desktop / interactive systems, I’d prefer something that doesn’t require any elevated privileges to enable or operate if the tasks will run as an unprivileged user.

snooze runs entirely within the process tree of the user, without any additional privileges in any of the parts involved. It is even suitable for usage inside containers.

The audit uncovered four minor findings, but no huge security issues. I remain confident in my design and approach. It is good to have a second pair of eyes reviewing these. The following is a short summary of these findings.

Inappropriate file mode

[permalink]

The data in vdir storage is marked executable. Items themselves are well-escaped, making it difficult to make use of this, however the display name of a collection is not, so an attacker controlling the display name can effectively control the contents of an executable file.

This was a mistake on my part; I used the constant Mode::RWXU, which was not what I’d intended. I replaced with with 0o600, which is much easier to read.

In theory, an attacker with access to a remote calendar could have injected a malicious binary and tricked a local user to double click on the file, triggering the execution.

The issue by itself is quite minor, but could have been exploited with a combination of (1) write access to a remote CalDav or CardDav instance and (2) social engineering.

This issue was fixed in 302be5cc7b2c755b91311cc84a89f8f36f8733cd.

Potential panic in error handling

[permalink]

The application will panic if a storage provider such as a WebDAV server returns a malformed item.

Vdirsyncer could be made to panic if a WebDAV server served a completely empty file. I fixed this issue in the vparser library, and included a unit test to avoid future regressions.

This issue could have resulted in a remote server performing a denial of service on vdirsyncer by making it crash continuously. No data corruption or data leak could have occurred because of this.

This issue was fixed in a2a25d881c00f23b5dd60e3c7752c327ca11806a.

Following symlinks

[permalink]

Symlinks are followed by the vdirsyncer implementation even if they fall outside the target sync directory.

Vdirsyncer performs atomic writes by creating a new file and overwriting the previous one. Because of this, it could never write to a location outside the vdir, even in the presence of a symlink.

The only scenario where this can potentially be exploited, is where Alice has write permissions to a vdir, but Bob runs vdirsyncer on it. In such a scenario, Alice could potentially leak files which are only readable by Bob by uploading them to a remote storage.

The most likely course of action here will be to treat symlinks as unreadable files.

This is currently tracked in https://todo.sr.ht/~whynothugo/vdirsyncer-rs/94

No data validation

[permalink]

An item in a vdir can be replaced with any data, and vdirsyncer will attempt to submit it to the sync target.

This is an intentional design choice; if a user has data that is technically invalid, we still want to permit them to synchronise it in any direction. If we prohibit synchronising invalid data, users may become incapable of operating on this data in order to fix the invalidity.

It is also possible that a user is using a calendar client which handles this “technically invalid” data just fine, and we don’t want to get in the way of users synchronising this.

While I don’t consider this an issue in itself, the fact that it was reported as a security finding reveals that this behaviour was not properly documented. I had added a mention on this topic to the documentation so the behaviour is clear for all users.

Other findings

[permalink]

Early during the audit some security issues were found in dependencies. These were unlikely to be exploitable via vdirsyncer, but the dependencies where promptly updated anyway.

I also added rules to the several of the libraries that fall under this project to prohibit unsafe code. These are higher level libraries, and there’s no need to escape Rust’s typical safety rules. Some dependencies, however, contain unsafe code. This is inevitable for lower level dependencies.

The auditor and I also had some chats to confirm that my approach on TLS configuration made sense and that there were no gaps in my reasoning. Their general feedback mentioned that […] the simplicity of the design and choice of Rust that simultaneously reduces attack surface and averts common memory- safety bugs.

Closing words

[permalink]

Security is an ongoing process. While no greater issues have been found at this time, work on vdirsyncer continues, and the security status of the project needs to be reviewed at a future time.

I am considering naming the new project vdirsyncer2, so that both can be co-installed.

Right now my main blocker for an initial stable release is documentation: there’s some documentation, but not enough for a complete newcomer to figure out how to fully configure and use vdirsyncer. I can’t call this non-beta with the current state of the documentation.

Work on the documentation is ongoing, so that I can unblock a stable release. I decided to consolidate documentation into man pages, so that they are easily accessible offline in the same format as any other utility. Of course, I shall publish these same pages online too in HTML form.

Security considerations have been moved into these same man pages for better visibility. On the topic of security considerations, the ongoing security audit is coming to a close, and I am quite pleased with the results.

Splitting subprojects

[permalink]

I started writing vdirsyncer in Rust in a single repository along with its dependant libraries. This probably sped up initial development by a marginal amount, but as the project stabilised, this didn’t prove as convenient.

The main problem was when making backwards-incompatible changes to libdav. I suddenly found that I couldn’t commit these changes without breaking vdirsyncer, implying that I had to implement changes to multiple projects at once (and in a single commit) even if those changes were not entirely related and had different scopes.

This was not the first that this happened, and it became clear that as pieces mature, I want to be able to release dependencies before I deal with upgrading vdirsyncer for them.

I therefore split libdav and davcli into separate repositories. I believe this also improves visibility for these projects, since they’re now more than simply “a subdirectory in some other project”. Both of these are usable in scenarios that don’t involve vdirsyncer at all.

Protections

[permalink]

I implemented protections for emptied collections in vdirsyncer. This means that if you delete a collection entirely, it won’t delete the collection on the other storage by default: it will show a warning and skip it instead.

While I prefer emptied collections to be emptied on the other side for my own usage, this behaviour is somewhat dangerous and doesn’t feel like safe default. So the protection is enabled by default, and users will have to opt-out of it with the new on_empty = "sync" configuration directive.

Start-up performance

[permalink]

At start-up, vdirsyncer will no longer load pairs and storages that are not being synchronised (e.g.: if only a single pair was explicitly requested). This improves usability in two ways:

• If a disabled pair has some invalid configuration, that won’t block users from synchronising another pair.
• Vdirsyncer won’t prompt for credentials for storages that are not being operated upon.

Other minor tweaks

[permalink]

libdav, davcli and vdirsyncer now all properly handle situations where a CalDav server has a home set with multiple entries. This is not a frequent situation; most common servers use a single home set. But the edge case had already come up, so it made sense to properly support it. It’s also well covered by the CalDav specification.

Vdirsyncer now uses internal locks to avoid concurrent access to a same storage. This makes a difference for scenarios like storage A being synchronised with storage B and storage B being synchronised with storage C. The locks prevents both of these operations from running concurrently. Running them concurrently would often result in errors (although these errors would not result in data loss).

Several configuration errors now print the full path to the configuration path and clearer error messages. This helps understand what’s wrong when configuring vdirsyncer.

Username and password are now both optional, so vdirsyncer can be used in servers which don’t expect authentication.

Finally, I tested the project on OpenBSD. The objective was not only to ensure that vdirsyncer works well on this platform, but to also ensure good portability. Given the positive results, I don’t expect any issues on other BSDs or Unix-like platforms.

My intention is to produce long text by speaking it out loud and then doing some minor refinement by changing potential typos.

Preparation

[permalink]

The first thing that was on the whisper.cpp repository.

git clone https://github.com/ggerganov/whisper.cpp/

Then I built the main example which just takes a wave file as input and produces text as output.

make main

Next, I had to download the models for recognizing English speech.

The script did not work at first. I had to patch it to use curl instead of wget, because it uses few specific wget flags which don’t work on busybox.

--- a/models/download-ggml-model.sh
+++ b/models/download-ggml-model.sh
@@ -101,11 +101,7 @@ if [ -f "ggml-$model.bin" ]; then
     exit 0
 fi

-if [ -x "$(command -v wget2)" ]; then
-    wget2 --no-config --progress bar -O ggml-"$model".bin $src/$pfx-"$model".bin
-elif [ -x "$(command -v wget)" ]; then
-    wget --no-config --quiet --show-progress -O ggml-"$model".bin $src/$pfx-"$model".bin
-elif [ -x "$(command -v curl)" ]; then
+if [ -x "$(command -v curl)" ]; then
     curl -L --output ggml-"$model".bin $src/$pfx-"$model".bin
 else
     printf "Either wget or curl is required to download models.\n"

I then downloaded the model files using the patched script.

sh ./models/download-ggml-model.sh base.en

Usage

[permalink]

Next, I needed to record audio snippets with me speaking the actual text. For this, I used the arecord utility. My first example was rejected because the audio needs to be recorded at 16 kilohertz. I quickly found the flag to change the rate at which the audio is recorded.

I used the following command to record showed audio snippets, usually one paragraph at a time. I would terminate recording by pressing Ctrl+C.

arecord --format=cd -r 16000 file.wav

Finally, I used the following command to transcribe the audio files into text.

./main -m ./models/ggml-base.en.bin -f file.wav

This would usually take about perhaps a second or two. I then copied the text into this file and simply remove the timestamps because they are not relevant.

Results

[permalink]

All prose text in this article was dictated by me to my laptop and converted by whisper with steps mentioned above. Only the URLs and code samples were copy-pasted from elsewhere.

I only had to fix a few minor typos as described below.

• cycles -> typos
• it uses new specific wget lives -> it uses a few specific wget flags
• busybooks -> busybox
• A record -> arecord
• Ctrl C -> Ctrl+C (I also added <kbd> around this myself)
• Finally, I use -> Finally, I used

I also had to apply some editorial fixes which were more related to my own grammar than the transcription process itself.

I was quite impressed by the fact that wget and curl were actually spelled correctly.

I found the quality of the transcriptions to be quite reliable and convenient to use. I will experiment further with using them to take notes. At this stage there is minor annoyance of having two separate steps for recording and converting audio. This is simply because I am using an example binary for my test.

I expect that I should be able to write a small program which uses whisper and simply pipes input from my microphone straight to whisper so I can have a continuous stream of text which I would then copy paste perhaps into articles and maybe refine small typos and mistakes.

Future ideas

[permalink]

I would also like to set up an automated messaging account such that I can send an instant message with an audio recording to it, and it will relay the transcript to my email. This would allow me to take spoken notes on my phone, which I can later process as text on my laptop.

By implementing this as an input method, it works on any application where typing regular text is possible.

I knew this was not something that I’d be able to complete in a day, but I at least wanted to get the basic scaffold and general structure set up on this first day. Spoiler: I think this went quite well.

I decided to use Hare for this. It’s a simple language, quite low level, with a solid approach to error management. Compilation is fast, and I only need two dependencies which compile in milliseconds. I used hare-wayland in the past for wlhc, and found myself quite comfortable developing with it.

I started my project today by generating client code for the Wayland protocols that I expect to need. I copied some bits from wlhc for this, and came to this initial Makefile:

DESTDIR=
PREFIX=/usr/local

.PHONY: build
build:
	mkdir -p wayland/xdg wayland/wlr wayland/wp

	hare-wlscanner \
		< $(shell pkg-config --variable=pkgdatadir wayland-protocols)/stable/xdg-shell/xdg-shell.xml \
		> wayland/xdg/shell.ha
	hare-wlscanner \
		< $(shell pkg-config --variable=pkgdatadir wayland-protocols)/staging/single-pixel-buffer/single-pixel-buffer-v1.xml \
		> wayland/wp/single_pixel_buffer.ha
	hare-wlscanner \
		< $(shell pkg-config --variable=pkgdatadir wayland-protocols)/staging/fractional-scale/fractional-scale-v1.xml \
		> wayland/wp/fractional_scale.ha
	hare-wlscanner \
		< $(shell pkg-config --variable=pkgdatadir wlr-protocols)/unstable/wlr-layer-shell-unstable-v1.xml \
		> wayland/wlr/layer_shell.ha
	hare-wlscanner \
		< input-method-unstable-v2.xml \
		> wayland/zwp/input_method.ha

	hare build -o emoji-im

install: build
	install -m755 emoji-im $(DESTDIR)$(PREFIX)/bin/emoji-im

This generates code to unmarshall events and send requests to a wayland server. The generated files are not intended to be edited manually, and the functions that they provide are a direct representation of the underlying protocol.

My code is in separate files, but it’s all compiled together as one big project.

I find this approach quite convenient. I can actually inspect the generated code to double check the names and types of functions and their method. The library itself handles all the wire protocol serialisation and deserialisation.

Coming from Python, I’ve used to libraries that generate abstractions to call some remote method at runtime. In these situations, even though I’d be using a proxy object that wraps some underlying IPC there’s no source code for the client objects; it’s all generated at runtime. So double checking types, or figuring out the little details was not as straightforward as checking the code for the function I was calling – there was no code!

In this sense, I find that Rust proc_macros tend to behaves too closely to Python. They generate code at compile them, and feed it straight to the compiler. It’s not as straightforward to double check details in this code.

I started out writing code to register the Wayland global objects that I’d need. I wrote code to register registered a new input method. Each execution was a simple:

make && WAYLAND_DEBUG=1 ./emoji-im

Most of this was some straightforward scaffold that I was pleased to have finished quickly.

The next step was to intercept keyboard input and determine emoji suggestions based on that. Because my program would be intercepting all keyboard input, I wouldn’t be able to press Ctrl+c to kill it. In order to ensure that I didn’t lock myself out of my own session, I ran:

watch -n 10 pkill emoji-im

This would kill the emoji keyboard every ten seconds, so if I ended up in a state where it was eating up all keystrokes, this would save the day. Occasionally it gets killed at the wrong time. I could have just ran tests in a nested compositor.

The wayland protocol for input methods is pretty well designed, and integrating with it was quite uneventful. Until I had to process keystrokes. Keymaps and Key events -presses are transmitted using the Xorg. These are quite non-trivial to parse. Instead of actually parsing the properly, I hard-coded the key sequences for all letters, Space and Tab on my own keyboard.

Some time later I was pointed at hare-xkb, which I will have to integrate eventually. I preferred to continue the wave of implementing new bits rather than sinking time into polishing something that can be left for a final stage. Adding tiny improvements helps keep motivation up; perfecting inconsequential details before everything else works does not.

Being able to recognise keystrokes, I started intercepting them and building a string with the characters. This is literally just concatenating a character to a string. This is the preedit string, renders in text input areas only as a placeholder. Depending on this string, I intend to show emoji suggestions. When Space is pressed, I intend to insert the selected emoji. At this early stage, if the input string is smile, I’ll insert the 🙂. At this point, I had to leave to deal with some personal stuff.

That’s basically it for day 1. This was a pretty fun project, and there’s plenty of work left to do. At this point, I have an emoji input method which, when active, lets me insert the smile emoji.

Black text on white background is relatively easy to read in a highly lit environment. Conversely, trying to read white text on dark background is a futile exercise in a sunny room: I can’t see anything with all that glare.

On the other hand, sunset here in the winter is at around 17hs, and most of the winter is dark and gloomy. In a dimly lit environment with just some warm lights, looking at a screen with a white background and black text makes my eyes hurt. It’s terrible for my eyes.

Effectively, I need dark mode when my working environment is a dark one.

While dark mode and light mode are often thought of as an aesthetic preference, I find them to be about usability.

Since the last status update, a lot of polishing has happened, and I’ve completed support for failure scenarios that could lead to data corruption.

Code that writes data into a vdir no uses atomic writes. Data is first written to a new, temporary, file, and then that file is linked to the appropriate path. If a catastrophic failure happens mid-way (e.g.: a power failure), then the file with either remain in its original state, or in its final state, but it will never end up with half of the data written and half missing.

Right now, the main “protections” that are missing are to avoid deleting all items when a collection is fully emptied on one side and disable deletion of deleted collections. During everyday usage, none of these should be relevant, but if you ever delete a calendar entirely with the expectation that “vdirsyncer will just download it again”, the current result might not be what you expect. Personally, I often create short-term todo lists which I later delete, so I collection auto-deletion has been convenient for me.

davcli now supports communicating with a server with a username and no password, or with no authentication at all.

I’ve dropped support for “item properties” in the storage abstractions. The intent for these was to eventually support synchronising emails. The subtle differences between emails and calendars has led me to conclude that this is a bad idea. Mailboxes don’t have properties, but emails do. However, calendars have properties, and events do not. Additionally, emails are immutable, whereas events are mutable. Having a single sync algorithm that supports both adds unwanted complexity, and a unsatisfactory implementation for both.

I have also implemented a simple readiness notification mechanism, so that a service manger knows when vdirsyncer has transitioned from “starting up” to “running”. Once it has reached its running state it should not exit and will continuously (try to) synchronise periodically.

Finally, I’ve tagged a 0.1.0 release of ab-tidy, the little helper which renames vcard files to match the name of the person inside. I have a few other tiny tool in mind that I’d like to implement in the near future.

Security audit

[permalink]

The grant provided by NLNet and NGI Zero also includes a security audit of vdirsyncer.

During our initial discussion of scope and expectations, I realised that service discovery assumes that your local DNS resolver is a validating one. In cases where the local resolver is not a validating one, vdirsyncer is vulnerable to an active MITM attack via DNS spoofing. I have since included a clear explanation of this expectation in a SECURITY.md file which will eventually become part of the documentation. I have also included a mention of this in libdav API documentation in and in davcli’s readme.

Aside from this, the security audit has found another interesting issue which has now been addressed: Files created in a vdir were created with the executable bit set. It’s hard to imagine how this could be exploited in practice, but it’s still a valid bug, and has since been fixed.

• I want to keep the scope and code simple, reach a stable state, and stop adding new features.
• I want to support a variety of use cases, like being able to use client TLS certificates, custom http transports, etc.

After thinking about this for along time, I’ve concluded that support for Unix domain sockets is the best compromise to satisfy both goals.

Supported connection transports

[permalink]

Essentially, vdirsyncer will support two transports to talk to a CalDav server:

• Using HTTP(s) with the default route, default connection parameters and the system certificate authorities. This is the typical use case.
• Using a Unix domain socket, suitable for any other use case which requires additional flexibility.

The first transport is straightforward, so I’ll focus on the second one in the rest of this article.

Potential usages of this approach

[permalink]

By having vdirsyncer communicate with a Unix domain socket, traffic can be routed wherever necessary. The socket must be set up by an external process. Here as some examples of what that process could be:

• A local CalDav server which is only exposed via a socket.
• An HTTP proxy that handles client-side TLS certificates.
• An HTTP proxy that uses a custom TLS validation mechanism.
• A special proxy service that re-writes authentication to something non-standard.
• An SSH client forwarding a port to a remote host.
• A proxy that handles TLS with a specific, approved, implementation.

The ability to connect to a local socket becomes an escape hatch for infinite possibilities without having to grow vdirsyncer’s codebase by infinite complexity.

Scope

[permalink]

I believe that this approach keeps a sane scope for vdirsyncer. It doesn’t require implementing all sorts of TLS flags, but allows integrating equivalent functionality.

I find that this approach aligns well with the Unix philosophy of “do one thing”. E.g.: vdirsyncer just does the synchronisation; some external process handles establishing and encrypting connections. I am not choosing this approach because it aligns with the Unix philosophy: I’m choosing it because it allow supporting all kind of special scenarios while minimising the amount of code that vdirsyncer needs to in order to support each scenario.

Personally, I would consider it cleaner design to only support sockets and require that even HTTP(s) connections be handled by a dedicated program. Such an approach would likely be impractical for the typical use cases.

Alternatives

[permalink]

I also considered making vdirsyncer take file descriptors as input on invocation. Vdirsyncer would use these for network communication. This would require complex scaffolding to use, and I fear would quickly become too complex when using a few different storages. That aside, it requires a lot of custom glue code for almost any use case.

I considered using a connection-cmd, a command that is executed to obtain file descriptors for a socket to the target server. This provides similar flexibility as both of the above. This alternative is more unorthodox and doesn’t allow leveraging existing tools as easily (e.g.: you’d require special glue code to communicate to a local server listening on a Unix socket).

Impact on security

[permalink]

If you need to use vdirsyncer with a specific (e.g.: audited) TLS implementation, then you can use that external implementation without vdirsyncer requiring any changes and without needing to link to it.

This also means that I can drop all custom code for custom TLS configurations. Less custom security-related code correlates to less chances of a security bugs.

My solution works as follows:

• I copy into my CLIPBOARD selection the data that I want to transfer.
• From my launcher, I pick an entry to render clipboard as a QR code.
• I scan the QR code from the phone.

This is convenient and globally accessible. It requires not connectivity between the devices; only line of sight.

Implementation

[permalink]

I created the following desktop entry in ~/.local/share/applications/clip2qr.desktop so it shows up in my launcher, fuzzel:

[Desktop Entry]
Version=1.0
Name=QR code for CLIPBOARD selection
TryExec=qrencode
Exec=sh -c "wl-paste | qrencode -s 30 -o - | imv -"
Icon=terminal
Keywords=qr;clipboard;selection
Type=Application
Terminal=false

This will pipe the clipboard into qrencode using wl-paste, and then render the image using imv. I can then scan it on the phone.

My launcher does fuzzy matching, so just typing qr shows my entry shows up as the first result. The whole sequence of keys is Super+d q r Return ¹.

Dependencies

[permalink]

The necessary software can be installed on Alpine Linux with apk add wl-clipboard libqrencode-tools imv.

Alternative Graph

[permalink]

The AltGr (Alternative Graph) key is typically located on the right of the space bar. Some keyboard don’t have an AltGr key but have a second Alt key instead. In this case, the right Alt key can be configured to work as AltGr.

The AltGr key provide quick access a few symbols common in Western European languages. These are some examples that work on my current configuration:

• AltGr + a = á
• AltGr + s = ß
• AltGr + e = é
• AltGr + n = ñ
• AltGr + p = ö
• AltGr + k = œ
• AltGr + 5 = €
• AltGr + Shift + a = Á
• AltGr + Shift + s = §

It is not possible to type the ° symbol nor ǔ with AltGr on an English keyboard.

Dead letters

[permalink]

Dead letters are those that don’t immediately insert a character when pressed. The naming comes from the days of typewriters.

Any keyboard (typically a “US English” one) can be configured to be used with the “US English International” variant, which enables some dead keys.

Dead keys are used in sequences: they change the result of the following key. The , notation in the examples below means “one key after another”:

• ’ = Dead key (doesn’t type anything)
• ’, Space = '
• ’, a = à
• ’, p = ḱ
• ’, p = ṕ
• " = Dead key (doesn’t type anything)
• ", Space = "
• ", u = ü
• ", Shift + u = Ü
• ~ = Dead key (doesn’t type anything)
• ~, Space = ~
• ~, n = ñ
• ~, y = ỹ

This approach makes it easy to write Latin characters, but requires an extra keystroke to type quotes and double quotes. For a Spanish speaker this can be a reasonable compromise, especially if when replacing the keyboard is not feasible.

AltGr is also available in this layout. This provides access to some extra characters (like ß), but also provides redundant access to others:

• AltGr + a = á
• AltGr + s = ß

Dead keys are usually easier to reach than AltGr, which make them easier to use when touch-typing.

Compose

[permalink]

The ⎄ (Compose) key is extremely rare in modern keyboards. Windows does not support the Compose key¹, so most keyboard manufacturers don’t include it.

Those who use the Compose key usually configure some other key to function as Compose, typical candidates being:

• CapsLock
• Menu: Often seen on left side of the right Control key.
• AltGr: Compose provide access to a superset of what AltGr can type.
• Esc: This only really makes sense when some other key is mapped to Esc.
• ScrollLock: Usage of this is extremely rare nowadays.²

Pressing the compose key signals to the computer that the following keystrokes are to insert alternate characters. The keys are pressed one after the other, and not all at once.

• ⎄, ’, a = á
• ⎄, s, s = ß (this is usually thought of as a “double s”)
• ⎄, o, o = °
• ⎄, +, - = ±
• ⎄, v, c = č
• ⎄, _, u = ū
• ⎄, ’, u = ú
• ⎄, `, u = ù
• ⎄, c, u = ǔ (u with charon)
• ⎄, u, u = ŭ (u with breve)
• ⎄, ~, n = ñ
• ⎄, ", _, u = ṻ

On most Unix-like systems, these mappings are defined in /usr/share/X11/locale/en_US.UTF-8/Compose.

The compose key requires learning some sequences and mnemonics, but sequences are consistent. The example given above should be enough to infer how to type ǐ or É.

The compose key allows provides access to all letters from European languages, Chinese tonal marks (for pinyin) and many scientific and mathematical symbols.

Eastern Asian languages (CJK)

[permalink]

Typing Chinese, Japanese or Korean requires using an Input Method. An input method is a program that assists in converting keystrokes into the desired characters. Usually, this includes a pop-up window to select the right characters.

For example, to type 雨果 on an Keyboard, I need to know the pinyin representation of these characters. The pinyin for 雨果 is yǔ gǔo. With a Chinese input method active, I type yuguo and a little pop-up window shows me suggestions that match this:

雨果
雨过
与国
于国
[...]

I can cycle through these with Tab and confirm my selection with Space.

I can switch back to English input by pressing Ctrl + Space.

The general idea is similar to code completion in code editors.

Unicode character points

[permalink]

The above approaches all require reconfiguring the keyboard layout (or running an input method program).

It is also possible to type any character using Unicode code points. On Unix-like operating systems, this is typically done with Ctrl + Shift + u, followed by the hexadecimal Unicode code point. For example:

• Ctrl + Shift + u, e, 1 = á
• Ctrl + Shift + u, 1, f, 4, 3, c = 🐼 (panda emoji)
• Ctrl + Shift + u, 6, 7, 9, c = 果

Clearly this is far less convenient and requires memorising or looking up Unicode character points.

German, Norwegian and other European keyboards

[permalink]

Some country-specific keyboards include extra keys with the extra letters commonly used in the local language. The are convenient for bilingual usage (e.g.: English and German), but are limited when needing to type multiple languages (e.g.: English, German and Spanish).

Arabic, Cyrillic

[permalink]

Any keyboard can be configured to have an Arabic or Cyrillic layout. Each key will insert a single letter. Typing English requires switching to an English layout, so a global key combination is usually used to switch back and forth.

Emoji

[permalink]

I intend to write an emoji-picker input method in future. The intended usage is:

• Press some key combination to enable it (e.g.: ⌘ + ,).
• Type sparkles.
• Input method shows a pop-up with suggestions, in this case: ✨.
• Press the same key combination or Ctrl + Space to switch back.

wlhc is a small program to enable “hot corners” on wayland desktops. It takes a corner and a command as argument. When the mouse (or touchpad) pointer hits that corner of the screen, wlhc executes the provided command.

The implementation uses the wlr-layer-shell protocol to detect when the mouse hits a corner. This is a bit of an abuse of this protocol (i.e.: it’s not really designed for this), but it works.

The whole project is available at https://git.sr.ht/~whynothugo/wlhc and released under the ISC licence. The README contains all the documentation.

Right now there’s only one annoyance: the Makefile hard-codes paths to the wayland protocol files. I think most distributions ship these in the same location, but the build script may require tinkering on distributions with unusual filesystem layouts.

Working on a small project like this in Hare has been a pleasure. Hare doesn’t add any layers over the underlying abstractions, so it’s always quite straightforward to understand what’s going on. The dopamine hits of completing little milestones while advancing on this project have been invaluable in my current state of health.

The main downside of Hare right now is that the ecosystem is still immature, and I had to apply patches (now merged upstream) to both dependencies in order for them to build with the latest compiler. I’ll wait for these to trickle down into stable releases before tagging a release of wlhc itself.

Today I came across vidir, from the moreutils package. It lets me edit a filenames and directories in Vim.

The idea is pretty simple; when I want to edit the filenames inside a directory, I run vidir path/to/directory/. This opens my text editor¹ with a temporary file where each line contains a number and a filename. I can edit filenames, exit Vim, and vidir will rename these files to match my changes. Deleting lines deletes the corresponding files.

Here’s an example (when using vidir photos/Unsorted/:

0001	photos/20120718_002.jpg
0002	photos/IMG_0021.JPEG
0003	photos/IMG_0166.JPEG
0004	photos/IMG_0191.JPEG
0005	photos/IMG_1168.JPEG
0006	photos/IMG_1170.JPEG
0007	photos/IMG_1465.JPEG
0008	photos/IMG_1796.JPEG
0009	photos/IMG_1802.JPEG
0010	photos/IMG_1902.JPEG
0011	photos/IMG_2032.JPG
[...]

I can quickly replace all instances of \.JPEG$ and \.jpg$ with .jpeg using Vim’s replacement command². Once I exit Vim, vidir will rename all these files at once. If I give multiple files the same name, one of them will include a trailing ~, the next one a trailing ~1, and so forth. This prevents unintentionally overwriting files.

The numbers at the beginning of each line are used internally by vidir to match which line corresponds to which original file.

My own idea was pretty similar, but for some reason I had planned on using inode numbers instead of sequential numbers. I now see that it the design could be much simpler.

fd -t d  | xargs vidir  # all files in current directory tree
fd 'py$' | xargs vidir  # all python files in the current directory tree

Last month I mentioned a bug when items are renamed in a collection. After renaming an item, vdirsyncer would check the entire content on both sides during every single execution. I have now fixed this.

I have now renamed all my vcard files to match the name of the person they describe. This lets me use a terminal or file manager to easily visualise files. In the future, I’d like to write on a GUI vcard viewer, so I can just double click on vcard files when using a file manager.

I still haven’t released ab-tidy since it relies on yet-unreleased features of Hare itself. I will tag a release once upstream ships a new release.

Properties

[permalink]

I have implemented synchronisation of properties. For calendars, these are DisplayName, Description, Colour and Order. The latter two are not formally standardised, but are widely supported de-facto standards.

For calendar stored in a vdir, these are supported by tools like khal and todoman. For calendars in CalDav, these properties are supported by a great deal of software, including Nextcloud, DAVx5, iOS’s calendars and reminders app.

Synchronising properties is subject to race conditions. For example, when synchronising, vdirsyncer will read the colour on both storages, and plan to copy the value from one storage onto the other. If another process changed the colour of the target storage in the meantime, that value would be lost. This is really unlikely to happen in practice, and the data loss is minimal (in this example, only the newly applied colour is lost).

This issue is due to limitations of the CalDav protocol, so this is unlikely to change in future. I will simply document it well and move on.

Reading Etags

[permalink]

Previously the Storage abstraction didn’t always return an Etag (because CalDav servers sometimes don’t return an Etag). I have changed this logic to always return an Etag, or an error if a race condition occurred reading it.

This has simplified logic during synchronisation and was required in order to handle renamed items.

Focus

[permalink]

I’ve been focusing a lot on actually completing development milestones rather than attempting to polish everything into a perfect state (nothing is really perfect anyway). This has worked pretty well so far, and having a working version quickly allows some level of polish. As long as no bugs are introduced, I’m fine with keeping this “good enough” until there is an actual need to improve them.

Memory usage is a good example; I’ve been focusing a lot on not using too much memory but vdirsyncer uses about 16MiB of ram when synchronising my collections 3500 items. This could be better, but it’s a bit of an unnecessary improvement at this stage.

Properties for items

[permalink]

I’ve kept properties for items scope for quite some time now, but I am concerned this might have been a mistake. Calendars and contacts don’t have properties for items, they only have properties for collections. This feature was only in scope for potentially synchronising emails in future (where “properties” would basically be “flags”).

This has added some complexity that I’m not certain will ever pay off. Synchronisation of properties currently only works for collections, since dealing with the added complexity of item-properties would have substantially delayed properties for calendars and address books.

Documentation

[permalink]

I have split the user documentation, developer documentation and contributor documentation. This provides clearer starting points for each mode of usage.

The user documentation still needs more work, especially configuring and setting up from scratch.

Vdir safety checks

[permalink]

The vdir (filesystem) storage had a lot of pending FIXME comments regarding security. Most of them were to prevent malicious input from writing elsewhere in the filesystem.

I have since cleared all of those and implemented the appropriate checks.

That’s it for this month. Cheers!

I like to imagine IRC servers as virtual co-working spaces. The kind of co-working space with lots of meeting rooms. Each channel is a one of these meeting room, and anyone can walk in at any time. If you were there first in a room, you can ask others to leave or kick them out. If you reserve a room (e.g.: register a channel), you can have the staff kick out unwanted guests. If somebody misbehaves, the moderators or administrator will throw them out of the building entirely.

IRC isn’t end to end encrypted, but it’s encrypted at flight. This is in the same vein that meeting rooms are private, but you trust that the owner hasn’t installed hidden microphones inside the walls. If you need tighter security, you want your own building (e.g.: your own IRC server).

You can also be sure that only people in the room can hear your conversations. If someone walks in, they can’t suddenly know what was spoken before they walked in. If someone is keeping logs, they might leak what you said, just like someone having a recorder in the coat pocket can record anything that happens in the room.

For a public or semi-public meeting point of an open source project, or open community, all this is exactly what’s needed; no more, no less. The protocol is well-established, lightweight and standardised. And there exist plenty of client implementations for a huge variety of devices and platforms.

A core dump can later be used with tools such as gdb to generate a backtrace of the program and debug the root cause of the crash.

Relevant runtime parameters

[permalink]

The RLIMIT_CORE resource limit defines a maximum size for core dump. Like most resource limits, it has a soft limit (which can be adjusted by an unprivileged user) and a hard limit (the maximum value to which an unprivileged user can adjust the soft limit). These can be queried via ulimit -Hc and ulimit -c respectively. ulimit is a shell built-in, consult your shell’s documentation for more details¹. The default value is zero, which means that no core dumps are generated.

The kernel.core_pattern kernel attribute defines where core dumps will be stored. If the first character of this pattern is |, then the rest of the attribute is interpreted as the path to an program which will be executed with the core dump piped into its stdin. The default value is core, so core dumps would be generated into a file named core in the process’s working directory.

If the value of kernel.core_pattern does not start with a pipe, the core dump is only saved if the terminated process had permission to write to the resulting path. However, if the pattern does start with a pipe, the new process started by the kernel runs with elevated privileges (e.g.: as root).

Saving core dumps

[permalink]

I take the simplest approach, saving all core dumps into a common directory. I want any user to be able to write dumps, but only members of the wheel group can read dumps, and only the ones that they own.

mkdir /var/coredump/
chown root:wheel /var/coredump
chmod 1775 /var/coredump

The 1 in the above permissions is the sticky bit: only the owner of a file within the directory can delete or rename it. This setup allows any local user to fill the disk completely by forking and crashing continuously.

Next, I’ll set the kernel.core_pattern attribute (this can be made permanent via a configuration file in /etc/sysctl.d):

sysctl -w kernel.core_pattern='/var/coredump/%t_%P_%u_%s_%f'

From the above pattern:

• %t: UNIX time of dump
• %P: Global PID
• %u: Global UID
• %s: Signal number
• %f: Executable filename

I initially tried %E (executable path) instead of %f first. It saves the full path to the executable replacing / with !. I didn’t find the full path of any use, so opted for the simpler variation.

Testing the setup

[permalink]

First, I’ll create a quick helper that segfaults right away. C makes this easier than any other language!

int main(int argc, char *argv[]) {
	int *i = 0;
	*i = 42;
}

I’ll then raise the core size resource limit:

ulimit -c 256

And then run my test program that segfaults:

cc segfault.c -o segfault
./segfault
doas ./segfault

I now see both core dumps in /var/coredump²:

.rw------- 201k hugo hugo 2024-06-24 13:07 1719227265_11806_1000_11_segfault
.rw------- 201k root root 2024-06-24 13:07 1719227268_11866_0_11_segfault

• 1719227265 is the timestamp.
• 11806 is the PID.
• 1000 is the UID.
• 11 is the signal number
• segfault is the name of my test program.

I also confirm that permissions are as expected:

> file 1719227265_11806_1000_11_segfault 1719227268_11866_0_11_segfault
1719227265_11806_1000_11_segfault: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from './segfault'
1719227268_11866_0_11_segfault:    regular file, no read permission

I can also debug this dump by pointing gdb to the original binary and the coredump itself:

gdb ./segfault /var/coredump/1719227677_18833_1000_11_segfault

Portability

[permalink]

• RLIMIT_CORE is standard and exists in the BSDs as well.
• kernel.core_pattern seems to be Linux-specific.

Piping to a dedicated handler

[permalink]

A more sophisticated setup could pipe to a dedicated handler. This handler could extract additional process information, drop privileges, and save the dump along with this extra information into a location where it has exclusive write permissions.

For saving the metadata, a .info file seems like the obvious choice, but one could also use extended file attributes³. Extended attributes would ensure that if a coredump is deleted, no orphan metadata is left behind.

For my use-case, I don’t really need any more information. Even the filename patterns described above already have more data than I need.

The design is not ideal, and also drags in a few limitations:

1. The whole user session needs to retain capabilities to perform privilege escalation.
2. They don’t work when running an entire user session in a restricted user namespace.
3. setuid binaries introduce limitations on how the whole system is secured.

An interesting alternative with a is s6-sudod, which splits the program into two parts: a privileged server and an unprivileged client.

This is a summary of an experiment from a few weeks ago where I experimented with using ssh locally to perform the same role as sudo, without exposing this sshd instance to the network.

Goals

[permalink]

• Enable authorised users (and only authorised users) to run commands as root.
• Don’t use privilege escalation.

Implementation

[permalink]

First, I configured a dedicated SSH key that will be authorised for authentication as root. This key is not in the regular authorized_keys file, but in a separate file which will only be used for this purpose:

mkdir /root/.ssh/
echo ssh-ed25519 AAAAC3Nza... > /root/.ssh/local_keys

I then ran an sshd server instance bound to a unix domain socket. Permissions are tightened so unauthorised users cannot even access the socket. This instance overrides the PermitRootLogin option to enable logging in as root¹ and uses the newly created /root/.ssh/local_keys as a source for authorised keys:

mkdir /run/sshd/
chown root:wheel /run/sshd/
chmod 750 /run/sshd/
s6-ipcserver /run/sshd/sshd.sock sshd -ie -o AuthorizedKeysFile=/root/.ssh/local_keys -o PermitRootLogin=yes

The root account was locked to disallow logging in via any mechanism. This was done by prefixing the password’s hash with ! (so no password’s hash can ever match this value). sshd interprets this special prefix as the account being locked and won’t allow logging in as root.

I changed the root password in /etc/passwd and replaced the ! with an *. sshd won’t give new value any special interpretation, and will allow logging in as root. The value * will never match the hash of any password either, so logging in via password remains effectively disabled.²

I then needed to connect to the local sshd instance. While sshd has a -i flag which allows passing an existing socket to it, ssh has no equivalent flag. The ProxyCommand option can be (ab)used for this:

ssh -o ProxyCommand='socat STDIO UNIX-CONNECT:/run/sshd/sshd.sock' \
    -i .ssh/root-key.pub \
    -t \
    root@root \
    "cd $(pwd); '$SHELL' --login"

I’m using a hardware-bound SSH key in this case, which means that I need to tap the physical device to authorise this connection. I could also use an ssh-agent that requires explicit approval before disclosing keys (e.g.: hissh-agent).

A little caveat here is that socat will read all input from ssh, and then write it into the socket, effectively duplicating the overhead of the connection. I read the relevant manual pages a few more times, and couldn’t find a solution. I came across ProxyUseFdpass, but wasn’t entirely sure how to make it work.

After some more research online (and some major frustration), I found a clear usage example from 2016. It turnes out that ProxyUseFdpass was quite straightforward and allows me to specify a command that sends the socket file descriptor (via stdout) to ssh, and ssh then connects over this socket.

I saved the following script into /home/hugo/tmp/passfd.py:

#!/usr/bin/env python3
# From: https://www.gabriel.urdhr.fr/2016/08/07/openssh-proxyusefdpass/

import sys
import socket
import array

# Create the file descriptor:
s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
s.connect("/run/sshd/sshd.sock")

# Pass the file descriptor:
fds = array.array("i", [s.fileno()])
ancdata = [(socket.SOL_SOCKET, socket.SCM_RIGHTS, fds)]
socket.socket(fileno = 1).sendmsg([b'\0'], ancdata)

And the command to connect to ssh now becomes:

ssh -o ProxyCommand='/home/hugo/tmp/passfd.py' \
    -i .ssh/root-key.pub \
    -o ProxyUseFdpass=yes \
    -t \
    root@root \
    "cd $(pwd); '$SHELL' --login"

The linked article mentioned using nc (for a somewhat different use case). Initially, it would seem that nc -FU /run/sshd/sshd.sock would work, but the manual page actually specifies that this is not supported:

-F		Pass the first connected socket using sendmsg(2) to stdout and exit. This
		is useful in conjunction with -X to have nc perform connection setup
		with a proxy but then leave the rest of the connection to another
		program (e.g. ssh(1) using the ssh_config(5) ProxyUseFdpass option).
		Cannot be used with -c or -U.

Conclusion

[permalink]

This technique works. It relies mainly on OpenSSH for all the sensitive security details. Not only does OpenSSH have a great track record, but it also enables various forms of authentication including using a hardware-based SSH key.

Configuring this on a new host has no complex steps, and the above ipcserver command can just be executed via the system’s service manager.

The above passfd.py script is a quick hack to move the experiment forward; for daily usage it would be best to write a tiny executable that does the same thing and put it into /usr/local/bin. The whole ssh command could also be placed in a tiny wrapper.

vdirsyncer alpha release

[permalink]

I have finally tagged an alpha release of vdirsyncer 2. This version is stable enough that I’ve been using it for over a month now no issues. I generally keep vdirsyncer daemon running in background my calendars & contacts simply synchronise across devices continuously.

While some features are missing, none of the ones that I use are missing, so it might be suitable for you as well. I’d appreciate feedback from testers, but please do make a full backups of your data just in case!

In hindsight, the code quality is much better than what I’d usually call alpha or beta quality, but I tagged it as alpha as an indicator that user-facing aspects of program may well change before the final release (in fact, some command line arguments have already changed since then).

Caveats on daemon mode

[permalink]

For now, the daemon mode simply synchronises at a given interval. I intend to work on implementing storage monitoring in future, so as to trigger a synchronisation only when a storage indicates that an item has changed. The “fixed interval” approach is only a temporarily solution until storage monitoring is fully ironed out.

ab-tidy

[permalink]

ab-tidy is a tiny tool to tidy up an address book. It operates on a directory where each contact is an individual vcard file (e.g.: a “vdir”) and renames each file to match the name of the contact inside of it.

After running it, the directory is easy to browse with a file manager, terminal, or even ls.

I wrote this in Hare in one afternoon, and I have to say that writing Hare has been a pleasant (and fun) experience so far. The standard library provides abstractions right on top of the primitives that the operating system offers. The standard library is also readable and approachable code, even for a Hare newbie like myself. The layer at which the it provides abstractions are not too far from C itself, but the error handling is lovely (it uses tagged unions much like Rust, Zig and other recent languages).

This tool also helped me find another bug in vdirsyncer: when a file is renamed locally, the status database isn’t updated properly. As a result, the file get read from both storages each time a synchronisation occurs. The end result is the exact same, but it adds unnecessary network traffic on each run.

darkman v2.0.0

[permalink]

Another release that has been delayed more than necessary.

The main breaking change is that geoclue is no longer used by default. Geoclue automatically communicates external services to determine the current location by default, so I want users to explicitly opt into it to ensure that here is clear consent of what is going on. Geoclue also requires some manual configuration to work properly on distributions, so for many use cases, just putting latitude and longitude manually into the configuration file is good enough.

See the changelog for a full list of all changes.

Resuming from hibernation from this partition doesn’t entirely work. Alpine’s initfs can’t re-use the same passphrase to unlock more than one partition. Well, it can, but I’d be prompted to type the same twice password every time the system boots. LVM was also an option, but that seems like one layer of complexity too many for such a simple setup.

I eventually set up hibernation to use a swapfile inside the main partition. But the original swap partition is still there, idle and unused.

Having run out of space yesterday trying to compile the main/rust package, I decided that it’s time to recover this idle space¹.

Overview

[permalink]

My disk layout is as follows:

# fdisk -l
Found valid GPT with protective MBR; using GPT

Disk /dev/nvme0n1: 500118192 sectors, 2534M
Logical sector size: 512
Disk identifier (GUID): XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
Partition table holds up to 128 entries
First usable sector is 2048, last usable sector is 500118158

Number  Start (sector)    End (sector)  Size Name
     1            2048         1230847  600M
     2         1230848        34785279 16.0G
     3        34785280       500117503  221G

I put the swap first, because that saved me doing the math (a single subtraction) required to put it at the end. Removing the swap partition is easy, but growing the main system partition to the left is a bit more work.

Deleting the swap partition

[permalink]

I delete the swap partition with:

sfdisk --delete /dev/nvme0n1 2

It is effectively gone:

# fdisk -l /dev/nvme0n1
Found valid GPT with protective MBR; using GPT

Disk /dev/nvme0n1: 500118192 sectors, 2534M
Logical sector size: 512
Disk identifier (GUID): XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
Partition table holds up to 128 entries
First usable sector is 2048, last usable sector is 500118158

Number  Start (sector)    End (sector)  Size Name
     1            2048         1230847  600M
     3        34785280       500117503  221G

It low-key bothers me that now partitions are 1 and 3 with no 2. Hopefully I can fix this later.

Backup

[permalink]

This whole operation can potentially destroy all data on my disk. Skipping a backup would tempt the fates to actually make this whole operation fail.

I make a backup of the whole disk by just dumping an image of it into an external drive:

1. Reboot.
2. Choose to boot from an Alpine bootable USB drive.
3. Log in as root.
4. Plug in USB drive.
5. modprobe ext4 && mount /dev/sdb1 /mnt
6. dd if=/dev/nvme0n1 of=/mnt/dump_2024-05-27.img bs=512

Moving the partition to the left

[permalink]

This isn’t as trivial because the partition that I’m moving is my main partition and it needs to be unmounted before an operation like this. I’ll use a gparted bootable USB drive for this.

After downloading gparted, I flash it onto a USB drive using:

dd if=gparted-live-1.6.0-3-amd64.iso of=/dev/sda bs=512

I reboot the laptop again and boot into this USB drive. The touchpad seems to work for moving around the cursor, but not for clicking. The easiest solution is to just grab a wired USB mouse, which works right away.

I right click on the main partition, and choose “Resize/Move”. I type 999999 on the “New size” field and it automatically sets it to occupy all available space.

I expected this operation to take a long time since every single byte in this partition needs to be moved leftwards. It took 44 minutes.

Once gparted has done its job, I proceed to grow the filesystem inside the partition. First, I unlock the encrypted partition with:

cryptsetup luksOpen /dev/nvme0n1p3 root

The btrfs tool operates on a mounted filesystem, so I mount my filesystem and then resize it.

mount -t btrfs /dev/mapper/root /mnt
btrfs filesystem resize max /mnt/

It is instantaneous. Nice.

I can not unmount the filesystem:

umount /mnt

Fixing the partition number

[permalink]

My disk now has partitions 1 and 3. I want to rename the latter to 2.

This is one by deleting the partition from the partition table and creating a new one with the exact same size. This operation doesn’t touch the actual partition itself at all, only its definition in the partition table.

First, I print the current details for the partition, since I’ll need to replicate those:

# fdisk -l
Found valid GPT with protective MBR; using GPT

Disk /dev/nvme0n1: 500118192 sectors, 2534M
Logical sector size: 512
Disk identifier (GUID): XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
Partition table holds up to 128 entries
First usable sector is 2048, last usable sector is 500118158

Number  Start (sector)    End (sector)  Size Name
     1            2048         1230847  600M
     3         1230848       500117503  237G

I then delete and recreate the partition as number 2:

sfdisk --delete /dev/nvme0n1 3
echo '1230848' | sfdisk --append /dev/nvme0n1

I only specify the start sector for the new partition, and the default uses all available space (which matches the previously existing one). Because only partition 1 exist, the new partition is assigned number 2. The final result look as expected:

# fdisk -l
Found valid GPT with protective MBR; using GPT

Disk /dev/nvme0n1: 500118192 sectors, 2534M
Logical sector size: 512
Disk identifier (GUID): XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
Partition table holds up to 128 entries
First usable sector is 2048, last usable sector is 500118158

Number  Start (sector)    End (sector)  Size Name
     1            2048         1230847  600M
     2         1230848       500117503  237G

I reboot the system, provide my passphrase and… it works! I have effectively reclaimed those 16GiB.

The xdg-desktop-portal is a kitchen-sink service¹, which implements dozens of unrelated functionalities mashed together. Some of these features are implemented directly in the portal, while others are delegated to portal implementations.

One of the features implemented by the xdg-desktop-portal is the settings interface. This interface exposes settings to local user applications. Amongst a few other values, it exposes a color-scheme preference, which can be either dark mode, light mode, or “no preference”.

However, the xdg-desktop-portal itself isn’t the source of this information; it queries portal implementations for the correct value. Which portal implementation it uses requires careful configuration, and this is the main source of confusion.

There are several ways to configure which portal implementation is used.

Configuration files can exist in system-wide locations, or user-specific locations. I’ll only cover user-specific locations here; if you need to configure something system-wide for multiple users, consult man 5 portals.conf for additional details.

Depending on the filename given to the configuration file, it will apply only to specific desktop environments (e.g.: kde-portals.conf would only apply on KDE) or apply to any environment (e.g.: portals.conf will be used by default on any desktop environment unless a desktop-specific configuration is found)

I prefer using portals.conf, since my user account always runs sway and I don’t need per-DE functionality for it. If you use your same user account with different compositors or DEs, you’ll need to take the other approach.

A single configuration file

[permalink]

The most straightforward location for a user configuration is ~/.config/xdg-desktop-portal/portals.conf, which overrides other directories.

To use darkman as a source for settings (which results in its dark / light mode value being exposed), the following configuration would suffice:

[preferred]
org.freedesktop.impl.portal.Settings=darkman

In my case, I also configure the xdg-desktop-portal to use xdp-wlr for screen casting (this in turn is used by Firefox). My real configuration file currently looks like this:

[preferred]
org.freedesktop.impl.portal.ScreenCast=wlr
org.freedesktop.impl.portal.Settings=darkman

If you are using a distribution with a pre-configured desktop environment, when you create a configuration file it will completely override the system-wide configuration. You may need to copy-paste any lines from the system-wide configuration file into your own to retain existing integrations.

Per-DE configuration files

[permalink]

I mentioned before that it is possible to create per-DE configuration files.

For example, if you need to support both sway and KDE, you’ll need two configuration files²:

• ~/.config/xdg-desktop-portal/sway-portals.conf
• ~/.config/xdg-desktop-portal/kde-portals.conf

Creating the files alone isn’t enough: the xdg-desktop-portal needs to know which desktop you are currently running. This is configured via the XDG_CURRENT_DESKTOP environment variable, which must be set for the xdg-desktop-portal process itself. If you are using a service manager, you need to inject this variable into the environment that it creates for services. If you are starting the service manually, you can just export it with the usual mechanisms.

For the above two example, this variable should be defined as XDG_CURRENT_DESKTOP=sway or XDG_CURRENT_DESKTOP=kde respectively.

Darkman and XDG_CURRENT_DESKTOP

[permalink]

Darkman doesn’t care about the value of XDG_CURRENT_DESKTOP. It will expose its portal implementation bus in org.freedesktop.impl.portal.desktop.darkman. When the xdg-desktop-portal is configured with org.freedesktop.impl.portal.Settings=darkman, it will query darkman’s bus for settings.

Darkman will only respond to queries for the color-scheme preference.

Start-up order

[permalink]

The XDG_CURRENT_DESKTOP variable needs to be defined in the execution environment of xdg-desktop-portal. If it starts during early session initialisation, ensure that the ordering is as expected.

The xdg-desktop-portal will try to communicate with configured portal implementations (e.g.: darkman) as soon as it starts.

When dbus-daemon is configured to auto-start services (this is the default), then it will start darkman immediately.

If you are using a service manager to start these service, you should consider darkman a dependency of xdg-desktop-portal. In this case, darkman must start and be ready before the xdg-desktop-portal starts.

As I mentioned before, when synchronising two storages, the sync algorithm keeps around a local “status” with some basic metadata. Future executions use this metadata to understand which side has changed and which side needs updating.

I had to re-write most of my previous status implementation due to the issues that I mentioned in my previous status update. The new implementation keeps track of the metadata for the latest synchronised version, instead of the latest seen version. I’ve also made sure that all operations are atomic and race-free. Atomicity ensures that if the process is ever interrupted, the status never ends up in a partially updated state. Race-free queries ensure that if another process applied some other operation concurrently, the current one will bail rather than overwrite conflicting information.

Sync errors

[permalink]

I also split synchronisation errors into two groups: non-fatal and fatal.

Non-fatal errors imply that the whole operation can continue, leaving just the failing item out of sync. Some examples of non-fatal errors are transient network error, or a server rejecting a calendar element for some server-specific reason.

Fatal errors imply that the entire operation must abort. Fatal errors occur mainly when writing to the status file fails for some unexpected reason.

Error reporting during sync

[permalink]

I need to provide a mechanism to report non-fatal errors to the end user. My current scope includes a command line tool, but an important goal of the sync library is that it has to provide enough functionality to build a proper GUI as well.

Initially, I tried accumulating non-fatal errors and later returning them together. This became rather entangled as fatal errors can interrupt the flow in various code paths, and the non-fatal errors need taking into consideration in each place. It also means that output would not be available until after the entire process completes.

I settled on a on_error(SyncError) parameter for the Plan::execute method. This parameter is a function that gets called each time an error happens. The error is not a plain string, but actually a proper error type with all the details necessary to explain to the user what went wrong.

Vdirsyncer itself passes an on_error function that merely logs the errors.

Collection auto-creation

[permalink]

I sometimes create a new todo list on another device (or on my phone), and I want these to replicate automatically without reconfiguring vdirsyncer.

When synchronising two storages, my new implementation will replicate new collections (e.g.: calendars, address books) to the other side too. It will also delete from one side a collection that was deleted on the other.

I later intend to make this feature optional; I have no doubt that many users would prefer no auto-creating or auto-deletion. Disabling this feature would result in the same behaviour as the previous vdirsyncer.

I also intend to add a feature to prevent emptying a collection if it was emptied on the other side. Again, this is a safety measure, and also exists in the previous vdirsyncer.

Conflict resolution

[permalink]

I thought long and hard about conflict resolution, and tried a couple of prototypes. Most dedicated APIs end up having too much complexity and various limitations.

After enough thought, I settled on a simpler approach.

The module that handles synchronisation, vstorage::sync creates a plan. This plan has all the individual actions that need to happen to synchronise the storages. It also includes details about the items that are in conflict (although no explicit action is actually taken for these).

The details for in-conflict items are enough to manually resolve the conflict. E.g.: they include the path to the item on both sides as well as their UID and Etag.

Instead of using a dedicated API for “conflict resolution” provided by this library, consumers can read details on conflicting items from the plan, and resolve the conflict externally by updating one or both storages with the desired final version of the item. Most of the work involved here was ensuring that enough information is made available for consumers to resolve conflicts in a race-free way (e.g.: by exposing the current Etags).

Vdirsyncer is built on top of this library, so it will follow these steps when resolving conflicts:

• Prepare a plan by using vstorage::sync.
• Inspect all items that are in conflict.
• If the conflict resolution strategy is keep a or keep b, vdirsyncer replace the “conflict” actions with CopyToB or CopyToA respectively.
• If the conflict resolution is a custom command, vdirsyncer will download both sides of the conflicting item and call the custom command. If the command exits with a success exit code, vdirsyncer will update both storages with the returned content (by using the Storage::update_item method.
  • In the latter case, during the next synchronisation the vdirsyncer will identify that the item is in sync again, resolving the conflict.

Providing enough information to resolve conflicts manually puts little burden in implementing conflict resolution while maximising flexibility.

This also means that conflict resolution can continue in the foreground (potentially, with user interaction) while continuing synchronisation in the background.

In the coming weeks, I will continue working on the configuration aspect of conflict resolution (e.g.: reading the configured command and executing it). Unless any new issues pop up, I intend to publishing an alpha release after this.

Current state

[permalink]

I have used this version of vdirsyncer myself for these last several months, and feel quite pleased with two large improvements over the previous version:

• If any single item fails for any reason, all others continue to synchronise.
• I can run it in the background and it keeps things in sync without need for restarts or prompts for credentials.

On the downside, the lack of conflict resolution soon became an issue.

s6-rc can be used to manage system service, user-session service, or an ad-hoc collection of services. s6 is equally flexible and can be used for any sort of service.

The official documentation does a good job of explaining every component in great details. This is a high level overview.

s6

[permalink]

s6 is a small suite of programs to supervise services (or really any other type of process). Understanding this suite first helps understand s6-rc later on.

s6-log

[permalink]

s6-log reads input from stdin and saves logs, handling storage and rotation.

Log rotation needs to be handled by the same process that writes logs; any separation between processes eventually leads to race conditions, requiring inter-process locks or other similar issues.

Generally, another service is executed piping its output to s6-log. Service processes themselves don’t ever need access to the log files.

Like most tools in this list, it can be used completely independently of the s6 suite. This implies, amongst other things, that it is easy to test s6-log by itself to get a better feeling of how it works.

The easiest usage of s6-log is to pipe output from a process to it:

offlineimap 2>&1 | s6-log T /path/to/logs/offlineimap/

s6-log takes an output directory, given that it will handles log rotation inside that of it. It also keeps a lock to avoid concurrent instances writing to the same directory.

s6-log focuses on ensuring that all logs are saved and no messages are lost. This causes frequent writes to disk and may not be ideal when the primary storage is an SD card or some other device that wears out quickly with frequent writes. Consider using logbookd for scenario which require reduced writes and losing logs in an acceptable compromise.

s6-supervise

[permalink]

s6-supervise is the service supervisor program. It runs as direct parent of a service process. It handles restarting the service when it dies, terminating it (when explicitly requested), and notifies registered processed when it is up or down.

The definition of a single service is represented by a directory called a service directory. For advanced configurations, it is possible to programmatically generate these directories. If you’re curious what these look like, I have a collection of service directories that I use for my desktop sessions.

A running s6-supervise instance can be manually controlled via s6-svc.

Much like s6-log, s6-supervise (together with s6-svc) can be used independently of the rest of the s6 suite. It is perfectly feasible to write or adapt some other service manager and use s6-supervise for service supervision without using any other pieces of the s6.

s6-svscan

[permalink]

s6-svscan reads a scan directory; a directory with multiple service directories. It then runs an instance of s6-supervise for each one. s6-svscan is often the root of the service process tree.

s6-svscan can be manually controlled via s6-svscanctl.

s6-rc

[permalink]

s6-rc is a service manager. It is a collection of programas that builds on top of s6 and implements dependency management, bundles (these are similar to OpenRC runlevels) and one-shot services.

While s6-rc sounds like the obvious choice to use on top of s6, it is perfectly feasible to use a separate service manager which uses s6 for service supervision under the hood. An example of this is anopa.

s6-rc doesn’t actually run the services itself; it delegates this to s6-svscan under the hood. s6-rc merely mutates the scan directory, and makes s6-svccan reload, start, or stop services as needed. It also uses its own “live” directory for its runtime data and state.

s6-rc-init initialises the scan directory with all services disabled by default. Further state changes (including starting up an initial set of services) can be triggered via the s6-rc command.

s6-rc keeps its definitions in a compiled database, which must first be generated using s6-rc-compile. The compiled database is then re-usable, so it is possible to keep versioned copies, as well as re-use the last-compiled database after system reboots.

While compiling a database initially sounded like a tedious step at first, the compilation step helps ensure that the directory used by s6-rc contains valid data before it is used. For example, when I accidentally added a non-existent service as a dependency for another, compilation failed. This ensures that the source directory can’t have completely bogus data (which could yield terrible results after a reboot). It also prevents re-doing all the parsing work on each start-up: it is done once by an individual tool and the pre-digested data is loaded in subsequent runs.

I find this last trait an unusual approach, but it is a solid design choice.

Caveat: Note that the $SERVICENAME/log pattern supported by s6-svscan is not supported by s6-rc. Loggers must be declared via provider_for and consumer_for.

Process tree

[permalink]

s6-svscan is the parent of all s6-supervise processes, and s6-supervise process is the parent of a different service. This results in a process tree that is intuitively quite intuitive to understand (ps auxf will even render this in a way that can be visually comprehended).

Readiness notification

[permalink]

Services supervised by s6-supervise can implement service startup notifications, often referred to as readiness notification. This allows services to notify their supervisor when they are ready. While the exact definition of ready varies for each service, this usually indicates that the service is ready to accept client connections. For example, a service that sets up a unix domain socket on which it listens to connections will only send a readiness notification after the socket is ready to receive client connections.

Readiness notification allows ordering dependant service and having each one start only when their dependency is ready. For example, darkman will connect to dbus as soon as it starts, so darkman needs to be started after dbus is ready. Starting darkman after dbus has merely started will inevitably lead to race conditions where darkman tries to connect to dbus before the latter has set up its socket.

Usage via wrappers

[permalink]

Personally, I find that using the s6 utilities directly tends be too low level. Most of my usage so far has been using own wrapper scripts. I expect that other high-level tools will show up over time, and the components mentioned above will likely not be used directly by most users.

See also

[permalink]

• As mentioned above: the official documentation is well detailed, albeit a bit terse and it takes a while to figure out how to navigate it properly.
• The same documentation is available (from a third party) as man pages. These are also available in the alpine repositories as s6-man-pages and s6-rc-man-pages. I do wish that upstream would focus on man pages first, and then use those as a base for the HTML pages.
• The gentoo wiki has an s6 article with further information.

There are numerous additional utilities in the s6 and s6-rc suites. It’s worth getting to know the basics of most of them.

Locking the system before going to sleep ensures that the system never shows an unlocked desktop when waking up from sleep. If the screen locker fails to start for some reason, this fact becomes clear sooner rather than later.

Going back to sleep automatically prevents the system from remaining active if woken up by accident. For example, if someone (some person, or some cat) presses any key during the night the system wakes up. I don’t want it to remain awake indefinitely in such situations.

When the system wakes up, it won’t always reset the idle timer. If I press the space-bar when the system is asleep, this key-press event is “swallowed” by the firmware (which wakes up the systems) and never delivered to the operating system or the compositor. The idle timer continues running since it originally started, and the compositor never send an event indicating that the system has resumed from idle.

The script

[permalink]

This script also exists in my dotfiles repository.

#!/usr/bin/env python3
#
# Requires swaylock > 1.7.2.
import os
import sys
import subprocess

(r, w) = os.pipe()
os.set_inheritable(r, False)
swaylock = subprocess.Popen(["swaylock", "--ready-fd", str(w)], pass_fds=(w,))
os.close(w)

# Wait for swaylock to be ready.
with os.fdopen(r, "rb") as ready_pipe:
    while True:
        pipe_data = ready_pipe.read(1)
        if pipe_data == b"":
            print("fatal: swaylock closed fd before locking")
            sys.exit(1)
        if pipe_data == b"\n":
            break

while True:
    subprocess.call(["powerctl", "mem"])  # blocks until wakeup from sleep.

    wayidle = subprocess.Popen(["wayidle", "--timeout", "10"])

    (pid, returncode) = os.wait()
    if pid == swaylock.pid:
        wayidle.kill()
        sys.exit(returncode)
    if pid == wayidle.pid and returncode != 0:
        print("fatal: wayidle exited non-zero")
        sys.exit(returncode)  # swaylock will continue running

Let’s go over it bit by bit.

swaylock > 1.7.2

[permalink]

This script requires swaylock > 1.7.2. It relies on swaylock’s readiness notification, which indicates once swaylock has finished locking the screen.

With previous releases, there exists no trivial way to do this and wait for swaylock to exit (more on this below).

Running swaylock

[permalink]

First comes the initial block which runs swaylock:

(r, w) = os.pipe()
os.set_inheritable(r, False)
swaylock = subprocess.Popen(["swaylock", "--ready-fd", str(w)], pass_fds=(w,))
os.close(w)

os.pipe() creates a pipe, a pair of file descriptors, w and r. Any data written to w gets read via r. After closing all the copies of w, r will yield an EOF (end of file).

The child process (swaylock) inherits a copy of w, and the Python script keeps the original copy. If the Python script keeps its the copy of the w descriptor open, then the r one will never reach its end. The script explicitly closes it with the os.close(w) call.

When swaylock writes to on one of these file descriptors (w), this Python script will read from the other one (r).

swaylock will then start up, and as soon as it locks the system, it will write \n into w and then close it.

Waiting for the system to lock

[permalink]

The next section now becomes relevant:

with os.fdopen(r, "rb") as ready_pipe:
    while True:
        pipe_data = ready_pipe.read(1)
        if pipe_data == b"":
            print("fatal: swaylock closed fd before locking")
            sys.exit(1)
        if pipe_data == b"\n":
            break

This reads from r in an infinite loop (while True). If swaylock writes a \n, this indicates that locking the screen has finished. The loop breaks and the program continues executing in the next section.

If swaylock does not print a \n, this indicates that it has exited before locking the screen. If this happens, the script exits with an error. I avoid putting the system to sleep in this case, since that would hide the fact that something went wrong with the screen locker.

The main loop

[permalink]

Lastly, the “main loop”. Again, an infinite loop:

while True:
    subprocess.call(["powerctl", "mem"])  # blocks until wakeup from sleep.

    wayidle = subprocess.Popen(["wayidle", "--timeout", "10"])

    (pid, returncode) = os.wait()
    if pid == swaylock.pid:
        wayidle.kill()
        sys.exit(returncode)
    if pid == wayidle.pid and returncode != 0:
        print("fatal: wayidle exited non-zero")
        sys.exit(returncode)  # swaylock will continue running

This starts off by calling powerctl mem, which puts the system to sleep. This command exits after the system has resumed from sleep. This guarantees that the next line of this script executes after the system has woken up again.

At this point, wayidle runs. wayidle sets an idle timer (for 10 seconds in this case) and exits once the timer expires. I tried using swayidle here, but it didn’t quite fit, because swayidle doesn’t know when the system wakes up again, so it won’t run another timer and the system will remain awake.

The script will then wait for one of its children (wayidle or swaylock) to exit.

If swaylock exits first, then it has unlocked the system (or crashed). In this case wayidle gets explicitly killed, and the script exits (with the status code of swaylock, to reflect any errors).

If wayidle exits first and without any errors, this means that the system reached the idle timeout. The loop starts over, executing powerctl mem and putting the system back to sleep.

If wayidle exits first but with an error, then the script exits with the same error. This shouldn’t happen, and implies an unexpected error with the idle timeout. The systems stays awake; if it went to sleep again, then it could end up in an irrecoverable sleep-loop.

Closing notes

[permalink]

I configured sway to run this script whenever I tap the power button on my computer:

bindsym --release XF86PowerOff exec lock-and-sleep

I don’t think I can make this script any simpler, and I welcome any feedback on it.

Feel free to re-use any bits or ideas to your liking.

Vdirsyncer keeps a “status” file locally. The status file contains metadata from items on both storages the last time they were synchronised. My current implementation runs the synchronisation process, and then returns a new status, which should be saved to disk until next time.

I decided on this API since it made the whole codebase easier to reuse in different scenarios, including situation where the local system has no persistent storage (and the status could be saved onto a storage server, or wherever feasible).

This idea turned out to be problematic.

Handling interruptions

[permalink]

If synchronisation suffers a fatal interruption (e.g.: power failure, SIGKILL, etc), none of the updated status is saved, which could result in problems during the next synchronisation. Specifically, if content changes on either side before another synchronisation, it would result in conflict. Deleted content might be mistakenly restored on the other side. While annoying, I don’t think it can ever result in data being deleted.

I initially decided to leave this in its current state for the alpha0 release, and to address it before a stable beta release.

Regrettably, this wasn’t the only issue with the status management.

Recovery after one-time failures

[permalink]

Due to a transient network issue, a single synchronisation failed mid-way (it had plenty of operations pending). Upon retrying, there were no operations to run. Something was wrong.

The problem turned out to be that I was updating the status file with the new metadata even if synchronisation failed. So the next time that synchronisation runs, it looks like the file hasn’t changed, and there is nothing to do. In reality, it hasn’t changed since the last failed synchronisation, but it has changed since the last successful one.

Unlike the problem above, I feel that this one is problematic enough to delay an alpha0 release.

To address I am considering using the previous state as input when synchronising, and only updating status data for items which are successfully synchronised. However, during the first run, there is no previous state, and for items that are identical on both sides, no action needs to be taken. Currently, the status gets updated based on the actions that are executed, so “no action” means that items never end up in the status database.

To work around this last issue, a new action is needed, a no-op in terms of synchronisation that merely saves the item into the status file. This would only really be relevant during the first-time synchronisation, or if some other tool synchronised both storages.

The approach has somewhat worked, but has some issues. The experiment has been a success: it has resulted in useful findings. Xendmail itself, however, is likely not the way forward.

To summarise, my goals with Xendmail were:

• Provide a sendmail-compatible interface for any application to send messages.
  • This cancels the need to configure email credentials repeatedly in every other application.
• SMTP credentials are not exposed or resting in plain-text.
• No emails are accidentally dispatched without being aware of it.
• Enable non-root users to configure the setup, so it is feasible on shared hosts.

These goal were well met, but additional requirements should have been in scope too…

Problem 1: Not offline mode or outbox

[permalink]

Usually I write an email message (or prepare a git patch) and then hand it over to xendmail, which sends it out immediately. Whenever there is no internet connection available, xendmail outright fails, and I have to figure out what to do with the message myself. I usually save it into drafts and need to remember to manually dispatch it later.

The solution for this issue would be to save messages into a local outbox, and have it dispatched later when I am online again. This is pretty much what old-school monolithic email clients did.

Problem 2: Poor integration with automated emails

[permalink]

Not all emails are prepared manually: Tools like rss-email², prepare emails automatically and attempt to dispatch them.

Xendmail tries to use the secret store to read the SMTP credentials, which results in the secret store prompting for user consent before disclosing credentials. This results in random interruptions during the day.

I have considered an approach where xendmail has some form of privileged access to the secret store and can read serets without my approval, but have decided against the idea. Aside from a lot of complexity issues, any misconfigured or misbehaving client can start sending out unwanted emails without indication.

It is clear that xendmail, in its current form, is a bad fit for this type of automated tools.

Using something like opensmtpd

[permalink]

The idea of putting emails into an queue and dispatching them later when network is available again is nothing new. Classic SMTP servers to exactly this.

There are mainly two obstacles with locally running a classic SMTP server like opensmtpd or alike:

1. They need to be configured and executed by the root user. The whole point of these experiments is to find a solution where each user can configure their own email on a system with no root access (e.g.: on a shared machine).

2. Credential management is still an issue. There are two possible approaches:

   • If the locally running SMTP server requires a password, then automated tools cannot queue messages without unexpected interruptions.
   • If credentials are stored in plain-text form or don’t require user approval before access, then misbehaving processes can send out messages without limitations (and without any awareness of it).

The first of these items (requiring root for configuration and operation) is enough to make this approach non-viable for the given goals.

Future endeavours

[permalink]

Based on these learnings, I’m still thinking of the next approach to try. For the moment, key observations are:

• Try sending an email, and save it into an outbox maildir if sending fails.
• A status bar indicator with the amount of messages in queue.
• A simple UI to review the outbox and send all messages.

I will continue using Xendmail for the time being, but my intent is to eventually deprecate it in favour of one or more tools that can fit this description.

I opted to run soju (mirror) on my personal server running OpenBSD. soju is what powers chat.sr.ht. It is well tested, I know it fits my needs, and has support for connecting to multiple networks. Its bouncer/network support integrates nicely with senpai, a modern irc terminal client.

The OpenBSD port for soju seems rather new, and there doesn’t seem to be a binary package for it in the latest stable release. I opted to upgrade from -stable to -current, which was really as simple as running sysupgrade -s, rebooting, and finally running pkg_add -U.

Installing soju is as simple as running pkg_add soju.

I also needed to create a DNS entry for irc.whynothugo.nl, and point that to the same server.

Since soju uses TLS, I need certificates for it. I added the new domain to /etc/acme-client.conf and then executed acme-client irc.whynothugo.nl:

domain irc.whynothugo.nl {
        domain key "/etc/ssl/private/irc.whynothugo.nl.key"
        domain full chain certificate "/etc/ssl/irc.whynothugo.nl.fullchain.pem"
        sign with letsencrypt
}

To avoid acme-client from messing up permissions for the certificates, I’ll copy them to another location on renewal. My renewal script is then:

#!/bin/sh
set -e
acme-client irc.whynothugo.nl
install -Dm 644 -o _soju /etc/ssl/irc.whynothugo.nl.fullchain.pem /etc/soju/
install -Dm 600 -o _soju /etc/ssl/private/irc.whynothugo.nl.key /etc/soju/
rcctl reload soju

And the configuration file at /etc/soju/config reads:

db sqlite3 /var/soju/main.db
message-store fs /var/soju/logs/
listen ircs://
tls /etc/soju/irc.whynothugo.nl.fullchain.pem /etc/soju/irc.whynothugo.nl.key

As a on-off, I need to manually copy the certificates into the right location, and manually start and enable soju:

# These two lines copied from the above script:
install -Dm 644 -o _soju /etc/ssl/irc.whynothugo.nl.fullchain.pem /etc/soju/
install -Dm 600 -o _soju /etc/ssl/private/irc.whynothugo.nl.key /etc/soju/

rcctl start soju
rcctl enable soju

With soju running, I reconfigured senpai locally and connected to it. In order to connect the bouncer to a new network, I used:

/msg BouncerServ network create -addr irc.libera.chat

It works!

There is a lot of unexplored potential in this feature. For example, it is perfectly feasible to have some desktop-wide special menu with actions that apply on it. This menu could be opened with Ctrl+RightClick, or maybe Ctrl+Alt+M and could have entries such as:

• Show selection in dictionary.
• Read selected text out load.
• Read dictionary definition of selected word out loud.
• Translate selection into another language.
• Save selection into snippets / notes.
• Added 2024-01-16: Render a QR code with the current selection.

Implementing such features is pretty feasible and doesn’t require any new architectural changes or modifications to existing compositors. An early prototype could map a global hotkey directly to an action, for example Super+D to open a given word in a dictionary, or Super+R to read the selection out load with an existing text-to-speech engine.

This idea has been floating in my head for a long time, but I haven’t had the opportunity to experiment with it. Specifically, I am not familiar with any desktop dictionary or desktop translation application which I can be executed programmatically. E.g.: it should be possible for a script to execute the application and provide some input text (the current primary selection) programmatically.

Eventually I want --daemon to monitor for changes and immediately synchronise when changes occur. Until monitoring of storages is actually implemented, it simply syncs every five minutes.

This solves a long standing issue in vdirsyncer: there’s no way to continuously sync in background. Not unless credentials are available in plain-text form or available without user intervention in a similar way.

The results so far have been quite satisfactory. I’ve created multiple events on other devices, and they’ve shown up soon on my desktop. I even have a misconfigured storage that fails every time. Vdirsyncer keeps synchronising everything else just fine.

An alpha release

[permalink]

I still have heavy backups of my calendars and collections. And some more refinement is still required before I can publish an alpha release for others to use. But I’m still happy to have reached this point.

I’ll be winding down for the remainder of this year, but my general roadmap right now is:

• Draft a migration guide, mentioning all the little things that change between the previous implementation and this one.
• Finish writing clear instructions on how the configuration file needs to be upgraded (I’ve tried to keep these changes to a minimum).
• Write a lot more unit tests for some storages.
• Handle a failures to write the state to disk (this is a fatal error anyway).
• Ensure that all file-system writes are atomic.
• Publish an alpha release for others to test.

The initial alpha release will be in a state where it may lose or corrupt your data. I hope it doesn’t, but the word alpha means exactly this: it’s not a final version.

vparser

[permalink]

I have also published v1.0.0 of vparser, the low level parser for icalendar and vcard. I have a few other project for which I’ll use this in future too, including normalising calendar entries and merging contacts.

Specification for valarmd

[permalink]

The basic usage is:

valarmd --socket /path/to/socket --vdir /path/to/vdir

valarmd shall read icalendar components (both events and todos) from a local file-system directory, extracting their respective dates, alarms and summaries. It will also monitor the directory for any changes to these files and load any changes.

At the time of each alarm, valarmd will write a single event into its socket. An event is a JSON object, encoded into a single line:

{"summary": "Meeting with Alice", "alarm": 1700811229, "type": "event", "start": 1700813029}

Programs that read events from this socket can perform all kind of actions:

• Show a notification via a notification daemon.
• Play an audible notification
• Read the event summary via text-to-speech.
• Send an email, print job or fax with the details of the alarm.
• Emit a D-Bus signal.

Clearly, anything can be done with the information. How these alarms are presented is up to end users.

Development status

[permalink]

The current work in progress is written in Rust. Parsing icalendar timezones and recurrence rules is a bit not trivial, and existing Rust libraries have some limitations that need to be addressed:

• chrono: It is not possible to create a Vec<DateTime> where datetimes have a different timezones. There is an issue for this upstream. This is required to properly handle all form of recurring events.
• rrule: Doesn’t support using custom timezone implementations. A custom timezone implementation is needed to represent data read from icalendar components. I attempted to address the issue but it’s more complicated that it seems.

The current (incomplete and broken) implementation is in the valarmd repository. There is also a short thread covering its status in further detail.

Given limitations of existing libraries (and being focused elsewhere), this project remains paused. A few workarounds have been mentioned, but they only work for some events in my personal calendar. Missing some alarms is not a reasonable compromise.

Considering for golang

[permalink]

I had considered golang before using Rust, but its timezone implementation is annoying to work with. Golang’s timezone implementation is represented by the Location type. Creating a new Location instance cannot be done directly; it can only be done by providing IANA Time Zone database-formatted data as input.

So in order to create a Location instance from an icalendar VTIMEZONE, it needs to be converted into IANA Time Zone database-format first, and then into Location.

This is an awful API (which won’t be fixed), and only a single example. In general, I don’t consider golang’s datetime and timezone interfaces flexible enough for tools that need to interoperate with icalendar data.

See also

[permalink]

• An issue from 2015 requesting exactly this.

Requirements

[permalink]

The main goal of this parser (now called vparser) is to parse only the basic structure of icalendar and vcard files. The main requirements for vdirsyncer’s use case are:

• Extracting the UID from an entry (calendar component or vcard).
• Normalise a component:
  • Sort its properties into a deterministic order.
  • Recognise and ignore specific properties (e.g.: PRODID).
• Avoid copying data in memory wherever possible. This will run once for every single item , so needs to be well-performant.
• Handle potentially invalid components. The general structure needs to be reasonably valid:
  • The syntax of content lines must be correct.
  • The syntax of individual parameter types or values may be invalid.

Focusing on a this use case

[permalink]

I consider code re-usability important, and therefore wanted this parser to be reusable for other applications that need to parse icalendar data.

My original intent was for this implementation to be usable in other sorts of applications, including something like a calendar GUI application. However, such an application would have somewhat conflicting requirements, and trying to design a parser that would fit such a use case resulted in a design that was not ideal for vdirsyncer.

Designing a lower level library optimised for all usages turned out to be unrealistic. The design on which I have finally settled is optimal for vdirsyncer, and quite suitable for other low-level tools, for example:

• icalendar-funnel: a tool that shall read two distinct calendars and copy their items into a third (this is a special kind of one-way sync).
• vcard-merge: a tool that shall merge two vcard entries into a single one.

However, its performance may not be optimal for something like a calendar UI.

Quick intro to content lines

[permalink]

As a quick reference, content lines look something like this:

DTSTART;TZID=America/New_York:19970902T090000

• The name is DTSTART.
• The only parameter here is: TZID=America/New_York.
  • The parameter’s name is TZID.
  • The parameter’s value is America/New_York
• The value is 19970902T090000

Note that a single content line may have zero, one or more parameters.

Designs considered

[permalink]

I already had quite a few designs in mind for my parser, and quite a few notes spread across different places. I’ve kept around and present here only a few of these (most others were really regurgitations of the same ideas presented below).

The following all share a same general approach: export a trait¹ which consumers must implement. The parser will parse the entire input data. As it encounters elements on the input being parsed, it will call the appropriate functions on the trait handler implementation.

The general idea is inspired on Java’s SAX parsers. Generally, these handlers would implement a state pattern.

Version 1

[permalink]

trait ParseHandler {
    // TODO: maybe use an enum with variants KnownName and XName?
    fn name(&self, name: Cow<str>);

    fn param(&self, name: Cow<str>, value: &[Cow<str>]);

    fn value(&self, value: Cow<&str>);

    fn end_of_file(&self);

    /// Called when invalid data is found.
    ///
    /// Includes all input text until the next `\r\n`.
    fn invalid(&self, value: Cow<&str>);
}

A consumer of the parser would implement this trait and handle events as function calls. For the following example line:

BEGIN:VTODO

The following sequence would be invoked (this won’t strictly compile, and is simplified for readability):

handler.name("BEGIN");
handler.value("VTODO");

Pros:

• Skipping a content lines is as simple as ignoring all events until a corresponding handle.name("END") is received.
• Using Cow allows passing a reference most of the time, but owned data in only when a content line is folded.

Cons:

• Using Cow for params implies creating copies of parameters even in situations which don’t care about them (which is every single scenario in the case of vdirsyncer).
• Likewise, the array of parameters always needs to be allocated, even if unused.
  • As I review this, I realise that re-using the same Vec for each call is feasible, although I suspect that lifetimes would be quite unpleasant to deal with for consumers that make use of parameters.
• For usages that need to further interpret the calendar component data, this API is quite tedious to use.
• Requires additional logic to reconstruct content lines; which means vdirsyncer would be decomposing them and the re-composing them again, adding pointless complexity.

I didn’t consider this terrible, but it’s definitely not good either.

Version 2

[permalink]

I tried using an enum as a type handed over to the handler, and a single event per line:

#[non_exhaustive]
pub enum ContentLine<'input> {
    Begin {
        params: Vec<Cow<'input, &'input str>>,
        value: Cow<'input, &'input str>,
    },
    End {
        params: Vec<Cow<'input, &'input str>>,
        value: Cow<'input, &'input str>,
    },
    // ... TODO: maybe variants for each standard name too?
    Other {
        name: Cow<'input, &'input str>,
        params: Vec<Cow<'input, &'input str>>,
        value: Cow<'input, &'input str>,
    },
    Invalid(Cow<'input, &'input str>),
}

trait ParseHandler {
    fn content_line(&self, line: ContentLine);
}

Compared to the previous approach, usage is (mildly) easier for consumers that want to actually interpret the calendar component’s data, but doesn’t provide great improvements for the vdirsyncer use case.

Pros:

• Skipping content lines as simple as the previous approach.
• Each line is a single event, which is simpler to handle for consumers that need to interpret the calendar component data.

Cons:

• Still parses parameters for lines that we don’t care, and allocates lots of Vecs that may never be used.
  • Again, maybe using the same Vec that merely gets repopulated for each line was feasible. The above type would have a &[Cow<str>] instead of Vec<Cow<str>>. However, this sounds like lifetimes could be even more complicated, given that the array has a shorter lifetime than the references (I’m not even 100% sure that this is allowed).
• Requires different enum types for vcard and icalendar. Keeping all the variants updated would be quite tedious, especially given that most won’t even be used in vdirsyncer’s use case.
• Also requires additional logic to reconstruct content lines as the previous approach.
  • On the other hand, given that this approach uses one event per line, it is possible to simply provide a reference to the original line.

Version 3

[permalink]

pub struct ContentLine<'input> {
    data: Cow<'input, str>,
}

impl<'input> ContentLine<'_> {
    pub fn name(&self) -> Cow<str> {
        // ...
    }
    pub fn params(&self) -> Cow<str> {
        // ...
    }
    pub fn value(&self) -> Cow<str> {
        // ...
    }
    pub fn raw(&self) -> Cow<str> {
        // ...
    }
}

trait ParseHandler {
    fn content_line(&self, line: ContentLine);
    fn invalid_line(&self, line: Cow<&str>);
}

This approach does away with the enum type used above, but keeps the idea of a single event per content line.

Pros:

• Skipping content lines still simple as the previous approach.
• Doesn’t create Vecs with pointers to parameters that are never used.
• Requires no extra complexity to access the original content line unfolded.

Cons:

• Requires parsing a lot of portions twice; once when extracting the content line, and again when accessing the name, value or parameters.
• Using Cow to hold data in ContentLine means that we copy around every line when unfolding, even if the unfolded portions are never used.
• Requires extra complexity to re-create the original content line in its folded form.

Overall, this is a minor improvement over previous approaches. Mostly the API provides a cleaner API (for all use cases).

Version 4

[permalink]

pub enum Name<'input> {
    Begin,
    End,
    // ... TODO: variants for each standard name
    Other(Cow<'input, &'input str>),
    Invalid(Cow<'input, &'input str>),
}

pub struct ContentLine<'input> {
    name: Name<'input>,
    data: &'input str,
}

// Implements impl Iterator<Item = &str>
pub struct Params {}

impl<'input> ContentLine<'input> {
    pub fn name(&'input self) -> &'input Name {
        &self.name
    }

    pub fn params(&self) -> Params {
        // ...
    }
    pub fn value(&self) -> Cow<str> {
        // ...
    }
}

trait ParseHandler {
    fn content_line(&self, line: ContentLine);
    fn invalid_line(&self, line: Cow<&str>);
}

This API is more convenient for situations which want to interpret the data inside calendar components. It also avoids parsing the name portion of each line twice.

Pros:

• Skipping a content lines is as simple as ignoring all events until a matching Name::End is found.
• Does not unfold and copy data that won’t be used.
• Accessing parameters has a much more ergonomic interface.
• The cost of accessing parameters is only paid when they are actually required.
• No Cow for data that won’t be used.

Cons:

• Still requires a second pass to access parameters.
• The enum type for the name might be tedious to maintain, and their value is questionable.
  • On the other hand: it is perfectly feasible to replace it with a &'str instead; another version of the handler that is omitted here did exactly that. We’ll see more of this idea below.

Taking a step back

[permalink]

At this point, a few patterns where clear:

• Whenever I avoided unfolding and re-allocating data that vdirsyncer doesn’t care about (mostly parameters), interpreting that data requires parsing that same portion of data twice.
  • Implying that I should prioritise the vdirsyncer use case, rather than try to make this too generic.
• Using a Cow<str> as a field in the struct type means unfolding and copying the entire line, including portions that would never be used.
• Using enums for the line’s name doesn’t seem to provide a worthwhile advantage, and always brings in maintenance overhead.

Current Version

[permalink]

After the above experiments, I set out to design something that would be a perfect fit for vdirsyncer and other low level tools. Higher level tools will not have the same level of consideration.

The resulting data type merely includes references (pointers) to the original data:

/// A valid content line.
///
/// May be folded via a CRLF immediately followed by a single linear
/// white-space character (i.e., SPACE or HTAB).
pub struct ContentLine<'input> {
    raw: &'input str,
    // Everything before the first colon or semicolon (whichever comes first).
    name: &'input str,
    // Everything after the unquoted semicolon and before the first unquoted colon.
    params: &'input str,
    // Everything after the first unquoted colon.
    value: &'input str,
}

trait ParseHandler {
    fn content_line(&self, line: ContentLine);
}

Pros:

• Skipping a component is still as simple as ignoring all lines until a matching END:XXX event.
• Parses everything that vdirsyncer cares about in one pass.
• No eager unfolding and copying of data that won’t be used.
• There is no need to reconstruct the original line; a reference to it is passed.

Cons (sort of):

• Accessing individual parameters still needs a second pass.
  • But vdirsyncer doesn’t actually access those at all.
• For usages that need to further interpret the calendar component data, this still requires a second pass to access individual parameters.
  • Again, vdirsyncer doesn’t need this.

All the disadvantages here affect other type of tools, but not vdirsyncer specifically.

Accessors

[permalink]

The data in the structure above points to the original input data, which may contain folded lines. Accessing it is implemented via methods that unfold content on-demand:

impl<'input> ContentLine<'input> {
    fn raw(&self) -> &'input str {
        // ...
    }

    fn name(&self) -> Cow<'input, str> {
        // ...
    }

    fn params(&self) -> Cow<'input, str> {
        // ...
    }

    fn value(&self) -> Cow<'input, str> {
        // ...
    }

    fn re_folded(&self) -> Cow<'input, str> {
        // ...
    }
}

Some notes on this:

• vdirsyncer doesn’t even read parameters, so that won’t even be used.
• Names are seldom folded (due to being at the beginning of the line), so name() will usually return a Cow::Borrowed (e.g.: a reference).
  • Names are accessed for every line, so maybe I’ll escape them prematurely and avoid a second pass for them too.
• Accessing values is only done for a small minority of lines, and only those will be unfolded.
• The re_folded method can be used to normalise how lines are folded. I won’t be using this initially and want to evaluate whether it is ever of any real impact in real-life conflict resolution.
• Explicit lifetimes here allow keeping refernces beyond the lifetime of the parser itself.

Inversion of control

[permalink]

An issue with this approach is that parsing can’t be stopped immediately. There’s no way to stop the parser immediately and ignore the rest of a file.

I addressed this by simply using a Parser type which can be used to iterates over lines:

struct Parser<'data> {
    data: &'data str,
    index: usize,
}

impl<'data> Parser<'data> {
    fn new(data: &'data str) -> Parser<'data> {
        // ...
    }
}

impl<'data> Iterator for Parser<'data> {
    type Item = ContentLine<'data>;

    fn next(&mut self) -> Option<Self::Item> {
        // ...
    }
}

This is also a more idiomatic Rust API, which actually turned out to be more pleasant to use.

Implementing the parser

[permalink]

Implementing the parser is relatively straightforward. Given that I want to handle invalid input gracefully, I only really need to concearn myself with a subset of the rules for icalendar content lines.

In general terms, I’ll parse each line until finding one of the following:

• A semicolon: indicates that the name ends here and parameters.
• A colon: indicates that the name/parameters ends and the value begins.
• A \r\n:
  • When followed by whitespace indicates a continuation line.
  • Otherwise, it indicates the end of this content line.

(these are the same rules as those in the comments in the sample code above)

In my head, this looked like a decision tree, and this is pretty much what the implementation looks like.

The parsing code is one huge function. I usually consider functions this long an indicator of poor code quality, but I find this to be an exception. The code’s indentation actually reflects depth in this decision tree that I mentioned. And the lack of fancy macros or “smart” helpers makes it straightforward to read and follow.

When writing this, I initially represented the main branches with a todo!() for each one. I and then filled in the missing parts, usually by writing tests that would reach this unreachable code and refining until those passed.

I also looked into criterion for performance benchmarks (I only used it for one specific improvement so far). It turns out to be quite simple to use. I intend to use this for a few pending improvements after the first working releases.².

The rest of this month

[permalink]

It’s been a few weeks since the last vdirsyncer status update. The design work mentioned above took only a couple of days. I’m still not very happy with the time consumed on it. A less polished approach could have sufficed for an initial implementation, and this won’t be as flexible as I hoped it would be.

The implementation itself didn’t take long. I already had a pretty clear mental picture on what it would look like, so it was mostly a day or two of writing tests, writing code, and iterating on that.

I’m now finalising integrating this into the vdirsyncer codebase and dealing with (what I hope are) the final bits of the basics in conflict resolution.

All this aside, I’ve been enjoying some time off recently, and also a lot of time working around more networking issues than I had anticipated.

Until next month!

First, run the tests with instrumentation enabled:

> RUSTFLAGS="-C instrument-coverage" cargo test -p vparser
   Compiling vparser v0.1.0 (/home/hugo/src/git.sr.ht/~whynothugo/vdirsyncer-rs/vparser)
    Finished test [unoptimized + debuginfo] target(s) in 0.33s
     Running unittests src/lib.rs (target/debug/deps/vparser-5204a1fc06fb7f74)
...

In some cases, you might need to merge multiple profraw files into a single one. This was not required in this particular case, but should be kept in mind if multiple files were generated.

llvm-profdata merge -sparse vparser/default_*.profraw -o vparser.profdata

Finally, show coverage with:

llvm-cov show --instr-profile vparser.profdata --object target/debug/deps/vparser-5204a1fc06fb7f74 --show-line-counts-or-regions

Note that the path target/debug/deps/vparser-5204a1fc06fb7f74 should match the one printed by cargo test above.

Both llvm-profdata and llvm-cov are provided by the llvm package.

It would be nice to find a plugin that reflects these results inside neovim (to quickly spot lines that are not covered).

In the case of my laptop, the grand majority of my usages happens while plugged into a power source. Continuously charging the battery in this state merely serves to reduce its battery life and keep it warm, neither of which is desirable.

I decided to set the maximum charge to 80% to avoid this. Sadly, I don’t have a second identical laptop to use as control group, so I’ll never know if this has the impact that I expect. The theory indicates that it should.

Setting the threshold

[permalink]

The Linux battery driver exposes a value to configure the charge threshold. Changing it is as simple as writing a number into the charge_stop_threshold file:

echo 80 > /sys/class/power_supply/BAT0/charge_stop_threshold

The above won’t work unless it is executed as root. The following approach works as an unprivileged user:

echo 80 | doas tee /sys/class/power_supply/BAT0/charge_stop_threshold

However, when the system reboots, the driver is loaded in its default state, and this setting is not preserved. It needs to be specified at each start-up.

Running a command after each start-up

[permalink]

I could write a system service that does this, but it seems like an overkill for a one-liner.

Alpine currently uses OpenRC, which includes a very simple mechanism to run custom scripts during start-up. Basically, they just need to be placed in the /etc/local.d/ directory, with the extension .start.

The main caveat to keep in mind is that these scripts are in a blocking manner; the next service won’t start until they’re done. Fortunately, this specific script is near-instantaneous, so this isn’t a problem.

For a more in-depth explanation on this feature, see the relevant article in the Gentoo wiki.

Setting the threshold after each start-up

[permalink]

Create the file /etc/local.d/20-batery-threshold.start:

#!/bin/sh

for battery in /sys/class/power_supply/*/charge_stop_threshold; do
	echo 80 > $battery;
done

Make it executable:

doas chmod +x /etc/local.d/20-batery-threshold.start

Enable and start the local service:

doas rc-update add local default
doas service local start

Over-engineering this idea

[permalink]

It would be useful to have a privileged daemon that lets me change this at runtime (likely via a control socket). I could then have a widget in my status bar that indicates the current threshold and lets me toggle it.

For now, the above approach is sufficient.

This article covers the steps that I take when setting up Alpine on this device. It is not intended to cover all possible installation options, but is a reference of my particular setup.

My setup uses btrfs for the main partition, with a subvolume for the root partition. This allows me to create snapshots of it and roll back to one of these snapshots if necessary. I intend to build a system which allow atomic updates on top of this.

A minimal intro to btrfs

[permalink]

Skip this section if you are familiar with btrfs.

Btrfs supports subvolume. A subvolume is created for a directory inside a btrfs partition, and that subvolume can then be mounted as if it were a filesystem of its own.

If a partition contains a directory media/videos, and one creates a subvolume for that path, it is them possible to mount that media/videos directory into any location without the need to mount the rest of the filesystem. Subvolumes can somewhat be though of as sub-partitions, where the pool of free space is shared amongst all of them.

Btrfs also supports copy-on-write, which allows creating a copy of a file or directory without actually copying the data on disk. When /home/hugo/orig is copied to /home/hugo/dest, the filesystem will only mark the latter is a copy of the former. If later modifications are made to one of these, then the filesystem will write the modified version into a new location, taking care to leave the original file unchanged.

These two features combined are what allows implementing snapshots: a copy of a subvolume at a given time. The subvolume can be further modified, but the snapshot keeps an unchanged copy of the data.

Booting into the installer

[permalink]

Download Alpine’s installation image. Copy in into the USB drive as explained in the wiki. E.g.:

doas dd if=alpine-extended-3.18.4-x86_64.iso of=/dev/sdX bs=512

Plug the USB drive into the laptop, turn on the device and immediately start tapping F12 (about once per second until the laptop plays a short beep) to enter the boot menu. Select the USB drive.

Once Alpine boots, log in as root (there is no password in the installation media).

Preparing the disk

[permalink]

Use fdisk -l to inspect local disks. The internal drive is /dev/nvme0n1.

The setup-alpine command will do fine for most of the setup, but partitions and bootloader will be configured manually, since the intended setup diverges from what the setup scripts support.

setup-alpine will prompt for a few things:

• Keyboard: us.
• Keyboard variation: us-altgr-intl.
• Hostname: device-name.whynothugo.nl.
• Network adapter: wlan0.
• Wifi network.
• Wifi password. Care must be taken to avoid typos here; apparently if this is typed wrong the network ends up in a broken state and doesn’t outright fail.
• IP address: dhcp.
• Networks: done.
• Manual network configuration: n.
• The default DNS is the one provided by the local router. This is fine for now.
• root password: empty; this account will be locked later in this process.
• Time zone: Europe/Amsterdam.
• User account: hugo. This user will be a member of the wheel group, and has (by default) privileges to use doas.
• The defaults are fine for all remaining steps.

Next, partition the disk manually (to create the btrfs subvolume for the root and home partitions).

Also of note: the default swap partition created by the installer is way too small and might not work when suspending to disk. When the system is suspended-to-disk, it copies the contents of the RAM into the swap partition, so the swap should be at least as large as the amount of RAM available. The swap partition also needs to be encrypted, which the installation script do not support.

Use cfdisk to partition the disk and create three partitions:

• A 600M EFI System Partition (ESP).
• A swap equal to physical memory size.
• A main partition with all remaining space.

Next, format the partitions:

apk add btrfs-progs cryptsetup
modprobe btrfs

mkfs.vfat /dev/nvme0n1p1
# Create encrypted partitions for swap and main partition.
# Hint: use the same passphrase for both so that they can be unlocked in unison.
cryptsetup luksFormat --label luks-swap /dev/nvme0n1p2
cryptsetup luksFormat --label luks-main /dev/nvme0n1p3
# Unlock the newly created partitions:
cryptsetup open /dev/nvme0n1p2 swap
cryptsetup open /dev/nvme0n1p3 main
# Format the inner layer of the encrypted partitions:
mkfs.btrfs --label main /dev/mapper/main
mkswap /dev/mapper/swap

Temporarily mount the btrfs partition:

mount /dev/mapper/root /mnt/

Create two subvolumes inside of it:

# This implicitly creates the directories too.
btrfs subvolume create /mnt/root
btrfs subvolume create /mnt/home

And unmount the partition:

umount /mnt

Now mount the subvolume that will be used as the root partition. subvol=root here refers to the name of the subvolume created above (it is the directory path relative to the root of the partition).

mount /dev/mapper/fs -o subvol=root /mnt/

Then mount the ESP and the home subvolume inside of the above:

mkdir /mnt/home /mnt/efi
mount /dev/nvmen1p1 /mnt/efi
mount /dev/mapper/fs -o subvol=home /mnt/home

Finally, set up Alpine into the mounted partitions. This will also generates /mnt/etc/fstab.

setup-disk /mnt

Switching into the installation filesystem

[permalink]

chroot into the new installation. Also mount /proc and /dev inside the new root:

chroot /mnt
mount -t proc proc /proc
mount -t devtmpfs dev /dev

Enable the community and testing repositories, and switch to the Alpine Edge branch. This is done by editing /etc/apk/repositories and replacing its contents with:

http://dl-cdn.alpinelinux.org/alpine/edge/main
http://dl-cdn.alpinelinux.org/alpine/edge/community
http://dl-cdn.alpinelinux.org/alpine/edge/testing

Some regions may have networking issues reaching the global CDN mirror. If this is the case, a local mirror should be used instead.

Setting up the bootloader

[permalink]

When the firmware is done initialising, it will load a bootloader: a program responsible for loading the actual operating system itself. I will be using a UEFI bundle (nowadays also known as a UKI) to boot the system. This is a executable binary which bundles the following:

• The kernel itself
• The kernel’s command line parameters
• A small stub that execute the kernels with that command line
• The initramfs (a small read-only filesystem with the necessary userspace utilities to boot into the main system).

There are additional notes on this setup in the Alpine Wiki.

The stub itself is provided by the gummiboot package. It is considered deprecated, but no solid alternative is available (I do have one in progress, but still has some rough edges). The bundle itself is built by efi-mkuki, and secureboot-hook will rebuild the bundle after each kernel upgrade.

Install the mentioned packages:

apk add efi-mkuki secureboot-hook gummiboot

I use the linux-edge kernel:

apk add linux-edge
apk del linux-lts

Install blkid which will be used in a moment. This tool prints the uuid (and a few other details) for a specified partition. This is the recommended way to address a partition ambiguously:

apk add blkid

Edit /etc/kernel-hooks.d/secureboot.conf with the following. See below for how to determine the exact value for these UUIDs. secureboot.conf should look something like this:

cmdline="root=UUID=5021db58-cc3a-4829-a630-2d468f8d1761 ro rootflags=subvol=root cryptroot=UUID=0db973a0-1b95-4a23-a63f-cb6248fe2bf7 cryptdm=crypt modules=sd-mod,btrfs,nvme quiet rootfstype=btrfs hostname=destiny.whynothugo.nl"
signing_disabled=yes
output_dir="/efi/EFI/Boot"
output_name="bootx64.efi"

Signing is disabled only temporarily until I install the proper keys.

The output path for the above is /efi/EFI/Boot/bootx64.efi, which is the default path where UEFI firmware looks for a bootloader.

Note: While it is possible to configure additional paths for the firmware to run executables, support for this varies, and configuration data is often lost after firmware updates, so I prefer to avoid the approach. Note that this particular setup is incompatible with booting multiple operating systems on the same host.

• The root UUID can be determined with:

# Append this data at the end of the file; this prevents having to copy it
# manually. Edit the file, cut out the UUID and delete the rest of the line
# appended by `blkid`
blkid /dev/mapper/main nvme0n1p3 >> /etc/kernel-hooks.d/secureboot.conf

• The cryptroot UUID can be determined with:

# Same as above:
blkid /dev/nvme0n1p3 >> /etc/kernel-hooks.d/secureboot.conf

• hostname is optional, but specifying it here ensures that the hostname is set at the earliest moment possible.

During boot, the bootloader will hand over control to the kernel. At this stage, the only filesystem is the initramfs. The kernel needs the cryptsetup user-space utilities in order to mount the encrypted root partition, so they must be included in the initramfs.

Edit /etc/mkinitfs/mkinitfs.conf and add crypsetup to the features variable. While editing this line, it is also safe to delete virtio, which is used only in virtual machines. Also add kms to enable kernel mode setting.

Also disable the mkinitfs trigger. This trigger builds the initramfs into a standalone file, but that won’t be used for this setup; the initramfs is included in the UEFI bundle:

The resulting file should look something like this:

features="ata base cdrom btrfs keymap kms mmc nvme raid scsi usb cryptsetup cryptkey resume"
disable_trigger=yes

Remove GRUB. The generated configuration is broken for the current setup, so it won’t work as-is and will get in the way.

apk del grub-efi grub

Finally, trigger the newly created kernel hook so that all the right files are copied into /efi:

apk fix kernel-hooks

Setting zsh as the default shell

[permalink]

My shell of choice is zsh. It is ideal for interactive usage, having great support for auto-completion and a few other interactive features.

Install zsh and set it as the default shell for my user:

apk add zsh
vi /etc/passwd # locate user 'hugo' and replace `/bin/ash` with `/bin/zsh`.