beeping

An encrypted, deduplicating backup tool defined by one guarantee: a backup or restore can be suspended and resumed at any time — by a keypress, a signal, a kill -9, or a power failure — and resumed later by rerunning the same command, even if the filesystem has changed in between.

Highlights

• Suspend anywhere, resume anywhere. Work flows through durable on-disk queues between idempotent pipeline stages. Nothing is lost on interruption; resuming picks up where the queues left off, and every path is re-validated against the live filesystem (and the current exclude rules) at the moment it is processed, not when it was first discovered.
• Encrypted for hostile storage. Content is chunked (content-defined chunking), encrypted with ChaCha20-Poly1305, and addressed by keyed HMACs, so the server learns neither contents nor fingerprints. The threat model assumes you are handing the files to your enemy — which is why plain FTP is a supported transport.
• No stored keys. The passphrase is the key: everything is re-derived from it (Argon2id over a public salt in the repository header) on each run. There is no key file to protect or lose — disaster recovery is the remote repository plus the passphrase, and a lost passphrase is unrecoverable by design.
• Deduplicating. Identical content is stored once, within and across snapshots; an interrupted-and-restarted upload skips chunks that already made it.
• Bounded everywhere. Every stage is backpressured down to the backend’s write speed: the scanner pauses when the chunker backlog grows, the chunker blocks when uploads lag. It works with a nearly full disk and never rushes ahead of a slow network.
• Remote-first. Local directory, FTP, SFTP/SSH, and S3 (including S3-compatible stores such as Linode Object Storage and MinIO) backends, with connection pooling, lftp bookmark integration, and graceful handling of server connection limits.

Usage

$ beeping init --repository sftp://user@host/backups/home
$ beeping backup --repository sftp://user@host/backups/home ~/work
$ beeping snapshots --repository sftp://user@host/backups/home
$ beeping list SNAPSHOT --repository ...
$ beeping restore SNAPSHOT /some/empty/dir --repository ...
$ beeping restore SNAPSHOT /some/empty/dir 'docs/**/*.tex' --repository ...

Interrupt a backup or restore however you like — Ctrl-C, SIGTERM, SIGKILL, pulling the plug — and rerun the same command to resume it.

The repository location and standing excludes can live in ~/.config/beeping/config.toml instead of on the command line:

repository = "sftp://user@host/backups/home"
exclude = ["*.tmp", "/nobackup", ".cache"]
cross_mount_points = false
minimum_free_space = "20 GiB"
maximum_store_size = "500 GiB"

beeping --help documents the configuration file, the gitignore-style exclude pattern rules, every repository URL form, and the relevant environment variables in full.

Mount points are never crossed unless you opt in with cross_mount_points.

A store too small for the tree stops the backup instead of filling up. Two settings say where the bottom is, depending on what the store can tell you about itself: minimum_free_space is room to leave free, which suits a local disk or an SFTP server, and maximum_store_size is a size for the repository to stay under, which is the same limit from the other end and the one an FTP account with a quota can answer. A backup that reaches either publishes what it has stored and ends; --early-finish asks for the same thing outright.

The snapshot left by hitting a space limit or --early-finish is a finished one, not a partial: the run is over, so what it stored is its last word rather than a stopgap. Being over rather than suspended, the way on from a full store is to exclude (or delete, if they’re really junk) files that should not be part of the backup, and run again with --ignore-store-limits since the store is still at the limit that stopped the first run, then prune the first run’s snapshot. The second run stores only files that were not already stored by the first run, then the prune frees up the space used by the first run that is not needed by the second.

Exclude rules may be edited between suspending a backup and resuming it; the new rules apply to everything not yet processed, while anything already stored stays in the snapshot.

How it works

A backup is a pipeline of stages — scanner → chunker → uploader → finalizer — connected by persistent, crash-safe queues (and one bounded in-memory channel in front of the uploader). Each stage is idempotent and acknowledges work only after its results are durable, so the process can die at any instruction boundary and the replayed work converges: re-chunked files deduplicate to the chunks already stored, and snapshot IDs are deterministic, so even a crash during finalization redoes into the same snapshot.

The repository itself is a pile of write-once objects: a public header (salt, KDF parameters, chunking parameters, and a verifier that commits to all of them, so a tampered header is unusable), encrypted chunks named by keyed content HMACs, and snapshot manifests. No object is ever modified in place, which makes dumb transports like FTP viable.

Per-run working state (the queues, progress counters, and a marker recording what run they belong to) lives under the user state directory (~/.local/state/beeping by default, or --state-dir). Beeping automatically refuses to back up its own state or a repository living inside the tree being backed up.

Two commands take things out of a repository, and neither can be undone. beeping prune SNAPSHOT... removes whole snapshots. beeping excise ROOT PATTERN... removes files and directories from every snapshot of one tree — for something that should never have been backed up, and for getting the room it occupies back. Both sweep the chunks nothing needs afterwards; --dry-run shows what an excision would remove without removing it.

An excision refuses wherever sweeping could break a backup that is not finished with its work: while a run holds the repository, and while a run of that tree was killed rather than suspended, since what it stored is not all named by a snapshot yet. Against a backup that is suspended it goes ahead and corrects that too — rewriting the partial snapshot it published and dropping the selected paths from the work it still has queued — so the run resumes to a snapshot without them. A backup finds files by walking the tree, though, so anything still on disk under an excised path comes back on the next run unless it is excluded as well.

Caveats

• The passphrase cannot be changed (that would mean re-encrypting the repository), and there is no recovery path without it.
• Unix only, for now.

License

MIT — see LICENSE.