Home
Name Modified Size InfoDownloads / Week
README.md 2026-10-11 17.1 kB
0
mtdelta 2026-10-11 54.0 kB
0
Totals: 2 Items   71.1 kB 0

mtdelta

Block-level file and directory synchronization over TLS 1.3 with pinned public keys.

Version 0.8.2 (SHA1: b9d98e058a20215edeab152e07be0e4e6a5c56a8)

https://www.linkedin.com/in/daniel-j-garcia-fidalgo/

Disclaimer of Warranty and Liability

This software is provided free of charge, “AS IS” and without warranties of any kind, whether express or implied, including but not limited to warranties of merchantability, fitness for a particular purpose, and non-infringement.

USE THIS SOFTWARE AT YOUR OWN RISK.

The author(s) shall not be held liable for any damage or loss arising from the use of, inability to use, or misuse of this software, including but not limited to data loss, file corruption, unintended file deletion or overwriting, system failure, or any other direct or indirect damages.

You are solely responsible for ensuring that your data is properly backed up and for verifying the results of any operation performed by this software.

By using this software, you acknowledge and accept these terms.

Nothing in this disclaimer excludes or limits any liability that cannot legally be excluded or limited under applicable law.

MTDELTA works pretty much like rsync and similar tools. It copies a file or a dir structure to a remote server or to a local FS. The main difference is that MTDelta is a multithreaded tool, it uses multiple streams of data to copy a file. This is specially convenient to overcome the limitations imposed by high latencies in regards to data throughput. Therefore, it will saturate whatever bandwidth you count with, regardless of the latency between the client and the remote end.

It will also greatly improve copy throughput in local file systems by using multiple cores to copy a file. It is specially designed to copy huge files, like virtual machine disks, or to sync big dir structures. Normally 4 threads will be more than enough to reach the hardware limits.

It uses a rolling delta algorithm to detect changed blocks in such way that only the differential data is copied every time. It's also zero aware, so that zeros are ignored.

mtdelta copies only the blocks that changed. Each file is split into fixed-size blocks, every block is hashed (SHA-1), and only blocks whose hash differs from the other side's manifest are sent or written. Sparse holes are detected and never sent as data.

  • Local copy, or client/server over the network
  • Files or whole directory trees (constant memory, millions of files)
  • TLS 1.3 with mutual public-key pinning, no certificate authority needed
  • Optional --raw plaintext mode for trusted, isolated links
  • Distributed as a prebuilt Linux binary: no compiler, no build step

Note: mtdelta is proprietary software. This repository hosts binaries and documentation only.

Contents

Quick start

Two hosts: A (sender, client) and B (receiver, server).

# 1. On BOTH hosts: install runtime (Debian/Ubuntu shown) and the binary
sudo apt-get update && sudo apt-get install -y libssl3 openssl
sudo install -m 755 mtdelta /usr/local/bin/mtdelta

# 2. On BOTH hosts: generate a key pair
mkdir -m 700 ~/.mtdelta
openssl genpkey -algorithm ED25519 -out ~/.mtdelta/identity.key
chmod 600 ~/.mtdelta/identity.key
openssl pkey -in ~/.mtdelta/identity.key -pubout -out ~/.mtdelta/identity.pub

# 3. Cross-copy the PUBLIC keys
scp ~/.mtdelta/identity.pub user@B:~/.mtdelta/peer.pub     # run on A
scp ~/.mtdelta/identity.pub user@A:~/.mtdelta/peer.pub     # run on B

# 4. Server on B
mtdelta --server --port 9000

# 5. Client on A
mtdelta --source /data --ip 10.0.0.2 --port 9000 --target /backup/data

No encryption wanted (trusted link only)? Add --raw on both ends and skip the key steps. See Raw mode.

Man page

MTDELTA(1)

NAME
  mtdelta - block-level file/directory sync over TLS 1.3

SYNOPSIS
  mtdelta -s SRC -t DST [-n THREADS] [-b SIZE]
  mtdelta -S -p PORT [-r]
  mtdelta -s SRC -t DST -i IPV4 -p PORT [-r]
  mtdelta -V

DESCRIPTION
  Sends only changed blocks (SHA-1 manifests), keeps
  sparse holes. Network mode uses TLS 1.3 with pinned
  public keys on both ends. Needs OpenSSL 3 runtime.

OPTIONS
  -s, -t  source / target (file or directory)
  -i, -p  server IPv4 address / TCP port
  -S      run as server (exits when client finishes)
  -r      --raw: plaintext, NO authentication
          (give on BOTH client and server)
  -n      worker threads (default 4)
  -b      block size: 1M, 10M, 20M, 50M
  -V      print version and exit

FILES  ($MTDELTA_DIR, default ~/.mtdelta)
  identity.key  own PEM private key, mode 600
  peer.pub      other host's PEM public key

EXAMPLE
  B$ mtdelta -S -p 9000
  A$ mtdelta -s /data -i 10.0.0.2 -p 9000 -t /bak

Full manual

1. Overview

mtdelta is a block-based synchronizer for large files and directory trees. It splits each file into fixed-size blocks, hashes each block with SHA-1, and sends only the blocks whose hash differs from the other side's manifest. Filesystem holes are detected and sent as compact "hole" messages instead of data. At the destination they are turned into zeros, so a region that held data in an earlier run reads back as zeros after the sync. Ranges the destination already stores as holes are skipped, so sparse files stay sparse.

It is distributed as a prebuilt binary. You do not need a compiler or any development tools.

Mode Command shape
Local copy --source X --target Y
Network client --source X --ip A --port N --target Y
Network server --server --port N

In network mode, the transport is either TLS 1.3 with mutual public-key pinning (default) or plaintext (--raw).

2. Requirements

Platform

  • Linux only. The program relies on SEEK_DATA/SEEK_HOLE and user.* extended attributes.
  • The binary must match your CPU architecture (check with uname -m, for example x86_64 or aarch64) and your C library. A glibc build will not run on musl-based systems such as Alpine unless a musl build is supplied.

Runtime libraries

  • OpenSSL 3.0 or newer shared libraries (libssl.so.3 and libcrypto.so.3). OpenSSL 1.1.1 and older are not supported.
  • glibc and pthreads, which every mainstream distribution already includes.
  • If your copy of the binary has OpenSSL linked statically, the OpenSSL library requirement does not apply. Run ldd as shown in section 3 to find out.

Tools

  • The openssl command-line tool (version 3.x), only to generate key pairs. Not needed for --raw or local mode.
  • scp or any other trusted way to copy public key files between hosts.

Filesystem

  • Extended attribute support (user.*) is recommended. ext4, xfs and btrfs have it by default. Without it, manifests are trusted without certification.

3. Installation on a bare Linux box

Nothing needs to be compiled. Install the runtime libraries, then copy in the binary. Run the commands as root or with sudo.

Step 1: install the runtime libraries

Distribution Command
Debian 12+, Ubuntu 22.04+ apt-get update && apt-get install -y libssl3 openssl
RHEL, Rocky, Alma 9+, Fedora dnf install -y openssl openssl-libs
Arch pacman -S --needed openssl
Alpine Use a musl-specific build, then apk add openssl libssl3

On older distributions that ship only OpenSSL 1.1.x (Ubuntu 20.04, CentOS 7/8), upgrade the OS to a release with OpenSSL 3, or use a build that bundles OpenSSL statically.

Step 2: install the binary

install -m 755 mtdelta /usr/local/bin/mtdelta

Step 3: verify

uname -m                                           # must match the binary's architecture
file /usr/local/bin/mtdelta
ldd /usr/local/bin/mtdelta | grep -i "not found"    # no output = all libraries found
mtdelta --version
mtdelta --help

If ldd prints a not found line for libssl.so.3 or libcrypto.so.3, repeat Step 1.

Step 4 (recommended for server hosts): create an unprivileged account

useradd --system --create-home --shell /usr/sbin/nologin mtdelta

Run the server under this account, and generate its keys as that account (for example with sudo -u mtdelta) so that ~/.mtdelta belongs to it.

4. Command-line options

Option Meaning
-s, --source PATH Source file or directory.
-t, --target PATH Destination. For a single file, used verbatim as the file path. For a directory source, the root under which the relative tree is replicated.
-i, --ip ADDR Server address. Dotted IPv4 only: no hostnames, no IPv6. Presence of -i selects network mode.
-p, --port N TCP port. Always set it explicitly on the server: the default is 0, which binds a random port.
-n, --threads N Worker threads per file (default 4), capped at the number of blocks.
-b, --block-size S 1MB, 10MB, 20MB or 50MB (1M, 10M, ... also accepted). Default 1MB. Keep it constant between runs.
-r, --raw Disable encryption and authentication. Must be given on both client and server.
-S, --server Run as server.
-v, --verbose Accepted, currently no effect.
-V, --version Print the version number and exit. Needs no other options or keys.
-h, --help Usage. Unknown options also print usage.

5. Secure mode: two servers, two key pairs

Each machine owns its own key pair and holds a copy of the other machine's public key. Nothing key-related travels over the network. Below, A is the sending (client) host and B is the receiving (server) host.

Step 1: generate a key pair on each host (run on both A and B)

mkdir -m 700 ~/.mtdelta
openssl genpkey -algorithm ED25519 -out ~/.mtdelta/identity.key
chmod 600 ~/.mtdelta/identity.key
openssl pkey -in ~/.mtdelta/identity.key -pubout -out ~/.mtdelta/identity.pub

Step 2: cross-copy the public keys

# on A:
scp ~/.mtdelta/identity.pub user@B:~/.mtdelta/peer.pub
# on B:
scp ~/.mtdelta/identity.pub user@A:~/.mtdelta/peer.pub

Only identity.pub is copied. identity.key never leaves its host.

Important: verify the keys out-of-band (for example, compare sha256sum identity.pub over the phone or another trusted channel). The pin is the only trust anchor.

Step 3: start the server on B

mtdelta --server --port 9000

Open TCP 9000 in B's firewall, restricted to A's address if possible.

Step 4: run the client on A

mtdelta --source /data/vm.img --ip 10.0.0.2 --port 9000 --target /backup/vm.img
mtdelta --source /data         --ip 10.0.0.2 --port 9000 --target /backup/data

The server exits automatically when the client finishes (the client sends a "done" message even if some files failed). Start it again for each run, for example from a systemd unit or a wrapper script.

To test both roles on one machine, give each process its own directory, for example MTDELTA_DIR=/tmp/a and MTDELTA_DIR=/tmp/b.

6. Key and TLS requirements

Key directory: $MTDELTA_DIR, otherwise $HOME/.mtdelta.

identity.key (your private key)

  • PEM format, unencrypted. An encrypted key would trigger an interactive passphrase prompt, which breaks unattended servers.
  • Mode must have no group or other permission bits (600 or 400). Otherwise the program refuses to start.
Key type Status
Ed25519 Recommended
ECDSA (P-256, P-384, P-521) Accepted
RSA (2048-bit or larger) Accepted
Ed448 Not usable (signing fails)
X25519 / X448 Not usable (no signing capability)
DSA Not usable (not valid in TLS 1.3)
Keys too weak for the system's OpenSSL security level Rejected

peer.pub (the other host's public key)

  • PEM -----BEGIN PUBLIC KEY----- (SubjectPublicKeyInfo), as produced by openssl pkey -pubout.
  • Not accepted: PKCS#1 BEGIN RSA PUBLIC KEY, SSH authorized_keys format, or certificates.

Certificates: each process wraps its key in an ephemeral self-signed certificate (CN=mtdelta, valid 10 years, regenerated at every start). Chain, hostname and expiry checks are deliberately ignored. After the handshake, the peer's certificate public key must equal peer.pub exactly, otherwise the connection is dropped.

TLS levels: only TLS 1.3. There is no TLS 1.2 or earlier fallback. Cipher suites are OpenSSL's TLS 1.3 defaults (AES-256-GCM, ChaCha20-Poly1305, AES-128-GCM). Both sides authenticate: the server requires a client certificate and pins it.

7. Raw mode (--raw)

# server (B):
mtdelta --server --port 9000 --raw
# client (A):
mtdelta --source /data --ip 10.0.0.2 --port 9000 --target /backup/data --raw
  • No TLS, no key files needed, no authentication. Data, filenames and hashes travel in plaintext.
  • --raw must be on both ends. A raw peer talking to a TLS peer fails at the handshake.
  • It has no effect in local mode.
  • Use only on a trusted, isolated link (direct cable, VLAN, VPN) and firewall the port to the client's address.

8. Behavior reference

  • Single file vs. directory: a directory source is walked depth-first with constant memory. Only regular files are synced. Symlinks, devices, FIFOs and sockets are skipped.
  • Per-file connections: the client keeps one control connection for the whole run and opens one new data connection per file.
  • Resizing: if the destination size differs, it is resized and fully resynced.
  • Sparse files: a region that is a hole at the source is a run of zeros at the destination, in both local and network mode. If the destination held data there from an earlier run, it is overwritten with zeros. Regions the destination already stores as holes are left alone.
  • Manifests: hidden files next to the data.
  • Source side: .NAME.loc.sha1 and .NAME.rem.sha1. The source directory must be writable.
  • Destination: .NAME.rem.sha1 in the destination directory.
  • Certified with the xattrs user.loc_sha1_hash and user.rem_sha1_hash. Delete a manifest to force a full resync.
  • Output: *** Hash certified *** means the source and destination manifests match after the run. Failures print to stderr.
  • Exit code: 0 if all files succeeded, 1 if any failed.
  • Limits:
  • Paths up to 1023 bytes.
  • TLS needs file descriptors below 65536.
  • Control replies are capped at 256 MB (manifests for roughly 12 TB at 1 MB blocks).

9. Security notes and known limitations

  1. The server does not sandbox target paths. It writes to whatever path the client sends (including absolute paths or ..) with the server user's permissions. In TLS mode only the pinned client can do this. In --raw mode anyone who can reach the port can. Run the server as an unprivileged user, preferably in a container or chroot, and firewall it.
  2. Server-side write errors (disk full, I/O) are logged on the server but not reported to the client, so the client may still print "certified". Monitor the server's stderr.
  3. SHA-1 is used for change detection, not security. Integrity and authenticity in transit come from TLS.
  4. Hostnames and IPv6 are unsupported for --ip. Always pass --port to the server.

10. Troubleshooting

Message or symptom Cause
error while loading shared libraries: libssl.so.3 OpenSSL 3 runtime not installed. See section 3, Step 1.
cannot execute binary file: Exec format error Binary architecture does not match the machine (uname -m).
version 'GLIBC_x.y' not found The distribution's glibc is older than the one the binary was built against. Use a newer OS or a build for your platform.
identity.key is accessible by other users Run chmod 600 on the key.
peer.pub missing or not a valid PEM public key Wrong format or file not copied.
peer's public key does not match peer.pub You copied the wrong key, or a key was regenerated.
TLS handshake errors printed One side is --raw and the other is not, or the peer is not TLS 1.3.
could not connect to server Server not running, wrong port or firewall, or --ip is not a dotted IPv4.

License

Proprietary. All rights reserved. Redistribution and reverse engineering are not permitted without written permission from the copyright holder.

Source: README.md, updated 2026-10-11