hashsync
A tool that reduces SSH transfer volume by reusing data blocks already on the receiving host.
What does it do?
Install hashsync on both hosts and specify directories to index in the receiving host's configuration file. The receiver sends a list of block hashes from existing files; the sender finds matching blocks in a command's input or output and sends references instead of the actual data. It compresses the remaining data with zstd, and the receiver reconstructs the original byte stream. After the transfer, it summarizes transfer volume and deduplication results for each stream.
Key features
- Find blocks overlapping with existing files in SSH command input, output, or both to reduce duplicate transfers.
- Compress unmatched data with zstd.
- Index file blocks in specified directories on the receiving host.
- Apply to data streams from tar, rsync, Nix exports, and similar commands.
- Summarize bytes read, bytes sent, deduplication volume, and total traffic after transfer.