Learn Rsync - Batch Mode & Partial Transfer

#Introduction

Some scenarios demand more from rsync than just "copy files". Two of them are handled in episode 7: sending the same changes to many machines at once (batch mode), and surviving networks that frequently drop connections (partial transfer). Both mechanisms are often underestimated, yet they become lifesavers in large-scale deployments and unstable connections.

#Batch Mode: The Concept

Imagine managing 50 servers that must all receive identical updates. Running rsync 50 times from the source means computing the delta 50 times. Batch mode computes the delta once, saves it to a file, then "replays" that file on each host.

Batch mode flow

1. rsync --write-batch=batchfile src/ dest/   # compute delta, save it
2. copy the batchfile to the other hosts
3. rsync --read-batch=batchfile dest/          # replay the delta on each host

The advantage: the expensive checksum computation is done once, and transfers to additional hosts only move a small batch file relative to the full data.

#--write-batch

The first command produces a batch file containing the delta:

Write a delta batch

rsync -av --write-batch=myupdate src/ dest/

The result is several files: myupdate (the rsync-format batch file), myupdate.sh (a wrapper script for --read-batch), and myupdate.cmd (for Windows). The contents of myupdate.sh look roughly like:

Contents of myupdate.sh

#!/bin/sh
rsync --read-batch=myupdate /srv/dest

Note

A batch file contains deltas that depend on the destination's state at creation time. If a target host has already been changed by another source before the replay, the result is not guaranteed to be consistent. Use batches for machines that are identical and centrally managed — such as cluster nodes provisioned from the same template.

#--read-batch

On each target host, simply run the wrapper or use --read-batch directly:

Replay the delta on the target host

rsync --read-batch=myupdate /srv/dest

Because all hosts are identical (usually the result of the same provisioning), the same delta applies to all of them. This is the classic image/fleet update pattern from before the era of modern configuration tools like Ansible — and it's still useful for offline cases or very limited bandwidth.

#When to Use Batch Mode

Scenario	Suitable?
Updating image/state to many identical nodes	Yes
Very slow network, delta computed once	Yes
Offline (air-gapped) environments	Yes — transfer the batch file once
Hosts with different states	No — use regular rsync
Occasional updates, few hosts	No — the overhead isn't worth it

#Partial Transfer: --partial

When a transfer is interrupted, rsync's default behavior is to discard the partial file — the half-transferred file is deleted, so the next attempt starts from zero. --partial changes that:

Keep the partial file

rsync -avhP src/big-file.iso dest/

-P (which you met in episode 5) includes --partial + --progress. With --partial, a 10 GB file that breaks at 6 GB leaves a 6 GB partial file at the destination. On the next attempt, rsync uses that leftover as the delta basis — continuing from 6 GB instead of zero.

Warning

A partial file looks like a complete file in the destination directory (same name, smaller size). Scripts processing the destination could mistake a partial file for a full one. The solution: --partial-dir below, which isolates partial files into a separate directory.

#--partial-dir: Isolating Partial Files

Instead of letting partial files mix with complete ones, direct them into a hidden directory:

Partial files in a separate directory

rsync -avh --partial-dir=.rsync-partial src/ dest/

Interrupted files are now stored in dest/.rsync-partial/ — not confused with official files. On the next attempt, rsync finds and uses those partial files automatically. This is a far safer pattern for directories accessed by other applications.

#--timeout: Giving Up Gracefully

Bad networks often leave transfers hanging in silence. --timeout sets a limit in seconds without I/O before the connection is declared failed:

60 second timeout

rsync -avh --timeout=60 --partial-dir=.rsync-partial src/ user@host:/dest/

Meaning: if no data arrives for 60 seconds, rsync drops the connection with a clear exit code instead of hanging forever. Combine it with cron (episode 10) that automatically reruns: break → timeout → restart → continue from partial. This cycle lets large-file transfers over unstable connections eventually complete.

#The Combination for Unstable Networks

The full recipe for transferring large data over flaky connections:

Resilient transfer over a bad network

rsync -avhz --partial-dir=.rsync-partial --timeout=60 --bwlimit=500 \
  src/ user@host:/dest/

--partial-dir saves your progress.
--timeout prevents hanging.
--bwlimit keeps the connection alive with a steady flow (episode 12).
A while ! rsync ...; do sleep 30; done wrapper in cron makes it self-sufficient.

#Closing

In this episode you've mastered batch mode and partial transfer.

Key takeaways:

--write-batch computes the delta once; --read-batch replays it on many hosts.
Batches suit identical fleets, offline environments, and narrow bandwidth.
--partial keeps half-done files; --partial-dir isolates them into a separate directory.
--timeout prevents transfers from hanging on bad networks.
The partial + timeout + cron combination lets big transfers eventually finish.

In episode 8 we enter the real backup phase: incremental & full backup strategy — the concept of full vs incremental, why tar alone isn't enough, and the time-machine-style hardlink snapshot pattern with rsync -a --link-dest=../backup-1/ src/ backup-2/. See you in episode 8!