Rsync for Syncing Local and Remote Directories

• Prerequisites

• Rsync Syntax

• Use the Rsync tool to Sync with a Remote System

• Summary

Prerequisites

You need two machines to use rsync to sync files between a local and remote system. The machines act as your local computer and your remote machine.

You can use configured virtual private servers, personal computers, or virtual machines. Set up admin users and configure firewalls for each of the systems.

You also require SSH keys on both systems. Copy the server’s public key to the other server’s authorized_keys file.

You can use this guide for machines running Ubuntu 20.04. It can work with any Linux-based operating system that has rsync installed.

Rsync Syntax

The syntax for rsync is similar to other tools, ssh, scp, and cp.

Change into the home directory with the following command:

cd ~

Create a test directory:

mkdir dir1

Create one more test directory:

mkdir dir2

Then add some of the test files:

touch dir1/file{1..100}

You now have a directory called dir1 with 100 empty files in it. Confirm it by listing the files using the command below:

ls dir1

Output
file1    file20  file27  file45  file36  file54  file63  file72  file90  file81
file10   file19  file28  file37  file55  file46  file64  file73  file82  file91
file100  file2   file29  file38  file47  file56  file65  file74  file83  file92
file11   file18  file12  file93  file48  file57 file66  file75  file84  file39
file3    file21  file30  file49   file4  file58  file76  file67  file85  file94
file13   file22  file31  file40  file5   file59  file68  file77  file86  file95
file14   file40  file32  file23  file41  file6   file69  file78  file87  file96
file15   file24  file42  file33  file51  file60  file7   file79  file89  file97
file16   file25  file34  file43  file52  file62  file70  file8   file88  file98
file17   file26  file35  file44  file53  file61  file71  file80  file9   file99

There is also an empty directory called dir2. You can now sync the contents of dir1 to dir2 on the same system.

Run rsync and use the following -r flag, which means recursive and is essential for directory syncing:

rsync -r dir1/ dir2

Use the -a flag, which stands for “archive”. It is a recommended flag and more commonly used than -r.

The flag syncs recursively and preserves symbolic links. It includes special and device files, modification times, groups, owners, and permissions.

Run the same command using the -a flag:

rsync -a dir1/ dir2

Note: There is a trailing slash (/) at the end of the first argument in the syntax of the previous two commands.

This trailing slash showcases the contents of dir1.

Without the trailing slash, dir1 and the directory is placed in dir2.

The outcome will create a hierarchy as shown below:

~/dir2/dir1/[files]

Check your arguments before executing the rsync command.

Rsync offers a method of passing the -n or --dry-run options. The -v flag, which stands for verbose, is also required to get the proper output.

For the next command, you can combine the a, n, and v flags:

rsync -anv dir1/ dir2

Output
sending incremental file list
./
file1
file10
file100
file11
file12
file13
file14
file15
file16
file17
file18
. . .

You can also compare the output to when we remove the trailing slash:

rsync -anv dir1 dir2

Output
sending incremental file list
dir1/
dir1/file1
dir1/file10
dir1/file100
dir1/file11
dir1/file12
dir1/file13
dir1/file14
dir1/file15
dir1/file16
dir1/file17
dir1/file18
. . .

The output shows that the directory was transferred, not only the files in the directory.

Use the Rsync tool to Sync with a Remote System

You will require SSH access configured between the local and remote machines. Install rsync in both systems.

After the SSH access is verified between the two machines, sync the dir1 folder from the last section to a remote machine with the following syntax.

As we want to transfer the actual directory, we can remove the trailing slash as shown below:

rsync -a ~/dir1 username@remote_host:destination_directory

It is a push operation as it “pushes” a directory from the local system to a remote system. A pull operation syncs a remote directory to the local system.

If the dir1 directory is on the remote system instead of your local system, use the following syntax:

rsync -a username@remote_host:/home/username/dir1 place_to_sync_on_local_machine

The source is the first argument above, and the destination is the second.

Using Other Rsync Options

Rsync offers options to change the default behavior of the utility.

For example, you can reduce the network transfer by adding compression with the -z option. It is useful when transferring files that have not already been compressed.

rsync -az source destination

The -P flag is also helpful. It combines the flags --progress and --partial. The first flag offers a progress bar for the transfer. The second flag resumes interrupted transfers:

rsync -azP source destination

Output
sending incremental file list
created directory destination
source/
source/file1
              0 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=99/101)
sourcefile10
              0 100%    0.00kB/s    0:00:00 (xfr#2, to-chk=98/101)
source/file100
              0 100%    0.00kB/s    0:00:00 (xfr#3, to-chk=97/101)
source/file11
              0 100%    0.00kB/s    0:00:00 (xfr#4, to-chk=96/101)
source/file12
              0 100%    0.00kB/s    0:00:00 (xfr#5, to-chk=95/101)
. . .

You get a shortened output if you run the command again. Rsync uses modification times to see if changes are made:

rsync -azP source destination

Output
sending incremental file list
sent 818 bytes received 12 bytes 1660.00 bytes/sec
total size is 0 speedup is 0.00

You can update the modification time on some files:

touch dir1/file{1..10}

If you run rsync with -azP again, the output shows how Rsync intelligently re-copies only the modified files:

rsync -azP source destination

Output
sending incremental file list
file1
            0 100%    0.00kB/s    0:00:00 (xfer#1, to-check=99/101)
file10
            0 100%    0.00kB/s    0:00:00 (xfer#2, to-check=98/101)
file2
            0 100%    0.00kB/s    0:00:00 (xfer#3, to-check=87/101)
file3
            0 100%    0.00kB/s    0:00:00 (xfer#4, to-check=76/101)
. . .

By default, rsync does not delete any files from the destination directory. To keep the two directories in sync, delete the files from the destination directory if it's removed from the source.

You can change it with the --delete option. Before that, use the -n, the --dry-run option to perform a test:

rsync -an --delete source destination

You can exclude some files or directories located inside a directory while syncing. It also lets you specify them in a comma-separated list after the --exclude= option:

rsync -a --exclude=pattern_to_exclude source destination

Exclude a specified pattern by overriding the exclusion for files that match a different pattern. You can use the --include= option:

rsync -a --exclude=pattern_to_exclude --include=pattern_to_include source destination

The --backup option is used to store backups of important files. It’s used along with the --backup-dir option. It specifies the directory where the backup files are stored:

rsync -a --delete --backup --backup-dir=/path/to/backups /path/to/source destination

Summary

Rsync or Remote Sync lets you transfer files and directories to local and remote destinations.

Rsync is used for mirroring, backups, or migrating data to other servers. It is an efficient tool that copies only the changes from the source and offers customization options.

We covered how to sync local and remote directories with rsync. The tool is useful to streamline file transfers and perform file-level operations.

For more useful guides, check out CloudPanel tutorials.