What is the importance of a backup system? Everything.
No infallible system exists, the datacenter provider can fail (do you remember the OVH fire?), the hardware of your dedicated server, a disastrous data manipulation by a user in the back office, an out-of-control synchronization process, a failed Magento update… so many and so many possible causes with a disastrous end that whoever does not have an efficient backup system as close to the current moment as possible will sooner or later find themselves in serious trouble.
Agreeing on this point, let me explain my current backup system in case it is useful to another sysadmin. Before discovering ZFS, I used to use tools like BackupPC, rsync + various scripts and other ingenious solutions of different open source tools but all of them turned out to be far inferior to the simplicity and effectiveness of ZFS + snapshots + live compression + rsync.
As an example the backup system I implemented for a Magento hosting/cloud provider in France that has been in production for more than 2 years. It currently performs daily backup of more than 250 servers (between VPS, internal and dedicated servers from all over the world) which is performed simultaneously on two backup servers in different locations (Poland and Finland). Containing 7 “complete” backups one for each day of the week plus another on days 1 and 15 of each month that are kept and the oldest months are deleted while there is space on the backup server, the size of the ZFS pool is about 12T (zstd-5 compression) for a complete copy of all data from all servers, from there add the differential copy made each day. The backup completes daily in less than 7 hours for all servers with the result of a “complete” copy in each backup performed.
The basis of this backup system is ZFS snapshots and automatic compression in the filesystem itself (whoever does not know this filesystem should study it immediately). ZFS is not the best filesystem for all uses and I do not defend it as such but for the purpose of a backup system it seems excellent to me.
It is preferable to have a dedicated server for backup tasks and also with sufficient RAM, ZFS needs a lot of it to work optimally especially with large volumes of data. I also prefer the operating system to be FreeBSD, it is the system that best integrates with ZFS, with Linux I have also had memory management problems, something that has not yet happened to me with FreeBSD in its integration with ZFS.
Once your FreeBSD system is installed and the ZFS pool configured (there are numerous parameters to optimize) the backup system is quite simple. It basically consists of:
- Taking a snapshot of the dataset where the copy of the server to back up is stored. Using a simple nomenclature for the snapshot names it is easy to achieve, for example, a weekly rotation and thus have a complete week of snapshots. Each day (where X is the day of the week):
- zfs destroy vpsdataset@dowX
- zfs snapshot vpsdataset@dowX
- Performing an rsync (-a –numeric-ids -H –delete) from the remote server to the vpsdataset.
And that is it. Naturally there are more variables to take into account such as optimizing an “excludes” list for rsync, having locally on each server a process that makes you a local copy of the database or optimizing the SSH encryption protocol for rsync. But basically this is the idea.
The great advantage of snapshots is that their access in case of need is lightning fast and available instantly with de facto transparent compression. You can go to the vpsdataset/.zfs/snapshot/ directory and there have access to all the snapshots taken and access any file you need or do a complete rsync to the server and restore it.
# cd /vpsdataset/.zfs
# ls
snapshot/
# cd snapshot/
# ls -la
total 128
dr-xr-xr-x+ 17 root wheel 17B 20 jun. 02:36 ./
dr-xr-xr-x+ 3 root wheel 3B 28 ene. 2021 ../
drwxr-xr-x 14 root wheel 22B 22 sept. 2021 dom0201/
drwxr-xr-x 14 root wheel 22B 22 sept. 2021 dom0301/
drwxr-xr-x 14 root wheel 22B 22 sept. 2021 dom0401/
drwxr-xr-x 14 root wheel 22B 22 sept. 2021 dom0416/
drwxr-xr-x 14 root wheel 22B 22 sept. 2021 dom0501/
drwxr-xr-x 14 root wheel 22B 22 sept. 2021 dom0516/
drwxr-xr-x 14 root wheel 22B 22 sept. 2021 dom0601/
drwxr-xr-x 14 root wheel 22B 22 sept. 2021 dom0616/
drwxr-xr-x 14 root wheel 22B 22 sept. 2021 dow1/
drwxr-xr-x 14 root wheel 22B 22 sept. 2021 dow2/
drwxr-xr-x 14 root wheel 22B 22 sept. 2021 dow3/
drwxr-xr-x 14 root wheel 22B 22 sept. 2021 dow4/
drwxr-xr-x 14 root wheel 22B 22 sept. 2021 dow5/
drwxr-xr-x 14 root wheel 22B 22 sept. 2021 dow6/
drwxr-xr-x 14 root wheel 22B 22 sept. 2021 dow7/
# ls -la dow1/
total 75
drwxr-xr-x 14 root wheel 22B 22 sept. 2021 ./
dr-xr-xr-x+ 17 root wheel 17B 20 jun. 02:36 ../
-rw-r--r-- 1 root wheel 4,8K 22 jun. 2018 .BUILDINFO
-rw-r--r-- 1 root wheel 406B 25 jun. 2018 .INSTALL
-rw-r--r-- 1 root wheel 460B 22 jun. 2018 .MTREE
-rw-r--r-- 1 root wheel 406B 22 jun. 2018 .PKGINFO
lrwxrwxrwx 1 root wheel 7B 19 ene. 2021 bin@ -> usr/bin
drwxr-xr-x 4 root wheel 8B 13 mar. 2021 boot/
drwxr-xr-x 53 root wheel 148B 6 jun. 11:59 etc/
drwxr-xr-x 3 root wheel 3B 19 abr. 2021 home/
lrwxrwxrwx 1 root wheel 7B 19 ene. 2021 lib@ -> usr/lib
lrwxrwxrwx 1 root wheel 7B 19 ene. 2021 lib64@ -> usr/lib
drwx------ 2 root wheel 2B 28 ene. 2021 lost+found/
drwxr-xr-x 2 root wheel 2B 5 ene. 2018 mnt/
drwxr-xr-x 2 root wheel 2B 5 ene. 2018 opt/
drwxr-xr-x 4 root wheel 4B 28 ene. 2021 ovh/
drwxr-x--- 9 root wheel 25B 11 jun. 10:56 root/
drwxr-xr-x 2 root wheel 2B 29 mar. 2021 run/
lrwxrwxrwx 1 root wheel 7B 19 ene. 2021 sbin@ -> usr/bin
drwxr-xr-x 4 root wheel 4B 5 ene. 2018 srv/
drwxr-xr-x 8 root wheel 10B 6 jun. 11:59 usr/
drwxr-xr-x 13 root wheel 17B 4 jun. 02:44 var/
# zfs get all vpsdataset|grep compress
vpsdataset compressratio 3.55x -
vpsdataset compression zstd-5 inherited from backup
vpsdataset refcompressratio 2.27x -
* vpsdataset is used as an example name without actual utility