One of my Proxmox CT backup has been unable to be saved for several days, and I'm unsure about the reason. Based on my testing, the Stop-mode backup operates effectively, while the Suspend mode does not function as intended.
Here is the log of attempted vzdump:
INFO: starting final sync /proc/180682/root/ to /mnt/storage/dump/vzdumptmp209244_1006/
INFO: resume vm
INFO: guest is online again after 3 seconds
ERROR: Backup of VM 1006 failed - command 'rsync --stats -h -X -A --numeric-ids -aH --delete --no-whole-file --inplace --one-file-system --relative '--exclude=/var/lib/php/sessions/?*' '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' /proc/180682/root//./ /mnt/storage/dump/vzdumptmp209244_1006/' failed: exit code 23
INFO: Failed at 2023-07-10 22:10:06
INFO: Backup job finished with errors
job errors
By default, for systems based on Bullseye, the 'fs.protected_regular' sysctl is set to '2' instead of the previous value of '0'. This change poses a problem for rsync's `--inplace` mode when dealing with protected files, as even the root user cannot open them with O_CREAT.
An example of this issue can be observed in Debian (or Debian-based) containers that use PHP. In these containers, the session directory '/var/lib/php/sessions' has sticky permissions, is world-writable, owned by root, and contains session files typically owned by www-data. If any of these session files are modified between the first and second rsync runs, the second run and consequently the backup operation will fail.
The solution is set fs.protected_regular in Proxmox before backup LXC (or make it permanent in /etc/sysctl.conf):
# sysctl fs.protected_regular=0