Proxmox 7 LXC Backup Failed (exit code 23)

One of my Proxmox CT backup has been unable to be saved for several days, and I'm unsure about the reason. Based on my testing, the Stop-mode backup operates effectively, while the Suspend mode does not function as intended.

Here is the log of attempted vzdump:

INFO: starting final sync /proc/180682/root/ to /mnt/storage/dump/vzdumptmp209244_1006/

INFO: resume vm

INFO: guest is online again after 3 seconds

ERROR: Backup of VM 1006 failed - command 'rsync --stats -h -X -A --numeric-ids -aH --delete --no-whole-file --inplace --one-file-system --relative '--exclude=/var/lib/php/sessions/?*' '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' /proc/180682/root//./ /mnt/storage/dump/vzdumptmp209244_1006/' failed: exit code 23

INFO: Failed at 2023-07-10 22:10:06

INFO: Backup job finished with errors

job errors

By default, for systems based on Bullseye, the 'fs.protected_regular' sysctl is set to '2' instead of the previous value of '0'. This change poses a problem for rsync's `--inplace` mode when dealing with protected files, as even the root user cannot open them with O_CREAT.

An example of this issue can be observed in Debian (or Debian-based) containers that use PHP. In these containers, the session directory '/var/lib/php/sessions' has sticky permissions, is world-writable, owned by root, and contains session files typically owned by www-data. If any of these session files are modified between the first and second rsync runs, the second run and consequently the backup operation will fail.

The solution is set fs.protected_regular in Proxmox before backup LXC (or make it permanent in /etc/sysctl.conf):

# sysctl fs.protected_regular=0