Proxmox help needed - proxmox backup ends with broken pipe

Support
1

I configured a proxmox backup which was running without errors for a while. Now I guess the dump was growing too big for /var/lib/vz. Here’s the log:

INFO: starting new backup job: vzdump 100 --mode stop --quiet 1 --mailnotification always --compress zstd --storage B200 --mailto admin@xxx.de
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2021-06-28 23:00:02
INFO: status = running
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: nethp100
INFO: include disk ‘scsi0’ ‘local-lvm:vm-100-disk-0’ 360G
INFO: stopping vm
INFO: snapshots found (not included into backup)
INFO: creating vzdump archive ‘/media/backup/dump/vzdump-qemu-100-2021_06_28-23_00_02.vma.zst’
INFO: starting kvm to execute backup task
INFO: started backup task ‘fb002d20-b543-435f-aa68-e1c13a194ce6’
INFO: resuming VM again after 15 seconds
INFO: 0% (210.2 MiB of 360.0 GiB) in 3s, read: 70.1 MiB/s, write: 69.6 MiB/s
INFO: 1% (4.8 GiB of 360.0 GiB) in 8s, read: 949.5 MiB/s, write: 22.0 MiB/s
INFO: 2% (8.1 GiB of 360.0 GiB) in 11s, read: 1.1 GiB/s, write: 0 B/s
INFO: 3% (11.0 GiB of 360.0 GiB) in 15s, read: 761.5 MiB/s, write: 528.0 KiB/s
INFO: 4% (15.1 GiB of 360.0 GiB) in 22s, read: 592.8 MiB/s, write: 0 B/s
INFO: 5% (18.6 GiB of 360.0 GiB) in 25s, read: 1.2 GiB/s, write: 0 B/s
INFO: 6% (21.9 GiB of 360.0 GiB) in 40s, read: 223.6 MiB/s, write: 2.4 MiB/s
INFO: 7% (25.3 GiB of 360.0 GiB) in 45s, read: 707.1 MiB/s, write: 1.2 MiB/s
INFO: 8% (29.1 GiB of 360.0 GiB) in 49s, read: 967.2 MiB/s, write: 9.3 MiB/s
INFO: 9% (32.5 GiB of 360.0 GiB) in 1m 7s, read: 190.3 MiB/s, write: 40.6 MiB/s
INFO: 10% (36.1 GiB of 360.0 GiB) in 2m, read: 69.3 MiB/s, write: 68.9 MiB/s
INFO: 11% (39.6 GiB of 360.0 GiB) in 3m 10s, read: 51.9 MiB/s, write: 51.3 MiB/s
INFO: 12% (43.2 GiB of 360.0 GiB) in 4m 16s, read: 56.6 MiB/s, write: 56.6 MiB/s
INFO: 13% (46.8 GiB of 360.0 GiB) in 5m 10s, read: 68.1 MiB/s, write: 65.2 MiB/s
INFO: 14% (50.4 GiB of 360.0 GiB) in 6m, read: 73.3 MiB/s, write: 72.6 MiB/s
INFO: 15% (54.1 GiB of 360.0 GiB) in 7m 5s, read: 57.5 MiB/s, write: 57.5 MiB/s
INFO: 16% (57.6 GiB of 360.0 GiB) in 8m 14s, read: 52.5 MiB/s, write: 52.4 MiB/s
INFO: 17% (61.2 GiB of 360.0 GiB) in 9m 11s, read: 65.1 MiB/s, write: 64.4 MiB/s
INFO: 18% (64.9 GiB of 360.0 GiB) in 10m 18s, read: 55.7 MiB/s, write: 55.0 MiB/s
INFO: 19% (68.4 GiB of 360.0 GiB) in 11m 14s, read: 65.1 MiB/s, write: 64.9 MiB/s
INFO: 20% (72.0 GiB of 360.0 GiB) in 12m 16s, read: 59.3 MiB/s, write: 59.3 MiB/s
INFO: 21% (75.8 GiB of 360.0 GiB) in 12m 29s, read: 296.9 MiB/s, write: 44.8 MiB/s
INFO: 22% (79.9 GiB of 360.0 GiB) in 12m 33s, read: 1.0 GiB/s, write: 4.4 MiB/s
INFO: 23% (83.2 GiB of 360.0 GiB) in 12m 36s, read: 1.1 GiB/s, write: 134.7 KiB/s
INFO: 24% (87.0 GiB of 360.0 GiB) in 12m 40s, read: 947.8 MiB/s, write: 55.0 KiB/s
INFO: 25% (90.3 GiB of 360.0 GiB) in 12m 44s, read: 849.1 MiB/s, write: 13.9 MiB/s
INFO: 26% (94.2 GiB of 360.0 GiB) in 12m 48s, read: 1011.7 MiB/s, write: 760.0 KiB/s
INFO: 27% (98.2 GiB of 360.0 GiB) in 12m 52s, read: 1011.1 MiB/s, write: 4.7 MiB/s
INFO: 28% (101.4 GiB of 360.0 GiB) in 12m 55s, read: 1.1 GiB/s, write: 78.7 KiB/s
INFO: 29% (104.8 GiB of 360.0 GiB) in 12m 58s, read: 1.1 GiB/s, write: 121.3 KiB/s
INFO: 30% (108.1 GiB of 360.0 GiB) in 13m 1s, read: 1.1 GiB/s, write: 436.0 KiB/s
INFO: 31% (112.6 GiB of 360.0 GiB) in 13m 5s, read: 1.1 GiB/s, write: 2.0 KiB/s
INFO: 32% (115.3 GiB of 360.0 GiB) in 13m 35s, read: 89.9 MiB/s, write: 45.9 MiB/s
INFO: 33% (118.8 GiB of 360.0 GiB) in 14m 39s, read: 57.4 MiB/s, write: 57.1 MiB/s
INFO: 34% (122.4 GiB of 360.0 GiB) in 15m 52s, read: 50.5 MiB/s, write: 50.4 MiB/s
INFO: 35% (126.0 GiB of 360.0 GiB) in 17m, read: 53.7 MiB/s, write: 53.0 MiB/s
INFO: 36% (129.6 GiB of 360.0 GiB) in 18m 2s, read: 59.8 MiB/s, write: 59.5 MiB/s
INFO: 37% (133.2 GiB of 360.0 GiB) in 19m 6s, read: 57.7 MiB/s, write: 57.5 MiB/s
INFO: 38% (136.9 GiB of 360.0 GiB) in 20m 10s, read: 58.0 MiB/s, write: 58.0 MiB/s
INFO: 39% (140.4 GiB of 360.0 GiB) in 21m 6s, read: 64.9 MiB/s, write: 64.7 MiB/s
INFO: 40% (144.1 GiB of 360.0 GiB) in 22m 6s, read: 62.2 MiB/s, write: 62.1 MiB/s
INFO: 41% (147.6 GiB of 360.0 GiB) in 23m 8s, read: 59.1 MiB/s, write: 59.1 MiB/s
INFO: 42% (151.2 GiB of 360.0 GiB) in 24m 11s, read: 58.6 MiB/s, write: 58.6 MiB/s
INFO: 43% (154.8 GiB of 360.0 GiB) in 25m 18s, read: 54.4 MiB/s, write: 54.3 MiB/s
INFO: 44% (158.4 GiB of 360.0 GiB) in 26m 18s, read: 61.9 MiB/s, write: 61.5 MiB/s
INFO: 45% (162.0 GiB of 360.0 GiB) in 27m 29s, read: 51.6 MiB/s, write: 51.6 MiB/s
INFO: 46% (165.7 GiB of 360.0 GiB) in 28m 23s, read: 69.5 MiB/s, write: 69.5 MiB/s
INFO: 47% (169.2 GiB of 360.0 GiB) in 29m 31s, read: 53.5 MiB/s, write: 53.5 MiB/s
INFO: 48% (172.9 GiB of 360.0 GiB) in 30m 22s, read: 73.4 MiB/s, write: 68.8 MiB/s
INFO: 49% (176.5 GiB of 360.0 GiB) in 30m 30s, read: 461.7 MiB/s, write: 19.3 MiB/s
INFO: 50% (180.8 GiB of 360.0 GiB) in 30m 37s, read: 635.3 MiB/s, write: 13.3 MiB/s
INFO: 51% (184.2 GiB of 360.0 GiB) in 30m 40s, read: 1.1 GiB/s, write: 1.3 KiB/s
INFO: 52% (187.6 GiB of 360.0 GiB) in 30m 43s, read: 1.1 GiB/s, write: 32.0 KiB/s
INFO: 53% (191.0 GiB of 360.0 GiB) in 30m 46s, read: 1.1 GiB/s, write: 4.0 KiB/s
INFO: 54% (195.5 GiB of 360.0 GiB) in 30m 50s, read: 1.1 GiB/s, write: 81.0 KiB/s
INFO: 55% (198.0 GiB of 360.0 GiB) in 31m 31s, read: 63.5 MiB/s, write: 50.6 MiB/s
INFO: 56% (201.6 GiB of 360.0 GiB) in 32m 35s, read: 57.5 MiB/s, write: 57.4 MiB/s
INFO: 57% (205.2 GiB of 360.0 GiB) in 33m 36s, read: 60.3 MiB/s, write: 60.3 MiB/s
INFO: 58% (208.8 GiB of 360.0 GiB) in 34m 28s, read: 71.4 MiB/s, write: 71.4 MiB/s
INFO: 59% (212.5 GiB of 360.0 GiB) in 35m 31s, read: 59.2 MiB/s, write: 58.6 MiB/s
INFO: 60% (216.0 GiB of 360.0 GiB) in 36m 20s, read: 74.0 MiB/s, write: 74.0 MiB/s
INFO: 61% (219.7 GiB of 360.0 GiB) in 37m 29s, read: 54.4 MiB/s, write: 54.2 MiB/s
INFO: 62% (223.2 GiB of 360.0 GiB) in 38m 22s, read: 68.8 MiB/s, write: 68.8 MiB/s
INFO: 63% (226.9 GiB of 360.0 GiB) in 39m 31s, read: 53.8 MiB/s, write: 53.7 MiB/s
INFO: 64% (230.4 GiB of 360.0 GiB) in 40m 23s, read: 69.9 MiB/s, write: 69.9 MiB/s
INFO: 65% (234.0 GiB of 360.0 GiB) in 41m 20s, read: 65.5 MiB/s, write: 62.7 MiB/s
INFO: 66% (237.6 GiB of 360.0 GiB) in 42m 18s, read: 63.4 MiB/s, write: 63.0 MiB/s
INFO: 67% (242.1 GiB of 360.0 GiB) in 42m 55s, read: 122.7 MiB/s, write: 44.3 MiB/s
INFO: 68% (244.9 GiB of 360.0 GiB) in 43m 10s, read: 190.5 MiB/s, write: 63.1 MiB/s
INFO: 69% (249.3 GiB of 360.0 GiB) in 43m 19s, read: 509.0 MiB/s, write: 34.2 MiB/s
INFO: 70% (252.4 GiB of 360.0 GiB) in 43m 22s, read: 1.0 GiB/s, write: 193.3 KiB/s
INFO: 71% (255.7 GiB of 360.0 GiB) in 43m 29s, read: 479.7 MiB/s, write: 6.5 MiB/s
INFO: 72% (260.1 GiB of 360.0 GiB) in 43m 33s, read: 1.1 GiB/s, write: 1.6 MiB/s
INFO: 73% (262.9 GiB of 360.0 GiB) in 43m 36s, read: 963.5 MiB/s, write: 6.9 MiB/s
INFO: 74% (267.1 GiB of 360.0 GiB) in 43m 40s, read: 1.0 GiB/s, write: 691.0 KiB/s
INFO: 75% (270.5 GiB of 360.0 GiB) in 43m 43s, read: 1.1 GiB/s, write: 2.7 KiB/s
INFO: 76% (274.7 GiB of 360.0 GiB) in 43m 47s, read: 1.0 GiB/s, write: 3.4 MiB/s
INFO: 77% (277.9 GiB of 360.0 GiB) in 43m 50s, read: 1.1 GiB/s, write: 1.1 MiB/s
INFO: 78% (280.9 GiB of 360.0 GiB) in 44m 40s, read: 59.7 MiB/s, write: 59.2 MiB/s
INFO: 79% (284.4 GiB of 360.0 GiB) in 45m 53s, read: 50.2 MiB/s, write: 50.2 MiB/s
INFO: 80% (288.0 GiB of 360.0 GiB) in 46m 59s, read: 55.9 MiB/s, write: 55.3 MiB/s
INFO: 81% (291.6 GiB of 360.0 GiB) in 48m 1s, read: 59.2 MiB/s, write: 58.6 MiB/s
INFO: 82% (295.2 GiB of 360.0 GiB) in 49m 6s, read: 56.8 MiB/s, write: 56.7 MiB/s
INFO: 83% (298.9 GiB of 360.0 GiB) in 50m 8s, read: 60.2 MiB/s, write: 60.1 MiB/s
INFO: 84% (302.4 GiB of 360.0 GiB) in 51m 8s, read: 60.9 MiB/s, write: 59.3 MiB/s
zstd: error 25 : Write error : No space left on device (cannot write compressed block)
INFO: 84% (303.3 GiB of 360.0 GiB) in 51m 34s, read: 34.7 MiB/s, write: 34.7 MiB/s
ERROR: vma_queue_write: write error - Broken pipe
INFO: aborting backup job
ERROR: Backup of VM 100 failed - vma_queue_write: write error - Broken pipe
INFO: Failed at 2021-06-28 23:51:56
INFO: Backup job finished with errors

TASK ERROR: job errors

df -hl /

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/pve-root 96G 7.1G 89G 8% /

I assume the pve-root should be increased? If so, what would be the best approach?

Thanks for any hint.

regards,
stefan

2

@schulzstefan

Hello Stefan

Here:

nano /etc/vzdump.conf

you can set the proxmox internal TEMP, this is where proxmox actually does the “zipping” (=compression) of the backup file.

tmpdir: /tmp

Note: this can be on an additional partition / disk if you’re short of space and do not want to risk reformatting everything.

If not defined, Proxmox will use the target storage for creating the TGZ files, very much slower over eg NFS than locally, besides, you can run out of space…

So it’s better to be defined, and you know there’s space to do your jobs, like the important backups!

My 2 cents
Andy

2 Likes
3

Hello Andy,

thank’s for jumping in.

I followed your advise - but nevertheless I think, that the 360 GB cannot be any longer compressed in the 89 (96) GB of pve-root. Therefore the pipe is breaking. Should I attach an USB drive for compression?

4

A USB drive would work, or maybe, if possible a simple SATA disk. if you have a possibility to connect the drive (Like using the SATA CD/DVD connection…).

The USB is kinda slow, unless the hardware has full USB3 support…

You can format the drive as an ext4 or xfs partition, and add it in to the fstab of proxmox.
(So it’s there after the next reboot…)!

My 2 cents
Andy

5

I’ll give it a try.

regards,
stefan

1 Like
6

Back on track again. Backup went through, everything fine again. Thank’s for yout help.

regards,
stefan

1 Like
7

@schulzstefan

Hi Stefan

Please mark this topic as solved, it may help others in future with similiar issues…

Thanks!

My 2 cents
Andy

1 Like