SOLVED - XFS "The structure needs cleaning"

michelandre · April 8, 2020, 9:40pm

Hi all,

NethServer Version: 7.6.1810
Module: Backup Data

Backup Data gives error because XFS is corrupted ???

OSError: [Errno 117] Structure needs cleaning: ’ /var/lib/nethserver/vmail/toto@toto.com/Maildir/ .file_name:2,Sac’.

Imposible to delete/rename that file.

# df -h
Sys. de fichiers             Taille Utilisé Dispo Uti% Monté sur
/dev/mapper/VolGroup-lv_root   457G     25G  433G   6% /
devtmpfs                       3,8G       0  3,8G   0% /dev
tmpfs                          3,8G     72K  3,8G   1% /dev/shm
tmpfs                          3,8G    9,0M  3,8G   1% /run
tmpfs                          3,8G       0  3,8G   0% /sys/fs/cgroup
/dev/sda1                     1014M    330M  685M  33% /boot
tmpfs                          776M       0  776M   0% /run/user/0
#

# cat /proc/partitions
major minor  #blocks  name

   8        0  488386584 sda
   8        1    1048576 sda1
   8        2  487336960 sda2
  11        0    1048575 sr0
 253        0  479203328 dm-0
 253        1    8126464 dm-1
#

# ls -als /dev/mapper/
control           VolGroup-lv_root  VolGroup-lv_swap
#

# ls -als /dev/mapper/*
0 crw------- 1 root root 10, 236  8 avril 16:56 /dev/mapper/control
0 lrwxrwxrwx 1 root root       7  8 avril 16:56 /dev/mapper/VolGroup-lv_root -> ../dm-0
0 lrwxrwxrwx 1 root root       7  8 avril 16:56 /dev/mapper/VolGroup-lv_swap -> ../dm-1
#

After Googling for a while, it looks like I have to use:

/sbin/xfs_repair /dev/mapper/VolGroup-lv_root

I burned a Live CD of OpenSuse, boot with it and run the command -> permisson denied…
Do I need special parameter to xfs_repair ???
Maybe I have to 'su - root" but I prefer to wait for a NethServer advice from a pro.

What is the best way to repair?

Michel-André

mrmarkuz · April 8, 2020, 11:15pm

Did you have root permissions when running the command?

Redhat way:

Safe repair by dumping partition to a file to test before doing the operation on the partition (good when you got dirty logs):

michelandre · April 9, 2020, 4:57am

Hi Markus,

Thank you very much for your answer.

From the second link:

@MichaelKohne Restoring from backup, of course. You shouldn’t get anywhere near this level of hell unless you haven’t got backups. – Michael Hampton♦ May 19 '16 at 7:31

I have backups, so I restored the complete content of the directory containing the corrupted file.
-Return message from the restoration: “Restored to the original position”.
-The corrupted file is still there with the restored original file and, both have the exact same name:

# ls -ls cur/
ls: unable to access cur/.file_name:2,Sac:  The structure needs cleaning
total 8
8 -rw------- 1 vmail clamupdate 6471 15 juin   2012 .file_name:2,Sac:2,Sac
? -????????? ? ?     ?             ?              ? .file_name:2,Sac:2,Sac
#

I think I will have to go “near this level of hell”

Again, thank you very much for your answer. I will keep you informed,

Michel-André

Andy_Wismer · April 9, 2020, 8:19am

Hi

I can confirm that the NethServer (rarely) needs a repair, and when it does, xfs_repair on an unmounted system WORKS !!!

XFS will not do an automatic FS-check like Ext4 does during boot. (I still consider XFS better than Ext4 in almost all cases).

Doing a Proxmox image before is prudent (My NethServers run almost ALL in Proxmox), otherwise a clone with eg clonezilla is a very good (and safe) idea.

I use SystemRescueCD, that has sofar worked well (3 times used on different servers).
Here, LVM is recognized from the start, you only need to run this:

xfs_repair /dev/mapper/VolGroup-lv_root

This is fairly fast, even for a large filesystem.

Note: After the power outage which caused this at one of my clients, the client finally bought a new UPS (The old one had died a year earlier at 11 years of age)!

My 2 cents
Andy

filippo_carletti · April 9, 2020, 9:38am

You can use the Rescue option booting the NethServer (or CentOS) CD.
When presented with the option to find the system and mount the rootfs you could go to a shell, start raid, lvm etc and then run xfs_repair.
Or, easier, use rescue option to mount / read-only and run xfs_repair -d /dev/mapper/VolGroup-lv_root

BTW, I needed this only once (and I see a lot of NethServer), xfs is really robust.

Andy_Wismer · April 9, 2020, 10:31am

@filippo_carletti

I quite agree XFS is really robust.

The 3 NethServers all were involved in either a Power Outage, or in one case a NAS Outage (Replaced by warranty), but the System working, but badly corrupted FileSystem.

The system worked, but no access to Server-manager or updates (Even with yum update!).
File, Mail and even NextCloud worked.

xfs_repair repared that 500 GB virtual system without any issues!

I still prefer using SystemRescueCD, as that starts RAID and LVM, but does not mount them.
xfs_repair needs to run on an unmounted system to repair it, just saves a few steps…

Amusing sidenote:
At the time the NAS died early in the morning at the hotel, there were 100 guests to a conference titeled “The risks of digitalization” - and Internet wasn’t working due to the outage…
Those are the moments you need and are really glad for fast recovery!

My 2 cents
Andy

michelandre · April 9, 2020, 4:58pm

Hi André,

I booted with SystemRescueCD an executed the command:

xfs_repair /dev/mapper/VolGroup-lv_root

Marvellous, it worked perfectly. It found the corrupted file and repaired it.

I did a new full backup and SUCCESS.

Thousand thanks for your help,

Michel-André

Andy_Wismer · April 9, 2020, 5:00pm

Always glad to be able to help others who use FOSS!

PS: The first time it happened, since everything “seemed” to work, except backup, I didn’t notice how severe the problem actually was…
Several other tries failed, my trusty SystemRescueCD saved the day easily!