Diskusage-module ends in error

rolf · November 13, 2016, 1:00pm

Hi all,
when updating the ‘disk-usage’ module, this ends in an error:
Task completed with errors
S20nethserver-duc-index #1 (exit status 256)
(that’s on the still white web-page).
/var/log/messages tells me:

Nov 13 13:56:58 helium esmith::event[6192]: Event: nethserver-duc-save
Nov 13 13:56:59 helium esmith::event[6192]: Traceback (most recent call last):
Nov 13 13:56:59 helium esmith::event[6192]: File “/usr/share/duc/xml2json.py”, line 184, in
Nov 13 13:56:59 helium esmith::event[6192]: main()
Nov 13 13:56:59 helium esmith::event[6192]: File “/usr/share/duc/xml2json.py”, line 179, in main
Nov 13 13:56:59 helium esmith::event[6192]: out = xml2json(input, options, strip_ns, strip)
Nov 13 13:56:59 helium esmith::event[6192]: File “/usr/share/duc/xml2json.py”, line 151, in xml2json
Nov 13 13:56:59 helium esmith::event[6192]: elem = ET.fromstring(xmlstring)
Nov 13 13:56:59 helium esmith::event[6192]: File “”, line 124, in XML
Nov 13 13:56:59 helium esmith::event[6192]: cElementTree.ParseError: not well-formed (invalid token): line 3366, column 40
Nov 13 13:56:59 helium esmith::event[6192]: Action: /etc/e-smith/events/nethserver-duc-save/S20nethserver-duc-index FAILED: 1 [0.816849]
Nov 13 13:56:59 helium esmith::event[6192]: Event: nethserver-duc-save FAILED

I run NS on 1 disk, no custom-partitioning.
What more info can I provide?

alefattorini · November 14, 2016, 11:06am

@edoardo_spadoni would you mind taking a look at this?

edoardo_spadoni · November 14, 2016, 11:24am

Hi,
are duc.db and tree.xml files correctly created in /var/cache/duc?

Can you provide the output of ls -la /var/cache/duc/?

Thanks.

rolf · November 14, 2016, 6:46pm

The files are there:

root@helium:duc> $ ls -la
total 12072
drwxr-xr-x. 2 root root 49 Oct 26 11:19 .
drwxr-xr-x. 12 root root 4096 Oct 26 12:50 …
-rw-r–r–. 1 root root 12112384 Nov 14 19:40 duc.db
-rw-r–r–. 1 root root 0 Nov 14 19:40 duc.json
-rw-r–r–. 1 root root 236090 Nov 14 19:40 tree.xml

When I look into the xml I see files I recognise.
When I move them out of this dir, and push the button again, the files are recreated, resulting in the same error.
Anymore I can do to troubleshoot?

edoardo_spadoni · November 15, 2016, 8:14am

Ok, it seems that all files are correct try to execute:

/usr/share/duc/xml2json.py /var/cache/duc/tree.xml

and paste the output (Should be a valid JSON string)

rolf · November 15, 2016, 9:00am

OK, I’ll do tonight when I’m home.
Till then: I’m serving a share with music, and the filenames sometimes have special characters in them (like songs from Michael Bublé)
Is it possible this module trips on that?

edoardo_spadoni · November 15, 2016, 9:40am

The special character like é is encoded in JSON file in \u00e9.

Theoretically the parsing is not broken and in the graphical view, the character is properly decoded.

I’m waiting your output

rolf · November 15, 2016, 6:37pm

here’s the output:

root@helium:~> $ /usr/share/duc/xml2json.py /var/cache/duc/tree.xml
Traceback (most recent call last):
File “/usr/share/duc/xml2json.py”, line 184, in
main()
File “/usr/share/duc/xml2json.py”, line 179, in main
out = xml2json(input, options, strip_ns, strip)
File “/usr/share/duc/xml2json.py”, line 151, in xml2json
elem = ET.fromstring(xmlstring)
File “”, line 124, in XML
cElementTree.ParseError: not well-formed (invalid token): line 3366, column 40

looks familiar; it’s the same as the logged error…

edoardo_spadoni · November 16, 2016, 8:32am

Ok, this is the same output of /var/log/messages, try to change the /usr/share/duc/xml2json.py like this:

Step 1
At the top of file after the imports remove this

import xml.etree.cElementTree as ET

and add:

from lxml import etree

Step 2
Line 151 remove

elem = ET.fromstring(xmlstring)

and add

parser = etree.XMLParser(recover=True)
elem = etree.fromstring(xmlstring, parser=parser)

Step 3
Finally install python-lxml with yum install python-lxml

After this, repeat the parsing procedure

/usr/share/duc/xml2json.py /var/cache/duc/tree.xml

If the output is a valid JSON I’ll open a issue, you find a bug

PS. Check also if the Dashboard widget is correctly displayed.

Thanks

rolf · November 16, 2016, 12:22pm

OK, did as instructed:
when running

/usr/share/duc/xml2json.py /var/cache/duc/tree.xml

it ended in an indentationerror:

File “/usr/share/duc/xml2json.py”, line 153
return elem2json(elem, options, strip_ns=strip_ns, strip=strip)
^

When adding some blanks before the lines you quoted, I get another error:

root@helium:tmp> $ /usr/share/duc/xml2json.py /var/cache/duc/tree.xml
Traceback (most recent call last):
File “/usr/share/duc/xml2json.py”, line 185, in
main()
File “/usr/share/duc/xml2json.py”, line 180, in main
out = xml2json(input, options, strip_ns, strip)
File “/usr/share/duc/xml2json.py”, line 153, in xml2json
return elem2json(elem, options, strip_ns=strip_ns, strip=strip)
File “/usr/share/duc/xml2json.py”, line 145, in elem2json
return json.dumps(elem_to_internal(elem, strip_ns=strip_ns, strip=strip))
File “/usr/share/duc/xml2json.py”, line 78, in elem_to_internal
v = elem_to_internal(subelem, strip_ns=strip_ns, strip=strip)
File “/usr/share/duc/xml2json.py”, line 78, in elem_to_internal
v = elem_to_internal(subelem, strip_ns=strip_ns, strip=strip)
File “/usr/share/duc/xml2json.py”, line 78, in elem_to_internal
v = elem_to_internal(subelem, strip_ns=strip_ns, strip=strip)
File “/usr/share/duc/xml2json.py”, line 78, in elem_to_internal
v = elem_to_internal(subelem, strip_ns=strip_ns, strip=strip)
File “/usr/share/duc/xml2json.py”, line 78, in elem_to_internal
v = elem_to_internal(subelem, strip_ns=strip_ns, strip=strip)
File “/usr/share/duc/xml2json.py”, line 78, in elem_to_internal
v = elem_to_internal(subelem, strip_ns=strip_ns, strip=strip)
File “/usr/share/duc/xml2json.py”, line 74, in elem_to_internal
for key, value in list(elem.attrib.items()):
File “lxml.etree.pyx”, line 2320, in lxml.etree._Attrib.items (src/lxml/lxml.etree.c:56138)
File “apihelpers.pxi”, line 570, in lxml.etree._collectAttributes (src/lxml/lxml.etree.c:18179)
File “apihelpers.pxi”, line 479, in lxml.etree._attributeValue (src/lxml/lxml.etree.c:17000)
File “apihelpers.pxi”, line 1313, in lxml.etree.funicode (src/lxml/lxml.etree.c:24405)
UnicodeDecodeError: ‘utf8’ codec can’t decode byte 0xe9 in position 12: invalid continuation byte

And there my knowledge ends again…

On the dashboard I see a simple green bar-graph, displaying my root partition is 26% full.
Is that what I’m supposed to see?

edoardo_spadoni · November 16, 2016, 1:14pm

Could you share your tree.xml file or are there any personal data?

If you want I try to replicate the issue on my machine.

rolf · November 16, 2016, 1:48pm

I’ll send it to you via mail/message. Not thát sensitive, but rather not have it laying around the interwebs forever.

edoardo_spadoni · November 16, 2016, 2:10pm

Ok, message received, what type of encoding you use? Can you provide the output of command
locale?

rolf · November 16, 2016, 2:26pm

Nothing changed around that since install:
root@helium:~> $ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

edoardo_spadoni · November 16, 2016, 2:33pm

If you look at your file, there are any other songs with special character like é, but the one in Michael Bublé is quite strange.

Open the tree.xml file with less and go to line 3366, as you can see, the é is not properly well encoded.

I don’t know why, but for any reason the name of that file is corrupted.

Restore the version of /usr/share/duc/xml2json.py with the original one (you can found it here: Original)
Open with vim the tree.xml file and delete the two è in line 3366
Close the file
Re-open with vim the tree.xml and write the è again
Try to execute /usr/share/duc/xml2json.py /var/cache/duc/tree.xml
The output should be valid now

rolf · November 16, 2016, 8:05pm

Indeed, after doing so, there was reasonable output. But the website still didn’t output a nice graph.
I ended up deleting the folder (sorry Michael), and then it gave me a graph.
I’ll rerip the CD and call him Michael Bubble then.
Edit: I can’t find the "yes, this answer did it!"flag…?

dnutan · November 16, 2016, 8:34pm

It’s only for the Support category.

edoardo_spadoni · November 17, 2016, 8:23am

It’s a very Strange Thing because other songs indexed have the è.

Anyway we found the solution, great @rolf

alefattorini · November 17, 2016, 9:11am

Moved, now you can