Diskusage-module ends in error

Hi all,
when updating the ‘disk-usage’ module, this ends in an error:
Task completed with errors
S20nethserver-duc-index #1 (exit status 256)
(that’s on the still white web-page).
/var/log/messages tells me:

Nov 13 13:56:58 helium esmith::event[6192]: Event: nethserver-duc-save
Nov 13 13:56:59 helium esmith::event[6192]: Traceback (most recent call last):
Nov 13 13:56:59 helium esmith::event[6192]: File “/usr/share/duc/xml2json.py”, line 184, in
Nov 13 13:56:59 helium esmith::event[6192]: main()
Nov 13 13:56:59 helium esmith::event[6192]: File “/usr/share/duc/xml2json.py”, line 179, in main
Nov 13 13:56:59 helium esmith::event[6192]: out = xml2json(input, options, strip_ns, strip)
Nov 13 13:56:59 helium esmith::event[6192]: File “/usr/share/duc/xml2json.py”, line 151, in xml2json
Nov 13 13:56:59 helium esmith::event[6192]: elem = ET.fromstring(xmlstring)
Nov 13 13:56:59 helium esmith::event[6192]: File “”, line 124, in XML
Nov 13 13:56:59 helium esmith::event[6192]: cElementTree.ParseError: not well-formed (invalid token): line 3366, column 40
Nov 13 13:56:59 helium esmith::event[6192]: Action: /etc/e-smith/events/nethserver-duc-save/S20nethserver-duc-index FAILED: 1 [0.816849]
Nov 13 13:56:59 helium esmith::event[6192]: Event: nethserver-duc-save FAILED

I run NS on 1 disk, no custom-partitioning.
What more info can I provide?

@edoardo_spadoni would you mind taking a look at this?

Hi,
are duc.db and tree.xml files correctly created in /var/cache/duc?

Can you provide the output of ls -la /var/cache/duc/?

Thanks.

The files are there:

root@helium:duc> $ ls -la
total 12072
drwxr-xr-x. 2 root root 49 Oct 26 11:19 .
drwxr-xr-x. 12 root root 4096 Oct 26 12:50 …
-rw-r–r–. 1 root root 12112384 Nov 14 19:40 duc.db
-rw-r–r–. 1 root root 0 Nov 14 19:40 duc.json
-rw-r–r–. 1 root root 236090 Nov 14 19:40 tree.xml

When I look into the xml I see files I recognise.
When I move them out of this dir, and push the button again, the files are recreated, resulting in the same error.
Anymore I can do to troubleshoot?

Ok, it seems that all files are correct :slight_smile: try to execute:

/usr/share/duc/xml2json.py /var/cache/duc/tree.xml

and paste the output (Should be a valid JSON string)

OK, I’ll do tonight when I’m home.
Till then: I’m serving a share with music, and the filenames sometimes have special characters in them (like songs from Michael Bublé)
Is it possible this module trips on that?

The special character like é is encoded in JSON file in \u00e9.

Theoretically the parsing is not broken and in the graphical view, the character is properly decoded.

I’m waiting your output :wink:

here’s the output:

root@helium:~> $ /usr/share/duc/xml2json.py /var/cache/duc/tree.xml
Traceback (most recent call last):
File “/usr/share/duc/xml2json.py”, line 184, in
main()
File “/usr/share/duc/xml2json.py”, line 179, in main
out = xml2json(input, options, strip_ns, strip)
File “/usr/share/duc/xml2json.py”, line 151, in xml2json
elem = ET.fromstring(xmlstring)
File “”, line 124, in XML
cElementTree.ParseError: not well-formed (invalid token): line 3366, column 40

looks familiar; it’s the same as the logged error…

Ok, this is the same output of /var/log/messages, try to change the /usr/share/duc/xml2json.py like this:

Step 1
At the top of file after the imports remove this

import xml.etree.cElementTree as ET

and add:

from lxml import etree

Step 2
Line 151 remove

elem = ET.fromstring(xmlstring)

and add

parser = etree.XMLParser(recover=True)
elem = etree.fromstring(xmlstring, parser=parser)

Step 3
Finally install python-lxml with yum install python-lxml

After this, repeat the parsing procedure

/usr/share/duc/xml2json.py /var/cache/duc/tree.xml

If the output is a valid JSON I’ll open a issue, you find a bug :wink:

PS. Check also if the Dashboard widget is correctly displayed.

Thanks

1 Like

OK, did as instructed:
when running

/usr/share/duc/xml2json.py /var/cache/duc/tree.xml

it ended in an indentationerror:

File “/usr/share/duc/xml2json.py”, line 153
return elem2json(elem, options, strip_ns=strip_ns, strip=strip)
^

When adding some blanks before the lines you quoted, I get another error:

root@helium:tmp> $ /usr/share/duc/xml2json.py /var/cache/duc/tree.xml
Traceback (most recent call last):
File “/usr/share/duc/xml2json.py”, line 185, in
main()
File “/usr/share/duc/xml2json.py”, line 180, in main
out = xml2json(input, options, strip_ns, strip)
File “/usr/share/duc/xml2json.py”, line 153, in xml2json
return elem2json(elem, options, strip_ns=strip_ns, strip=strip)
File “/usr/share/duc/xml2json.py”, line 145, in elem2json
return json.dumps(elem_to_internal(elem, strip_ns=strip_ns, strip=strip))
File “/usr/share/duc/xml2json.py”, line 78, in elem_to_internal
v = elem_to_internal(subelem, strip_ns=strip_ns, strip=strip)
File “/usr/share/duc/xml2json.py”, line 78, in elem_to_internal
v = elem_to_internal(subelem, strip_ns=strip_ns, strip=strip)
File “/usr/share/duc/xml2json.py”, line 78, in elem_to_internal
v = elem_to_internal(subelem, strip_ns=strip_ns, strip=strip)
File “/usr/share/duc/xml2json.py”, line 78, in elem_to_internal
v = elem_to_internal(subelem, strip_ns=strip_ns, strip=strip)
File “/usr/share/duc/xml2json.py”, line 78, in elem_to_internal
v = elem_to_internal(subelem, strip_ns=strip_ns, strip=strip)
File “/usr/share/duc/xml2json.py”, line 78, in elem_to_internal
v = elem_to_internal(subelem, strip_ns=strip_ns, strip=strip)
File “/usr/share/duc/xml2json.py”, line 74, in elem_to_internal
for key, value in list(elem.attrib.items()):
File “lxml.etree.pyx”, line 2320, in lxml.etree._Attrib.items (src/lxml/lxml.etree.c:56138)
File “apihelpers.pxi”, line 570, in lxml.etree._collectAttributes (src/lxml/lxml.etree.c:18179)
File “apihelpers.pxi”, line 479, in lxml.etree._attributeValue (src/lxml/lxml.etree.c:17000)
File “apihelpers.pxi”, line 1313, in lxml.etree.funicode (src/lxml/lxml.etree.c:24405)
UnicodeDecodeError: ‘utf8’ codec can’t decode byte 0xe9 in position 12: invalid continuation byte

And there my knowledge ends again…

On the dashboard I see a simple green bar-graph, displaying my root partition is 26% full.
Is that what I’m supposed to see?

Could you share your tree.xml file or are there any personal data?

If you want I try to replicate the issue on my machine.

1 Like

I’ll send it to you via mail/message. Not thát sensitive, but rather not have it laying around the interwebs forever.

Ok, message received, what type of encoding you use? Can you provide the output of command
locale?

Nothing changed around that since install:
root@helium:~> $ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

If you look at your file, there are any other songs with special character like é, but the one in Michael Bublé is quite strange.

Open the tree.xml file with less and go to line 3366, as you can see, the é is not properly well encoded.

I don’t know why, but for any reason the name of that file is corrupted.

  • Restore the version of /usr/share/duc/xml2json.py with the original one (you can found it here: Original)
  • Open with vim the tree.xml file and delete the two è in line 3366
  • Close the file
  • Re-open with vim the tree.xml and write the è again
  • Try to execute /usr/share/duc/xml2json.py /var/cache/duc/tree.xml
  • The output should be valid now :pray:
1 Like

Indeed, after doing so, there was reasonable output. But the website still didn’t output a nice graph.
I ended up deleting the folder (sorry Michael), and then it gave me a graph.
I’ll rerip the CD and call him Michael Bubble then. :stuck_out_tongue_winking_eye:
Edit: I can’t find the "yes, this answer did it!"flag…?

1 Like

It’s only for the Support category. :slight_smile:

It’s a very Strange Thing because other songs indexed have the è. :thinking:

Anyway we found the solution, great @rolf :smiley:

Moved, now you can :slight_smile: