Nextcloud Low performance & High CPU issues since 7.6

[pagaille (Matthieu Gaillet)]

Hi,

Does anyone else remarked that the performances of a nethserver instance was severely impacted by NS 7.6 release ?

I was investigating today and it seems related to MySQL and Nextcloud. The system stays all the time under load with any real change in the way the server is used.

Here is a screen cap of the server load :

You see that the load suddenly augmented on Week 50 (when I upgraded to 7.6), and that when the users are coming back from holidays around week 02 the server beings to struggle to keep in pace.

The main load is nextcloud (10 users), and besides this there is only the mail and webmaill server.

The main CPU load is share by php-fm and mysql. There is some continuous disk activity too.

Stopping Apache stops almost all CPU & Disk activity.

Any comment / help / pointers ?

Thanks folks

Matthieu

[giacomo (Giacomo Sanchietti)]

I can’t reproduce on my production machine.
Do you often see the nextclud cron?

If yes, you could try to temporary disable it from the admin UI.
Then try to profile it to see what it does.

[LayLow]

Anything in the nextcloud log file? Do you use onlyoffice?

[pagaille (Matthieu Gaillet)]

Thanks @giacomo, I’ll check.

Thanks @LayLow, nothing wrong in the logs. Indeed I use collabora (not only office). It is true that I installed it on 9/01/19. Could it be the culprit ?

[LayLow]

Sorry, I don;t use Collabora. Maybe other readers can speak up. I believe that Collabora module was updated after that date.

[pike (Michael Kicks)]

Would you please try to use an older kernel?

[pagaille (Matthieu Gaillet)]

I didn’t know it was possible to switch the kernel. I’ll get some information to know how to do it and will report back.

[pike (Michael Kicks)]

Not uninstall. Choose at boot time. If i’m not wrong, 5 kernels are kept by default config.

[pagaille (Matthieu Gaillet)]

Right. I always wondered why now I know.

[mrmarkuz (Markus Neuberger)]

@pagaille

Indeed I use collabora (not only office). It is true that I installed it on 9/01/19. Could it be the culprit ?

There’s a new version of collabora, see NethServer Collabora Online

[pagaille (Matthieu Gaillet)]

Oho ! I’ll try first. Should I uninstall your package first ?

[mrmarkuz (Markus Neuberger)]

Yes and remove the collabora repo.

[pagaille (Matthieu Gaillet)]

So sad Let’s do it

[davidep (Davide Principi)]

@pagaille

the load suddenly augmented on Week 50 (when I upgraded to 7.6)

If ns runs as VM in HyperV check out this topic

Issues with 7.6 and HyperV 16

Hi, Unfortunately I fell into the same trap. My system - although locked - installed 7.6 final but did not upgrade the Kernel. Cleaning the yum cache helped. But know with the new Kernel the system still seem to have issues such as Chron / logrotate creates files with xxx.log.1 instead of xxx.log-20181217 The system in bridge mode does not show the external IP in diagnostics I need to reboot my switch everytime I restart nethserver (seems to be someting strange with the network interface) TL…

[pagaille (Matthieu Gaillet)]

That’s indeed a VM but running on Proxmox 5.3.5. The server is a 8 x Intel® Xeon® CPU E3-1230 v6 @ 3.50GHz

[pagaille (Matthieu Gaillet)]

It looks like removing collabora helped, @mrmarkuz

Didn’t reinstalled the nethforge one yet. I’ll look into it tonight.

[pagaille (Matthieu Gaillet)]

Spoke too fast : removing your module and trying to install the new one somehow broke my nextcloud installation. Therefore the above graph is probably irrelevant. Investigating.

[pagaille (Matthieu Gaillet)]

Fixed now, indeed Nextcloud wasn’t running and as far as I can see @mrmarkuz Collabora isn’t the culprit : CPU load still too high.

@pike I tried to load kernel 3.10.0-862.14.4, no change.

Any other idea ?

[pagaille (Matthieu Gaillet)]

Update.

I updated and power cycled both nethserver (VM) and proxmox yesterday and the issue looks solved, see the latest part of the graph :

It looks like @davidep was on the right track.

Thanks all,

Matthieu

[pagaille (Matthieu Gaillet)]

Update.

I finally had the time to figure out what when wrong with that issue that came back and plagued us more and more :

It looks like it is related to a bug that happens during nextcloud updates. The table oc_jobs, that contains all the periodic cron jobs that have to be run by the sql server every 15 minutes, was flooded by tons (-14000 in my case) of pointless jobs related to an obscure module used to hep discovering federated servers.

Since we don’t use federation currently, the fix was to delete all those jobs in the DB :

DELETE FROM `nextcloud`.`oc_jobs` WHERE `oc_jobs`.`class` LIKE 'OCA\\LookupServerConnector\\BackgroundJobs\\RetryJob';

And removing the offeding app

mkdir -p /root/backup_NC_lookup_server_issue/usr/share/nextcloud/apps/lookup_server_connector
mv /usr/share/nextcloud/apps/lookup_server_connector/ /root/backup_NC_lookup_server_issue/usr/share/nextcloud/apps/lookup_server_connector/

Credits to https://help.nextcloud.com/t/high-sql-load-since-update/43588/7?u=pagaille