Nextcloud Low performance & High CPU issues since 7.6

Hi,

Does anyone else remarked that the performances of a nethserver instance was severely impacted by NS 7.6 release ?

I was investigating today and it seems related to MySQL and Nextcloud. The system stays all the time under load with any real change in the way the server is used.

Here is a screen cap of the server load :

You see that the load suddenly augmented on Week 50 (when I upgraded to 7.6), and that when the users are coming back from holidays around week 02 the server beings to struggle to keep in pace.

The main load is nextcloud (10 users), and besides this there is only the mail and webmaill server.

The main CPU load is share by php-fm and mysql. There is some continuous disk activity too.

image
Stopping Apache stops almost all CPU & Disk activity.

Any comment / help / pointers ?

Thanks folks

Matthieu

I can’t reproduce on my production machine.
Do you often see the nextclud cron?

If yes, you could try to temporary disable it from the admin UI.
Then try to profile it to see what it does.

Anything in the nextcloud log file? Do you use onlyoffice?

Thanks @giacomo, I’ll check.

Thanks @LayLow, nothing wrong in the logs. Indeed I use collabora (not only office). It is true that I installed it on 9/01/19. Could it be the culprit ?

Sorry, I don;t use Collabora. Maybe other readers can speak up. I believe that Collabora module was updated after that date.

1 Like

Would you please try to use an older kernel?

1 Like

I didn’t know it was possible to switch the kernel. I’ll get some information to know how to do it and will report back.

Not uninstall. Choose at boot time. If i’m not wrong, 5 kernels are kept by default config.

Right. I always wondered why :wink: now I know.

There’s a new version of collabora, see NethServer Collabora Online

Oho ! I’ll try first. Should I uninstall your package first ?

Yes and remove the collabora repo.

So sad :slight_smile: Let’s do it :slight_smile:

1 Like

If ns runs as VM in HyperV check out this topic

That’s indeed a VM but running on Proxmox 5.3.5. The server is a 8 x Intel® Xeon® CPU E3-1230 v6 @ 3.50GHz

It looks like removing collabora helped, @mrmarkuz

image

Didn’t reinstalled the nethforge one yet. I’ll look into it tonight.

Spoke too fast : removing your module and trying to install the new one somehow broke my nextcloud installation. Therefore the above graph is probably irrelevant. Investigating.

Fixed now, indeed Nextcloud wasn’t running and as far as I can see @mrmarkuz Collabora isn’t the culprit : CPU load still too high.

@pike I tried to load kernel 3.10.0-862.14.4, no change.

Any other idea ?

Update.

I updated and power cycled both nethserver (VM) and proxmox yesterday and the issue looks solved, see the latest part of the graph :


It looks like @davidep was on the right track.

Thanks all,

Matthieu

2 Likes

Update.

I finally had the time to figure out what when wrong with that issue that came back and plagued us more and more :

It looks like it is related to a bug that happens during nextcloud updates. The table oc_jobs, that contains all the periodic cron jobs that have to be run by the sql server every 15 minutes, was flooded by tons (-14000 in my case) of pointless jobs related to an obscure module used to hep discovering federated servers.

Since we don’t use federation currently, the fix was to delete all those jobs in the DB :

DELETE FROM `nextcloud`.`oc_jobs` WHERE `oc_jobs`.`class` LIKE 'OCA\\LookupServerConnector\\BackgroundJobs\\RetryJob';

And removing the offeding app

mkdir -p /root/backup_NC_lookup_server_issue/usr/share/nextcloud/apps/lookup_server_connector
mv /usr/share/nextcloud/apps/lookup_server_connector/ /root/backup_NC_lookup_server_issue/usr/share/nextcloud/apps/lookup_server_connector/

Credits to https://help.nextcloud.com/t/high-sql-load-since-update/43588/7?u=pagaille

4 Likes