Nextcloud fails after update

[wbilger (Wayne Bilger)]

NethServer Version: 7.9.2009 (final)
Module: Nextcloud 1.17.0

Since updating NethServer with the latest community updates yesterday, Nexctloud fails to load.
I receive this email every 5 minutes.

Subject: Cron /usr/bin/scl enable rh-php73 – php -d memory_limit=512M -f /usr/share/nextcloud/cron.php

PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 20480 bytes) in /usr/share/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php on line 133

When it initially came up it wanted to click the button to update to NC 22 which failed and now I am stuck in maintenance mode.

occ maintenance:mode --off gives;
PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 20480 bytes) in /usr/share/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php on line 133

[giacomo (Giacomo Sanchietti)]

Try to install from testing, it could fix your issue:

yum --enablerepo=nethsever-testing update nethserver-nextcloud

[wbilger (Wayne Bilger)]

@giacomo

yum --enablerepo=nethsever-testing update nethserver-nextcloud

That installed correctly, and now I am backup to the Nextcloud will be updated to version 21.0.3
with the Start update button, which failed before.
Is this safe to run, or should I run occ upgrade from cli?

[giacomo (Giacomo Sanchietti)]

Everything should be automatic, unless you had a previous interrupted upgrade.
By the way, if the database has been migrated to the new db instance, you can safely run occ.
If you’re stuck, please provide an extract of /var/log/messages.

[wbilger (Wayne Bilger)]

I am not sure if database is updated, I did nothing manually, I just ran the updates from the Software Center. I had no interrupted upgrades, unless this one did initially and it did not inform me.
I am getting a little different emails every 5 minutes now;

Subject: Cron /usr/bin/scl enable rh-php73 – php -d memory_limit=512M -d apc.enable_cli=1 -f /usr/share/nextcloud/cron.php

/var/tmp/sclObH2UM: line 8: 7115 Bus error “php” “-d” “memory_limit=512M” “-d” “apc.enable_cli=1” “-f” “/usr/share/nextcloud/cron.php”

/var/log/messages pasted here;
https://zerobin.net/?fae4977052f9690c#Wg/G7naHPQD18I1WYmLzw2z+UYEqumDNuF5T/T51K4s=

[giacomo (Giacomo Sanchietti)]

I’ve never seen such an error.
Try to manually execute the command and see if it goes on segfault or something like this.

I think you need to hack a little on the system to find the cause of the problem

[wbilger (Wayne Bilger)]

This is on a live system in use by about 10 users at the moment, for Domain Controller, file server, webtop email, and guacamole. Is it safe to run the occ upgrade command, or should I wait until the evening when noone is using?
Not sure what you mean to hack a little on the system, but will try anything that is safe on live, or can try something later.

[dnutan (Marc)]

Off topic: check if your backups are done correctly:

Jul 4 06:51:45 lrtserv-data esmith::event[4413]: Failed to insert into /var/lib/samba/private/sam.ldb.d/DC=DOMAINDNSZONES,DC=xxxxxxxxxx,DC=LAN.ldb.bak.tmpLONDONROOFTRUSS
Jul 4 06:51:45 lrtserv-data esmith::event[4413]: failed to copy /var/lib/samba/private/sam.ldb.d/DC=DOMAINDNSZONES,DC=xxxxxxxxxx,DC=LAN.ldb
Jul 4 06:51:45 lrtserv-data esmith::event[4413]: Action: /etc/e-smith/events/pre-backup-config/S40nethserver-dc-pre-backup FAILED: 1 [0.540734]

[wbilger (Wayne Bilger)]

Ok, how do I do that, it is not something I have done before or can find documentation for.

[dnutan (Marc)]

About Nextcloud, what is the output of these commands?

systemctl status -l rh-mariadb105-mariadb@nextcloud.service
mysqlshow
/opt/rh/rh-mariadb105/root/bin/mysqlshow --socket="/var/run/rh-mariadb105-mariadb/nextcloud-mysql.sock"
grep \'db /usr/share/nextcloud/config/config.php |grep -v dbpass

[wbilger (Wayne Bilger)]

@dnutan

systemctl status -l rh-mariadb105-mariadb@nextcloud.servic

rh-mariadb105-mariadb@nextcloud.service - MariaDB 10.5 database server
Loaded: loaded (/etc/systemd/system/rh-mariadb105-mariadb@.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/rh-mariadb105-mariadb@nextcloud.service.d
└─nethserver.conf
Active: active (running) since Mon 2021-07-05 07:28:57 EDT; 5h 29min ago
Docs: man:mysqld(8)
systemd - MariaDB Knowledge Base
Process: 2229 ExecStartPost=/usr/bin/scl enable $RH_MARIADB105_SCLS_ENABLED – /opt/rh/rh-mariadb105/root/usr/libexec/mysql-check-upgrade --defaults-group-suffix=.%I (code=exited, status=0/SUCCESS)
Process: 1707 ExecStartPre=/usr/bin/scl enable $RH_MARIADB105_SCLS_ENABLED – /opt/rh/rh-mariadb105/root/usr/libexec/mysql-prepare-db-dir --defaults-group-suffix=.%I %n (code=exited, status=0/SUCCESS)
Process: 1419 ExecStartPre=/usr/bin/scl enable $RH_MARIADB105_SCLS_ENABLED – /opt/rh/rh-mariadb105/root/usr/libexec/mysql-check-socket --defaults-group-suffix=.%I (code=exited, status=0/SUCCESS)
Process: 1296 ExecStartPre=/usr/bin/scl enable $RH_MARIADB105_SCLS_ENABLED – /usr/bin/scl_enabled rh-mariadb105 (code=exited, status=0/SUCCESS)
Main PID: 1779 (mysqld)
Status: “Taking your SQL requests now…”
CGroup: /system.slice/system-rh\x2dmariadb105\x2dmariadb.slice/rh-mariadb105-mariadb@nextcloud.service
└─1779 /opt/rh/rh-mariadb105/root/usr/libexec/mysqld --defaults-group-suffix=.nextcloud --basedir=/opt/rh/rh-mariadb105/root/usr

Jul 05 07:28:49 lrtserv.mydomain.com systemd[1]: Starting MariaDB 10.5 database server…
Jul 05 07:28:51 lrtserv.mydomain.com scl[1707]: Database MariaDB is probably initialized in /var/opt/rh/rh-mariadb105/lib/mysql-nextcloud already, nothing is done.
Jul 05 07:28:51 lrtserv.mydomain.com scl[1707]: If this is not the case, make sure the /var/opt/rh/rh-mariadb105/lib/mysql-nextcloud is empty before running mysql-prepare-db-dir.
Jul 05 07:28:53 lrtserv.mydomain.com mysqld-scl-helper[1779]: 2021-07-05 7:28:53 0 [Note] /opt/rh/rh-mariadb105/root/usr/libexec/mysqld (mysqld 10.5.8-MariaDB) starting as process 1779 …
Jul 05 07:28:57 lrtserv.mydomain.com systemd[1]: Started MariaDB 10.5 database server.

@dnutan

mysqlshow

±-------------------+
| Databases |
±-------------------+
| information_schema |
| guacamole |
| mysql |
| performance_schema |
| roundcubemail |
±-------------------+

@dnutan

/opt/rh/rh-mariadb105/root/bin/mysqlshow --socket="/var/run/rh-mariadb105-mariadb/nextcloud-mysql.sock"

±-------------------+
| Databases |
±-------------------+
| information_schema |
| mysql |
| nextcloud |
| performance_schema |
±-------------------+

@dnutan

grep \'db /usr/share/nextcloud/config/config.php |grep -v dbpass

‘dbtype’ => ‘mysql’,
‘dbname’ => ‘nextcloud’,
‘dbhost’ => ‘localhost:/run/rh-mariadb105-mariadb/nextcloud-mysql.sock’,
‘dbport’ => ‘’,
‘dbtableprefix’ => ‘oc_’,
‘dbuser’ => ‘nextcloud’,

[dnutan (Marc)]

It seems the nextcloud database was imported to the new mariadb.

From the command line run:

occ upgrade

[wbilger (Wayne Bilger)]

Ok, thanks.
I will update tonight when no users are online, and I’ll update here after.

[wbilger (Wayne Bilger)]

I figured out what was happening.
I had a 1GB /tmp partition, and during the NC update it was filling, and something was going wrong but nothing reported, it just finished. I extended the /tmp to 2GB and reinstalled NC and now seems to be all working again.
Thanks for all the help.

[dnutan (Marc)]

OFF TOPIC

@wbilger

I figured out what was happening.
I had a 1GB /tmp partition, and during the NC update it was filling

So the full partition could have been the cause of other errors, like:

sssd[krb5_child[6702]]: Credentials cache I/O operation failed

Remember to check the log for this error on the next backup. If it doesn’t occur anymore then it was probably caused by the space issue:

@dnutan

Off topic: check if your backups are done correctly:

Jul 4 06:51:45 lrtserv-data esmith::event[4413]: Failed to insert into /var/lib/samba/private/sam.ldb.d/DC=DOMAINDNSZONES,DC=xxxxxxxxxx,DC=LAN.ldb.bak.tmpLONDONROOFTRUSS
Jul 4 06:51:45 lrtserv-data esmith::event[4413]: failed to copy /var/lib/samba/private/sam.ldb.d/DC=DOMAINDNSZONES,DC=xxxxxxxxxx,DC=LAN.ldb
Jul 4 06:51:45 lrtserv-data esmith::event[4413]: Action: /etc/e-smith/events/pre-backup-config/S40nethserver-dc-pre-backup FAILED: 1 [0.540734]

The next one is related to samba shares:

Jul 4 22:41:22 lrtserv-data smbd[14869]: [2021/07/04 22:41:22.900593, 0] …/…/source3/smbd/uid.c:448(change_to_user_internal)
Jul 4 22:41:22 lrtserv-data smbd[14869]: change_to_user_internal: chdir_current_service() failed!

Smbd: change_to_user_internal: chdir_current_service() failed!

NethServer Version: 7.7.1908 Module: File Server Hello, Everything seems to work in file server, but still the log is flooded with the following error messages: change_to_user_internal: chdir_current_service() failed! smbd [2020/02/12 13:06:23.863870, 0] ../source3/smbd/uid.c:417(change_to_user_internal) smbd I get these error messages only if there is a Windows 10 Pro computer running that is joined the domain. There is no error message if I access the file shar…

The following could be due to failed service or duplicate port:

Jul 4 23:40:40 lrtserv-data daemon.sh: 04-Jul-2021 23:40:40.031 SEVERE [main] org.apache.catalina.core.StandardService.initInternal Failed to initialize connector [Connector[HTTP/1.1-8080]]
Jul 4 23:40:40 lrtserv-data daemon.sh: org.apache.catalina.LifecycleException: Protocol handler initialization failed
(…)
Jul 4 23:40:40 lrtserv-data daemon.sh: Caused by: java.net.BindException: Address already in use

Another one refers to a firmware bug (you can search for firmware updates for your hardware, if any), and could be related to time skew issues.

kernel: [Firmware Bug]: TSC doesn’t count with P0 frequency!

More info:

The AMD processor has the x86 constant TSC feature enabled but the firmware has not set bit 24 of the MSR MSR_K7_HWCR and so the processor TSC does not count in the P0 frequency level.

Some errors/warnings might be harmless and could be ignored, but I don’t know which ones.