Everything looks ok. Piler has started, as has mariadb. I am currently restoring from the last backup from yesterday. Then it should actually work again. Actually…
As I already suspected, that doesn’t work either. I use SSDs from Crucial. In another context (OMV on Proxmox), @Andy_Wismer advised me to use Enterprise SSDs.
Maybe that is the cause. Here is an excerpt from the logs after restarting Piler:
2025-03-02T17:14:24+01:00 [1:piler1:agent@piler1] task/module/piler1/df014708-be5a-4b4f-adcc-e8f993f235a3: get-name/50get_name is starting
2025-03-02T17:14:24+01:00 [1:piler1:agent@piler1] task/module/piler1/f964ee64-182f-4307-a985-6a0ab87dab11: get-configuration/20read is starting
2025-03-02T17:14:24+01:00 [1:piler1:agent@piler1] task/module/piler1/b9441eb4-f25c-4436-92df-8ec245aaaf65: get-status/20read is starting
2025-03-02T17:14:24+01:00 [1:piler1:agent@piler1] task/module/piler1/df014708-be5a-4b4f-adcc-e8f993f235a3: action “get-name” status is “completed” (0) at step 50get_name
2025-03-02T17:14:24+01:00 [1:piler1:agent@piler1] task/module/piler1/b9441eb4-f25c-4436-92df-8ec245aaaf65: action “get-status” status is “completed” (0) at step validate-output.json
2025-03-02T17:14:25+01:00 [1:piler1:agent@piler1] task/module/piler1/f964ee64-182f-4307-a985-6a0ab87dab11: action “get-configuration” status is “completed” (0) at step validate-output.json
2025-03-02T17:14:28+01:00 [1:piler1:agent@piler1] task/module/piler1/b6a3dc05-e23e-40c9-8085-22ecea5ce112: get-configuration/20read is starting
2025-03-02T17:14:29+01:00 [1:piler1:agent@piler1] task/module/piler1/b6a3dc05-e23e-40c9-8085-22ecea5ce112: action “get-configuration” status is “completed” (0) at step validate-output.json
2025-03-02T17:14:32+01:00 [1:piler1:agent@piler1] task/module/piler1/212c95d0-c180-4095-9085-03dd74283914: configure-module/01_validate_domain is starting
2025-03-02T17:14:32+01:00 [1:piler1:agent@piler1] task/module/piler1/212c95d0-c180-4095-9085-03dd74283914: configure-module/01_validate_mail_server is starting
2025-03-02T17:14:32+01:00 [1:piler1:agent@piler1] task/module/piler1/212c95d0-c180-4095-9085-03dd74283914: configure-module/10configure is starting
2025-03-02T17:14:32+01:00 [1:piler1:agent@piler1] task/module/piler1/212c95d0-c180-4095-9085-03dd74283914: configure-module/20traefik is starting
2025-03-02T17:14:34+01:00 [1:piler1:agent@piler1] task/module/piler1/212c95d0-c180-4095-9085-03dd74283914: configure-module/80start_services is starting
2025-03-02T17:14:34+01:00 [1:piler1:systemd] Reloading.
2025-03-02T17:14:34+01:00 [1:piler1:systemd] Stopping Podman piler-app.service…
2025-03-02T17:14:44+01:00 [1:piler1:piler1] time=“2025-03-02T17:14:44+01:00” level=warning msg=“StopSignal SIGTERM failed to stop container piler-app in 10 seconds, resorting to SIGKILL”
2025-03-02T17:14:44+01:00 [1:piler1:piler1] 10bab3ebb92e2978b9d7198ab9dd0811ab9df6f1d30b6d35b40098353370e8f8
2025-03-02T17:14:44+01:00 [1:piler1:systemd] piler-app.service: Main process exited, code=exited, status=137/n/a
2025-03-02T17:14:44+01:00 [1:piler1:piler1] 10bab3ebb92e2978b9d7198ab9dd0811ab9df6f1d30b6d35b40098353370e8f8
2025-03-02T17:14:44+01:00 [1:piler1:systemd] piler-app.service: Failed with result ‘exit-code’.
2025-03-02T17:14:44+01:00 [1:piler1:systemd] Stopped Podman piler-app.service.
2025-03-02T17:14:44+01:00 [1:piler1:systemd] piler-app.service: Consumed 1.244s CPU time.
2025-03-02T17:14:44+01:00 [1:piler1:systemd] Stopping Podman mariadb-app.service…
2025-03-02T17:14:44+01:00 [1:piler1:systemd] Stopping Podman memcached-app.service…
2025-03-02T17:14:44+01:00 [1:piler1:mariadb-app] 2025-03-02 16:14:44 0 [Note] mariadbd (initiated by: unknown): Normal shutdown
2025-03-02T17:14:44+01:00 [1:piler1:mariadb-app] 2025-03-02 16:14:44 0 [Note] InnoDB: FTS optimize thread exiting.
2025-03-02T17:14:44+01:00 [1:piler1:mariadb-app] 2025-03-02 16:14:44 0 [Note] InnoDB: Starting shutdown…
2025-03-02T17:14:44+01:00 [1:piler1:mariadb-app] 2025-03-02 16:14:44 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
2025-03-02T17:14:44+01:00 [1:piler1:mariadb-app] 2025-03-02 16:14:44 0 [Note] InnoDB: Buffer pool(s) dump completed at 250302 16:14:44
2025-03-02T17:14:44+01:00 [1:piler1:piler1] f675adbc3ac57cb1aa78434a5c6c708dff89154fcabee3ae4de073ee878cf290
2025-03-02T17:14:44+01:00 [1:piler1:piler1] f675adbc3ac57cb1aa78434a5c6c708dff89154fcabee3ae4de073ee878cf290
2025-03-02T17:14:44+01:00 [1:piler1:systemd] Stopped Podman memcached-app.service.
2025-03-02T17:14:44+01:00 [1:piler1:mariadb-app] 2025-03-02 16:14:44 0 [Note] InnoDB: Removed temporary tablespace data file: “./ibtmp1”
2025-03-02T17:14:44+01:00 [1:piler1:mariadb-app] 2025-03-02 16:14:44 0 [Note] InnoDB: Shutdown completed; log sequence number 15988396; transaction id 17199
2025-03-02T17:14:44+01:00 [1:piler1:mariadb-app] 2025-03-02 16:14:44 0 [Note] mariadbd: Shutdown complete
2025-03-02T17:14:44+01:00 [1:piler1:mariadb-app]
2025-03-02T17:14:44+01:00 [1:piler1:piler1] 93ad1cf9d4a16daca0b6fe8ea5fe4e492a3f3952e12cdfabdbda5215acba38c9
2025-03-02T17:14:44+01:00 [1:piler1:piler1] 93ad1cf9d4a16daca0b6fe8ea5fe4e492a3f3952e12cdfabdbda5215acba38c9
2025-03-02T17:14:44+01:00 [1:piler1:systemd] Stopped Podman mariadb-app.service.
2025-03-02T17:14:44+01:00 [1:piler1:systemd] Stopping Podman piler.service…
2025-03-02T17:14:44+01:00 [1:piler1:systemd] Removed slice cgroup user-libpod_pod_f6b2b16fa2781181f460a4b149db31d6f04160f74c48b57647733b63d41eac1c.slice.
2025-03-02T17:14:44+01:00 [1:piler1:systemd] user-libpod_pod_f6b2b16fa2781181f460a4b149db31d6f04160f74c48b57647733b63d41eac1c.slice: Consumed 1.385s CPU time.
2025-03-02T17:14:44+01:00 [1:piler1:podman] piler
2025-03-02T17:14:44+01:00 [1:piler1:systemd] user-libpod_pod_f6b2b16fa2781181f460a4b149db31d6f04160f74c48b57647733b63d41eac1c.slice: Failed to open /run/user/1010/systemd/transient/user-libpod_pod_f6b2b16fa2781181f460a4b149db31d6f04160f74c48b57647733b63d41eac1c.slice: No such file or directory
2025-03-02T17:14:44+01:00 [1:piler1:podman] f6b2b16fa2781181f460a4b149db31d6f04160f74c48b57647733b63d41eac1c
2025-03-02T17:14:44+01:00 [1:piler1:systemd] Stopped Podman piler.service.
2025-03-02T17:14:44+01:00 [1:piler1:systemd] Starting Podman piler.service…
2025-03-02T17:14:45+01:00 [1:piler1:systemd] Created slice cgroup user-libpod_pod_299eae025d59ed15f5e577444cd5f376256d983a2b28ee122a30e1f86ae77d96.slice.
2025-03-02T17:14:45+01:00 [1:piler1:podman] 299eae025d59ed15f5e577444cd5f376256d983a2b28ee122a30e1f86ae77d96
2025-03-02T17:14:45+01:00 [1:piler1:systemd] Started libcrun container.
2025-03-02T17:14:45+01:00 [1:piler1:podman] piler
2025-03-02T17:14:45+01:00 [1:piler1:systemd] Started Podman piler.service.
2025-03-02T17:14:45+01:00 [1:piler1:systemd] Starting Podman mariadb-app.service…
2025-03-02T17:14:45+01:00 [1:piler1:systemd] Starting Podman memcached-app.service…
2025-03-02T17:14:45+01:00 [1:piler1:agent@piler1] Error: no container with name or ID “piler-app” found: no such container
2025-03-02T17:14:45+01:00 [1:piler1:systemd] Started libcrun container.
2025-03-02T17:14:45+01:00 [1:piler1:systemd] Started libcrun container.
2025-03-02T17:14:45+01:00 [1:piler1:piler1] 254154b4018337606541a29f2509ce91a7332b2a7cc6672a96d6d1cfb30bae3e
2025-03-02T17:14:45+01:00 [1:piler1:systemd] Started Podman memcached-app.service.
2025-03-02T17:14:45+01:00 [1:piler1:piler1] e9f9b53e7617392be99c83ebe468e3d5dbe98b7360c4bcff2f5765f990b43307
2025-03-02T17:14:45+01:00 [1:piler1:mariadb-app] 2025-03-02 16:14:45+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.11.7+maria~ubu2204 started.
2025-03-02T17:14:45+01:00 [1:piler1:systemd] Started Podman mariadb-app.service.
2025-03-02T17:14:45+01:00 [1:piler1:systemd] Starting Podman piler-app.service…
2025-03-02T17:14:45+01:00 [1:piler1:mariadb-app] 2025-03-02 16:14:45+00:00 [Warn] [Entrypoint]: /sys/fs/cgroup///memory.pressure not writable, functionality unavailable to MariaDB
2025-03-02T17:14:45+01:00 [1:piler1:mariadb-app] 2025-03-02 16:14:45+00:00 [Note] [Entrypoint]: Switching to dedicated user ‘mysql’
2025-03-02T17:14:45+01:00 [1:piler1:mariadb-app] 2025-03-02 16:14:45+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.11.7+maria~ubu2204 started.
2025-03-02T17:14:45+01:00 [1:piler1:mariadb-app] 2025-03-02 16:14:45+00:00 [Note] [Entrypoint]: MariaDB upgrade not required
2025-03-02T17:14:45+01:00 [1:piler1:mariadb-app] 2025-03-02 16:14:45 0 [Note] Starting MariaDB 10.11.7-MariaDB-1:10.11.7+maria~ubu2204 source revision 87e13722a95af5d9378d990caf48cb6874439347 as process 1
2025-03-02T17:14:45+01:00 [1:piler1:mariadb-app] 2025-03-02 16:14:45 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2025-03-02T17:14:45+01:00 [1:piler1:mariadb-app] 2025-03-02 16:14:45 0 [Note] InnoDB: Number of transaction pools: 1
2025-03-02T17:14:45+01:00 [1:piler1:mariadb-app] 2025-03-02 16:14:45 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2025-03-02T17:14:45+01:00 [1:piler1:mariadb-app] 2025-03-02 16:14:45 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
2025-03-02T17:14:45+01:00 [1:piler1:mariadb-app] 2025-03-02 16:14:45 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
2025-03-02T17:14:45+01:00 [1:piler1:mariadb-app] 2025-03-02 16:14:45 0 [Note] InnoDB: Completed initialization of buffer pool
2025-03-02T17:14:45+01:00 [1:piler1:mariadb-app] 2025-03-02 16:14:45 0 [Note] InnoDB: Buffered log writes (block size=512 bytes)
2025-03-02T17:14:45+01:00 [1:piler1:mariadb-app] 2025-03-02 16:14:45 0 [Note] InnoDB: End of log at LSN=15988396
2025-03-02T17:14:45+01:00 [1:piler1:mariadb-app] 2025-03-02 16:14:45 0 [Note] InnoDB: 128 rollback segments are active.
2025-03-02T17:14:45+01:00 [1:piler1:mariadb-app] 2025-03-02 16:14:45 0 [Note] InnoDB: Setting file ‘./ibtmp1’ size to 12.000MiB. Physically writing the file full; Please wait …
2025-03-02T17:14:45+01:00 [1:piler1:mariadb-app] 2025-03-02 16:14:45 0 [Note] InnoDB: File ‘./ibtmp1’ size is now 12.000MiB.
2025-03-02T17:14:45+01:00 [1:piler1:mariadb-app] 2025-03-02 16:14:45 0 [Note] InnoDB: log sequence number 15988396; transaction id 17193
2025-03-02T17:14:45+01:00 [1:piler1:mariadb-app] 2025-03-02 16:14:45 0 [Note] Plugin ‘FEEDBACK’ is disabled.
2025-03-02T17:14:45+01:00 [1:piler1:mariadb-app] 2025-03-02 16:14:45 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2025-03-02T17:14:45+01:00 [1:piler1:mariadb-app] 2025-03-02 16:14:45 0 [Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds work.
2025-03-02T17:14:45+01:00 [1:piler1:mariadb-app] 2025-03-02 16:14:45 0 [Note] Server socket created on IP: ‘0.0.0.0’.
2025-03-02T17:14:45+01:00 [1:piler1:mariadb-app] 2025-03-02 16:14:45 0 [Note] Server socket created on IP: ‘::’.
2025-03-02T17:14:45+01:00 [1:piler1:mariadb-app] 2025-03-02 16:14:45 0 [Note] mariadbd: ready for connections.
2025-03-02T17:14:45+01:00 [1:piler1:mariadb-app] Version: ‘10.11.7-MariaDB-1:10.11.7+maria~ubu2204’ socket: ‘/run/mysqld/mysqld.sock’ port: 3306 mariadb.org binary distribution
2025-03-02T17:14:45+01:00 [1:piler1:mariadb-app] 2025-03-02 16:14:45 0 [Note] InnoDB: Buffer pool(s) load completed at 250302 16:14:45
2025-03-02T17:14:46+01:00 [1:piler1:systemd] Started libcrun container.
2025-03-02T17:14:46+01:00 [1:piler1:piler1] 351b1f0ecfd677f8630866d164f4c8c72f710dc5d3f068b63c924f9711d33a9b
2025-03-02T17:14:46+01:00 [1:piler1:systemd] Started Podman piler-app.service.
2025-03-02T17:14:46+01:00 [1:piler1:piler-app] Database
2025-03-02T17:14:46+01:00 [1:piler1:piler-app] information_schema
2025-03-02T17:14:46+01:00 [1:piler1:piler-app] piler
2025-03-02T17:14:46+01:00 [1:piler1:piler-app] DEBUG: 127.0.0.1 is ready
2025-03-02T17:14:46+01:00 [1:piler1:piler-app] DEBUG: metadata table exists
2025-03-02T17:14:46+01:00 [1:piler1:piler-app] * Starting periodic command scheduler cron
2025-03-02T17:14:46+01:00 [1:piler1:piler-app] …done.
2025-03-02T17:14:46+01:00 [1:piler1:piler-app] * Starting nginx nginx
2025-03-02T17:14:46+01:00 [1:piler1:piler-app] …done.
2025-03-02T17:14:46+01:00 [1:piler1:piler-app] starting searchd . . .
2025-03-02T17:14:46+01:00 [1:piler1:piler-app] [Sun Mar 2 16:14:46.534 2025] [58] using config file ‘/etc/piler/manticore.conf’ (10595 chars)…
2025-03-02T17:14:46+01:00 [1:piler1:piler-app] Manticore 6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)
2025-03-02T17:14:46+01:00 [1:piler1:piler-app] Copyright (c) 2001-2016, Andrew Aksyonoff
2025-03-02T17:14:46+01:00 [1:piler1:piler-app] Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
2025-03-02T17:14:46+01:00 [1:piler1:piler-app] Copyright (c) 2017-2023, Manticore Software LTD (https://manticoresearch.com)
2025-03-02T17:14:46+01:00 [1:piler1:piler-app]
2025-03-02T17:14:46+01:00 [1:piler1:piler-app] starting daemon version ‘6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)’ …
2025-03-02T17:14:46+01:00 [1:piler1:piler-app] listening on 127.0.0.1:9312 for sphinx and http(s)
2025-03-02T17:14:46+01:00 [1:piler1:piler-app] listening on 127.0.0.1:9306 for mysql
2025-03-02T17:14:46+01:00 [1:piler1:piler-app] Manticore 6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)
2025-03-02T17:14:46+01:00 [1:piler1:piler-app] Copyright (c) 2001-2016, Andrew Aksyonoff
2025-03-02T17:14:46+01:00 [1:piler1:piler-app] Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
2025-03-02T17:14:46+01:00 [1:piler1:piler-app] Copyright (c) 2017-2023, Manticore Software LTD (https://manticoresearch.com)
2025-03-02T17:14:46+01:00 [1:piler1:piler-app]
2025-03-02T17:14:46+01:00 [1:piler1:piler-app] precaching table ‘main1’
2025-03-02T17:14:46+01:00 [1:piler1:piler-app] precaching table ‘main2’
2025-03-02T17:14:46+01:00 [1:piler1:piler-app] precaching table ‘main3’
2025-03-02T17:14:46+01:00 [1:piler1:piler-app] precaching table ‘main4’
2025-03-02T17:14:46+01:00 [1:piler1:piler-app] precaching table ‘dailydelta1’
2025-03-02T17:14:46+01:00 [1:piler1:piler-app] precaching table ‘delta1’
2025-03-02T17:14:46+01:00 [1:piler1:piler-app] precaching table ‘tag1’
2025-03-02T17:14:46+01:00 [1:piler1:piler-app] precaching table ‘note1’
2025-03-02T17:14:46+01:00 [1:piler1:piler-app] Manticore 6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)
2025-03-02T17:14:46+01:00 [1:piler1:piler-app] Copyright (c) 2001-2016, Andrew Aksyonoff
2025-03-02T17:14:46+01:00 [1:piler1:piler-app] Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
2025-03-02T17:14:46+01:00 [1:piler1:piler-app] Copyright (c) 2017-2023, Manticore Software LTD (https://manticoresearch.com)
2025-03-02T17:14:46+01:00 [1:piler1:piler-app]
2025-03-02T17:14:46+01:00 [1:piler1:piler-app] starting piler-smtp . . .
2025-03-02T17:14:46+01:00 [1:piler1:piler-app] starting piler . . .
2025-03-02T17:14:47+01:00 [1:piler1:agent@piler1] task/module/piler1/212c95d0-c180-4095-9085-03dd74283914: configure-module/90add-relay-rule is starting
2025-03-02T17:14:48+01:00 [1:piler1:agent@piler1] task/module/piler1/212c95d0-c180-4095-9085-03dd74283914: action “configure-module” status is “completed” (0) at step 90add-relay-rule
2025-03-02T17:14:48+01:00 [1:piler1:agent@piler1] task/module/piler1/e490f0ed-693a-45bc-a7dc-ad0f8ca7ba6b: get-name/50get_name is starting
2025-03-02T17:14:48+01:00 [1:piler1:agent@piler1] task/module/piler1/4063b3af-8cc4-4f16-86e3-54feb5c886b4: get-configuration/20read is starting
2025-03-02T17:14:48+01:00 [1:piler1:agent@piler1] task/module/piler1/e490f0ed-693a-45bc-a7dc-ad0f8ca7ba6b: action “get-name” status is “completed” (0) at step 50get_name
2025-03-02T17:14:49+01:00 [1:piler1:agent@piler1] task/module/piler1/4063b3af-8cc4-4f16-86e3-54feb5c886b4: action “get-configuration” status is “completed” (0) at step validate-output.json
2025-03-02T17:14:52+01:00 [1:piler1:agent@piler1] task/module/piler1/1b02678b-1a36-4878-948e-c8bfb89a4139: get-configuration/20read is starting
2025-03-02T17:14:52+01:00 [1:piler1:agent@piler1] task/module/piler1/ef2d4e1f-5146-4e0d-ba34-665b73745d69: get-status/20read is starting
2025-03-02T17:14:53+01:00 [1:piler1:agent@piler1] task/module/piler1/ef2d4e1f-5146-4e0d-ba34-665b73745d69: action “get-status” status is “completed” (0) at step validate-output.json
2025-03-02T17:14:53+01:00 [1:piler1:agent@piler1] task/module/piler1/1b02678b-1a36-4878-948e-c8bfb89a4139: action “get-configuration” status is “completed” (0) at step validate-output.json
2025-03-02T17:17:46+01:00 [1:piler1:agent@piler1] task/module/piler1/23fe8bc5-1d11-4258-adf9-47c140131277: get-name/50get_name is starting
2025-03-02T17:17:46+01:00 [1:piler1:agent@piler1] task/module/piler1/6a0c7e0e-4ab8-4df9-9d1b-b9728faf8a4a: get-configuration/20read is starting
2025-03-02T17:17:46+01:00 [1:piler1:agent@piler1] task/module/piler1/3ad9e95d-c265-4622-85b0-43ecbff6945e: get-status/20read is starting
2025-03-02T17:17:47+01:00 [1:piler1:agent@piler1] task/module/piler1/23fe8bc5-1d11-4258-adf9-47c140131277: action “get-name” status is “completed” (0) at step 50get_name
2025-03-02T17:17:47+01:00 [1:piler1:agent@piler1] task/module/piler1/3ad9e95d-c265-4622-85b0-43ecbff6945e: action “get-status” status is “completed” (0) at step validate-output.json
2025-03-02T17:17:47+01:00 [1:piler1:agent@piler1] task/module/piler1/6a0c7e0e-4ab8-4df9-9d1b-b9728faf8a4a: action “get-configuration” status is “completed” (0) at step validate-output.json
You are still logged in as user piler but need to run this as root in the container, so you need another exit to leave su and be root in the container again, then the /etc/init.d/rc.searchd start should work.
It all works, but doesn’t change the result. It no longer wants to work. I am now trying another backup. If it doesn’t work then either, I’ll just stick with Mailstore.
I have tried all five backups I have had available since installing Piler. Not a single one has enabled me to get Piler up and running again. I have now reinstalled the update before installing Piler. Something is mighty mysterious