Service Elasticsearch wont start

Support
1

NethServer Version: 7.9.2009
Module: Zammad

Hello friends,

after I installed Zammad, the associated service Elasticsearch refuses to start working. Multiple reboots of the server brings nothing. The service is enabled but does not start.
Attached is the error message:

  • elasticsearch.service - Elasticsearch
    Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
    Active: failed (Result: timeout) since Sun 2022-03-13 13:39:52 CET; 1min 9s ago
    Docs: https://www.elastic.co
    Process: 1413 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=143)
    Main PID: 1413 (code=exited, status=143)

Mar 13 13:38:37 server.xxx.com systemd[1]: Starting Elasticsearch…
Mar 13 13:39:52 server.xxx.com systemd[1]: elasticsearch.service start operation timed out. Terminating.
Mar 13 13:39:52 server.xxx.com systemd[1]: Failed to start Elasticsearch.
Mar 13 13:39:52 server.xxx.com systemd[1]: Unit elasticsearch.service entered failed state.
Mar 13 13:39:52 server.xxx.com systemd[1]: elasticsearch.service failed.

Screenshot 2022-03-13 133623

Where is the error?

Regards…

Uwe

2

Please check if there’s enough free memory.

You may also check /var/log/elasticsearch/elasticsearch.log to get more information about the error.

EDIT:

I’m working on a nethserver-zammad5 docker version, the current module is bound to version 4 because native install of Zammad 5 isn’t possible anymore, see Zammad ticketing/helpdesk on Nethserver available - #169 by mrmarkuz

1 Like
3

Hi

Maybe this helps:

/usr/share/elasticsearch/bin/elasticsearch-plugin remove ingest-attachment
/usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-attachment

then

systemctl enable elasticsearch
systemctl restart elasticsearch
systemctl status elasticsearch

Hope this helps

My 2 cents
Andy

3 Likes
4

Hello mrmarkuz,

here is an excerpt from the log:

[2022-03-13T13:17:55,182][INFO ][o.e.n.Node ] [xxx.com] version[7.11.1], pid[24311], build[default/rpm/ff17057114c2199c9c1bbecc727003a907c0db7a/2021-02-15T13:44:09.394032Z], OS[Linux/3.10.0-1160.59.1.el7.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
[2022-03-13T13:17:55,184][INFO ][o.e.n.Node ] [xxx.com] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2022-03-13T13:17:55,184][INFO ][o.e.n.Node ] [xxx.com] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-15851077864622207663, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms7853m, -Xmx7853m, -XX:MaxDirectMemorySize=4118806528, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
[2022-03-13T13:17:56,988][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [aggs-matrix-stats]
[2022-03-13T13:17:56,988][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [analysis-common]
[2022-03-13T13:17:56,989][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [constant-keyword]
[2022-03-13T13:17:56,989][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [flattened]
[2022-03-13T13:17:56,989][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [frozen-indices]
[2022-03-13T13:17:56,989][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [ingest-common]
[2022-03-13T13:17:56,989][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [ingest-geoip]
[2022-03-13T13:17:56,989][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [ingest-user-agent]
[2022-03-13T13:17:56,989][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [kibana]
[2022-03-13T13:17:56,989][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [lang-expression]
[2022-03-13T13:17:56,990][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [lang-mustache]
[2022-03-13T13:17:56,990][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [lang-painless]
[2022-03-13T13:17:56,990][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [mapper-extras]
[2022-03-13T13:17:56,990][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [mapper-version]
[2022-03-13T13:17:56,990][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [parent-join]
[2022-03-13T13:17:56,990][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [percolator]
[2022-03-13T13:17:56,990][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [rank-eval]
[2022-03-13T13:17:56,990][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [reindex]
[2022-03-13T13:17:56,990][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [repositories-metering-api]
[2022-03-13T13:17:56,991][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [repository-url]
[2022-03-13T13:17:56,991][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [search-business-rules]
[2022-03-13T13:17:56,991][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [searchable-snapshots]
[2022-03-13T13:17:56,991][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [spatial]
[2022-03-13T13:17:56,991][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [systemd]
[2022-03-13T13:17:56,991][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [transform]
[2022-03-13T13:17:56,991][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [transport-netty4]
[2022-03-13T13:17:56,991][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [unsigned-long]
[2022-03-13T13:17:56,991][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [vectors]
[2022-03-13T13:17:56,992][INFO ][o.e.p.PluginsService ] [xxxcom] loaded module [wildcard]
[2022-03-13T13:17:56,992][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [x-pack-aggregate-metric]
[2022-03-13T13:17:56,992][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [x-pack-analytics]
[2022-03-13T13:17:56,992][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [x-pack-async]
[2022-03-13T13:17:56,992][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [x-pack-async-search]
[2022-03-13T13:17:56,992][INFO ][o.e.p.PluginsService ] [xxxcom] loaded module [x-pack-autoscaling]
[2022-03-13T13:17:56,992][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [x-pack-ccr]
[2022-03-13T13:17:56,992][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [x-pack-core]
[2022-03-13T13:17:56,992][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [x-pack-data-streams]
[2022-03-13T13:17:56,993][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [x-pack-deprecation]
[2022-03-13T13:17:56,993][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [x-pack-enrich]
[2022-03-13T13:17:56,993][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [x-pack-eql]
[2022-03-13T13:17:56,993][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [x-pack-fleet]
[2022-03-13T13:17:56,993][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [x-pack-graph]
[2022-03-13T13:17:56,993][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [x-pack-identity-provider]
[2022-03-13T13:17:56,993][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [x-pack-ilm]
[2022-03-13T13:17:56,993][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [x-pack-ingest]
[2022-03-13T13:17:56,993][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [x-pack-logstash]
[2022-03-13T13:17:56,993][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [x-pack-ml]
[2022-03-13T13:17:56,994][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [x-pack-monitoring]
[2022-03-13T13:17:56,994][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [x-pack-ql]
[2022-03-13T13:17:56,994][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [x-pack-rollup]
[2022-03-13T13:17:56,994][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [x-pack-runtime-fields]
[2022-03-13T13:17:56,994][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [x-pack-security]
[2022-03-13T13:17:56,994][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [x-pack-sql]
[2022-03-13T13:17:56,994][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [x-pack-stack]
[2022-03-13T13:17:56,994][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [x-pack-voting-only-node]
[2022-03-13T13:17:56,994][INFO ][o.e.p.PluginsService ] [xxx.com] loaded module [x-pack-watcher]
[2022-03-13T13:17:56,995][INFO ][o.e.p.PluginsService ] [xxx.com] loaded plugin [ingest-attachment]
[2022-03-13T13:17:57,018][ERROR][o.e.b.Bootstrap ] [xxx.com] Exception
java.lang.IllegalStateException: failed to obtain node locks, tried [[/var/lib/elasticsearch]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:292) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.node.Node.(Node.java:351) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.node.Node.(Node.java:278) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:216) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:216) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:387) [elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) [elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) [elasticsearch-cli-7.11.1.jar:7.11.1]
at org.elasticsearch.cli.Command.main(Command.java:79) [elasticsearch-cli-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) [elasticsearch-7.11.1.jar:7.11.1]
Caused by: java.io.IOException: failed to obtain lock on /var/lib/elasticsearch/nodes/0
at org.elasticsearch.env.NodeEnvironment$NodeLock.(NodeEnvironment.java:214) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:262) ~[elasticsearch-7.11.1.jar:7.11.1]
… 12 more
Caused by: java.nio.file.AccessDeniedException: /var/lib/elasticsearch/nodes/0/node.lock
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:181) ~[?:?]
at java.nio.channels.FileChannel.open(FileChannel.java:292) ~[?:?]
at java.nio.channels.FileChannel.open(FileChannel.java:345) ~[?:?]
at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:125) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:35:28]
at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:35:28]
at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:35:28]
at org.elasticsearch.env.NodeEnvironment$NodeLock.(NodeEnvironment.java:207) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:262) ~[elasticsearch-7.11.1.jar:7.11.1]
… 12 more
[2022-03-13T13:17:57,023][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [teamserver.tiedt-online.com] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/var/lib/elasticsearch]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.11.1.jar:7.11.1]
at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.11.1.jar:7.11.1]
Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/var/lib/elasticsearch]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:292) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.node.Node.(Node.java:351) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.node.Node.(Node.java:278) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:216) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:216) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:387) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.11.1.jar:7.11.1]
… 6 more
Caused by: java.io.IOException: failed to obtain lock on /var/lib/elasticsearch/nodes/0
at org.elasticsearch.env.NodeEnvironment$NodeLock.(NodeEnvironment.java:214) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:262) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.node.Node.(Node.java:351) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.node.Node.(Node.java:278) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:216) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:216) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:387) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.11.1.jar:7.11.1]
… 6 more
Caused by: java.nio.file.AccessDeniedException: /var/lib/elasticsearch/nodes/0/node.lock
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:181) ~[?:?]
at java.nio.channels.FileChannel.open(FileChannel.java:292) ~[?:?]
at java.nio.channels.FileChannel.open(FileChannel.java:345) ~[?:?]
at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:125) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:35:28]
at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:35:28]
at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45) ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 - atrisharma - 2020-10-29 19:35:28]
at org.elasticsearch.env.NodeEnvironment$NodeLock.(NodeEnvironment.java:207) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:262) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.node.Node.(Node.java:351) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.node.Node.(Node.java:278) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:216) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:216) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:387) ~[elasticsearch-7.11.1.jar:7.11.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.11.1.jar:7.11.1]
… 6 more

5

Hi Andy, my Swiss IT pocketknife,

thanks for this, but it dont help.

6

This seems to be the relevant error:

[2022-03-13T13:17:57,018][ERROR][o.e.b.Bootstrap ] [xxx.com] Exception java.lang.IllegalStateException: failed to obtain node locks, tried [[/var/lib/elasticsearch]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?

It happens immediately after loading the ingest-attachment, did you try to start elasticsearch without the plugin?

Please also check permissions:

ls -ld /var/lib/elasticsearch

After searching I found that others solved this issue by deleting lock files. If you don’t need elasticsearch for other apps, you could try it:

7

[root@server ~]# ls -ld /var/lib/elasticsearch
drwxr-s— 3 elasticsearch elasticsearch 19 Feb 23 23:26 /var/lib/elasticsearch

1 Like
8

Permissions are ok.

9

What commands are these in the language of Nethserver

10

I never tested but it should work to delete write.lock files under /var/lib/elasticsearch.

cd to relevant directory:

cd /var/lib/elasticsearch

Find and delete write.lock files:

find -name write.lock -delete

11

No way.
I removed zammad from the system. Case closed

Thanks…

2 Likes