Elasticsearch stopped service

capote · March 15, 2021, 1:02pm

Hi my Elasticsearch Engine stoped his service


[root@ns-srv01 ~]# systemctl status elasticsearch
● elasticsearch.service - Elasticsearch

Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)

Active: failed (Result: exit-code) since Mon 2021-03-15 13:51:16 CET; 7min ago

Docs: https://www.elastic.co

Process: 2418 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)

Main PID: 2418 (code=exited, status=1/FAILURE)

Mar 15 13:51:15 ns-srv01.dargels.de systemd-entrypoint[2418]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareComma…va:75) Mar 15 13:51:15 ns-srv01.dargels.de systemd-entrypoint[2418]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) Mar 15 13:51:15 ns-srv01.dargels.de systemd-entrypoint[2418]: at org.elasticsearch.cli.Command.main(Command.java:79) Mar 15 13:51:15 ns-srv01.dargels.de systemd-entrypoint[2418]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) Mar 15 13:51:15 ns-srv01.dargels.de systemd-entrypoint[2418]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) Mar 15 13:51:15 ns-srv01.dargels.de systemd-entrypoint[2418]: For complete error details, refer to the log at /var/log/elasticsearch/elastic…ch.log Mar 15 13:51:16 ns-srv01.dargels.de systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE Mar 15 13:51:16 ns-srv01.dargels.de systemd[1]: Failed to start Elasticsearch. Mar 15 13:51:16 ns-srv01.dargels.de systemd[1]: Unit elasticsearch.service entered failed state. Mar 15 13:51:16 ns-srv01.dargels.de systemd[1]: elasticsearch.service failed. Hint: Some lines were ellipsized, use -l to show in full.


[root@ns-srv01 ~]# journalctl -xe
Mar 15 13:51:15 ns-srv01.dargels.de systemd-entrypoint[2418]: uncaught exception in thread [main]
Mar 15 13:51:15 ns-srv01.dargels.de systemd-entrypoint[2418]: java.lang.IllegalArgumentException: Plugin [ingest-attachment] was built for Elasticsea
Mar 15 13:51:15 ns-srv01.dargels.de systemd-entrypoint[2418]: at org.elasticsearch.plugins.PluginsService.verifyCompatibility(PluginsService.java:342
Mar 15 13:51:15 ns-srv01.dargels.de systemd-entrypoint[2418]: at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:620)
Mar 15 13:51:15 ns-srv01.dargels.de systemd-entrypoint[2418]: at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:477)
Mar 15 13:51:15 ns-srv01.dargels.de systemd-entrypoint[2418]: at org.elasticsearch.plugins.PluginsService.(PluginsService.java:158)
Mar 15 13:51:15 ns-srv01.dargels.de systemd-entrypoint[2418]: at org.elasticsearch.node.Node.(Node.java:335)
Mar 15 13:51:15 ns-srv01.dargels.de systemd-entrypoint[2418]: at org.elasticsearch.node.Node.(Node.java:278)
Mar 15 13:51:15 ns-srv01.dargels.de systemd-entrypoint[2418]: at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:216)
Mar 15 13:51:15 ns-srv01.dargels.de systemd-entrypoint[2418]: at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:216)
Mar 15 13:51:15 ns-srv01.dargels.de systemd-entrypoint[2418]: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:387)
Mar 15 13:51:15 ns-srv01.dargels.de systemd-entrypoint[2418]: at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
Mar 15 13:51:15 ns-srv01.dargels.de systemd-entrypoint[2418]: at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
Mar 15 13:51:15 ns-srv01.dargels.de systemd-entrypoint[2418]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:7
Mar 15 13:51:15 ns-srv01.dargels.de systemd-entrypoint[2418]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
Mar 15 13:51:15 ns-srv01.dargels.de systemd-entrypoint[2418]: at org.elasticsearch.cli.Command.main(Command.java:79)
Mar 15 13:51:15 ns-srv01.dargels.de systemd-entrypoint[2418]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
Mar 15 13:51:15 ns-srv01.dargels.de systemd-entrypoint[2418]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)
Mar 15 13:51:15 ns-srv01.dargels.de systemd-entrypoint[2418]: For complete error details, refer to the log at /var/log/elasticsearch/elasticsearch.lo
Mar 15 13:51:16 ns-srv01.dargels.de systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Mar 15 13:51:16 ns-srv01.dargels.de systemd[1]: Failed to start Elasticsearch.
– Subject: Unit elasticsearch.service has failed

– Defined-By: systemd

– Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

– Unit elasticsearch.service has failed.

– The result is failed.

Mar 15 13:51:16 ns-srv01.dargels.de systemd[1]: Unit elasticsearch.service entered failed state. Mar 15 13:51:16 ns-srv01.dargels.de systemd[1]: elasticsearch.service failed.

I do not remember a context of action to reproduce the error.
Does anyone have an idea?
Best regards, Marko

Andy_Wismer · March 15, 2021, 1:08pm

@capote

Hi Marko

If I recall rightly, inside Nextcloud was an partial update for one of the Fulltextsearch components of NextCloud. Maybe updating that makes it work again… (See under Nextcloud Apps / Search for anything…).

My 2 cents
Andy

capote · March 15, 2021, 1:21pm

Hi @Andy_Wismer
There is nothing suspicious. (updates, disabled apps…) Or, I’m not able to identify it.
I disabled / enabled some properties - no other result.

Sincerely, MArko

Andy_Wismer · March 15, 2021, 1:30pm

Hi Marko

See if you have the same versions, I can’t recall whch of these 4 was updated… (Could be the 20.01…)

capote · March 15, 2021, 1:36pm

Andy_Wismer · March 15, 2021, 1:48pm

I’m sure you alos tried a reboot of your server.
BTW: My Elasticsearch IS working but again not giving any results…

capote · March 15, 2021, 4:37pm

of course - reboot tut immer gut. Nur diesmal nicht

mrmarkuz · March 15, 2021, 4:52pm

Did you already check /var/log/elasticsearch/elasticsearch.log ?

capote · March 15, 2021, 5:06pm

Yes, I have in the meantime:


[root@ns-srv01 ~]# cat /var/log/elasticsearch/elasticsearch.log | grep Exception

[2021-03-15T13:45:28,009][INFO ][o.e.n.Node ] [ns-srv01.dargels.de] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-15548305381735193353, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms8022m, -Xmx8022m, -XX:MaxDirectMemorySize=4206886912, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true] [2021-03-15T13:45:30,675][ERROR][o.e.b.Bootstrap ] [ns-srv01.dargels.de] Exception java.lang.IllegalArgumentException: Plugin [ingest-attachment] was built for Elasticsearch version 7.10.1 but version 7.11.1 is running [2021-03-15T13:45:30,680][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [ns-srv01.dargels.de] uncaught exception in thread [main] org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: Plugin [ingest-attachment] was built for Elasticsearch version 7.10.1 but version 7.11.1 is running Caused by: java.lang.IllegalArgumentException: Plugin [ingest-attachment] was built for Elasticsearch version 7.10.1 but version 7.11.1 is running [2021-03-15T13:51:13,724][INFO ][o.e.n.Node ] [ns-srv01.dargels.de] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-14831757749294032337, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms8022m, -Xmx8022m, -XX:MaxDirectMemorySize=4206886912, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true] [2021-03-15T13:51:15,932][ERROR][o.e.b.Bootstrap ] [ns-srv01.dargels.de] Exception java.lang.IllegalArgumentException: Plugin [ingest-attachment] was built for Elasticsearch version 7.10.1 but version 7.11.1 is running [2021-03-15T13:51:15,937][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [ns-srv01.dargels.de] uncaught exception in thread [main] org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: Plugin [ingest-attachment] was built for Elasticsearch version 7.10.1 but version 7.11.1 is running Caused by: java.lang.IllegalArgumentException: Plugin [ingest-attachment] was built for Elasticsearch version 7.10.1 but version 7.11.1 is running [root@ns-srv01 ~]#

If I knew what that was supposed to tell me…

One thing could be:

[ns-srv01.dargels.de] Exception
java.lang.IllegalArgumentException: Plugin [ingest-attachment] was built for Elasticsearch version 7.10.1 but version 7.11.1 is running

I updated ingest-attachment_
[root@ns-srv01 ~]# sudo /usr/share/elasticsearch/bin/elasticsearch-plugin remove ingest-attachment -> removing [ingest-attachment]…


[root@ns-srv01 ~]# sudo /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-attachment

-> Installing ingest-attachment

-> Downloading ingest-attachment from elastic

[=================================================] 100%

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

@     WARNING: plugin requires additional permissions     @

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@


java.lang.RuntimePermission accessClassInPackage.sun.java2d.cmm.kcms
java.lang.RuntimePermission accessDeclaredMembers
java.lang.RuntimePermission getClassLoader
java.lang.reflect.ReflectPermission suppressAccessChecks

java.security.SecurityPermission createAccessControlContext See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html for descriptions of what these permissions allow and the associated risks. Continue with installation? [y/N]y -> Installed ingest-attachment

The service is now active again.

Thank you for your attention.

capote · March 15, 2021, 5:46pm

Right now he indexes a lot of crap

capote · March 15, 2021, 7:46pm

HI Andy, now I have the same problem…no search results.

Andy_Wismer · March 15, 2021, 7:47pm

I put in a fresh pdf to test, it’s still indexing, but i’m getting strange errors at the input box…

capote · March 15, 2021, 7:50pm

Exactly the same at me. Look above at my screenshot.

Andy_Wismer · March 15, 2021, 7:52pm

capote · March 15, 2021, 7:53pm

no, that’s not at me. I only get 0 results.

Andy_Wismer · March 15, 2021, 7:54pm

I also get 0 results…
I just get more errors!

About 10 of the messages above…

capote · March 15, 2021, 7:56pm

you are the winner. I get nothing, you get at least errors.

capote · March 16, 2021, 8:20am

I’m wondering about the memory consumption.

The dent appeared when I deactivated the Elasticsearch service. After reactivating, the memory consumption jumped up again.

Andy_Wismer · March 16, 2021, 8:39am

@capote

My memery usage isn’t that severe…

grafik

But it can’t find files, only everything else (Adresses, Calendar works)…

My 2 cents
Andy

capote · March 16, 2021, 9:29am

@Andy_Wismer
Also rebooting didn’t reduce the memory consumption.
@mrmarkuz
Do you have an idea how to deal with?
I stopped and disabled the service now.

Sincerely, MArko