Elasticsearch stopped service

[Andy_Wismer (André Wismer)]

Hi Marko

See if you have the same versions, I can’t recall whch of these 4 was updated… (Could be the 20.01…)

[capote (Marko)]

[Andy_Wismer (André Wismer)]

I’m sure you alos tried a reboot of your server.
BTW: My Elasticsearch IS working but again not giving any results…

[capote (Marko)]

@Andy_Wismer

I’m sure you alos tried a reboot of your server.

of course - reboot tut immer gut. Nur diesmal nicht

[mrmarkuz (Markus Neuberger)]

@capote

For complete error details, refer to the log at /var/log/elasticsearch/elastic…ch.log

Did you already check /var/log/elasticsearch/elasticsearch.log ?

[capote (Marko)]

Yes, I have in the meantime:

[root@ns-srv01 ~]# cat /var/log/elasticsearch/elasticsearch.log | grep Exception

[2021-03-15T13:45:28,009][INFO ][o.e.n.Node ] [ns-srv01.dargels.de] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-15548305381735193353, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms8022m, -Xmx8022m, -XX:MaxDirectMemorySize=4206886912, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
[2021-03-15T13:45:30,675][ERROR][o.e.b.Bootstrap ] [ns-srv01.dargels.de] Exception
java.lang.IllegalArgumentException: Plugin [ingest-attachment] was built for Elasticsearch version 7.10.1 but version 7.11.1 is running
[2021-03-15T13:45:30,680][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [ns-srv01.dargels.de] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: Plugin [ingest-attachment] was built for Elasticsearch version 7.10.1 but version 7.11.1 is running
Caused by: java.lang.IllegalArgumentException: Plugin [ingest-attachment] was built for Elasticsearch version 7.10.1 but version 7.11.1 is running
[2021-03-15T13:51:13,724][INFO ][o.e.n.Node ] [ns-srv01.dargels.de] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-14831757749294032337, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms8022m, -Xmx8022m, -XX:MaxDirectMemorySize=4206886912, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
[2021-03-15T13:51:15,932][ERROR][o.e.b.Bootstrap ] [ns-srv01.dargels.de] Exception
java.lang.IllegalArgumentException: Plugin [ingest-attachment] was built for Elasticsearch version 7.10.1 but version 7.11.1 is running
[2021-03-15T13:51:15,937][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [ns-srv01.dargels.de] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: Plugin [ingest-attachment] was built for Elasticsearch version 7.10.1 but version 7.11.1 is running
Caused by: java.lang.IllegalArgumentException: Plugin [ingest-attachment] was built for Elasticsearch version 7.10.1 but version 7.11.1 is running
[root@ns-srv01 ~]#


If I knew what that was supposed to tell me…

One thing could be:

[ns-srv01.dargels.de] Exception
java.lang.IllegalArgumentException: Plugin [ingest-attachment] was built for Elasticsearch version 7.10.1 but version 7.11.1 is running

I updated ingest-attachment_

[root@ns-srv01 ~]# sudo /usr/share/elasticsearch/bin/elasticsearch-plugin remove ingest-attachment
-> removing [ingest-attachment]…

[root@ns-srv01 ~]# sudo /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-attachment
-> Installing ingest-attachment
-> Downloading ingest-attachment from elastic
[=================================================] 100%
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: plugin requires additional permissions @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

• java.lang.RuntimePermission accessClassInPackage.sun.java2d.cmm.kcms
• java.lang.RuntimePermission accessDeclaredMembers
• java.lang.RuntimePermission getClassLoader
• java.lang.reflect.ReflectPermission suppressAccessChecks
• java.security.SecurityPermission createAccessControlContext
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.
Continue with installation? [y/N]y
-> Installed ingest-attachment


The service is now active again.

Thank you for your attention.

[capote (Marko)]

Right now he indexes a lot of crap

[capote (Marko)]

@Andy_Wismer

BTW: My Elasticsearch IS working but again not giving any results

HI Andy, now I have the same problem…no search results.

[Andy_Wismer (André Wismer)]

I put in a fresh pdf to test, it’s still indexing, but i’m getting strange errors at the input box…

[capote (Marko)]

Exactly the same at me. Look above at my screenshot.

[Andy_Wismer (André Wismer)]

[capote (Marko)]

no, that’s not at me. I only get 0 results.

[Andy_Wismer (André Wismer)]

I also get 0 results…
I just get more errors!

About 10 of the messages above…

[capote (Marko)]

you are the winner. I get nothing, you get at least errors.

[capote (Marko)]

I’m wondering about the memory consumption.

The dent appeared when I deactivated the Elasticsearch service. After reactivating, the memory consumption jumped up again.

[Andy_Wismer (André Wismer)]

@capote

My memery usage isn’t that severe…

But it can’t find files, only everything else (Adresses, Calendar works)…

My 2 cents
Andy

[capote (Marko)]

@Andy_Wismer
Also rebooting didn’t reduce the memory consumption.
@mrmarkuz
Do you have an idea how to deal with?
I stopped and disabled the service now.

Sincerely, MArko

[mrmarkuz (Markus Neuberger)]

Elasticsearch uses a lot of RAM.

Maybe you can limit the RAM usage:

Zammad ticketing/helpdesk on Nethserver available

It may be too much memory used by elasticsearch, I tested on a VM with 8GB RAM. You could try to limit the memory usage: Don’t cross the 32 GB limit — if you have servers with a lot of memory, it is generally better to run more Elasticsearch nodes than going over the 32 GB limit for maximal heap size. In short, using -Xmx32g or higher results in the JVM using larger, 64-bit pointers that need more memory. If you don’t go over -Xmx31g , the JVM will use smaller, 32-bit pointers by using comp…

[capote (Marko)]

But this memory eating started with the update of ingest-attachment.

That doesn’t seem normal to me. Together with the faulty function, as with Andy, all this does not look healthy.

I’m not motivated to enable the service if he is unusable.

[mrmarkuz (Markus Neuberger)]

Did you already try an older elasticsearch version with according ingest-attachment?