I’m running the latest version of Webtop. No updates have been applied to my Nethserver for a couple of weeks now so nothing has changed on my Nethserver. We been running Webtop successfully for the past 2 years without issues.
On Monday (yesterday) my users started to notice that mail was slow. Slow to login and slow to navigate around folders and open mail. On Monday evening (last night) I noticed in the Webtop log an error that said:
java.lang.outof memory error
In Webtop Settings > Advanced Settings I saw a minimum of 512 MB and a maximum of 1024 MB. I doubled these to a minimum of 1024 MB and Maximum of 2048 MB. My Nethserver has 12 GB of Ram allocated to it (running virtually on Proxmox). Nothing has changed on Nethserver and nothing has changed on my Proxmox server. Last evening mail performance improved after I increased the memory settings in Webtop.
Our Webtop has the chat app enabled (more than a year ago now) and I’ve recently (2 weeks ago) added the Jitsi integration app to Webtop. Again nothing has changed recently on my Nethserver or my Webtop.
This morning my users are once again reporting that mail is very slow to use again. I’ve looked into the Webtop log file and I don’t see memory errors. There is nothing in the Webtop Log to indicate a problem:
2021-05-17 22:56:26 [INFO ] c.s.webtop.core.app.ServiceManager - Initialized
2021-05-17 22:56:26 [INFO ] c.s.webtop.core.app.I18nManager - Initialized
2021-05-17 22:56:26 [INFO ] c.sonicle.webtop.core.app.WebTopApp - WTA initialization completed [webtop]
2021-05-17 22:56:31 [INFO ] c.sonicle.webtop.core.app.WebTopApp - Checking domains homes structure…
2021-05-17 22:56:31 [INFO ] c.sonicle.webtop.core.app.WebTopApp - Checking webapp version…
2021-05-17 22:56:31 [WARN ] c.sonicle.webtop.core.app.WebTopApp - No configuration found for TomcatManager [tomcat.manager.uri]
2021-05-17 22:56:31 [WARN ] c.sonicle.webtop.core.app.WebTopApp - Webapp version automatic check will NOT be performed!
2021-05-17 22:56:31 [INFO ] c.sonicle.webtop.core.app.WebTopApp - This webapp [webtop] is the latest
2021-05-17 22:57:31 [INFO ] c.sonicle.webtop.core.app.WebTopApp - Scheduling JobServices tasks…
2021-05-17 22:57:31 [INFO ] c.sonicle.webtop.core.app.WebTopApp - Created javax.mail.Session for localhost:10587
Can the webtop team suggest any reasons why this slowdown has suddenly occurred? Where else I can look for why there is slowdown errors.
A bit more information on this issue my users are seeing.
I did implement through crontab manager new cron jobs to cleanup my shared folder recycle bins. I’ve created a script per recycle bin (5 jobs) and implemented these cron jobs to run weekly. I setup these cron jobs over a week ago so they were not recent. I’ve disabled these cron jobs and immediately my users are now reporting that mail is fast once again. Here is a section of the logs I see from crontab manager. It appears that cron is running often which I didn’t expect. I thought I created my cron jobs to run only at night on the weekend:
Here is a screenshot of a cronjob I setup. Perhaps I’ve set this up to run too often. Again my intention was to run these cronjobs only on Saturday at 11:35 pm.
Sadly my Webtop slowness issues continues today even with cron jobs stopped. Users are reporting they can’t login to mail and when they do it’s so slow to navigate it’s unusable. There’s definetly something wrong within Webtop struggling for memory. I am not seeing these errors in my Webtop log:
2021-05-19 08:51:40 [ERROR] com.sonicle.webtop.mail.Service - Exception
java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-05-19 08:55:19 [ERROR] c.s.webtop.calendar.CalendarManager - Error handling event instances’ reminder alert
java.lang.OutOfMemoryError: GC overhead limit exceeded
2021-05-19 08:55:32 [ERROR] com.sonicle.webtop.mail.Service - Exception
jakarta.mail.MessagingException: folder cannot contain messages
at com.sun.mail.imap.IMAPFolder.open(IMAPFolder.java:1047)
at com.sun.mail.imap.IMAPFolder.open(IMAPFolder.java:977)
at com.sonicle.webtop.mail.FolderCache.open(FolderCache.java:1001)
at com.sonicle.webtop.mail.FolderCache._getMessages(FolderCache.java:1618)
at com.sonicle.webtop.mail.FolderCache.refresh(FolderCache.java:781)
at com.sonicle.webtop.mail.FolderCache.getMessages(FolderCache.java:899)
at com.sonicle.webtop.mail.Service$MessageListThread.run(Service.java:6812)
at java.lang.Thread.run(Thread.java:748)
2021-05-19 08:55:38 [ERROR] com.sonicle.webtop.mail.Service - Exception
java.lang.OutOfMemoryError: GC overhead limit exceeded
I’m also seeing a bit of swap used on my Nethserver which I wouldn’t expect to see. I have 12 GB allocated to my Nethserver and only 6 GB is being used today.
I’ve increased the webtop Min and Max memory allocated (1024 and 2048 respectively) but this has not solved my memory problem it seems.
I’ve done a bit of searching and found this thread:
This mentions that increasing the memory java uses as being a temporary solution. It appears to me that perhaps a memory leak is happening within Webtop.
Hi Charles, and forgive the delay with which I answer you
These days we are analyzing some other cases similar to yours with the development team and we have probably been able to identify the problems that can trigger loops in some particular conditions.
Thank you for this reply @lucag. Very much appreciative of your help with this issue! I would be interested to hear what root cause is with this problem as well. My friend also runs Webtop on Nethserver and has had no issues so far. He has more users than I have but he has not added the integrations I have (I’ve installed Webchat and Jitsi and Nextcloud integrations on my Webtop and he hasn’t).
I’ve been noticing that my Nethserver SWAP has MB allocated to it recently. My Nethserver runs virtually on Proxmox and I have 4 CPU’s and 12 GB of Ram allocated to my Nethserver. I wouldn’t expect any swap space to be used as my Ram is well under half used. Can you suggest why SWAP would be used on my server or what I can do to check why it’s needed? Could SWAP space be related to this memory leak that Webtop is having where it runs out of memory and crashes? When I restart Webptop/tomcat after an out of memory message my SWAP goes back to zero.
I’d like to ensure my Nethserver is running optimally. Please let me know what I can do on Nethserver to confirm where SWAP is being used.
I am writing to notice that we have just released the packages of the 5.12.3 release of WebTop which contains some fixes that solve the problem reported by @greavette
Just wanted to mark this post as solved and comment on the positive experience I’ve received when working with Nethesis and the @webtop_team.
Our office is small but our reliance on Nethserver and WebTop (mail) is a critical part of our business.
Services that are down could mean money is lost so uptime is of critical importance to our business. @lucag responded to my support request through Nethesis and was very responsive to the issue we were facing. He regularly checked into our server when slowness was reported and was quick to reply to my questions. I was very happy with the support I’ve received working on this problem and I’m very happy that the WebTop Team was able to find resolution for our server and any other server that may come across this same issue.
