Could not connect to accounts provider!

[MrE (Enrique D)]

NethServer Version: release 7.6.1810 (final)
Module: Domain Accounts
Since previous update I’m observing a slow/lag in response on the section “Domain Accounts”, I click that and then after some angst moments, this message.

Two days ago if I click on “Samba audit” it shows a blank page, today it shows correctly (I hope)

That other day, this was the load, when it goes to normal I can see the two sections (domain account & sambaStatus):

And this CPU was at full usage that day, but not today:

This NS runs in a proxmox server with ZFS.
This is the load on this proxmox server, it looks normal, on that day I need to reboot twice, the second time I need to stop the vm because it doesn’t reboot

About ZFS file system, I know a little about it, just that it is intensive when it does the scrub, but I don’t see there is big activity daily (need to investigate more on ZFS), the hard disks aren’t fast are RED NAS drives.

What more I can look to detect and fix this behavior ?
Regards

---- Edit
The problem is that today, a lot of users included myself, cannot be authenticated and can’t use some services in our lan, mostly ws need authentication to use applications that use sql server.
I reboot the NS and then we can work again, the effect not occur in all, in some other PC it works, but maybe, because the “ticket” was still valid.

Something new in our network, I got a fortinet (FG) that I want to learn how to use, but of course is clean, not have rules just 3 interfaces connected for WAN (internet), one LAN to managed and share the Internet, when I reboot the NS, I disconnect the FG too… Just to explain a little.

Other thing, is that our antivirus (Avast) is updating (new version) and it just create a huge traffic in our lan. The bandwidth is controlled with a trick (using an old software for proxy) just to redirect the upgrade of the antivirus to another PC (not joined in the domain)

This are little details just to be sure.

I’ll try to use some tool to try to monitor our lan, need to know what (ethereal, wireshark) - researching about it.

---- Edit 2
That Tuesday I read this posts:
After reading this:

Account provider connection refused

Ok I am about to apply the same to another affected system. But its S|DNS box only has 2 entries the 1st pointing to the server and the 2nd 8.8.8.8. So will just try the command line arguments and report back

Could not connect to accounts provider

NethServer Version: NethServer release 7.6.1810 (final) Module: Account Provider I fill the form for Account Provider, every seems fine, but if I go to “Domain accounts” on the Web GUI it says: “Could not connect to accounts provider!” However I can query about a domain user from a cli: id ldap uid=1042601118(ldap@example.com) gid=1042600513(domain users@example.com) groups=1042600513(domain users@example.com) More configs: account-provider-test dump { "BindDN" : "NAOS\\TINY-FISHWIFE…

Samba is not working (SSSD accountsprovider error)

Ok, I’m playing the last cards… shorewall status shorewall clear And repeat dig check Then shorewall start

I see that LdapURI was empty and manage to fix it. But, the problem remains, even after a reboot of the VM nethserver in proxmox. And after a few hours later the load return to normal.

---- Edit 3
At this time 20:47, there is nobody at work.
The NS is responding normally … Good! but how/why?
This NS is only used as an authentication, it has some shared folder… I wonder… I wonder if that other module (Samba Audit) is the cause. Yesterday I see a lot of (and I mean a LOT) of pages/records… I take pictures before deleting that logs, and now I see 97 pages from the “24 06:45” to “26 11:15”

So, because I upgrade this Proxmox I will need to shutdown NS and reboot the Proxmox server. (fingers crossed, I’ll report later)

But I will disable the Samba audit of the shares to observe and watch the NS behavior.
---- Edit 4
Systems rebooted, this are the graph of the NS server

So I will see if tomorrow all works as need to be, or maybe just at Monday when all the crew is here.
Regards!
— Edit 5
The module with the huge amounts of logs was “Samba audit” not “SambaStatus”, sorry my mistake.

[MrE (Enrique D)]

Ok, today Saturday, the NS server is running and very fine, of course is too soon to speak if the issue is fixed.
I’m suspecting about the “Samba Audit” application on the “Shared folders/auditing” mixed with the Antivirus that is scanning all.

I disable the auditing for all the shared folders.
Is not an error of “Samba Audit” but of the antivirus, because we need to configure the AV to scan all the shared folders, after we where hit with some kind of virus.

Then when we join new PCs on the domain, the AV just scans and maybe it is the cause.

I’ll keep an eye this Monday when all our people returns to work.

Regards!

[MrE (Enrique D)]

Monday: The issues isn’t repeated, the server shows a decent load:

Then I’m almost sure, that all my issues was the mixture of antivirus + Sambaaudit (not SambaStatus) logs.
If I need to choose about an stable system or “known” who-what-when is accessing files, I always choose stability. Of course if the antivirus was not needed in this world, this will be another story.

Now I check the ‘interface’ activity, and looks: good ? :

---

I’ll keep an eye for this week on the NS system, so far this looks good.

Regards

[MrE (Enrique D)]

The DoS hasn’t occurred this week.

Some graphs of the load:

Hopefully the issue doesn’t appear never.

[robb (Rob Bosch)]

Hi @MrE
This is some weird story! Is your conclusion that AV and SambaAudit don’t go well together? I think I have them both running on my fileserver too, but I only have a few users and the shares are not accessed that often. But I have never seen such behavior.

[MrE (Enrique D)]

I’m trying to collect all the elements to get a conclusion. But looks the real conclusion.

Avast published a new version some weeks ago, we are still migrating users to the NS/domain and upgrading the AV.

The AV, is configured (isn’t by default) to scan the network shares, and when the AV is upgraded it scans and “hits” each .exe file on the shared folders.

This shared folders has a lot of program files, one is the “setup” share for a lot of software that we need, our ERP by ex. need them (activeX files, dll, setup, you name it).

The “lucky hint” was that some new joined users don’t need to use those files (ever), so I began to wonder the reason. That is when the AV came to the scene.

After some updates-reboots of the (Proxmox/NethServer) servers, and seeing the hits of that users, I remember to check the SambaAudit logs, yikes! thousands of entries in the logs, not, I mean hundred thousands of entries, that’s “normal”? I don’t think so.

Lucky me, the first entries in the SambaAudit logs are users that don’t need the ERP software and they where the last on the AV upgrade process.

Quickly, I disable the auditing of the shared folders. (Friday night)

On Saturday, the load now is normal, yes!, On Monday, the same, I can see network load graph a little heavy, but I don’t see the audit logs be bombed by the AV artillery.

I can’t disable the AV network scan, we where hit twice, and I found the guilty users (both wife and husband), but I can’t make then to pay for their sins, yet.

I will create some filters on the AV, to disable the scan on that network shares, by another pinch of luck that shares are accessed but not need to be mounted.

Regards

—Note # 1:
This graph show the year in the interface, you can see the traffic grows when users and added and how the AV/Audit is ruining my day:

—Note # 2:
AV filters to ignored the folder shared has been created.

[robb (Rob Bosch)]

So, if I interpret the situation correctly:
You use a non default AV solution: Avast
Avast gives a shitload of false positives because it points any .exe / .dll / ActiveX file as (possible) unwanted file. Because every user that has access to your shares initiates this behavior, you end up with huge load and accumulation of SambaAudit log files…
Can’t Avast be tweaked to not generate those (false) positives? Why use Avast (in my experience a resource hungry AV solution, at least on Win clients) while we have clamav by default on NethServer?

[MrE (Enrique D)]

Greetings, well,

@robb

You use a non default AV solution: Avast

We need the AV, because we have a lot of windows PC, I don’t use all the NS’s features. Mainly I use it for authentication, later I plan to use to apply GPOs and some other services joined on the AD.

From some years ago I use (Citadel) as our mail server, is very good and “simple” to setup, but I got an old version that I plan to upgrade. This mail server has clamav + spamassassin, but some nasty stuff can go though and is there when the AV is a need, the infections we where having occurs via USB sticks, and there clamav can’t help.

@robb

Can’t Avast be tweaked to not generate those (false) positives?

It can be adjusted to be more or less strict; but yesterday I add some scan exceptions for the shared folders on NS, that I’m still monitoring.

I don’t want to overload the principal role of this NS (Active Directory / Authentication) and want to keep this NS as simple as I can, for easy backup and recovery; I likely want to install more slaves of NS for the other features. But, I want to use a NAS for our users files, currently evaluating if buying a Synology or just buy hardware/server to install Nas4Free now Xigmanas.

Yesterday, after the upgrading NS, I got this issue: LDAP client internal error (AccountProvider_Error_82), that’s why I want a simple and stable NS, without many services to monitor.

I wonder if having so many servers is more that I can chew .

(Proxmox) + (NS)
(~70 windows clients)
(1 sql server/w2012r2) + (ERP, Payroll system, and a lot of apps)
(2 mail servers)
(3 small NAS for backups)
(NAS for files and storage buy or build)
(2 web servers | DNS servers | )
(The old w2k server, soon to be decommissioned - hopefully)

Regards

[robb (Rob Bosch)]

@MrE

We need the AV, because we have a lot of windows PC, I don’t use all the NS’s features. Mainly I use it for authentication, later I plan to use to apply GPOs and some other services joined on the AD.

You still can use Avast for your client pc’s. (although personally I would not choose Avast because of the resource hog it is) You could opt to use default ClamAV for NS

[MrE (Enrique D)]

@robb

Avast because of the resource hog it is

Alas! yes it is.

So, @robb; I’m guessing right that after some ssd updates, something need to be rebuilding like a database?
This because when the AV was scanning the shared folders that process takes a long time to finish.
And yesterday, after the update, I see that something was happening that I don’t see the full information of the Domain accounts. And today morning the whole info is there again.

So, I wonder what log file need to check to see if this process is happening. Any guess?

Regards

[davidep (Davide Principi)]

Please look at the memory usage graph: I’m afraid your system is in short of memory and random processes get killed by the kernel.

[MrE (Enrique D)]

Thank you @davidep, It was using 10GB now increased to 12GB, I will schedule a reboot later to apply this 12GB assigned.
Or maybe I need to add more RAM, but, how much can you recommend?

This is the yearly memory graph:

wow!

[MrE (Enrique D)]

@davidep, OK RAM increased to 12GB and server restarted.

Now to wait and get some

[davidep (Davide Principi)]

Just to clarify, I don’t think more ram is a solution. There could be a memory leakage that slows down the system at a certain point. You should identify the memory consuming process when the load starts to grow.

Could you post the swap graph in the same period?

[MrE (Enrique D)]

Let me connect remotely to get it.
And yes, I see the memory graph today before the reboot (today) and on those days
It show the “cached” memory on blue at high levels just as above.

---

But at this moment, there is no significative memory usage:

Each gap is a restart event.

Ooops you mean the swap (auch!), there is, nothing “bad” I see:

[MrE (Enrique D)]

Two weeks are passed, the previous one, is reported here.

So, after this time, I know that the issues that NS was having from some months ago where caused by the savage antivirus we use, not a bad thing (but worse ), mental note to remember that security tools can create issues in a local and weak network: Deny Of Service galore.

Using some filters for the antivirus ‘fix’ the thing.

Maybe avast treat/see the file system in different way on Nethserver (samba shares) than in windows (shared folders) because I don’t see the old file server stressed by the scanning (right now). Ah some users report slow response from time to time on the network.

When our new NAS arrive, the issue can reappear but in another device.

Time to learn how to scan our network traffic, maybe with wire shark or some tool related. Any ideas?

Thanks and regards.

[pike (Michael Kicks)]

@MrE

Any ideas ?

Change the AV configuration about automated scanning network drives, disabling it.
Consider the opportunity for a full filesystem check from ONE client on a timed schedule (once a month during the night?)

[MrE (Enrique D)]

@pike

check from ONE client on a timed schedule (once a month during the night?)

Hey! Thank you @pike , I was thinking the same after my previous post. The problem we live some months ago was this:

• 9AM - user put usb stick with “old” virus/trojan
• User open false ‘document’ (was an executable), we have some lemmings PEBKAC.
• The virus, put their exe hidden in the local disks, seeks for office files, create access links to this files, and do the same on the shared folders that it can see.
• 10AM - other users start to work in the shared folders, open their ‘documents’ that open the virus and repeat the process.
• 11AM, I don’t know what, someone calls, or just blind luck that I see a lot of files opened by some users. Then I start learning what is happening.
• The antivirus, IGNORE the treat just because it was old, even malware bytes (now part of avast) reject the reported sample, because is ‘old’. WTH!
• After reporting to avast and maybe some hours later (or a day), it began to recognize the threat and blocks the executable. But before the ‘real’ antivirus was myself (search, detect, move, delete).

Lessons learned:

1. Deactivate the usb storage use on each computer
2. Need to activate to scan the shared folders at runtime
3. Create filters to ignore shared folders (nethserver) that we I.T. knows are safe-clean and read-only
4. (Today) Choose a few users to Scan always the network shares (the writable ones)
5. Avast, needs to create a good update program. The new version came some months ago and it can’t be upgraded/installed if we don’t remove the old one

The difficult part is choose the right ones, when some or all stops scanning and something similar appear, Oh my!, not need another week like that.

Scanning just at night or on timed scheduled looks pretty dangerous.

Regards

[MrE (Enrique D)]

By the way, I pretend to use snapshots in the new NAS.
But I wonder, is you guys, that use Nextcloud (NC), think is more safe NC that only use a network shared folder for the users.
I know some of the benefits of NC; and of course if the user get hit by virus, they can ‘lost’ their files (unless backups exists);

So if I install the NC on the NAS with snapshots (yes, with backup prevention); I wonder how the restore process can be made. Because I barely knows NC, I recall it uses a database, but alas! I can’t grasp the process to full protect our users documents in NC+snapshot+recovery. Maybe NC see that there are files restored (with previous dates) and it starts updating its database, don’t know.

Regards (LOL I hijacked my own post )

[pike (Michael Kicks)]

@MrE

9AM - user put usb stick with “old” virus/trojan

You can’t solve this issue. But you can use a different AV.

IMVHO Avast is not a good choice, but i used only the “desktop product”, without the admin console.
I had experiences on McAfee VSE (with ePO Orchestrator), ESET Business, Symantec Enterprise, Sophos Home (web console), GData, TrendMicro OfficeScan. Most of them have issues, others do not recognize every detail. Most of these experiences are quite old, but i do not want to start to know Avast.

IMVHO this issue appeared tell me to look for another option for AV. Some are quite cheap (Sophos is quite affordable, as far as I can remember).
If you have to workaround servers for the issues for the Antivirus, maybe this product is not the best option, therefore you should at least try another one.