Exiting due to repeated, frequent failures - squid

Anton_Viktorovich · January 30, 2020, 6:01am

Squid is constantly falling, how to solve the problem?

PRIORITY	1
SYSLOG_FACILITY	20
SYSLOG_IDENTIFIER	squid
SYSLOG_PID	3039
_BOOT_ID	7af8bf74106241278930158abe824e53
_CAP_EFFECTIVE	1fffffffff
_CMDLINE	/usr/sbin/squid -f /etc/squid/squid.conf
_COMM	squid
_EXE	/usr/sbin/squid
_GID	23
_HOSTNAME	srv-isa.ies.local
_MACHINE_ID	b920bf55cad4468891566b3fd8eab239
_PID	3039
_SOURCE_REALTIME_TIMESTAMP	1580363113968307
_SYSTEMD_CGROUP	/system.slice/squid.service
_SYSTEMD_SLICE	system.slice
_SYSTEMD_UNIT	squid.service
_TRANSPORT	syslog
_UID	23
__CURSOR	s=0c849b8fc7b949e385f29edb542252eb;i=2599;b=7af8bf74106241278930158abe824e53;m=aa0b8b3e4;t=59d54f6063424;x=a2b1646d7efba84e
__MONOTONIC_TIMESTAMP	45646132196
__REALTIME_TIMESTAMP	1580363113968676

Anton_Viktorovich · January 30, 2020, 6:06am

before that, such an error

(squid-1)

Too many queued negotiateauthenticator requests

PRIORITY	1
SYSLOG_FACILITY	1
SYSLOG_IDENTIFIER	(squid-1)
_BOOT_ID	7af8bf74106241278930158abe824e53
_CAP_EFFECTIVE	400
_CMDLINE	(squid-1) -f /etc/squid/squid.conf
_COMM	squid
_EXE	/usr/sbin/squid
_GID	23
_HOSTNAME	srv-isa.local
_MACHINE_ID	b920bf55cad4468891566b3fd8eab239
_PID	14756
_SOURCE_REALTIME_TIMESTAMP	1580363113961509
_SYSTEMD_CGROUP	/system.slice/squid.service
_SYSTEMD_SLICE	system.slice
_SYSTEMD_UNIT	squid.service
_TRANSPORT	syslog
_UID	23
__CURSOR	s=0c849b8fc7b949e385f29edb542252eb;i=2596;b=7af8bf74106241278930158abe824e53;m=aa0b89962;t=59d54f60619a2;x=f8ae88f31b4b560c
__MONOTONIC_TIMESTAMP	45646125410
__REALTIME_TIMESTAMP	1580363113961890

dnutan · January 30, 2020, 10:28am

It’s similar to these problems (check if anything from them helps you):

Anton_Viktorovich · January 30, 2020, 11:46am

Here is a config squid now, what needs to be added or corrected to get rid of the problem?

# ================= DO NOT MODIFY THIS FILE =================
# 
# Manual changes will be lost when this file is regenerated.
#
# Please read the developer's guide, which is available
# at NethServer official site: https://www.nethserver.org
#
# 

# Uncomment this to enable debug
#debug_options ALL,1 33,2 28,9

# Sites not cached
acl no_cache dstdomain "/etc/squid/acls/no_cache.acl"
no_cache deny no_cache

# Allow access from green and trusted networks.
acl localnet src 192.168.18.0/24
acl localnet_dst src 192.168.18.0/24
acl localnet src 192.168.21.0/24
acl localnet_dst src 192.168.21.0/24
acl localnet src 192.168.22.0/24
acl localnet_dst src 192.168.22.0/24
acl localnet src 192.168.23.0/24
acl localnet_dst src 192.168.23.0/24
acl localnet src 192.168.24.0/24
acl localnet_dst src 192.168.24.0/24
acl localnet src 192.168.25.0/24
acl localnet_dst src 192.168.25.0/24
acl localnet src 192.168.26.0/24
acl localnet_dst src 192.168.26.0/24
acl localnet src 192.168.27.0/24
acl localnet_dst src 192.168.27.0/24
acl localnet src 192.168.28.0/24
acl localnet_dst src 192.168.28.0/24
acl localnet src 192.168.29.0/24
acl localnet_dst src 192.168.29.0/24

# Safe ports
acl SSL_ports port 443
acl SSL_ports port 980		# httpd-admin (server-manager)
acl SSL_ports port 9090		# Cockpit Web UI
acl Safe_ports port 80		# http
acl Safe_ports port 21		# ftp
acl Safe_ports port 443		# https
acl Safe_ports port 70		# gopher
acl Safe_ports port 210		# wais
acl Safe_ports port 1025-65535	# unregistered ports
acl Safe_ports port 280		# http-mgmt
acl Safe_ports port 488		# gss-http
acl Safe_ports port 591		# filemaker
acl Safe_ports port 777		# multiling http
acl Safe_ports port 980		# httpd-admin (server-manager)
acl CONNECT method CONNECT

#
# 20acl_00_portscustom
#

# Authentication required


# GSSAPI auth in ADS mode
auth_param negotiate program /usr/lib64/squid/negotiate_kerberos_auth -i
auth_param negotiate children 10
auth_param negotiate keep_alive on

# BASIC PAM auth (fallback) 
auth_param basic program  /usr/lib64/squid/basic_pam_auth
auth_param basic children 5
auth_param basic realm ies-prikame.local
auth_param basic credentialsttl 1 hours
auth_param basic casesensitive on
acl authenticated proxy_auth REQUIRED

# exclude localhost from logging
# these lines have to go before any logging acl
access_log none to_localhost
#
# 20acl_95_localnet_log
# Make sure logs go to access.log
# Put custom logging config above this section
#
access_log daemon:/var/log/squid/access.log squid localnet


# Allow access from localhost
http_access allow localhost

# Deny requests to certain unsafe ports
http_access deny !Safe_ports

# Deny CONNECT to other than secure SSL ports
http_access deny CONNECT !SSL_ports

# Only allow cachemgr access from localhost
http_access allow localhost manager
http_access deny manager

#
# Skip URL rewriter for local addresses
#
acl self dst 192.168.23.22
acl self_port port 80
acl self_port port 443
url_rewrite_access deny self localnet  self_port


# Authentication required on green and trusted networks
http_access allow localnet authenticated


# And finally deny all other access to this proxy
http_access deny all

cache_mem 256 MB


# Leave coredumps in the first cache dir
coredump_dir /var/spool/squid

#
# Add any of your own refresh_pattern entries above these.
#
refresh_pattern ^ftp:		1440	20%	10080
refresh_pattern ^gopher:	1440	0%	1440
refresh_pattern -i (/cgi-bin/|\?) 0	0%	0
refresh_pattern .		0	20%	4320
refresh_pattern ([^.]+.|)(download|(windows|)update|).(microsoft.|)com/.*\.(cab|exe|msi|msp) 4320 100% 43200 reload-into-ims

# Always enable manual proxy
http_port 3128


acl https_proto proto https
always_direct allow https_proto
ssl_bump none localhost
sslproxy_options NO_SSLv2,NO_SSLv3,No_Compression
sslproxy_cipher ALL:!SSLv2:!ADH:!DSS:!MD5:!EXP:!DES:!PSK:!SRP:!RC4:!IDEA:!SEED:!aNULL:!eNULL
# TLS/SSL bumping definitions
acl tls_s1_connect at_step SslBump1
acl tls_s2_client_hello at_step SslBump2
acl tls_s3_server_hello at_step SslBump3
# TLS/SSL bumping steps
ssl_bump peek tls_s1_connect all
ssl_bump splice all
# peek at TLS/SSL connect data
# splice: no active bumping

#
# 45marks
#



# Enable squidGuard 
url_rewrite_program /usr/sbin/ufdbgclient -l /var/log/squid
url_rewrite_children 20 startup=5 idle=5 concurrency=0
url_rewrite_extras "%>a/%>A %un %>rm bump_mode=%ssl::bump_mode sni=\"%ssl::>sni\" referer=\"%{Referer}>h\""
icap_enable on
icap_send_client_ip on
icap_send_client_username on
icap_client_username_encode off
icap_client_username_header X-Authenticated-User
icap_preview_enable on
icap_preview_size 1024

#
# 90options
#
forward_max_tries 25
shutdown_lifetime 1 seconds
buffered_logs on
max_filedesc 16384
logfile_rotate 0
icap_service clamav_req reqmod_precache bypass=1 icap://127.0.0.1:1344/squidclamav
adaptation_access clamav_req allow all
icap_service clamav_resp respmod_precache bypass=1 icap://127.0.0.1:1344/squidclamav
adaptation_access clamav_resp allow all

dnutan · January 30, 2020, 12:22pm

At least this:

Untested…

mkdir -p /etc/e-smith/templates-custom/etc/squid/squid.conf/
cp /etc/e-smith/templates/etc/squid/squid.conf/20acl_10_auth /etc/e-smith/templates-custom/etc/squid/squid.conf/20acl_10_auth_custom

Edit custom template:

vi /etc/e-smith/templates-custom/etc/squid/squid.conf/20acl_10_auth_custom

{
   use esmith::NetworksDB;
   use NethServer::SSSD;
   
   my $ndb = esmith::NetworksDB->open_ro();
   my $green_mode = $squid{'GreenMode'} || "manual";
   my $blue_mode = $squid{'BlueMode'} || "manual";
   my $sssd = new NethServer::SSSD();
   if ($green_mode eq 'authenticated' || (defined($ndb->blue()) && $blue_mode eq 'authenticated')) {
       $OUT .= "# Custom Authentication Parameters\n";

       if ($sssd->isAD()) {
           $OUT .= "auth_param negotiate children 20\n";
       }
   }
}

signal-event nethserver-squid-update

This will create a duplicate entry for auth_param negotiate children xx. If squid takes the last one then it will be effectively overwriting the default one.

Anton_Viktorovich · January 30, 2020, 3:49pm

OK, fixed it, after checking, I will write the result.

Anton_Viktorovich · February 3, 2020, 10:29am

passed 3 days and again squid fell

Anton_Viktorovich · February 3, 2020, 12:03pm

February 3, 2020

16:27 Exiting due to repeated, frequent failures
squid

15:21Too many queued negotiateauthenticator requests
(squid-1) 11

14:21 Exiting due to repeated, frequent failures
squid

14:21Too many queued negotiateauthenticator requests
(squid-1) 6

Anton_Viktorovich · February 3, 2020, 12:05pm

dnutan · February 3, 2020, 3:11pm

How many client requests?

This message is displayed once there are more simultaneous client requests waiting to be handled than helpers available to process the load. If you configure N helpers, this warning appears when 2N+1 clients are waiting for replies. Depending on the squid version the factor may be 2 or 5 queued per helper.

You can tweak the script, replacing the number of childrens to your needs (it might increase resource usage).
For example: auth_param negotiate children 40
Or: auth_param negotiate children 100 startup=20 idle=1

Squid 3.5.19 Configuration File: auth_param
children: The maximum number of authenticator processes to spawn. If you start too few Squid will have to wait for them to process a backlog of credential verifications, slowing it down. When password verifications are done via a (slow) network you are likely to need lots of authenticator processes.

The startup= and idle= options permit some skew in the exact amount run. A minimum of startup=N will begin during startup and reconfigure. Squid will start more in groups of up to idle=N in an attempt to meet traffic needs and to keep idle=N free above those traffic needs up to the maximum.

NOTE: NTLM and Negotiate schemes do not support concurrency in the Squid code module even though some helpers can.

Anton_Viktorovich · February 3, 2020, 4:25pm

what do I need to do now to make the problem disappear?

what to prescribe?

Anton_Viktorovich · February 3, 2020, 4:26pm

about 200

dnutan · February 3, 2020, 8:21pm

Edit the custom template file you created and replace:

       $OUT .= "auth_param negotiate children 20\n";

with:

       $OUT .= "auth_param negotiate children 200 startup=40 idle=5\n";

Then apply the changes with this command:

signal-event nethserver-squid-update

Anton_Viktorovich · February 4, 2020, 2:30pm

done, I’ll keep looking. Thanks.

Anton_Viktorovich · February 7, 2020, 7:10am

everything works thanks

Fernando · February 11, 2020, 2:27pm

@Anton_Viktorovich

How many users do you have? could you give me a screenshot?

THX!