Sendmsg: No buffer space available

Support
1

I have Neth 7.4 (with what should be all the most-current updates) running in a Contabo VPS, and have had it there for about a month. For most of that time, it’s been fine. But in the last week or so, it’s lost network connectivity three times–I can’t SSH to it or ping it. If I VNC into the console, I’m able to log in, but I can’t make any outbound network connections. When I try to ping, for example, I get this:

image
image.png1090×710 58.3 KB

Rebooting resolves the issue, but (in the last case) only for about 24 hours.

Edit: Nothing relevant in /var/log/messages that I can see–the most recent incident started at 22:35 tonight:

Feb 2 22:33:54 neth openvpn: [pfsense.familybrown.org] Inactivity timeout (--ping-restart), restarting
Feb 2 22:33:54 neth openvpn: SIGUSR1[soft,ping-restart] received, process restarting 
Feb 2 22:33:59 neth openvpn: TCP/UDP: Preserving recently used remote address: [AF_INET]96.68.219.29:1194 
Feb 2 22:33:59 neth openvpn: UDP link local: (not bound) 
Feb 2 22:33:59 neth openvpn: UDP link remote: [AF_INET]96.68.219.29:1194 
Feb 2 22:34:59 neth openvpn: [UNDEF] Inactivity timeout (--ping-restart), restarting 
Feb 2 22:34:59 neth openvpn: SIGUSR1[soft,ping-restart] received, process restarting 
Feb 2 22:35:44 neth openvpn: RESOLVE: Cannot resolve host address: pfSense.familybrown.org:1194 (Name or service not known) 
Feb 2 22:36:24 neth openvpn: RESOLVE: Cannot resolve host address: pfSense.familybrown.org:1194 (Name or service not known) 
Feb 2 22:36:24 neth openvpn: Could not determine IPv4/IPv6 protocol 
Feb 2 22:36:24 neth openvpn: SIGUSR1[soft,init_instance] received, process restarting

Edit: I can confirm that I have all the latest updates (yum update reports that none are available). Here’s the kernel version I’m running:

[root@neth ~]# uname -a
Linux neth.familybrown.org 3.10.0-693.17.1.el7.x86_64 #1 SMP Thu Jan 25 20:13:58 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@neth ~]#

Google finds a number of results, but none of them very recent or dealing with CentOS 7. This thread deals with CentOS 5, and suggests adjusting the sysctls for net.ipv4.neigh.default.gc_thresh1, thresh2, and thresh3 to 1024, 2048, and 4096, respectively. The current values are 128, 512, and 1024. I don’t suppose it’d hurt to make those changes temporarily…

2

Made those changes, no effect. It started again this morning at 0340 after about 28 hours’ uptime.

Following a suggestion made in the thread I linked above, I ran an strace on the ping command for a few seconds; the output is below:

execve("/bin/ping", ["ping", "8.8.8.8"], [/* 21 vars */]) = 0
brk(NULL)                               = 0x55f2b7770000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7faba9698000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=33219, ...}) = 0
mmap(NULL, 33219, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7faba968f000
close(3)                                = 0
open("/lib64/libcap.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 \26\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=20032, ...}) = 0
mmap(NULL, 2114112, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7faba9273000
mprotect(0x7faba9277000, 2093056, PROT_NONE) = 0
mmap(0x7faba9476000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7faba9476000
close(3)                                = 0
open("/lib64/libidn.so.11", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0000\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=208920, ...}) = 0
mmap(NULL, 2302416, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7faba9040000
mprotect(0x7faba9072000, 2093056, PROT_NONE) = 0
mmap(0x7faba9271000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x31000) = 0x7faba9271000
close(3)                                = 0
open("/lib64/libcrypto.so.10", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\317\6\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=2512448, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7faba968e000
mmap(NULL, 4588360, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7faba8bdf000
mprotect(0x7faba8e13000, 2097152, PROT_NONE) = 0
mmap(0x7faba9013000, 167936, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x234000) = 0x7faba9013000
mmap(0x7faba903c000, 13128, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7faba903c000
close(3)                                = 0
open("/lib64/libresolv.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\3209\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=111080, ...}) = 0
mmap(NULL, 2202264, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7faba89c5000
mprotect(0x7faba89db000, 2097152, PROT_NONE) = 0
mmap(0x7faba8bdb000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16000) = 0x7faba8bdb000
mmap(0x7faba8bdd000, 6808, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7faba8bdd000
close(3)                                = 0
open("/lib64/libm.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0pS\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1139680, ...}) = 0
mmap(NULL, 3150136, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7faba86c3000
mprotect(0x7faba87c4000, 2093056, PROT_NONE) = 0
mmap(0x7faba89c3000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x100000) = 0x7faba89c3000
close(3)                                = 0
open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\35\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=2127336, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7faba968d000
mmap(NULL, 3940800, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7faba8300000
mprotect(0x7faba84b8000, 2097152, PROT_NONE) = 0
mmap(0x7faba86b8000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1b8000) = 0x7faba86b8000
mmap(0x7faba86be000, 16832, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7faba86be000
close(3)                                = 0
open("/lib64/libattr.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320\23\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=19888, ...}) = 0
mmap(NULL, 2113904, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7faba80fb000
mprotect(0x7faba80ff000, 2093056, PROT_NONE) = 0
mmap(0x7faba82fe000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7faba82fe000
close(3)                                = 0
open("/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`\16\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=19776, ...}) = 0
mmap(NULL, 2109744, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7faba7ef7000
mprotect(0x7faba7ef9000, 2097152, PROT_NONE) = 0
mmap(0x7faba80f9000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7faba80f9000
close(3)                                = 0
open("/lib64/libz.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p!\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=90664, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7faba968c000
mmap(NULL, 2183688, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7faba7ce1000
mprotect(0x7faba7cf6000, 2093056, PROT_NONE) = 0
mmap(0x7faba7ef5000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x14000) = 0x7faba7ef5000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7faba968b000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7faba9689000
arch_prctl(ARCH_SET_FS, 0x7faba9689740) = 0
mprotect(0x7faba86b8000, 16384, PROT_READ) = 0
mprotect(0x7faba7ef5000, 4096, PROT_READ) = 0
mprotect(0x7faba80f9000, 4096, PROT_READ) = 0
mprotect(0x7faba82fe000, 4096, PROT_READ) = 0
mprotect(0x7faba89c3000, 4096, PROT_READ) = 0
mprotect(0x7faba8bdb000, 4096, PROT_READ) = 0
mprotect(0x7faba9013000, 114688, PROT_READ) = 0
mprotect(0x7faba9271000, 4096, PROT_READ) = 0
mprotect(0x7faba9476000, 4096, PROT_READ) = 0
mprotect(0x55f2b6eb1000, 4096, PROT_READ) = 0
mprotect(0x7faba9699000, 4096, PROT_READ) = 0
munmap(0x7faba968f000, 33219)           = 0
brk(NULL)                               = 0x55f2b7770000
brk(0x55f2b7791000)                     = 0x55f2b7791000
open("/etc/pki/tls/legacy-settings", O_RDONLY) = -1 ENOENT (No such file or directory)
access("/etc/system-fips", F_OK)        = -1 ENOENT (No such file or directory)
capget({_LINUX_CAPABILITY_VERSION_3, 0}, NULL) = 0
capget({_LINUX_CAPABILITY_VERSION_3, 0}, {1<<CAP_CHOWN|1<<CAP_DAC_OVERRIDE|1<<CAP_DAC_READ_SEARCH|1<<CAP_FOWNER|1<<CAP_FSETID|1<<CAP_KILL|1<<CAP_SETGID|1<<CAP_SETUID|1<<CAP_SETPCAP|1<<CAP_LINUX_IMMUTABLE|1<<CAP_NET_BIND_SERVICE|1<<CAP_NET_BROADCAST|1<<CAP_NET_ADMIN|1<<CAP_NET_RAW|1<<CAP_IPC_LOCK|1<<CAP_IPC_OWNER|1<<CAP_SYS_MODULE|1<<CAP_SYS_RAWIO|1<<CAP_SYS_CHROOT|1<<CAP_SYS_PTRACE|1<<CAP_SYS_PACCT|1<<CAP_SYS_ADMIN|1<<CAP_SYS_BOOT|1<<CAP_SYS_NICE|1<<CAP_SYS_RESOURCE|1<<CAP_SYS_TIME|1<<CAP_SYS_TTY_CONFIG|1<<CAP_MKNOD|1<<CAP_LEASE|1<<CAP_AUDIT_WRITE|1<<CAP_AUDIT_CONTROL|1<<CAP_SETFCAP|1<<CAP_MAC_OVERRIDE|1<<CAP_MAC_ADMIN|1<<CAP_SYSLOG|1<<CAP_WAKE_ALARM|1<<CAP_BLOCK_SUSPEND, 1<<CAP_CHOWN|1<<CAP_DAC_OVERRIDE|1<<CAP_DAC_READ_SEARCH|1<<CAP_FOWNER|1<<CAP_FSETID|1<<CAP_KILL|1<<CAP_SETGID|1<<CAP_SETUID|1<<CAP_SETPCAP|1<<CAP_LINUX_IMMUTABLE|1<<CAP_NET_BIND_SERVICE|1<<CAP_NET_BROADCAST|1<<CAP_NET_ADMIN|1<<CAP_NET_RAW|1<<CAP_IPC_LOCK|1<<CAP_IPC_OWNER|1<<CAP_SYS_MODULE|1<<CAP_SYS_RAWIO|1<<CAP_SYS_CHROOT|1<<CAP_SYS_PTRACE|1<<CAP_SYS_PACCT|1<<CAP_SYS_ADMIN|1<<CAP_SYS_BOOT|1<<CAP_SYS_NICE|1<<CAP_SYS_RESOURCE|1<<CAP_SYS_TIME|1<<CAP_SYS_TTY_CONFIG|1<<CAP_MKNOD|1<<CAP_LEASE|1<<CAP_AUDIT_WRITE|1<<CAP_AUDIT_CONTROL|1<<CAP_SETFCAP|1<<CAP_MAC_OVERRIDE|1<<CAP_MAC_ADMIN|1<<CAP_SYSLOG|1<<CAP_WAKE_ALARM|1<<CAP_BLOCK_SUSPEND, 0}) = 0
capget({_LINUX_CAPABILITY_VERSION_3, 0}, NULL) = 0
capset({_LINUX_CAPABILITY_VERSION_3, 0}, {0, 1<<CAP_NET_ADMIN|1<<CAP_NET_RAW, 0}) = 0
prctl(PR_SET_KEEPCAPS, 1)               = 0
getuid()                                = 0
setuid(0)                               = 0
prctl(PR_SET_KEEPCAPS, 0)               = 0
getuid()                                = 0
geteuid()                               = 0
open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=106070960, ...}) = 0
mmap(NULL, 106070960, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7faba17b8000
close(3)                                = 0
capget({_LINUX_CAPABILITY_VERSION_3, 0}, NULL) = 0
capget({_LINUX_CAPABILITY_VERSION_3, 0}, {0, 1<<CAP_NET_ADMIN|1<<CAP_NET_RAW, 0}) = 0
capset({_LINUX_CAPABILITY_VERSION_3, 0}, {1<<CAP_NET_RAW, 1<<CAP_NET_ADMIN|1<<CAP_NET_RAW, 0}) = 0
socket(AF_INET, SOCK_DGRAM, IPPROTO_ICMP) = -1 EACCES (Permission denied)
socket(AF_INET, SOCK_RAW, IPPROTO_ICMP) = 3
capget({_LINUX_CAPABILITY_VERSION_3, 0}, NULL) = 0
capget({_LINUX_CAPABILITY_VERSION_3, 0}, {1<<CAP_NET_RAW, 1<<CAP_NET_ADMIN|1<<CAP_NET_RAW, 0}) = 0
capset({_LINUX_CAPABILITY_VERSION_3, 0}, {0, 1<<CAP_NET_ADMIN|1<<CAP_NET_RAW, 0}) = 0
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 4
connect(4, {sa_family=AF_INET, sin_port=htons(1025), sin_addr=inet_addr("8.8.8.8")}, 16) = 0
getsockname(4, {sa_family=AF_INET, sin_port=htons(47044), sin_addr=inet_addr("173.249.19.149")}, [16]) = 0
close(4)                                = 0
setsockopt(3, SOL_RAW, ICMP_FILTER, ~(1<<ICMP_ECHOREPLY|1<<ICMP_DEST_UNREACH|1<<ICMP_SOURCE_QUENCH|1<<ICMP_REDIRECT|1<<ICMP_TIME_EXCEEDED|1<<ICMP_PARAMETERPROB), 4) = 0
setsockopt(3, SOL_IP, IP_RECVERR, [1], 4) = 0
setsockopt(3, SOL_SOCKET, SO_SNDBUF, [324], 4) = 0
setsockopt(3, SOL_SOCKET, SO_RCVBUF, [65536], 4) = 0
getsockopt(3, SOL_SOCKET, SO_RCVBUF, [131072], [4]) = 0
fstat(1, {st_mode=S_IFREG|0644, st_size=9916, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7faba9697000
setsockopt(3, SOL_SOCKET, SO_TIMESTAMP, [1], 4) = 0
setsockopt(3, SOL_SOCKET, SO_SNDTIMEO, "\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 16) = 0
setsockopt(3, SOL_SOCKET, SO_RCVTIMEO, "\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 16) = 0
getpid()                                = 2926
rt_sigaction(SIGINT, {0x55f2b6ca8dd0, [], SA_RESTORER|SA_INTERRUPT, 0x7faba8335270}, NULL, 8) = 0
rt_sigaction(SIGALRM, {0x55f2b6ca8dd0, [], SA_RESTORER|SA_INTERRUPT, 0x7faba8335270}, NULL, 8) = 0
rt_sigaction(SIGQUIT, {0x55f2b6ca8dc0, [], SA_RESTORER|SA_INTERRUPT, 0x7faba8335270}, NULL, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
ioctl(1, TCGETS, 0x7ffc073831f0)        = -1 ENOTTY (Inappropriate ioctl for device)
sendto(3, "\10\0.T\vn\0\1\236\365vZ\0\0\0\0\335\31\r\0\0\0\0\0\20\21\22\23\24\25\26\27"..., 64, 0, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("8.8.8.8")}, 16) = -1 ENOBUFS (No buffer space available)
poll([{fd=3, events=POLLIN|POLLERR}], 1, 500) = 0 (Timeout)
poll([{fd=3, events=POLLIN|POLLERR}], 1, 498) = 0 (Timeout)
sendto(3, "\10\0\343T\vn\0\1\237\365vZ\0\0\0\0'\31\r\0\0\0\0\0\20\21\22\23\24\25\26\27"..., 64, 0, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("8.8.8.8")}, 16) = -1 ENOBUFS (No buffer space available)
poll([{fd=3, events=POLLIN|POLLERR}], 1, 500) = 0 (Timeout)
poll([{fd=3, events=POLLIN|POLLERR}], 1, 499) = 0 (Timeout)
sendto(3, "\10\0\300M\vn\0\1\240\365vZ\0\0\0\0I \r\0\0\0\0\0\20\21\22\23\24\25\26\27"..., 64, 0, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("8.8.8.8")}, 16) = -1 ENOBUFS (No buffer space available)
poll([{fd=3, events=POLLIN|POLLERR}], 1, 500) = 0 (Timeout)
poll([{fd=3, events=POLLIN|POLLERR}], 1, 499) = 0 (Timeout)
sendto(3, "\10\0\234J\vn\0\1\241\365vZ\0\0\0\0l#\r\0\0\0\0\0\20\21\22\23\24\25\26\27"..., 64, 0, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("8.8.8.8")}, 16) = -1 ENOBUFS (No buffer space available)
poll([{fd=3, events=POLLIN|POLLERR}], 1, 500) = 0 (Timeout)
poll([{fd=3, events=POLLIN|POLLERR}], 1, 499) = 0 (Timeout)
sendto(3, "\10\0\275G\vn\0\1\242\365vZ\0\0\0\0J&\r\0\0\0\0\0\20\21\22\23\24\25\26\27"..., 64, 0, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("8.8.8.8")}, 16) = -1 ENOBUFS (No buffer space available)
poll([{fd=3, events=POLLIN|POLLERR}], 1, 500) = 0 (Timeout)
poll([{fd=3, events=POLLIN|POLLERR}], 1, 499) = 0 (Timeout)
sendto(3, "\10\0\344D\vn\0\1\243\365vZ\0\0\0\0\")\r\0\0\0\0\0\20\21\22\23\24\25\26\27"..., 64, 0, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("8.8.8.8")}, 16) = -1 ENOBUFS (No buffer space available)
poll([{fd=3, events=POLLIN|POLLERR}], 1, 500) = 0 (Timeout)
poll([{fd=3, events=POLLIN|POLLERR}], 1, 498) = 0 (Timeout)
sendto(3, "\10\0\340D\vn\0\1\244\365vZ\0\0\0\0%)\r\0\0\0\0\0\20\21\22\23\24\25\26\27"..., 64, 0, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("8.8.8.8")}, 16) = -1 ENOBUFS (No buffer space available)
poll([{fd=3, events=POLLIN|POLLERR}], 1, 500) = 0 (Timeout)
poll([{fd=3, events=POLLIN|POLLERR}], 1, 499) = 0 (Timeout)
sendto(3, "\10\0\10B\vn\0\1\245\365vZ\0\0\0\0\374+\r\0\0\0\0\0\20\21\22\23\24\25\26\27"..., 64, 0, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("8.8.8.8")}, 16) = -1 ENOBUFS (No buffer space available)
poll([{fd=3, events=POLLIN|POLLERR}], 1, 500) = 0 (Timeout)
poll([{fd=3, events=POLLIN|POLLERR}], 1, 499) = 0 (Timeout)
sendto(3, "\10\0N?\vn\0\1\246\365vZ\0\0\0\0\265.\r\0\0\0\0\0\20\21\22\23\24\25\26\27"..., 64, 0, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("8.8.8.8")}, 16) = -1 ENOBUFS (No buffer space available)
poll([{fd=3, events=POLLIN|POLLERR}], 1, 500) = 0 (Timeout)
poll([{fd=3, events=POLLIN|POLLERR}], 1, 498) = 0 (Timeout)
sendto(3, "\10\0.@\vn\0\1\247\365vZ\0\0\0\0\324-\r\0\0\0\0\0\20\21\22\23\24\25\26\27"..., 64, 0, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("8.8.8.8")}, 16) = -1 ENOBUFS (No buffer space available)
poll([{fd=3, events=POLLIN|POLLERR}], 1, 500) = 0 (Timeout)
poll([{fd=3, events=POLLIN|POLLERR}], 1, 499) = 0 (Timeout)
sendto(3, "\10\0I=\vn\0\1\250\365vZ\0\0\0\0\2700\r\0\0\0\0\0\20\21\22\23\24\25\26\27"..., 64, 0, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("8.8.8.8")}, 16) = -1 ENOBUFS (No buffer space available)
poll([{fd=3, events=POLLIN|POLLERR}], 1, 500) = 0 (Timeout)
poll([{fd=3, events=POLLIN|POLLERR}], 1, 499) = 0 (Timeout)
sendto(3, "\10\0\256:\vn\0\1\251\365vZ\0\0\0\0R3\r\0\0\0\0\0\20\21\22\23\24\25\26\27"..., 64, 0, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("8.8.8.8")}, 16) = -1 ENOBUFS (No buffer space available)
poll([{fd=3, events=POLLIN|POLLERR}], 1, 500) = 0 (Timeout)
poll([{fd=3, events=POLLIN|POLLERR}], 1, 499) = 0 (Timeout)
sendto(3, "\10\0<8\vn\0\1\252\365vZ\0\0\0\0\3035\r\0\0\0\0\0\20\21\22\23\24\25\26\27"..., 64, 0, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("8.8.8.8")}, 16) = -1 ENOBUFS (No buffer space available)
poll([{fd=3, events=POLLIN|POLLERR}], 1, 500) = 0 (Timeout)
poll([{fd=3, events=POLLIN|POLLERR}], 1, 498) = 0 (Timeout)
sendto(3, "\10\0\0309\vn\0\1\253\365vZ\0\0\0\0\3464\r\0\0\0\0\0\20\21\22\23\24\25\26\27"..., 64, 0, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("8.8.8.8")}, 16) = -1 ENOBUFS (No buffer space available)
poll([{fd=3, events=POLLIN|POLLERR}], 1, 500) = 0 (Timeout)
poll([{fd=3, events=POLLIN|POLLERR}], 1, 499) = 0 (Timeout)
sendto(3, "\10\0N6\vn\0\1\254\365vZ\0\0\0\0\2577\r\0\0\0\0\0\20\21\22\23\24\25\26\27"..., 64, 0, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("8.8.8.8")}, 16) = -1 ENOBUFS (No buffer space available)
poll([{fd=3, events=POLLIN|POLLERR}], 1, 500) = 0 (Timeout)
poll([{fd=3, events=POLLIN|POLLERR}], 1, 499) = 0 (Timeout)
sendto(3, "\10\0\2143\vn\0\1\255\365vZ\0\0\0\0p:\r\0\0\0\0\0\20\21\22\23\24\25\26\27"..., 64, 0, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("8.8.8.8")}, 16) = -1 ENOBUFS (No buffer space available)
poll([{fd=3, events=POLLIN|POLLERR}], 1, 500) = 0 (Timeout)
poll([{fd=3, events=POLLIN|POLLERR}], 1, 498) = 0 (Timeout)
sendto(3, "\10\0<3\vn\0\1\256\365vZ\0\0\0\0\277:\r\0\0\0\0\0\20\21\22\23\24\25\26\27"..., 64, 0, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("8.8.8.8")}, 16) = -1 ENOBUFS (No buffer space available)
poll([{fd=3, events=POLLIN|POLLERR}], 1, 500) = 0 (Timeout)
poll([{fd=3, events=POLLIN|POLLERR}], 1, 499) = 0 (Timeout)
sendto(3, "\10\0D0\vn\0\1\257\365vZ\0\0\0\0\266=\r\0\0\0\0\0\20\21\22\23\24\25\26\27"..., 64, 0, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("8.8.8.8")}, 16) = -1 ENOBUFS (No buffer space available)
poll([{fd=3, events=POLLIN|POLLERR}], 1, 500) = 0 (Timeout)
poll([{fd=3, events=POLLIN|POLLERR}], 1, 498) = 0 (Timeout)
sendto(3, "\10\0\0201\vn\0\1\260\365vZ\0\0\0\0\351<\r\0\0\0\0\0\20\21\22\23\24\25\26\27"..., 64, 0, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("8.8.8.8")}, 16) = -1 ENOBUFS (No buffer space available)
poll([{fd=3, events=POLLIN|POLLERR}], 1, 500) = 0 (Timeout)
poll([{fd=3, events=POLLIN|POLLERR}], 1, 499) = 0 (Timeout)
sendto(3, "\10\0\34.\vn\0\1\261\365vZ\0\0\0\0\334?\r\0\0\0\0\0\20\21\22\23\24\25\26\27"..., 64, 0, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("8.8.8.8")}, 16) = -1 ENOBUFS (No buffer space available)
dup(2)                                  = 4
fcntl(4, F_GETFL)                       = 0x8001 (flags O_WRONLY|O_LARGEFILE)
close(4)                                = 0
open("/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 4
fstat(4, {st_mode=S_IFREG|0644, st_size=2502, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7faba9696000
read(4, "# Locale name alias data base.\n#"..., 4096) = 2502
read(4, "", 4096)                       = 0
close(4)                                = 0
munmap(0x7faba9696000, 4096)            = 0
open("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en_US.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en_US/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
write(2, "ping: sendmsg: No buffer space a"..., 41ping: sendmsg: No buffer space available
) = 41
recvmsg(3, 0x7ffc07382270, 0)           = -1 EAGAIN (Resource temporarily unavailable)
sendto(3, "\10\0(&\vn\0\2\262\365vZ\0\0\0\0\317F\r\0\0\0\0\0\20\21\22\23\24\25\26\27"..., 64, 0, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("8.8.8.8")}, 16) = -1 ENOBUFS (No buffer space available)
write(2, "ping: sendmsg: No buffer space a"..., 41ping: sendmsg: No buffer space available
) = 41
recvmsg(3, 0x7ffc07382270, 0)           = -1 EAGAIN (Resource temporarily unavailable)
sendto(3, "\10\0\220!\vn\0\3\263\365vZ\0\0\0\0fJ\r\0\0\0\0\0\20\21\22\23\24\25\26\27"..., 64, 0, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("8.8.8.8")}, 16) = -1 ENOBUFS (No buffer space available)
write(2, "ping: sendmsg: No buffer space a"..., 41ping: sendmsg: No buffer space available
) = 41
recvmsg(3, 0x7ffc07382270, 0)           = -1 EAGAIN (Resource temporarily unavailable)
sendto(3, "\10\0n \vn\0\4\264\365vZ\0\0\0\0\207J\r\0\0\0\0\0\20\21\22\23\24\25\26\27"..., 64, 0, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("8.8.8.8")}, 16) = -1 ENOBUFS (No buffer space available)
write(2, "ping: sendmsg: No buffer space a"..., 41ping: sendmsg: No buffer space available
) = 41
recvmsg(3, 0x7ffc07382270, 0)           = -1 EAGAIN (Resource temporarily unavailable)
sendto(3, "\10\0\34\30\vn\0\5\265\365vZ\0\0\0\0\330Q\r\0\0\0\0\0\20\21\22\23\24\25\26\27"..., 64, 0, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("8.8.8.8")}, 16) = -1 ENOBUFS (No buffer space available)
write(2, "ping: sendmsg: No buffer space a"..., 41ping: sendmsg: No buffer space available
) = 41
recvmsg(3, 0x7ffc07382270, 0)           = -1 EAGAIN (Resource temporarily unavailable)
sendto(3, "\10\0\242\22\vn\0\6\266\365vZ\0\0\0\0QV\r\0\0\0\0\0\20\21\22\23\24\25\26\27"..., 64, 0, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("8.8.8.8")}, 16) = -1 ENOBUFS (No buffer space available)
write(2, "ping: sendmsg: No buffer space a"..., 41ping: sendmsg: No buffer space available
) = 41
recvmsg(3, 0x7ffc07382270, 0)           = -1 EAGAIN (Resource temporarily unavailable)
sendto(3, "\10\0\276\21\vn\0\7\267\365vZ\0\0\0\0004V\r\0\0\0\0\0\20\21\22\23\24\25\26\27"..., 64, 0, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("8.8.8.8")}, 16) = -1 ENOBUFS (No buffer space available)
write(2, "ping: sendmsg: No buffer space a"..., 41ping: sendmsg: No buffer space available
) = 41
recvmsg(3, 0x7ffc07382270, 0)           = -1 EAGAIN (Resource temporarily unavailable)
sendto(3, "\10\0y\20\vn\0\10\270\365vZ\0\0\0\0xV\r\0\0\0\0\0\20\21\22\23\24\25\26\27"..., 64, 0, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("8.8.8.8")}, 16) = -1 ENOBUFS (No buffer space available)
write(2, "ping: sendmsg: No buffer space a"..., 41ping: sendmsg: No buffer space available
) = 41
recvmsg(3, 0x7ffc07382270, 0)           = -1 EAGAIN (Resource temporarily unavailable)
sendto(3, "\10\0\177\17\vn\0\t\271\365vZ\0\0\0\0qV\r\0\0\0\0\0\20\21\22\23\24\25\26\27"..., 64, 0, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("8.8.8.8")}, 16) = -1 ENOBUFS (No buffer space available)
write(2, "ping: sendmsg: No buffer space a"..., 41ping: sendmsg: No buffer space available
) = 41
recvmsg(3, 0x7ffc07382270, 0)           = -1 EAGAIN (Resource temporarily unavailable)
sendto(3, "\10\0\277\16\vn\0\n\272\365vZ\0\0\0\0000V\r\0\0\0\0\0\20\21\22\23\24\25\26\27"..., 64, 0, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("8.8.8.8")}, 16) = -1 ENOBUFS (No buffer space available)
write(2, "ping: sendmsg: No buffer space a"..., 41ping: sendmsg: No buffer space available
) = 41
recvmsg(3, 0x7ffc07382270, 0)           = -1 EAGAIN (Resource temporarily unavailable)
sendto(3, "\10\0\311\r\vn\0\v\273\365vZ\0\0\0\0%V\r\0\0\0\0\0\20\21\22\23\24\25\26\27"..., 64, 0, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("8.8.8.8")}, 16) = -1 ENOBUFS (No buffer space available)
write(2, "ping: sendmsg: No buffer space a"..., 41ping: sendmsg: No buffer space available
) = 41
recvmsg(3, PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.

strace: Process 2926 detached
 <detached ...>
--- 8.8.8.8 ping statistics ---
11 packets transmitted, 0 received, 100% packet loss, time 29015ms
3

No idea, but when it fails you may check available memory and swap space.
Also:

ss -s
netstat -s

Some Internet search suggest to increase core wmem_max value:

# check current value
cat /proc/sys/net/core/wmem_max

# example: set new value at runtime (not permanently)
echo 2129920 > /proc/sys/net/core/wmem_max

Does local ping work?
pfSense: No buffer space available list some common causes for this message.

4

Aside from these buffers, it seems that it can also be just hardware or driver related:

In that topic, defective and slowly dieing hardware was the culprit.

5

There is no swap configured on the system; the instance has 24 GB of RAM. Any time I’ve checked the dashboard, at least 18 GB has been free.

This is a VPS instance running on Proxmox, using the virtio_net driver (for the NIC). No idea of the underlying hardware, though, and it isn’t under my control.

6

I’ve rebooted into the prior kernel version (693.11) to see if that makes a difference–it seems these issues started about the time I updated to 693.17.

7

As in pinging localhost? Yes, that works.

Increased 10x (using the value suggested) with no effect.

Total: 374 (kernel 1252)
TCP:   45 (estab 2, closed 3, orphaned 0, synrecv 0, timewait 0/0), ports 0

Transport Total     IP        IPv6
*	  1252      -         -        
RAW	  1         1         0        
UDP	  13        7         6        
TCP	  42        24        18       
INET	  56        32        24       
FRAG	  0         0         0        

Ip:
    81545196 total packets received
    0 forwarded
    0 incoming packets discarded
    81539325 incoming packets delivered
    141618211 requests sent out
    1659 outgoing packets dropped
    4 reassemblies required
    2 packets reassembled ok
    2 fragments failed
Icmp:
    1881 ICMP messages received
    108 input ICMP message failed.
    ICMP input histogram:
        destination unreachable: 1765
        timeout in transit: 37
        echo requests: 74
        echo replies: 5
    1238 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
        destination unreachable: 10
        echo request: 1154
        echo replies: 74
IcmpMsg:
        InType0: 5
        InType3: 1765
        InType8: 74
        InType11: 37
        OutType0: 74
        OutType3: 10
        OutType8: 1154
Tcp:
    95225 active connections openings
    22896 passive connection openings
    83662 failed connection attempts
    6829 connection resets received
    2 connections established
    81517510 segments received
    153580851 segments send out
    802349 segments retransmited
    3987 bad segments received.
    1667 resets sent
    InCsumErrors: 3
Udp:
    2215 packets received
    8 packets to unknown port received.
    0 packet receive errors
    2687 packets sent
    0 receive buffer errors
    183 send buffer errors
UdpLite:
TcpExt:
    577169 SYN cookies sent
    438 invalid SYN cookies received
    10 resets received for embryonic SYN_RECV sockets
    25 packets pruned from receive queue because of socket buffer overrun
    1 ICMP packets dropped because they were out-of-window
    9016 TCP sockets finished time wait in fast timer
    1756 packets rejects in established connections because of timestamp
    6479330 delayed acks sent
    2905 delayed acks further delayed because of locked socket
    Quick ack mode was activated 183019 times
    9004 times the listen queue of a socket overflowed
    1052630 SYNs to LISTEN sockets dropped
    2 packets directly queued to recvmsg prequeue.
    21963759 packet headers predicted
    10996442 acknowledgments not containing data payload received
    26673067 predicted acknowledgments
    10 times recovered from packet loss due to fast retransmit
    59637 times recovered from packet loss by selective acknowledgements
    24514 bad SACK blocks received
    Detected reordering 32 times using FACK
    Detected reordering 235 times using SACK
    Detected reordering 846 times using time stamp
    1419 congestion windows fully recovered without slow start
    3050 congestion windows partially recovered using Hoe heuristic
    2439 congestion windows recovered without slow start by DSACK
    29883 congestion windows recovered without slow start after partial ack
    TCPLostRetransmit: 2297
    7 timeouts after reno fast retransmit
    10851 timeouts after SACK recovery
    57714 timeouts in loss state
    94881 fast retransmits
    11901 forward retransmits
    108226 retransmits in slow start
    261247 other TCP timeouts
    TCPLossProbes: 126933
    TCPLossProbeRecovery: 53413
    6 classic Reno fast retransmits failed
    17209 SACK retransmits failed
    564 packets collapsed in receive queue due to low socket buffer
    186402 DSACKs sent for old packets
    115 DSACKs sent for out of order packets
    41469 DSACKs received
    55 DSACKs for out of order packets received
    75 connections reset due to unexpected data
    5 connections reset due to early user close
    8122 connections aborted due to timeout
    TCPSACKDiscard: 1
    TCPDSACKIgnoredOld: 39
    TCPDSACKIgnoredNoUndo: 18840
    TCPSpuriousRTOs: 19463
    TCPSackShifted: 23853
    TCPSackMerged: 133444
    TCPSackShiftFallback: 245701
    TCPDeferAcceptDrop: 18
    IPReversePathFilter: 16
    TCPReqQFullDoCookies: 577169
    TCPRetransFail: 117949
    TCPRcvCoalesce: 4754294
    TCPOFOQueue: 692981
    TCPOFOMerge: 86
    TCPChallengeACK: 4009
    TCPSYNChallenge: 3984
    TCPSpuriousRtxHostQueues: 3247
    TCPAutoCorking: 13579
    TCPFromZeroWindowAdv: 15
    TCPToZeroWindowAdv: 15
    TCPWantZeroWindowAdv: 230
    TCPSynRetrans: 251462
    TCPOrigDataSent: 58248626
    TCPHystartTrainDetect: 30
    TCPHystartTrainCwnd: 2567
    TCPHystartDelayDetect: 511
    TCPHystartDelayCwnd: 29663
    TCPACKSkippedSynRecv: 2
    TCPACKSkippedPAWS: 335
    TCPACKSkippedSeq: 373
    TCPACKSkippedChallenge: 5
IpExt:
    InNoRoutes: 1242
    InBcastPkts: 17711
    InOctets: 61947534607
    OutOctets: 136855907941
    InBcastOctets: 6506823
    InNoECTPkts: 82234548
    InECT1Pkts: 1623
    InECT0Pkts: 24135
    InCEPkts: 2982
8

It didn’t.

Another thing that was pretty close in time to when these problems started was that I started running a Tor relay node–wonder if that would be causing this. Seems odd, but…

9

You can try and let the NS instance up and running without the tor node and see if this happens again.
Also instead of rebooting try to bring down / up the net interface and see if it is working after the up/down cycle.
Looks like something creeps up and takes all the buffer space but the memory still looks available.
Maybe the host has some issues ?..

10

That’s what I’m doing now; I stopped the tor service when I posted my last. That was 16 hours ago and no issues so far. Edit: Now 27 hours and I’m still connected. This is looking like a good sign.

Edit 2: Now 39 hours uptime.

I should have thought to try that–thanks. Though I’m hopeful that stopping tor will fix it.

2 Likes