Nfnetlink queue full

Deckard · February 22, 2025, 8:03pm

Hello Folks,

I have these nice kernel messages on my virtualized NethSecurity 8 23.05.5-ns.1.4.1 system:

net_ratelimit: 978 callbacks suppressed
nfnetlink_queue: nf_queue: full at 4096 entries, dropping packets(s)
nfnetlink_queue: nf_queue: full at 4096 entries, dropping packets(s)
nfnetlink_queue: nf_queue: full at 4096 entries, dropping packets(s)
[...more...]

Looking into the nfnetlink_queue:

 cat /proc/net/netfilter/nfnetlink_queue
    4   2508     0 2  1518     0     0  8701082  1
    5 2571026936     0 2  1518     0     0  7904397  1
    6 2336528095     0 2  1518     0     0  3941436  1
    7 3875946501  1625 2  1518 110076     0 31682176  1

The docs say this means:

queue_number

peer_portid: good chance it is process ID of software listening to the queue

queue_total: current number of packets waiting in the queue

copy_mode: 0 and 1 only message only provide meta data. If 2, the message provides a part of packet of size copy range.

copy_range: length of packet data to put in message

queue_dropped: number of packets dropped because queue was full

user_dropped: number of packets dropped because netlink message could not be sent to userspace. If this counter is not zero, try to increase netlink buffer size. On the application side, you will see gap in packet id if netlink message are lost.

id_sequence: packet id of last packet

The last field is always ‘1’ and is ignored.

So I’m too late to the party, right now there are no queues that have dropped packets. But there were some recently, else there wouldn’t be the kernel messages…

Okay, perhaps more relavent is: what to do about full nfnetlink queues (4096 entries is the limit it seems)? Can one increase the number of queues? Or adjust something else? Or add more horse power? I can give the NethSec VM more vCPUs, but I am not so sure the firewall will scale to parallelize with more cores.

I’d be happy to hear your ideas.

Thnx.

Andy_Wismer · February 23, 2025, 3:29am

Hi @Deckard

Before giving any feedback / ideas / tips which would be plain guessing, it might be interesting to know what Hypervisor are you using?

And, of course, what hardware does that hypervisor have… CPU, Memory and Disk-Systems (HDs?, SSDs? NVMEs?)?

One “educated” guess might be Balloon Memory, eg in Proxmox / KVM…

My 2 cents
Andy

filippo_carletti · February 23, 2025, 3:52pm

Probably, your CPU is not fast enough to deal with the total amount of packets that snort has to process.
How many rules have you enabled?
Could you bypass some of the traffic?

The CPU may be fast enough, but, as @Andy_Wismer suspects, other VMs on the same hypervisor may be randomly stealing resources.

Deckard · February 23, 2025, 8:05pm

Hi Andy and Filippo,

you guys hit the nail on the head: I had signed up to Snort and turned IDS on, and had forgotten about it.

tl;dr: CPU not fast enough for IDS. Turned IDS off, things are back to normal.

Details, as this might be of general interest:

VM: 4x vCPU + 2GB RAM, QEMU disk image on XFS filesystem on NVME storage, virtio NICs and virtio disks.

Host: RHEL 9.5 Server on an ASRock Rack X470D4U mainboard with Ryzen 5 3600 CPU (6 cores x 12 threads), 64GB DDR4 RAM, OS drive on WD SN700 NVME, ZFS mirror on 2x 16TB Toshiba Enterprise HDDs with ZFS Special Device on SN700 NVME mirror. Important to note: the KVM VMs are not on ZFS, they run directly from the NVME OS drive, only the Data is on the ZFS pool. Networking is 10Gbit optical backbone with Mellanox SFP+ NICs and Omada SFP+ (DAC and optical) switches and 1Gbit copper for the rest.

The above hardware + software is very power efficient for what it is: the RHEL Server provides local application + media server, VM hypervisor, and NAS. This is what we at my office would implement on a much bigger scale as HCI - Hyper Converged Infrstructure.

OK, to NethSecurity: this is a KVM VM on said infrastructure. WAN is 1Gbit up 500Mbit down via GPON. The NethSec VM can handle everything as long as IDS is not on, that sent it over the edge.

WAN throughput, Snort off:

WAN throughput, Snort on:

Netdata from within the NethSec VM (left Snort off, right Snort on):

Packets:

CPU:

In general, in both cases 100% CPU utilization, two CPUs with user load, the other two however with primarily soft IRQ (Snort off) but evidently the much higher system and user load with Snort running.

This is also interesting to see how the 4x vCPUs are being utilized.

Interrupts:

It would seem that OpenWRT is not really using the 4 CPUs fully.

Performance take-away: It would seems that increasing core frequency and IPC are the only way to scale performance, unless there is some other KVM host and/or VM or OpenWRT driver or configuration optimization that can be done.

I like the Ryzen 5 3600 because it just sips power and is very performant for the amount of wattage. Ofcourse it is not a hot new CPU, it is socket AM4 and “only” DDR4 (which makes it very economical). Perhaps a newer Xen 3 CPU, possibly with a high frequency, would help. However I don’t see that throwing more virtual cores at OpenWRT will bring anything.

So, all-in-all I am quite happy with NethSecurity! This is a great showing. =)

filippo_carletti · February 24, 2025, 9:29am

Every CPU handles a queue, traffic is directed to the queue based on src and dst IP address (see NFQUEUE: introduce CPU fanout [LWN.net]).
Testing with a single speedtest tends to use a queue, while a LAN with a few clients browsing the web should be distributed to all cores.

Deckard · February 24, 2025, 9:50am

Hi Filippo,

you are right in that the queue full issue is not the same thing as the throughput issue, but they are related. I discovered the throughput problem because I was looking at the cause of the queue full errors.

The issue for both (perhaps, yes?) is the lack of parallelization in OpenWRT, or that my CPU does not have a high enugh single-core compute / frequency scaling / IPC to deal with IDS and other packet inspection workloads, since all I can do is give it more vCPUs, but not make them go faster.

I would have to test explicitly again, for example starting multiple downloads / streams, if the queue full issue comes back. However, with Snort enable, not even one stream is able to reach full bandwidth, and with it off things are OK again. So I am happy to have discovered that, along with the CPU scaling behaviour of OpenWRT.

It makes sense to test again, with Snort off, with 4 or more parallel full bandwidth downloads running, to see if the queue overuns show up again, and if they also impact full bandwidth delivery or not.

I’ll report back. =)