Hello everyone.
I’ve finalized my migration. I’ll try to summarize everything so far. These are my settings, so what I describe is just a suggestion.
It took me longer than expected due to a strange issue. I use IPTV. When starting with snort active, the TV stream would occasionally drop. After many experiments and tests on two different hardware platforms, I finally managed to reduce stream interruptions to 2-3 per day.
I’m using NethSecurity on a VM. VM OS: Proxmox. Primary VM router on a dedicated machine just for this purpose, backup VM router on my main hypervisor.
Initially, I started with Qotom Q20332G9 S20, CPU Intel(R) Atom™ CPU C3758 @ 2.20GHz (8 cores), 32GB ECC RAM. I wanted to get the C3758R, but it wasn’t available at the time. Currently testing on Beelink AZW SER8 with AMD Ryzen 7 8745HS w/ Radeon 780M Graphics (8 physical cores, 16 threads), 32GB RAM. VM for primary router - 14 cores, 8GB RAM.
Final Settings
Manual settings in /etc/config/network
Solution for ISP DHCP and MAC address issue
Set custom MAC address on network interface:
config device 'ns_NAME'
...
option macaddr 'AA:BB:CC:DD:EE:FF'
This setting prevents automatic interface activation when the backup router restarts. They are actually managed by keepalived using an external script /etc/keepalived/scripts/primary-backup.sh
Set interface to not come up after boot:
config interface 'NAME'
...
option auto '0'
The Script
#!/bin/sh
# Logging function
log_msg() {
logger -t "keepalive-script" "$1"
echo "$(date '+%Y-%m-%d %H:%M:%S') - $1"
}
# Service management function
services() {
local action="$1"
case "$action" in
start)
log_msg "Services start: starting odhcpd, dnsmasq, adblock"
/etc/init.d/odhcpd start
sleep 1
/etc/init.d/dnsmasq start
sleep 1
/etc/init.d/adblock start
;;
stop)
log_msg "Services stop: stopping adblock, dnsmasq, odhcpd"
/etc/init.d/adblock stop
sleep 1
/etc/init.d/dnsmasq stop
sleep 1
# Kill any remaining dnsmasq processes
killall dnsmasq 2>/dev/null
sleep 1
/etc/init.d/odhcpd stop
;;
esac
}
# Update resolv.conf with lock to prevent overwrites
update_resolv() {
local nameserver="$1"
log_msg "Updating /etc/resolv.conf with nameserver $nameserver"
# Remove immutable flag if exists
chattr -i /etc/resolv.conf 2>/dev/null
# Update resolv.conf
echo "nameserver $nameserver" > /etc/resolv.conf
# Make it immutable to prevent other services from changing it
chattr +i /etc/resolv.conf 2>/dev/null
# Verify the change
if grep -q "$nameserver" /etc/resolv.conf; then
log_msg "resolv.conf successfully updated and locked"
else
log_msg "ERROR: Failed to update resolv.conf"
fi
}
# Restore resolv.conf to normal state
restore_resolv() {
log_msg "Restoring resolv.conf to normal state"
chattr -i /etc/resolv.conf 2>/dev/null
log_msg "resolv.conf unlocked"
}
case "$1" in
primary)
log_msg "=== Transitioning to PRIMARY mode ==="
# Remove default route via gateway
log_msg "Removing default route via VR_IP_ADDRESS"
ip r d default via VR_IP_ADDRESS dev br0 2>/dev/null
# Restore resolv.conf (unlock it)
restore_resolv
# Bring up WAN interfaces
log_msg "Bringing up ISP1 and ISP2 interfaces"
ifup ISP1
ifup ISP2
# Start services
services start
log_msg "=== PRIMARY mode activated ==="
;;
backup)
log_msg "=== Transitioning to BACKUP mode ==="
# Bring down WAN interfaces
log_msg "Bringing down ISP1 and ISP2 interfaces"
ifdown ISP1
ifdown ISP2
# Stop services
services stop
# Add default route via gateway
log_msg "Adding default route via VR_IP_ADDRESS"
ip r a default via VR_IP_ADDRESS dev br0 2>/dev/null
# Update and lock DNS resolver
update_resolv "VR_IP_ADDRESS"
log_msg "=== BACKUP mode activated ==="
log_msg "Note: snort and banip are managed by NethSecurity HA system"
;;
*)
log_msg "ERROR: unknown state transition - received parameter: $1"
echo "Usage: $0 {primary|backup}"
exit 1
;;
esac
log_msg "Script execution completed with status: $?"
exit 0
Script Management in /etc/keepalived.user
Added the following:
state=$(grep state /tmp/keepalived.conf 2>/dev/null | awk '{print $2}')
ADMIN_EMAIL="user@domain.com"
MAIL_SUBJECT_PREFIX="$HOSTNAME [$state] keepalived"
case "$ACTION" in
NOTIFY_MASTER)
MSG="[$NAME] switched to MASTER – executing primary script"
logger -t keepalived "$MSG"
echo "$MSG" | mailx -s "$MAIL_SUBJECT_PREFIX MASTER" "$ADMIN_EMAIL"
/etc/keepalived/scripts/primary-backup.sh primary
;;
NOTIFY_BACKUP)
MSG="[$NAME] switched to BACKUP – executing backup script"
logger -t keepalived "$MSG"
echo "$MSG" | mailx -s "$MAIL_SUBJECT_PREFIX BACKUP" "$ADMIN_EMAIL"
/etc/keepalived/scripts/primary-backup.sh backup
;;
NOTIFY_FAULT)
MSG="[$NAME] entered FAULT state – forcing backup mode"
logger -t keepalived "$MSG"
echo "$MSG" | mailx -s "$MAIL_SUBJECT_PREFIX FAULT" "$ADMIN_EMAIL"
/etc/keepalived/scripts/primary-backup.sh backup
;;
NOTIFY_STOP)
MSG="[$NAME] STOP – keepalived stopped, switching to backup mode"
logger -t keepalived "$MSG"
echo "$MSG" | mailx -s "$MAIL_SUBJECT_PREFIX STOP" "$ADMIN_EMAIL"
/etc/keepalived/scripts/primary-backup.sh backup
;;
*)
# Ignore other events (e.g. NOTIFY)
;;
esac
/etc/config/nginx
Adding http2 to list listen '443 ssl', with a caveat - this method of adding http2 is deprecated. The warning: “the “listen … http2” directive is deprecated, use the “http2” directive instead in /etc/nginx/nginx.conf”
Should change from:
listen 443 ssl http2;
to:
listen 443 ssl;
http2 on;
Syntax:
...
list listen '443 ssl'
list listen '[::]:443 ssl'
option http2 'on'
...
This could really be a checkbox option for adding http2 on in the config file.
/etc/config/snort
I have several suggestions here (these are the changes I made):
option home_net - by default it’s 192.168.1.0/24, but it should be the LAN network.
In the current version 8-24.10.0-ns.1.6.0, an interface is specified, but snort is configured for nfq and listens to the forward chain. The interface parameter is actually redundant.
In version 8.7.0, snort is configured for pcap - if you have two active ISPs, how do you configure snort for two interfaces?
/etc/config/dhcp
config boot
option filename 'pxelinux.0'
option servername 'myserver'
option serveraddress '192.168.0.XX'
list dhcp_option 'option:root-path,192.168.0.XX:/'
/etc/rc.local
echo haltpoll | sudo tee /sys/devices/system/cpu/cpuidle/current_governor
printf "%x\n" $(( (1 << $(nproc)) - 1 )) | sudo tee /sys/class/net/eth0/queues/rx-*/rps_cpus
# RFS
echo 2048 | tee /sys/class/net/eth*/queues/rx-*/rps_flow_cnt
First setting: CPU governor optimization.
Second setting: This is important because all rps_cpus interrupts on the NIC are handled by a single CPU and can become saturated under heavy traffic. With this setting, I’m telling the OS to balance rps_cpus automatically.
Third setting: This sets how many network flows the kernel should remember for Receive Flow Steering (RFS) purposes - a mechanism that directs packets to the CPU core where the application is running for better performance.
/etc/sysctl.d/99-main-tweaks.conf
# Main options
net.netfilter.nf_conntrack_buckets = 524288
net.netfilter.nf_conntrack_max = 2097152
# Network buffer sizes
# 134217728 - 128MB
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
# 16777216 - 16MB
net.core.rmem_default = 16777216
net.core.wmem_default = 16777216
# Softirq budget
net.core.netdev_max_backlog = 30000
net.core.netdev_budget = 600
net.core.netdev_budget_usecs = 8000
# RFS global
net.core.rps_sock_flow_entries = 32768
# TCP tuning
net.ipv4.tcp_rmem = 4096 87380 134217728
net.ipv4.tcp_wmem = 4096 65536 134217728
#net.ipv4.tcp_congestion_control = bbr
# Maintains high throughput after brief pauses when set to 1
net.ipv4.tcp_slow_start_after_idle = 0
# Frees memory and conntrack slots faster, default is 30
net.ipv4.tcp_fin_timeout = 15
/etc/init.d/conntrackd
After: procd_set_param command $PROG -C /etc/conntrackd/conntrackd.conf
Add: procd_set_param nice -20
