I was looking at some issues streaming some data, which led me to run speedtest-cli on my NS instance which is connected directly to my cable modem. Running this a number of times, it was averaging a download speed of around 85 Mbps, which is less than half what my ISP is supposed to provide.
Remembering an issue I had when running Zentyal, where the Traffic Shaping module was throttling my downloads, I checked the same settings on NS. This by default was enabled, but I had no rules configured. I changed the setting to disabled, but this made no difference to my download speed.
Connecting a spare laptop, directly to the cable modem, I am able to get over 200 Mbps, as promised by my ISP. So NS is definitely throttling the downloads.
Any ideas where to look/investigate further as to what’s causing this.
But I did narrow it down. It’s the shorewall configuration doing it:
[root@NethServer ~]# ./speedtest_cli.py
Retrieving speedtest.net configuration...
Retrieving speedtest.net server list...
Testing from Time Warner Cable (76.91.205.244)...
Selecting best server based on latency...
Hosted by Time Warner Cable (Los Angeles, CA) [17.74 km]: 9.372 ms
Testing download speed........................................
Download: 83.91 Mbit/s
Testing upload speed..................................................
Upload: 23.16 Mbit/s
[root@NethServer ~]# shorewall stop > /dev/null
[root@NethServer ~]# ./speedtest_cli.py
Retrieving speedtest.net configuration...
Retrieving speedtest.net server list...
Testing from Time Warner Cable (76.91.205.244)...
Selecting best server based on latency...
Hosted by Time Warner Cable (Los Angeles, CA) [17.74 km]: 9.508 ms
Testing download speed........................................
Download: 229.21 Mbit/s
Testing upload speed..................................................
Upload: 22.14 Mbit/s
[root@NethServer ~]# ./speedtest_cli.py
Retrieving speedtest.net configuration...
Retrieving speedtest.net server list...
Testing from Time Warner Cable (76.91.205.244)...
Selecting best server based on latency...
Hosted by Time Warner Cable (Los Angeles, CA) [17.74 km]: 10.578 ms
Testing download speed........................................
Download: 230.89 Mbit/s
Testing upload speed..................................................
Upload: 23.26 Mbit/s
[root@NethServer ~]# shorewall start > /dev/null
[root@NethServer ~]# ./speedtest_cli.py
Retrieving speedtest.net configuration...
Retrieving speedtest.net server list...
Testing from Time Warner Cable (76.91.205.244)...
Selecting best server based on latency...
Hosted by Time Warner Cable (Los Angeles, CA) [17.74 km]: 9.778 ms
Testing download speed........................................
Download: 80.58 Mbit/s
Testing upload speed..................................................
Upload: 23.16 Mbit/s
[root@NethServer ~]#
Look at the download speeds before stopping shorewall, while it was stopped, and then after re-starting.
Really interesting. Before asking for advice in the shorewall mailing list, would you mind trying to configure traffic shaping?
That means adding a rule to red interface with “reasonable” values (as you noticed, setting shaping to enabled without rules is like having it disabled).
In your link, I’d set uplink to 22000 and down to 240000.
My setup to use that 2nd wan hasn’t been completed yet, so there is nothing in the firewall to mark packets to be routed via it.
[root@NethServer ~]# ./speedtest_cli.py --source=76.91.205.244
Retrieving speedtest.net configuration...
Retrieving speedtest.net server list...
Testing from Time Warner Cable (76.91.205.244)...
Selecting best server based on latency...
Hosted by WebNX (Los Angeles, CA) [17.74 km]: 17.858 ms
Testing download speed........................................
Download: 82.97 Mbit/s
Testing upload speed..................................................
Upload: 23.12 Mbit/s
[root@NethServer ~]# ./speedtest_cli.py --source=10.10.0.198
Retrieving speedtest.net configuration...
Retrieving speedtest.net server list...
Testing from LogicWeb Inc (64.64.123.7)...
Selecting best server based on latency...
Hosted by SoftLayer Technologies, Inc. (London) [0.89 km]: 164.705 ms
Testing download speed........................................
Download: 28.66 Mbit/s
Testing upload speed..................................................
Upload: 7.21 Mbit/s
[root@NethServer ~]#
Based on the server chosen by Speedtest, you can see I am always using my ethernet/cable modem.
OK, solved. Well, maybe not solved, but at least I now know the culprit.
Filippo, it was your question on the Shorewall mailing list from October 2014 about NFQUEUE that caught my eye. I’d been going through all the Shorewall configuration files looking to see exactly what was set up. I didn’t know what the action.NFQBY file was doing, so Googled it. That led me to your post, where you mentioned that it was for a snort/suricata setup.
So, as a test I turned off IPS via the GUI. Guess what, 200+Mbps again.
Just to make sure this wasn’t a fluke, I have now stopped/re-started IPS a number of times with consistent results. IPS on, 85Mbps. IPS off, 200Mbps.
Looks like some interaction between the 2 is doing this, because it’s only when both are turned on that I’m seeing this. Stopping either Shorewall or IPS restores my download speeds.
BTW. Looking at the Services tab on Dashboard, how do you have a service with a status of “Disabled” be in a “Running” state. I now have 2: snortd and xl2tpd.
I should have thought of that earlier, sorry. I was bitten by the same problem, using a weak cpu. I did some benchmarks, but I don’t have exact figures, too many variables at play.
snort is really resource intensive, it needs cpu power in proportion to the number of enabled rules.
Since I run a custom setup with more 20 thousand rules, I know I need a powerful cpu.
May I know what cpu and what snort profile you were using?
I’m not sure that Speedtest-cli is the best utility to run a real-world test, so I set my NewsReader on an internal WinBlows machine downloading a couple of large posts. Yes, some of us really do still use UseNet.
Here are a few graphs of the results. I think it’s fairly obvious which is the run with IPS and without. The NewsReader showed an average download speed of around 50 - 55Mbps with IPS and over 100Mbps peaking at around 130Mbps without.
@EddieA, could you please carry out some more tests to find the culprit?
I’d like you to enable IPS on the server-manager choosing whatever policy you want, then stop snort via cmdline with:
service snortd stop
Then, please, repeat the speed test.
If speed goes back to normal, we could be pretty sure that snort is throttling network performance.
From there, I’d try to find a snort option to improve performance.
That means restarting snort with different options and repeating the speedtest every time.
The option I hope will make a difference is: