NS Throttling Internet Connection

EddieA · March 3, 2016, 8:51am

I was looking at some issues streaming some data, which led me to run speedtest-cli on my NS instance which is connected directly to my cable modem. Running this a number of times, it was averaging a download speed of around 85 Mbps, which is less than half what my ISP is supposed to provide.

Remembering an issue I had when running Zentyal, where the Traffic Shaping module was throttling my downloads, I checked the same settings on NS. This by default was enabled, but I had no rules configured. I changed the setting to disabled, but this made no difference to my download speed.

Connecting a spare laptop, directly to the cable modem, I am able to get over 200 Mbps, as promised by my ISP. So NS is definitely throttling the downloads.

Any ideas where to look/investigate further as to what’s causing this.

Cheers.

Nas · March 3, 2016, 8:56am

Hi GRO i suppose.

EddieA · March 3, 2016, 9:26am

Naw, I tried that. No difference.

But I did narrow it down. It’s the shorewall configuration doing it:

[root@NethServer ~]# ./speedtest_cli.py
Retrieving speedtest.net configuration...
Retrieving speedtest.net server list...
Testing from Time Warner Cable (76.91.205.244)...
Selecting best server based on latency...
Hosted by Time Warner Cable (Los Angeles, CA) [17.74 km]: 9.372 ms
Testing download speed........................................
Download: 83.91 Mbit/s
Testing upload speed..................................................
Upload: 23.16 Mbit/s
[root@NethServer ~]# shorewall stop > /dev/null
[root@NethServer ~]# ./speedtest_cli.py
Retrieving speedtest.net configuration...
Retrieving speedtest.net server list...
Testing from Time Warner Cable (76.91.205.244)...
Selecting best server based on latency...
Hosted by Time Warner Cable (Los Angeles, CA) [17.74 km]: 9.508 ms
Testing download speed........................................
Download: 229.21 Mbit/s
Testing upload speed..................................................
Upload: 22.14 Mbit/s
[root@NethServer ~]# ./speedtest_cli.py
Retrieving speedtest.net configuration...
Retrieving speedtest.net server list...
Testing from Time Warner Cable (76.91.205.244)...
Selecting best server based on latency...
Hosted by Time Warner Cable (Los Angeles, CA) [17.74 km]: 10.578 ms
Testing download speed........................................
Download: 230.89 Mbit/s
Testing upload speed..................................................
Upload: 23.26 Mbit/s
[root@NethServer ~]# shorewall start > /dev/null
[root@NethServer ~]# ./speedtest_cli.py
Retrieving speedtest.net configuration...
Retrieving speedtest.net server list...
Testing from Time Warner Cable (76.91.205.244)...
Selecting best server based on latency...
Hosted by Time Warner Cable (Los Angeles, CA) [17.74 km]: 9.778 ms
Testing download speed........................................
Download: 80.58 Mbit/s
Testing upload speed..................................................
Upload: 23.16 Mbit/s
[root@NethServer ~]#

Look at the download speeds before stopping shorewall, while it was stopped, and then after re-starting.

Cheers.

filippo_carletti · March 3, 2016, 11:27am

Really interesting. Before asking for advice in the shorewall mailing list, would you mind trying to configure traffic shaping?
That means adding a rule to red interface with “reasonable” values (as you noticed, setting shaping to enabled without rules is like having it disabled).
In your link, I’d set uplink to 22000 and down to 240000.

EddieA · March 3, 2016, 3:46pm

@filippo_carletti
That didn’t have any effect. Still around 85 Mbps

Cheers.

filippo_carletti · March 3, 2016, 6:03pm

Ok, let’s try to collect data for shorewall support:
http://shorewall.net/support.htm

I’d start with:

shorewall dump > /tmp/shorewall_dump.txt

Then, please send me the dump.

EddieA · March 3, 2016, 6:38pm

Already reported on the Shorewall mailing list.

You can chime in if necessary.

Cheers.

filippo_carletti · March 3, 2016, 7:08pm

I see you have two wan, one over openvpn.
Could you try the speedtest forcing the connection via one connection, please?

./speedtest_cli.py --source=76.91.205.244

You could also try with 10.10.0.198.

EddieA · March 3, 2016, 7:31pm

My setup to use that 2nd wan hasn’t been completed yet, so there is nothing in the firewall to mark packets to be routed via it.

[root@NethServer ~]# ./speedtest_cli.py --source=76.91.205.244
Retrieving speedtest.net configuration...
Retrieving speedtest.net server list...
Testing from Time Warner Cable (76.91.205.244)...
Selecting best server based on latency...
Hosted by WebNX (Los Angeles, CA) [17.74 km]: 17.858 ms
Testing download speed........................................
Download: 82.97 Mbit/s
Testing upload speed..................................................
Upload: 23.12 Mbit/s
[root@NethServer ~]# ./speedtest_cli.py --source=10.10.0.198
Retrieving speedtest.net configuration...
Retrieving speedtest.net server list...
Testing from LogicWeb Inc (64.64.123.7)...
Selecting best server based on latency...
Hosted by SoftLayer Technologies, Inc. (London) [0.89 km]: 164.705 ms
Testing download speed........................................
Download: 28.66 Mbit/s
Testing upload speed..................................................
Upload: 7.21 Mbit/s
[root@NethServer ~]#

Based on the server chosen by Speedtest, you can see I am always using my ethernet/cable modem.

Cheers.

EddieA · March 5, 2016, 10:14pm

OK, solved. Well, maybe not solved, but at least I now know the culprit.

Filippo, it was your question on the Shorewall mailing list from October 2014 about NFQUEUE that caught my eye. I’d been going through all the Shorewall configuration files looking to see exactly what was set up. I didn’t know what the action.NFQBY file was doing, so Googled it. That led me to your post, where you mentioned that it was for a snort/suricata setup.

So, as a test I turned off IPS via the GUI. Guess what, 200+Mbps again.

Just to make sure this wasn’t a fluke, I have now stopped/re-started IPS a number of times with consistent results. IPS on, 85Mbps. IPS off, 200Mbps.

Looks like some interaction between the 2 is doing this, because it’s only when both are turned on that I’m seeing this. Stopping either Shorewall or IPS restores my download speeds.

BTW. Looking at the Services tab on Dashboard, how do you have a service with a status of “Disabled” be in a “Running” state. I now have 2: snortd and xl2tpd.

Cheers.

filippo_carletti · March 6, 2016, 6:33pm

I should have thought of that earlier, sorry. I was bitten by the same problem, using a weak cpu. I did some benchmarks, but I don’t have exact figures, too many variables at play.
snort is really resource intensive, it needs cpu power in proportion to the number of enabled rules.
Since I run a custom setup with more 20 thousand rules, I know I need a powerful cpu.

May I know what cpu and what snort profile you were using?

EddieA · March 6, 2016, 10:17pm

It’s a Dual-Core Celeron @ 2.7GHz with 16GB of memory.

Balanced profile.

Cheers.

Nas · March 6, 2016, 10:26pm

Please make some tests again and share CPU usage percentage and Memory consumption.
Thank you in advance.

EddieA · March 7, 2016, 11:38pm

I’m not sure that Speedtest-cli is the best utility to run a real-world test, so I set my NewsReader on an internal WinBlows machine downloading a couple of large posts. Yes, some of us really do still use UseNet.

Here are a few graphs of the results. I think it’s fairly obvious which is the run with IPS and without. The NewsReader showed an average download speed of around 50 - 55Mbps with IPS and over 100Mbps peaking at around 130Mbps without.

Cheers.

Nas · March 7, 2016, 11:54pm

You can try wget -o /dev/null its more preferable then speedtest

EddieA · March 7, 2016, 11:56pm

That wasn’t speedtest. It was a Usenet NewsReader pulling down a large, binary, article.

Cheers.

alefattorini · March 8, 2016, 8:21am

That’s cool, how did you do that?

EddieA · March 8, 2016, 5:46pm

Do what ??

Cheers.

filippo_carletti · March 14, 2016, 1:40pm

@EddieA, could you please carry out some more tests to find the culprit?
I’d like you to enable IPS on the server-manager choosing whatever policy you want, then stop snort via cmdline with:

service snortd stop

Then, please, repeat the speed test.
If speed goes back to normal, we could be pretty sure that snort is throttling network performance.
From there, I’d try to find a snort option to improve performance.
That means restarting snort with different options and repeating the speedtest every time.
The option I hope will make a difference is:

config detection: search-method lowmem

in /etc/snort/snort.conf.
You could try to change lowmem with ac, ac-q, ac-bnfa, ac-bnfa-q, ac-split (see http://manual-snort-org.s3-website-us-east-1.amazonaws.com/node16.html for all options).
To sum it up:

service snortd stop
change snort.conf
service snortd start
speedtest
repeat from the beginning

Thank you very much.

EddieA · March 14, 2016, 7:50pm

I think we can safely blame snort:

[root@NethServer ~]# ./speedtest_cli.py --simple
Ping: 11.433 ms
Download: 84.87 Mbit/s
Upload: 21.07 Mbit/s
[root@NethServer ~]# service snortd stop
Stopping snort:                                            [  OK  ]
[root@NethServer ~]# ./speedtest_cli.py --simple
Ping: 9.463 ms
Download: 228.27 Mbit/s
Upload: 21.87 Mbit/s
[root@NethServer ~]#

search-method ac

[root@NethServer ~]# ./speedtest_cli.py --simple
Ping: 11.069 ms
Download: 78.12 Mbit/s
Upload: 23.09 Mbit/s

search-method ac-q

[root@NethServer ~]# ./speedtest_cli.py --simple
Ping: 9.303 ms
Download: 85.52 Mbit/s
Upload: 23.38 Mbit/s

search-method ac-bnfa

[root@NethServer ~]# ./speedtest_cli.py --simple
Ping: 9.61 ms
Download: 86.33 Mbit/s
Upload: 22.69 Mbit/s

search-method ac-bnfa-q

[root@NethServer ~]# ./speedtest_cli.py --simple
Ping: 12.355 ms
Download: 90.75 Mbit/s
Upload: 23.16 Mbit/s

search-method ac-split

[root@NethServer ~]# ./speedtest_cli.py --simple
Ping: 9.508 ms
Download: 91.87 Mbit/s
Upload: 22.90 Mbit/s

Not a lot of difference with any.

Cheers.