The automatic calculation of the bandwidth with speedtest-cli lacks precision

In the multiwan user interface of cockpit we can use the speedtest-cli tools to measure the bandwidth and use it for each red NIC. This is asked by the manual when :

If the server has two or more configured red interfaces, it is required to correctly fill, Download bandwidth and Upload bandwidth fields from the Network page. Download and upload bandwidth can be automatically calculated using the Speedtest button.

I mean this UI

It seems until 500 megabits of download/upload, the speedtest tools of ookla or speedtest-cli that we bundle ourselves the rpm have similar values.

The issue issue comes when you have server with bandwidth in gigabits, speedtest-cli gets values really inferior to the speedtest tools of ookla and probably not true.

We try to bundle the right tool all the time and if a tool is wrong we can change it or also remove it

some ideas :

  • I am not sure we can use and distribute the speedtest tools of ookla due to a long license to read
  • sysadmin could use directly an external tool to measure the bandwidth and set it accordingly, hence we remove speedtest-cli…but I fear conversion errors
  • We could provide a warning banner, the remote server might not be able to answer to your 10 gigabits bandwidth, please set the correct value you think right.
  • find another tool for bandwidth
3 Likes

An alternative was mentioned in another thread but do not know if it can resolve the exposed problem.

It has cli and other versions (go, etc.)

3 Likes

I didn’t remember it, thank you for pointing it out.

@stephdl can we give it a try?

2 Likes

I’ve quickly tested it on a VPS, the result seems quite good:

{
  "timestamp": "2021-10-20T10:32:33.760174324+02:00",
  "server": {
    "name": "Alblasserdam, Netherlands (RamNode)",
    "url": "http://nl1.backend.librespeed.org"
  },
  "client": {
    "ip": "188.166.75.203",
    "hostname": "smtp-relay.nethesis.it",
    "city": "Amsterdam",
    "region": "North Holland",
    "country": "NL",
    "loc": "52.3740,4.8897",
    "org": "AS14061 DigitalOcean, LLC",
    "postal": "1012",
    "timezone": "Europe/Amsterdam"
  },
  "bytes_sent": 1795129344,
  "bytes_received": 1785109216,
  "ping": 3.6363636363636362,
  "jitter": 0.5,
  "upload": 920.42,
  "download": 914.68,
  "share": ""
}

To install and try it:

wget https://github.com/librespeed/speedtest-cli/releases/download/v1.0.9/librespeed-cli_1.0.9_linux_amd64.tar.gz
tar xvzf librespeed-cli_1.0.9_linux_amd64.tar.gz
./librespeed-cli
2 Likes

trop fort :smiley:

(really good)

[root@firewall ~]# ./librespeed-cli --source 192.168.0.3
Retrieving server list from https://librespeed.org/backend-servers/servers.php
Selecting the fastest server based on ping
Selected server: Amsterdam, Netherlands (Clouvider) [ams.speedtest.clouvider.net]
Sponsored by: Clouvider @ https://www.clouvider.co.uk/
You're testing from: 90.76.3.30 - Unknown ISP
Ping: 22 ms	Jitter: 1 ms
Download rate:	282.46 Mbps
Upload rate:	198.38 Mbps
[root@firewall ~]# ./librespeed-cli --source 192.168.0.3
Retrieving server list from https://librespeed.org/backend-servers/servers.php
Selecting the fastest server based on ping
Selected server: Strasbourg, France (Host Europe) [sxb.bandspeed.de]
Sponsored by: adminForge @ https://adminforge.de/
You're testing from: 90.76.3.30 - Orange S.A., FR (840 km)
Ping: 23 ms	Jitter: 0 ms
Download rate:	97.75 Mbps
Upload rate:	200.86 Mbps

Well a bit disappointed, we have here the same issue, the measure depends of the server you meet and the willingness of the server’sponsor to waste bytes for your only little pleasure. So in short like you can see I did some tests, most of time I went to Strasbourg and the measure is wrong, one time I went to Amsterdam and it is closest of the truth

I think I believe the issue we have is the same for speedtest-cli, maybe the free client cannot use the fast servers, this is a test I did on the same server to list the list of client. Speedtest-cli is supposed to use the speedtest.net servers like the not free speedtest tool

[root@ns7 ~]# ./speedtest --servers
Closest servers:

    ID  Name                           Location             Country
==============================================================================
 29032  FullSave                       Toulouse             France
 38494  SHPV FRANCE SAS                Toulouse             France
  2530  Andorra Telecom                Andorra La Vella     Andorra
  5161  Imatel                         Donostia-San Sebastian Spain
 37417  Inforcelra S.L.U.              PuigcerdĂ            Spain
 25731  BerguedĂ  Telecom              Guardiola de BerguedĂ  Spain
 17772  e.Telecom                      Lleida               Spain
 37356  Vunkerts IT Experts            Lleida               Spain
 29542  ORANGE FRANCE                  Bordeaux             France
 21415  LaFibre.info                   Bordeaux             France


[root@ns7 ~]# speedtest-cli --list
Retrieving speedtest.net configuration...
37356) Vunkerts IT Experts (Lleida, Spain) [185.11 km]
31869) Fibra Ă’ptica Centelles (Santa Maria d'OlĂł, Spain) [219.75 km]
29289) XTA - Xarxes de Telecomunicacions Alternatives (Vilafranca del Penedès, Spain) [248.12 km]
41844) e17 by J.C. Tècnics (Figueres, Spain) [258.08 km]
 5075) Aeris Navigo (Girona, Spain) [263.43 km]
 2254) CSUC (Barcelona, Spain) [267.46 km]
20672) apfutura (Barcelona, Spain) [267.46 km]
 1695) Adamo (Barcelona, Spain) [267.46 km]
40191) Vinga Fibra (Torroella de MontgrĂ­, Spain) [281.83 km]
24665) Fibra365 Vall-llobrega (Vall-llobrega, Spain) [291.11 km]

I am not sure the way to go, remove speedtest-cli because the measure could not be accurate, I am not sure, I would like to go to the warning, please if the measure does not satisfy your ego, use your tool :smiley:

1 Like

My question is: outside speedtest-cli and variants, which tool/task/setup can be accomplished for saturate bandwidth of ISP connection and determine the limits for BWM?

With commercial CPEs like residential/SOHO VDSL devices you can do some math on the negotiation (if is provided) but outside of that?
Fiber connections cannot achieve the same data, because negotiation if the connection works correctly should be like ethernet: the maximum capable for the media.
But for a GPON connection, which can be multiplexed from one OLT to 128 ONT… 2,5GBPS is completely theorical, because multiplexing splits the connection for all the ONT connected.
In a GPON connection with 128 ONTs connected, a theoretical top speed should be 20Mbit down/ 10Mbit up.
XG-PON, 80 mbit down, 20 up.
XGS-PON, 80 mbit down, 80 up
NG-PON2, 240 mbit down, 80 up.
Again, theoretical and with 128 ONTs active at the same time.

One way could raise a Bittorrent client, download 10 ISOs of most common Linux distros, disable bandwidth management and reupload it.
After 1 hour maybe statistics could theoretically provide some realistic value for setting the limits for BWM, but this could work only if the protocol is obfuscated, if the port is reachable, the ISP do not provide any kind of BWM and cannot detect BitTorrent.

And ISPs, guess what… They do BWM.
For voice, for video streaming, for IpTV. Italy started this year with “full internet broadcast” of soccer matches. And these are hitting quite hard on the connections, for what i can hear and read on newspapers.

Goal of BWM is… don’t stop services. First rules can be drafted and built with fewer data, but reading the reports of use and bandwidth peaks or loss for clients can help refine configuration.
Which is boring.
Because every time you have to

  1. re-evaluate status,
  2. find what can be improved
  3. write down the draft of “I am doing this for achieve that”
  4. understand how to create this in rules
  5. let the rule work for at least two weeks (unless something different happens)
  6. evaluate results
  7. go to 1 of this list.

Sometimes installing new serve could appear a bit more… appealing. :wink:

1 Like

May I try to summarize with “speedtest is almost useless”? :smiley:
I think the same! :slight_smile:

So I have a couple of proposal:

  1. totally remove the speed test from the UI
  2. keep the current speedtest-cli, remove from the UI the Use this settings button, and add a warning like “The results maybe highly inaccurate”
    Screenshot from 2021-10-22 09-24-08

For the record, speedtest-cli gives very different results even on my poor connections (20/2 Mbsp).

This will drive numerical error conversions, because you have to write them in kbits IIRC.

FWIW i vote for the 2, @giacomo. Inaccurate setting sometimes is better than no setting at all. And i stand to what @stephdl is saying about conversions. Moreover, end user products can sell easely Mbit/s because users want fast ideas.
Sysadmins (or anyone is willing to) has to stick to kbits because at least in Italy, 500kbits can be the difference between a slow and swifty setup than a fast and bump one.

It’s fine for me, so we can show results both in Mbits/s and Kbits. Somethine like 16.04Mbits (~16457 Kbits).

@davide_marini @filippo_carletti what do you think?

1 Like

In my opinion, ~16457 Kbits should be the only output, because as far as i can remember, BWM rules must be written in KBits.
And again, the disclaimer about Speedtest and accuracy should be anyway present.

1 Like

I’m trying to figure out what is the best thing to do among those that Giacomo has proposed.
The speedtest has a very wide variability of results that we cannot control (the client used, but also the servers used for the test) which leads to unreliable measurements and therefore not very useful or even harmful.

It’s supposed that everything that appears in the UI is believed to work reliably and the speedtest is no exception.
Inaccurate measurements can lead us to use less bandwidth than is actually available (from the tests we have seen deviations greater than 200 Mbps) or push us to open disputes with the provider or with the firewall’s admin because the results are not correct.

From the interface then it is not possible to know which speedtest you are using and not even which server (often chosen based on latency and not on the real bandwidth capacity) which makes things even worse.

For all these reasons I would remove the speedtest from the UI (and Giacomo knows that I’m the last person that wants to remove a feature).

Anyway the feature is not lost yet, it can be easily obtained from the character console by installing the desired client, so I would document how it can be done, in this way you get more information (type of test, server used) and the admin has greater awareness of what it is doing.

2 Likes

Thank you for your point of view Davide.

Just to clarify a bit, we pointed out this bug because we have received dozen of tickets where customers are saying something like “the firewall is slow, it cuts down my bandwidth”.
Usually is quite hard to justify the problem with something like “speedtest has a bug not the firewall”. The final user thinks that the whole firewall is buggy and not just a very little part of it. I think the user perception it’s totally fine on this one :confused:

So, while implementing option 2 is ok for me, I vote for solution 1: totally remove the speedtest.

2 Likes

So @davide_marini appears after 6 months for “announcing” that Speedtest feature is going to be deleted.

OOOOK…

Thanks for the heads up…

now that the ability to calculate the speedtest has been removed from cockpit, hopefully a quick replacement… :-/ (@davide_marini @giacomo I’m thinking about you :slight_smile: )

Sorry if I haven’t posted anything here, but all the work has been done in public: https://github.com/NethServer/dev/issues/6588.

There is no real replacement, but a bunch of alternatives for various cases:

  • speedtest-cli (already bundled in NS) is good for connections with a maximum download speed of 500Mbit/s
  • librespeed is much better than speedtest-cli for fast connections but the result heavily depends on the selected remote server
  • ookla is a closed-source and non-redistributable alternative, which is almost the de-facto standard

After selecting your preferred speedtest binary, execute as:

fireqos stop
<your_speedtest_binary>
fireqos start

I tried speedtest-cli:
How do I select the interface to test on?
I don’t seem to have seen an option that allows me to do this.