Transparent HTTPS proxy

Feature
1

I’d like to change the behavior of the Transparent HTTPS proxy dropping the MITM (Man In The Middle) feature that inspects all the encrypted traffic, substituting it with a new implementation that sniffs only the beginning of the connection to discover the destination website (for filtering purposes).

Brief recap: now, when the Web proxy is set to Transparent with SSL, NethServer intercepts traffic on port 443 (HTTPS), connects to the secure web server impersonating the client and decrypts all traffic for inspection before sending it back to the client browser.
This feature is primarily used to discover where a user is going and filter web access. There’s usually no need to look at the whole content of the connection to filter unwanted websites.
This feature can only be used in specific environments because all client browsers need to have a special certificate installed (the biggest drawback). Personally, I will never activate this feature and I will never connect to my bank website through such a proxy.
Finally, all web browser, today, are giving really visible warnings about untrusted certificates and sometimes they block browsing if spoofing is detected.

CentOS 7.3 will soon introduce squid 3.5 which has a new function to intercept an HTTPS connection, called peek and splice.
Basically, it means that we will peek at the beginning of the connection to discover the destination website (and block it if desired) and then let the traffic flow unaltered from the client pc to the secure website.

  • No certificate to install on all browsers
  • No untrusted certificate warning
  • No sniffing on bank account :slight_smile:
  • Seamless filter of unwanted websites, both HTTP and HTTPS

The web UI will not change (only the certificate download button will be removed), the behavior will change under the hood.
Keeping the MITM features as is now plus adding the new peek and splice will need a considerable development effort. Changing the transparent MITM to peek and splice is relatively easy, I plan to release it next week if CentOS releases 7.3.

We will loose a hard-to-use feature, with a lot of limitations (i.e. you can’t install the certificate on smartphones) and that poses privacy problems in exchange for a really transparent feature.

If you think I’ve forgotten a scenario where you absolutely need to be a MITM, please speak up now (unless you’re the NSA). :slight_smile:

24 Likes
2

@filippo_carletti amazing love nethserver

3

LOL :joy:

4

That would be so FU… Great !!! can’t wait to have it !!!

1 Like
5

As long as the data transfer remains encrypted and secure (and we can filter the websites if necessary), nothing prevents us to have life easier!
:clap::trophy::clap:

1 Like
6

great! ping me when it’s time to test it :slight_smile:

7

Superb,

This will be much easier to implement and manage.
NethServer responding quickly again to upstream advances, brilliant :slight_smile:

1 Like
8

@filippo_carletti looks that this new implementation is particularly appreciated 11 likes + 5 comments :sunglasses:

1 Like
9

I like it
I want it test

10

CentOS 7.3 hasn’t been officially released, but the new packages are already available in the cr repo.
So, here it is how to test the new Transparent HTTPS proxy.

Installation (both new systems or updates):

  1. download packages
    wget http://nethservice.nethesis.it/https_url_filter.tar.gz

  2. update/install squid
    yum --enablerepo=cr install squid

  3. install ufdbGuard (the new filter)
    yum install https://www.urlfilterdb.com/files/downloads/ufdbGuard-1.32.3-CentOS7.x86_64.rpm

  4. install packages

    tar -xvf https_url_filter.tar.gz
    yum install nethserver-squid*rpm

Substitute install with update if necessary. :slight_smile:

Known limitations:

  • Transparent URL filtering on HTTPS websites can only block whole domains, because ufdbGuard can only receive the domain name, not the full URL
  • Redirected HTTPS sites show the standard NethServer page instead of the block page
5 Likes
11

Life becomes a lot easier with that feature! :clap: :clap: :clap:
Love it. Will test it, if possible, allready today. :slight_smile:

12

Hi Filippo, I followed your instuctions on a clean NS7 VM. No packages from softwarecenter installed.

installed packages after your instructions:
ufdbGuard ver 1.32.3
squid ver 3.5.20
nethserver-squid-1.4.0-1.4.g867d56b.ns7.noarch
nethserver-squidguard-1.5.1-1.8.gf5497bb.ns7.noarch

and:

But facebook is still reachable (https) and block page (http) is directed to standard squidguard block page.
So squidguard works. But https is not blocked.

What did I do wrong? :disappointed_relieved:

13

Did you set it as transparent with ssl?

14

I tried that, but then every https-connection was blocked, because of the missing certificat. I understood it that way, that no certificat is needed anymore whith this feature and that it works with simple transparent proxy. This wood make live a lot easier.

[quote=“filippo_carletti, post:1, topic:5064”]* No certificate to install on all browsers

  • No untrusted certificate warning
    [/quote]

Did I understand wrong?

1 Like
15

You have to select Transparent with SSL (see inline help for details).
You got it right, no certificate to install.
Regarding the untrusted certificate warning, you could face two scenarios:

  1. permitted site: no warning
  2. blocked site: warning of cert not matching domain

Some tips for troubleshooting:

echo "http://bit.ly 10.10.0.1/ - - GET" | /usr/sbin/ufdbgclient -d
/etc/init.d/ufdb testconfig 2>&1 | grep FATAL
less /var/ufdbguard/logs/ufdbguardd.log
16

I did a reboot and selected Transparent with SSL again. Now it works.
I did all updates before without reboot, maybe that was the problem.

I have 3 scenarios:

  1. permitted https-site : no warning (green https)
  2. forbidden https-site : certificat error common name invalid (red https)
  3. forbidden http-site : normal squidguard block site

Another advantage of this new feature is, that you don’t violate privacy of users I think. Is This correct?
And BTW all internetradios are working now on my machine now (the boss is permitted to hear radio :slight_smile:)

As far as I can say, this works great. No certificats mean no problems with other SSL connections. No more digging for IPs to whitelist. I LOVE IT!! :heart_eyes:
Thank you very much for that feature. You deserve a gold medal. :clap: :medal: :clap:

2 Likes
17

A reboot is NOT needed AFAIK.
If someone encounters the same problem I’ll investigate further.
Point 1 is the most important: banks and other security sensitive sites work well.
For 2 we can’t do anything, it’s a browser decision. I’ll investigate different options in the future, the idea is to return an error so that the browser selects a different page to notify the user.

You’re correct, the connection is encrypted end to end: no privacy concerns.

Future improvements:

  • option to block tunnels on port 443 for closed environments
  • better block page

Last remark: I’ve some concerns about default state of Block access to web sites using ip address.
The idea behind having it enabled by default was to block malware which usually calls back home using ip addresses.
But with https intercept, many legit connections are made using ip address (we can’t know the requested url, it’s encrypted),
Do you have it enabled? Do you see legit blocks in the log file? Have you found sites or web app which don’t behave as expected?

2 Likes
18

A trully great idea ! I really appreciate reading such posts with a high sense of responsability. Thanks a lot :slight_smile:

1 Like
19

Yes, I’ve enabled that. (If it’s that what you mean. :slight_smile:)
I’ll have a look on that and digg the logs tomorrow.

1 Like
20

i also install and check this incredible feature. i got a problem with the ufdb service. it is deactivated by default and not starting if i boot the server. if i start it manually it looks ok so far. any solution for that?