Build web filter modules for ARM

filippo_carletti · December 18, 2016, 5:29pm

hostapd comes from epel repo, so you should simply type:
yum install hostpad
unless epel for arm still doesn’t contain the rpm. We may signal hostapd maintainer.
I’ll try to rebuild my RPI after we release NethServer 7 final.

Regarding ufdbguard, we could ask to the author (Marcus), but I’m not sure he wants to maintain one more package.
You should be able to build from the srpm, I can help.

denis.robel · December 18, 2016, 5:38pm

There is a src rpm available for Centos7

wget -c https://www.urlfilterdb.com/files/downloads/ufdbGuard-1.32.4.src.rpm

but there is an old init script used not a systemd service. I was able to compile the src rpm that’s no problem.

But as i know in Centos 7 systemd is the new standard…

For hostapd will check if I can compile from src rpm…

filippo_carletti · December 18, 2016, 5:40pm

Don’t worry about systemd, it can handle old sysv initscripts.
I agree with you that a systemd unit file would be better, we could probably build one as a future improvement.

denis.robel · December 18, 2016, 5:46pm

I know that systemd can handle sysv init scripts …
But I’m a German and all things should be in a clear structure

denis.robel · December 18, 2016, 7:59pm

well I’m edited /etc/e-smith/templates/etc/squid/squid.conf/40ports as you suggested and now it’s working well.

denis.robel · December 18, 2016, 8:01pm

okay here I’m again with my documentation to setup a nethserver 7 based router with raspi3:

Install CentOs7 on Raspi follow the Nethserver Howto:

http://wiki.nethserver.org/doku.php?id=ns_raspi2

ON FIRST BOOT CHANGE ROOT PASSWORD!!!

Install MC --> I’m not a fan of vi vim

yum install mc

change CPU frequency to 1000 Mhz - for faster compiling

mcedit /boot/config.txt

systemctl reboot

Install Nethserver

yum -y update
yum clean all

new nsswitch.conf is created as nsswitch.conf.rpmnew

cp /etc/nsswitch.conf.rpmnew /etc/nsswitch.conf
yum localinstall http://mirror.framassa.org/nethserver-arm/nethserver-release-7arm.rpm
nethserver-install

Install Development Tools

yum groupinstall "Development Tools"
yum install nethserver-devtools

Enable wifi as decribed in /root/README

curl --location https://github.com/RPi-Distro/firmware-nonfree/raw/master/brcm80211/brcm/brcmfmac43430-sdio.bin > /usr/lib/firmware/brcm/brcmfmac43430-sdio.bin
curl --location https://github.com/RPi-Distro/firmware-nonfree/raw/master/brcm80211/brcm/brcmfmac43430-sdio.txt > /usr/lib/firmware/brcm/brcmfmac43430-sdio.txt

systemctl reboot

Install Hostapd:

yum install libnl3-devel
yum install openssl-devel

download hostapd-src from epl

wget -c http://dl.fedoraproject.org/pub/epel/7/SRPMS/h/hostapd-2.4-3.el7.src.rpm
rpmbuild --rebuild hostapd-2.4-3.el7.src.rpm

yum install /root/rpmbuild/RPMS/armv7hl/hostapd-2.4-3.el7.centos.armv7hl.rpm

edit /etc/hostapd/hostapd.conf

########################################################################################################### begin hostapd.conf 
#
# This will give you a minimal, insecure wireless network.
# 
# DO NOT BE SATISFIED WITH THAT!!!
#
# A complete, well commented example configuration file is
# available here:
#
#       /usr/share/doc/hostapd/hostapd.conf
#
# For more information, look here:
#
#       http://wireless.kernel.org/en/users/Documentation/hostapd
#

ctrl_interface=/var/run/hostapd
ctrl_interface_group=wheel

# Some usable default settings...
macaddr_acl=0        # macaddr_acl will be managed from dhcp ...
auth_algs=1
ignore_broadcast_ssid=0

# Uncomment these for base WPA & WPA2 support with a pre-shared key
wpa=3
wpa_key_mgmt=WPA-PSK
wpa_pairwise=TKIP
rsn_pairwise=CCMP

# DO NOT FORGET TO SET A WPA PASSPHRASE!!
wpa_passphrase=YOURSECRETPASSWORD

# Most modern wireless drivers in the kernel need driver=nl80211
driver=nl80211

# Customize these for your local configuration...
interface=wlan0
hw_mode=g
channel=1
ssid=YOURSSID
########################################################################################################## end hostapd.conf

enable hotapd.service:

systemctl enable hostapd.service

systemctl start hostapd

now you should see your wifi but you wont get an IP address because dhcp is not ready yet

Build Nethserver-WebContentfilter stuff

download src rpms:

wget -c http://mirrorlist.nethserver.org/nethserver/7.3.1611/base/Source/SPackages/nethserver-squid-1.5.0-1.ns7.src.rpm
wget -c http://mirrorlist.nethserver.org/nethserver/7.3.1611/base/Source/SPackages/nethserver-squidguard-1.6.0-1.ns7.src.rpm
wget -c https://www.urlfilterdb.com/files/downloads/ufdbGuard-1.32.4.src.rpm

nethserver-squid:
rpmbuild --rebuild nethserver-squid-1.5.0-1.ns7.src.rpm
yum install /root/rpmbuild/RPMS/noarch/nethserver-squid-1.5.0-1.el7.centos.noarch.rpm

ufdbguard:
yum install bzip2-devel
create user ufdb - without htis user the RPM will not be packed - why???
useradd -r ufdb -d /var/ufdbguard -M -s /sbin/nologin
rpmbuild --rebuild ufdbGuard-1.32.4.src.rpm
yum install /root/rpmbuild/RPMS/armv7hl/ufdbGuard-1.32.4-CentOS7.armv7hl.rpm

nethserver-squidguard:
rpmbuild --rebuild nethserver-squidguard-1.6.0-1.ns7.src.rpm
yum install /root/rpmbuild/RPMS/noarch/nethserver-squidguard-1.6.0-1.el7.centos.noarch.rpm

9.login to nethserver web surface

set wlan0 to green network
set eth0 to red network

configure dhcp for wlan0

configure web-proxy
configure web-contenfilter

denis.robel · December 18, 2016, 8:25pm

okay I tested a little bit the filter:

ufdbguard does not working correctly. I’m using shalla as blacklist and no listed domain will be blocked.

There must be a problem deeper inside. I’ll investigate it later…

alefattorini · December 19, 2016, 10:07am

Wow I love your work and I’d like to involve also the @arm_team in this discussion!

denis.robel · December 19, 2016, 7:03pm

Okay I checked the urfdbguard reference manual but it’s not so easy to find a start point for debugging. Some hints would be helpful…

At the moment I’m using squid manually with debug level 5 and debugging for ufdbguard is not clear. So I’m not able to check if there are data in blacklist database etc…

mark_nl · December 19, 2016, 8:17pm

Hi @denis.robel, great work!

unfortunately there is no official epel for arm, here are unsigned packaged build by the centos arm-interest group.
http://armv7.dev.centos.org/repodir/epel-pass-1/hostapd/

that is odd, I will try to build it in mock at Christmas

I don’t know much about urfdbguard, so can’t help you right now…

mark_nl · December 19, 2016, 8:49pm

Do not kwon if it helps, I see more configuration files pointing to /usr/lib64/…

github.com

NethServer/nethserver-squid/blob/master/root/etc/e-smith/templates/etc/squid/squid.conf/20acl_10_auth

{
   use esmith::NetworksDB;
   use NethServer::SSSD;
   
   # check and set libdir 32/64bit (for armhfp)
   my $_libdir = ( -e '/usr/lib64/' ) ? "/usr/lib64" : "/usr/lib";

   my $ndb = esmith::NetworksDB->open_ro();
   my $green_mode = $squid{'GreenMode'} || "manual";
   my $blue_mode = $squid{'BlueMode'} || "manual";
   my $sssd = new NethServer::SSSD();
   if ($green_mode eq 'authenticated' || (defined($ndb->blue()) && $blue_mode eq 'authenticated')) {
       $OUT .= "# Authentication required\n\n";

       if ($sssd->isAD()) {
           $OUT .= "\n# GSSAPI auth in ADS mode\n";
           $OUT .= "auth_param negotiate program ${_libdir}/squid/negotiate_kerberos_auth -i\n";
           $OUT .= "auth_param negotiate children 10\n";
           $OUT .= "auth_param negotiate keep_alive on\n";
       }

This file has been truncated. show original

denis.robel · December 19, 2016, 9:22pm

okay the path problem seems to be a general problem for arm …

But maybe the config template for ufdbguard ist for a wrong version?
I’m using urfdbguard 1.32.4 and the nethserver-squidguard is working with which version of urfbguard???

denis.robel · December 20, 2016, 7:30am

Is there any chance to check that the import from shalla.tar.gz into the urfdbguard database is done?

It seems for me that the blacklist database is empty…

For my understanding: urfdbguard import the blacklist form SquidgGuard like from shalla into it’s own database right?

When I’m creating an own category the blocking mechanism is working well. So my thesis of empty blacklist database could be true.

mark_nl · December 20, 2016, 7:51am

Technically a problem with 32 bit architecture, on 64 bit arm (aarch64) this problem does not exist. Although I normally advise against it you could make a simlink /usr/lib64/ > /usr/lib/ for testing.

In the x86_64 nethserver repository is urfdbguard 1.32.4

Can not help you with questions regarding urfdbguard/SquidgGuard

filippo_carletti · December 20, 2016, 9:29am

Right. The conversion is done every night after the updated lists are downloaded.
All logs are in /var/ufdbguard/logs/.
In my logs I find:
2016-12-07 15:27:10 [9783] loading URL category defaulttable with creation date 20161207.0420
2016-12-07 15:27:10 [9783] loading URL table from “/var/squidGuard/blacklists/gamble/domains”
2016-12-07 15:27:10 [9783] loading URL category defaulttable with creation date 20161207.0420
2016-12-07 15:27:10 [9783] loading URL table from “/var/squidGuard/blacklists/science/chemistry/domains”

And:
2016-12-05 15:54:35 [7471] received TERM signal
2016-12-05 15:54:35 [7471] statistics: TERM
2016-12-05 15:54:35 [7471] statistics: 287 URL lookups (252 https). 115 URLs blocked. 0 tunnels detected. 0 safe searches. 0 Youtube
edufilter. 172 uncategorised URLs. 3 clients. 1 users.
2016-12-05 15:54:35 [7471] statistics: category gamble was matched 0 times and blocked 0 times

filippo_carletti · December 20, 2016, 2:27pm

Both refer to auth, the outcome is that squid in authenticated mode will not work on RPI, unless the path is changed in squid configuration.

denis.robel · December 20, 2016, 7:00pm

okay in my log files i can find some entries like this:

2016-12-20 19:46:17 [6382] loading URL category defaulttable with creation date 20161220.1846
2016-12-20 19:46:17 [6382] loading URL table from "/var/squidGuard/blacklists/custom/whitelist/domains"
2016-12-20 19:46:17 [6382] loading URL category defaulttable with creation date 20161220.1838
2016-12-20 19:46:17 [6382] loading URL table from "/var/squidGuard/blacklists/finance/banking/domains"
2016-12-20 19:46:17 [6382] loading URL category defaulttable with creation date 20161220.1838
2016-12-20 19:46:17 [6382] loading URL table from "/var/squidGuard/blacklists/automobile/boats/domains"
2016-12-20 19:46:17 [6382] loading URL category defaulttable with creation date 20161220.1838
2016-12-20 19:46:18 [6382] loading URL table from "/var/squidGuard/blacklists/news/domains"
2016-12-20 19:46:18 [6382] loading URL category defaulttable with creation date 20161220.1838
2016-12-20 19:46:18 [6382] loading URL table from "/var/squidGuard/blacklists/isp/domains"
2016-12-20 19:46:18 [6382] upload-crash-reports off
2016-12-20 19:46:18 [6382] url-lookup-delay-during-database-reload on
2016-12-20 19:46:18 [6382] url-lookup-result-during-database-reload allow
2016-12-20 19:46:18 [6382] redirect-loading-database "http://cgibin.urlfilterdb.com/cgi-bin/URLblocked.cgi?category=loading-database"
2016-12-20 19:46:18 [6382] category "downloads" {
2016-12-20 19:46:18 [6382]    domainlist     "/var/squidGuard/blacklists/downloads/domains"

in /var/squidGuard/blacklists/aggressive i see domains.ufdb - owner is squid ???

ls -ahslo aggressive/ total 32K 4.0K drwxr-xr-x 2 squid 4.0K Dec 20 19:38 . 4.0K drwxr-xr-x 58 squid 4.0K Dec 20 19:45 .. 12K -rw-r--r-- 1 squid 9.0K Dec 20 19:37 domains 8.0K -rw-r--r-- 1 squid 5.4K Dec 20 19:38 domains.ufdb 4.0K -rw-r--r-- 1 squid 2.7K Dec 20 19:37 urls

but the filter does not block anything

filippo_carletti · December 21, 2016, 8:54am

Try to check with:
echo "http://bit.ly 10.10.0.1/ - - GET" | /usr/sbin/ufdbgclient -d
substituing bit.ly with a website you know you have blocked.
Hint from:
https://github.com/NethServer/nethserver-squidguard

denis.robel · December 23, 2016, 7:20pm

it’s tricky:

when I open a site which is listed in my custom category blocking is working well and when I open a site which is listed in /var/log/squidGuard/blacklists/adult/domains (flat list) blocking is not working.

I recreated the the database with
ufdbConvertDB /var/squidGuard/blacklists

after deleting the database file.

I checked the file permissions, the owners …

So I’ve no idea what is going wrong. Maybe someone else can test it on arm device too and can share the the results…

filippo_carletti · December 24, 2016, 9:27am

I’ll try to find my RPI next week.
Meanwhile, you could ask to ufdbguard support (https://www.urlfilterdb.com/support/supportdesk.html).
Marcus usually answer very quickly, competently and kindly.
I see that Mageia has an arm package for ufdbguard, I think it should work.
https://www.rpmfind.net/linux/RPM/mageia/cauldron/armv7hl/media/core/release/ufdbguard-1.31-11.mga6.armv7hl.html