Build web filter modules for ARM

Both refer to auth, the outcome is that squid in authenticated mode will not work on RPI, unless the path is changed in squid configuration.

okay in my log files i can find some entries like this:

2016-12-20 19:46:17 [6382] loading URL category defaulttable with creation date 20161220.1846
2016-12-20 19:46:17 [6382] loading URL table from "/var/squidGuard/blacklists/custom/whitelist/domains"
2016-12-20 19:46:17 [6382] loading URL category defaulttable with creation date 20161220.1838
2016-12-20 19:46:17 [6382] loading URL table from "/var/squidGuard/blacklists/finance/banking/domains"
2016-12-20 19:46:17 [6382] loading URL category defaulttable with creation date 20161220.1838
2016-12-20 19:46:17 [6382] loading URL table from "/var/squidGuard/blacklists/automobile/boats/domains"
2016-12-20 19:46:17 [6382] loading URL category defaulttable with creation date 20161220.1838
2016-12-20 19:46:18 [6382] loading URL table from "/var/squidGuard/blacklists/news/domains"
2016-12-20 19:46:18 [6382] loading URL category defaulttable with creation date 20161220.1838
2016-12-20 19:46:18 [6382] loading URL table from "/var/squidGuard/blacklists/isp/domains"
2016-12-20 19:46:18 [6382] upload-crash-reports off
2016-12-20 19:46:18 [6382] url-lookup-delay-during-database-reload on
2016-12-20 19:46:18 [6382] url-lookup-result-during-database-reload allow
2016-12-20 19:46:18 [6382] redirect-loading-database "http://cgibin.urlfilterdb.com/cgi-bin/URLblocked.cgi?category=loading-database"
2016-12-20 19:46:18 [6382] category "downloads" {
2016-12-20 19:46:18 [6382]    domainlist     "/var/squidGuard/blacklists/downloads/domains"

in /var/squidGuard/blacklists/aggressive i see domains.ufdb - owner is squid ???

ls -ahslo aggressive/ total 32K 4.0K drwxr-xr-x 2 squid 4.0K Dec 20 19:38 . 4.0K drwxr-xr-x 58 squid 4.0K Dec 20 19:45 .. 12K -rw-r--r-- 1 squid 9.0K Dec 20 19:37 domains 8.0K -rw-r--r-- 1 squid 5.4K Dec 20 19:38 domains.ufdb 4.0K -rw-r--r-- 1 squid 2.7K Dec 20 19:37 urls

but the filter does not block anything

Try to check with:
echo "http://bit.ly 10.10.0.1/ - - GET" | /usr/sbin/ufdbgclient -d
substituing bit.ly with a website you know you have blocked.
Hint from:
https://github.com/NethServer/nethserver-squidguard

it’s tricky:

when I open a site which is listed in my custom category blocking is working well and when I open a site which is listed in /var/log/squidGuard/blacklists/adult/domains (flat list) blocking is not working.

I recreated the the database with
ufdbConvertDB /var/squidGuard/blacklists

after deleting the database file.

I checked the file permissions, the owners …

So I’ve no idea what is going wrong. Maybe someone else can test it on arm device too and can share the the results…

I’ll try to find my RPI next week.
Meanwhile, you could ask to ufdbguard support (https://www.urlfilterdb.com/support/supportdesk.html).
Marcus usually answer very quickly, competently and kindly.
I see that Mageia has an arm package for ufdbguard, I think it should work.
https://www.rpmfind.net/linux/RPM/mageia/cauldron/armv7hl/media/core/release/ufdbguard-1.31-11.mga6.armv7hl.html

okay thank you for feedback. Now I’m driving home for Christmas …

with warm regards and mery Christmas

Denis

1 Like

As first a happy new year to all.

during the meantime I contacted Markus from ufdbguard project. He told me that maybe there could be a bug when the ufdb database is created from the flat text list. Maybe it is depending from arm architecture…

I sent him my files for checking. When I’ll get some feedback from Markus I’ll update this post.

with warm regards from frosty Leipzig

Denis

4 Likes

Hallo,

here are some informations about the actual state:

Hi Denis,

I looked at the database that you sent and it has a severe corruption
that I cannot explain, but I am sure that it occurs only on the
ARM platform.
Since I do not have an ARM development system it is difficult
to find the root cause of the issue.

I like you to perform a test: go to …/src and
make t1 t2 t3
which produces 3 URL table files:
domains1.in.ufdb domains2.in.ufdb domains3.in.ufdb
and send me the 3 files.

Thanks,

Marcus

So it seems there are some problems under arm arch …

Denis

1 Like

Hallo,

so here are the brand new results of the last hour:

Markus from urfdb project found the issues an he created 2 new betas of ufdbGuard during our conversation.

The last beta V 1.26beta6 is working now on my raspi --> so that my project with a router for the kids is ready for daily use now :slight_smile:

I’ll ask Markus to provide the new sources for testing as soon as possible.
I was able to build a RPM with the original spec from ufdbGuard SRPM…

with warm regards from snowy and stormy Leipzig

Denis

2 Likes

Hallo,

Markus told me that I’m free to send a copy to the nethserver project for testing and he gave a short timeline for the next urfdbGuard release:

Hi Denis,

We plan to implement the last piece of missing IPv6 support into 1.32.5 and that will take a while.
I expect that in two months or so we will release it.
You are free to send of copy of the software that you received to the people of the nethserver project.

Best regards,

Marcus

@filippo_carletti: I’ll share it to the nethserver project with plesure. Please tell me an upload location.

Denis

2 Likes

Since ufdbguard is licensed under GPLv2 I think you could upload it to github (or ask Markus if he is willing to do that).
Or you could send it to me as a private message, but I will have to upload it to github to build it. :slight_smile:
Or you can send me a pastebin (or similar) link.

Anyway, thank you for your great work with this.
I’ll make sure we have an arm build asap.

2 Likes

Although I do use an RPI, arm acrhitecture is completely new and I honestly know less than little about it. I just want to give a HUGE shoutout to @denis.robel for his work on the webfilter module for arm architecture.
:clap: :clap: :clap::clap::clap::clap::clap:

3 Likes

Hallo Rob,

Thank you very much.

I tried to compile and tested it only. The issue was fixed by Marcus from ufdbGuard. So finally he did the job :wink: … I was only the bug hunter.

Small steps lead to big changes :slight_smile: Happy to see you jumping into the action :slight_smile:
@mark_nl will be more than happy to have you into the @arm_team

thanks to the great work of @denis.robel and @mark_nl , i’ve updated the wiki and some packages of NS7 on Raspberry…

if someone want to test it, this should be the howto to install ns7+hostapd+Proxy on rpi3 based on @denis.robel work, but in an easier way

install centos7 and nethserver as per wiki page (new one)
http://wiki.nethserver.org/doku.php?id=ns7_rasp

then configure the hostapd+proxy:

optional:
change CPU frequency to 1000 Mhz - for faster compiling
vi /boot/config.txt
systemctl reboot

  1. Enable wifi as decribed in /root/README

curl --location https://github.com/RPi-Distro/firmware-nonfree/raw/master/brcm80211/brcm/brcmfmac43430-sdio.bin > /usr/lib/firmware/brcm/brcmfmac43430-sdio.bin
curl --location https://github.com/RPi-Distro/firmware-nonfree/raw/master/brcm80211/brcm/brcmfmac43430-sdio.txt > /usr/lib/firmware/brcm/brcmfmac43430-sdio.txt

systemctl reboot

  1. Install Hostapd:

yum install hostapd
edit /etc/hostapd/hostapd.conf

########################################################################################################### begin hostapd.conf 
#
# This will give you a minimal, insecure wireless network.
# 
# DO NOT BE SATISFIED WITH THAT!!!
#
# A complete, well commented example configuration file is
# available here:
#
#       /usr/share/doc/hostapd/hostapd.conf
#
# For more information, look here:
#
#       http://wireless.kernel.org/en/users/Documentation/hostapd
#
ctrl_interface=/var/run/hostapd
ctrl_interface_group=wheel
# Some usable default settings...
macaddr_acl=0        # macaddr_acl will be managed from dhcp ...
auth_algs=1
ignore_broadcast_ssid=0
# Uncomment these for base WPA & WPA2 support with a pre-shared key
wpa=3
wpa_key_mgmt=WPA-PSK
wpa_pairwise=TKIP
rsn_pairwise=CCMP
# DO NOT FORGET TO SET A WPA PASSPHRASE!!
wpa_passphrase=YOURSECRETPASSWORD
# Most modern wireless drivers in the kernel need driver=nl80211
driver=nl80211
# Customize these for your local configuration...
interface=wlan0
hw_mode=g
channel=1
ssid=YOURSSID
########################################################################################################## end hostapd.conf

and enable hostapd.service:
systemctl enable hostapd.service
systemctl start hostapd

now you should see your wifi but you wont get an IP address because dhcp is not ready yet

  1. install web content filter from Software Center -> Extras and Testing

  2. login to nethserver web interface

set wlan0 to green network
set eth0 to red network

configure dhcp for wlan0

configure web-proxy
configure web-contenfilter
wait one day and check Reports->Web Proxy stats

##end

if there are error/problem, please let me know…
when confirmed to work, i’ll put the howto in the wiki.
tnx

I think we need a nethserver-hostapd, but this is another thread :slight_smile:

2 Likes

Such a great improvement. Thanks for your effort man! I would like to see the spin get his own way!

Hallo,

after the last updates I have some trouble with the wlan0 device of my raspi.

After rebooting there is no ip address assigned to this device.
It seems that /etc/sysconfig/network-scrips/ifcfg-wlan0 is ignored completely.
ifconfig shows the device without ip address.

When I’m assigning an IP manually to wlan0 all is working well again.

Where is the ifcfg-wlan0 called from? Is the networkmanager playing a role in this case too?

okay I found the problem:

for hostapd the config must be a little special:

DEVICE=wlan0 BOOTPROTO=static IPADDR=192.168.179.1 NETMASK=255.255.255.0 NM_CONTROLLED=no ONBOOT=yes #TYPE=Ethernet TYPE=Wireless MODE=AP USERCTL=no

The Wifi MODE must be defined different from what will be expected (managed, auto …) MODE=AP do that trick.

Wlan0 will get an static IP and it will not configured with a wrong wifi mode and so it’s not colliding with hostapd too.

Ehi any update on this? How are your tests going?

Hallo Alessio,

until today it’s working well. But I reduced the number of threats of ufdbguard by 50%…