Build web filter modules for ARM

denis.robel · December 19, 2016, 7:03pm

Okay I checked the urfdbguard reference manual but it’s not so easy to find a start point for debugging. Some hints would be helpful…

At the moment I’m using squid manually with debug level 5 and debugging for ufdbguard is not clear. So I’m not able to check if there are data in blacklist database etc…

mark_nl · December 19, 2016, 8:17pm

Hi @denis.robel, great work!

unfortunately there is no official epel for arm, here are unsigned packaged build by the centos arm-interest group.
http://armv7.dev.centos.org/repodir/epel-pass-1/hostapd/

that is odd, I will try to build it in mock at Christmas

I don’t know much about urfdbguard, so can’t help you right now…

mark_nl · December 19, 2016, 8:49pm

Do not kwon if it helps, I see more configuration files pointing to /usr/lib64/…

github.com

NethServer/nethserver-squid/blob/master/root/etc/e-smith/templates/etc/squid/squid.conf/20acl_10_auth

{
   use esmith::NetworksDB;
   use NethServer::SSSD;
   
   # check and set libdir 32/64bit (for armhfp)
   my $_libdir = ( -e '/usr/lib64/' ) ? "/usr/lib64" : "/usr/lib";

   my $ndb = esmith::NetworksDB->open_ro();
   my $green_mode = $squid{'GreenMode'} || "manual";
   my $blue_mode = $squid{'BlueMode'} || "manual";
   my $sssd = new NethServer::SSSD();
   if ($green_mode eq 'authenticated' || (defined($ndb->blue()) && $blue_mode eq 'authenticated')) {
       $OUT .= "# Authentication required\n\n";

       if ($sssd->isAD()) {
           $OUT .= "\n# GSSAPI auth in ADS mode\n";
           $OUT .= "auth_param negotiate program ${_libdir}/squid/negotiate_kerberos_auth -i\n";
           $OUT .= "auth_param negotiate children 10\n";
           $OUT .= "auth_param negotiate keep_alive on\n";
       }

This file has been truncated. show original

denis.robel · December 19, 2016, 9:22pm

okay the path problem seems to be a general problem for arm …

But maybe the config template for ufdbguard ist for a wrong version?
I’m using urfdbguard 1.32.4 and the nethserver-squidguard is working with which version of urfbguard???

denis.robel · December 20, 2016, 7:30am

Is there any chance to check that the import from shalla.tar.gz into the urfdbguard database is done?

It seems for me that the blacklist database is empty…

For my understanding: urfdbguard import the blacklist form SquidgGuard like from shalla into it’s own database right?

When I’m creating an own category the blocking mechanism is working well. So my thesis of empty blacklist database could be true.

mark_nl · December 20, 2016, 7:51am

Technically a problem with 32 bit architecture, on 64 bit arm (aarch64) this problem does not exist. Although I normally advise against it you could make a simlink /usr/lib64/ > /usr/lib/ for testing.

In the x86_64 nethserver repository is urfdbguard 1.32.4

Can not help you with questions regarding urfdbguard/SquidgGuard

filippo_carletti · December 20, 2016, 9:29am

Right. The conversion is done every night after the updated lists are downloaded.
All logs are in /var/ufdbguard/logs/.
In my logs I find:
2016-12-07 15:27:10 [9783] loading URL category defaulttable with creation date 20161207.0420
2016-12-07 15:27:10 [9783] loading URL table from “/var/squidGuard/blacklists/gamble/domains”
2016-12-07 15:27:10 [9783] loading URL category defaulttable with creation date 20161207.0420
2016-12-07 15:27:10 [9783] loading URL table from “/var/squidGuard/blacklists/science/chemistry/domains”

And:
2016-12-05 15:54:35 [7471] received TERM signal
2016-12-05 15:54:35 [7471] statistics: TERM
2016-12-05 15:54:35 [7471] statistics: 287 URL lookups (252 https). 115 URLs blocked. 0 tunnels detected. 0 safe searches. 0 Youtube
edufilter. 172 uncategorised URLs. 3 clients. 1 users.
2016-12-05 15:54:35 [7471] statistics: category gamble was matched 0 times and blocked 0 times

filippo_carletti · December 20, 2016, 2:27pm

Both refer to auth, the outcome is that squid in authenticated mode will not work on RPI, unless the path is changed in squid configuration.

denis.robel · December 20, 2016, 7:00pm

okay in my log files i can find some entries like this:

2016-12-20 19:46:17 [6382] loading URL category defaulttable with creation date 20161220.1846
2016-12-20 19:46:17 [6382] loading URL table from "/var/squidGuard/blacklists/custom/whitelist/domains"
2016-12-20 19:46:17 [6382] loading URL category defaulttable with creation date 20161220.1838
2016-12-20 19:46:17 [6382] loading URL table from "/var/squidGuard/blacklists/finance/banking/domains"
2016-12-20 19:46:17 [6382] loading URL category defaulttable with creation date 20161220.1838
2016-12-20 19:46:17 [6382] loading URL table from "/var/squidGuard/blacklists/automobile/boats/domains"
2016-12-20 19:46:17 [6382] loading URL category defaulttable with creation date 20161220.1838
2016-12-20 19:46:18 [6382] loading URL table from "/var/squidGuard/blacklists/news/domains"
2016-12-20 19:46:18 [6382] loading URL category defaulttable with creation date 20161220.1838
2016-12-20 19:46:18 [6382] loading URL table from "/var/squidGuard/blacklists/isp/domains"
2016-12-20 19:46:18 [6382] upload-crash-reports off
2016-12-20 19:46:18 [6382] url-lookup-delay-during-database-reload on
2016-12-20 19:46:18 [6382] url-lookup-result-during-database-reload allow
2016-12-20 19:46:18 [6382] redirect-loading-database "http://cgibin.urlfilterdb.com/cgi-bin/URLblocked.cgi?category=loading-database"
2016-12-20 19:46:18 [6382] category "downloads" {
2016-12-20 19:46:18 [6382]    domainlist     "/var/squidGuard/blacklists/downloads/domains"

in /var/squidGuard/blacklists/aggressive i see domains.ufdb - owner is squid ???

ls -ahslo aggressive/ total 32K 4.0K drwxr-xr-x 2 squid 4.0K Dec 20 19:38 . 4.0K drwxr-xr-x 58 squid 4.0K Dec 20 19:45 .. 12K -rw-r--r-- 1 squid 9.0K Dec 20 19:37 domains 8.0K -rw-r--r-- 1 squid 5.4K Dec 20 19:38 domains.ufdb 4.0K -rw-r--r-- 1 squid 2.7K Dec 20 19:37 urls

but the filter does not block anything

filippo_carletti · December 21, 2016, 8:54am

Try to check with:
echo "http://bit.ly 10.10.0.1/ - - GET" | /usr/sbin/ufdbgclient -d
substituing bit.ly with a website you know you have blocked.
Hint from:
https://github.com/NethServer/nethserver-squidguard

denis.robel · December 23, 2016, 7:20pm

it’s tricky:

when I open a site which is listed in my custom category blocking is working well and when I open a site which is listed in /var/log/squidGuard/blacklists/adult/domains (flat list) blocking is not working.

I recreated the the database with
ufdbConvertDB /var/squidGuard/blacklists

after deleting the database file.

I checked the file permissions, the owners …

So I’ve no idea what is going wrong. Maybe someone else can test it on arm device too and can share the the results…

filippo_carletti · December 24, 2016, 9:27am

I’ll try to find my RPI next week.
Meanwhile, you could ask to ufdbguard support (https://www.urlfilterdb.com/support/supportdesk.html).
Marcus usually answer very quickly, competently and kindly.
I see that Mageia has an arm package for ufdbguard, I think it should work.
https://www.rpmfind.net/linux/RPM/mageia/cauldron/armv7hl/media/core/release/ufdbguard-1.31-11.mga6.armv7hl.html

denis.robel · December 24, 2016, 11:12am

okay thank you for feedback. Now I’m driving home for Christmas …

with warm regards and mery Christmas

Denis

denis.robel · January 5, 2017, 5:19pm

As first a happy new year to all.

during the meantime I contacted Markus from ufdbguard project. He told me that maybe there could be a bug when the ufdb database is created from the flat text list. Maybe it is depending from arm architecture…

I sent him my files for checking. When I’ll get some feedback from Markus I’ll update this post.

with warm regards from frosty Leipzig

Denis

denis.robel · January 11, 2017, 4:03pm

Hallo,

here are some informations about the actual state:

Hi Denis,

I looked at the database that you sent and it has a severe corruption
that I cannot explain, but I am sure that it occurs only on the
ARM platform.
Since I do not have an ARM development system it is difficult
to find the root cause of the issue.

I like you to perform a test: go to …/src and
make t1 t2 t3
which produces 3 URL table files:
domains1.in.ufdb domains2.in.ufdb domains3.in.ufdb
and send me the 3 files.

Thanks,

Marcus

So it seems there are some problems under arm arch …

Denis

denis.robel · January 13, 2017, 5:07pm

Hallo,

so here are the brand new results of the last hour:

Markus from urfdb project found the issues an he created 2 new betas of ufdbGuard during our conversation.

The last beta V 1.26beta6 is working now on my raspi --> so that my project with a router for the kids is ready for daily use now

I’ll ask Markus to provide the new sources for testing as soon as possible.
I was able to build a RPM with the original spec from ufdbGuard SRPM…

with warm regards from snowy and stormy Leipzig

Denis

denis.robel · January 13, 2017, 9:46pm

Hallo,

Markus told me that I’m free to send a copy to the nethserver project for testing and he gave a short timeline for the next urfdbGuard release:

Hi Denis,

We plan to implement the last piece of missing IPv6 support into 1.32.5 and that will take a while.
I expect that in two months or so we will release it.
You are free to send of copy of the software that you received to the people of the nethserver project.

Best regards,

Marcus

@filippo_carletti: I’ll share it to the nethserver project with plesure. Please tell me an upload location.

Denis

filippo_carletti · January 13, 2017, 11:05pm

Since ufdbguard is licensed under GPLv2 I think you could upload it to github (or ask Markus if he is willing to do that).
Or you could send it to me as a private message, but I will have to upload it to github to build it.
Or you can send me a pastebin (or similar) link.

Anyway, thank you for your great work with this.
I’ll make sure we have an arm build asap.

robb · January 13, 2017, 11:53pm

Although I do use an RPI, arm acrhitecture is completely new and I honestly know less than little about it. I just want to give a HUGE shoutout to @denis.robel for his work on the webfilter module for arm architecture.

denis.robel · January 14, 2017, 4:05pm

Hallo Rob,

Thank you very much.

I tried to compile and tested it only. The issue was fixed by Marcus from ufdbGuard. So finally he did the job … I was only the bug hunter.