ufdbGuard Regex

[Vlad]

NethServer Version: 7.9.2009
Module: squidGuard

In the documention at: nethserver-squidguard — NethServer 7 documentation

There is an option for “Expressions” I assume referring to regular expressions, but I tried creating a custom category and using regex in the domain list but the following:
^.(cuevana[0-9]?).$
cuevana[0-9]?
cuevana
After applying the category to the filter, The sites are not blocked. So how is it that one can use the expressions to block sites if the url contains certain words?

Any help is appreciated,

Vlad

[m.traeumner (Michael Träumner)]

I’m also not able to get it working, but I don’t have a clue from regular expressions.
Perhaps @support_team can help.

[dnutan (Marc)]

I tried but unable to make it work.

Going to Applications → Web Proxy and Filter → Filter: Configuration - Edit, within the global options we find:

Expressions : if enabled, allow regular expression on blacklists categories.
And linked to the Expressions control we find UrlBlacklist and UrlWhitelist text fields.

What is the purpose of having these fields linked to Expressions control?

It seems that when “Enable expression matching on URLs” option is enabled then each filter category can make use of their own expression file, but I could be mistaken.
Was expression matching on URLs really implemented on the GUI or do we need to tweak some extra file(s)?

It is old but maybe it refreshes someone’s memory to come with a bright idea:

Using expressions matching to Web Proxy & Filter

@giacomo or somebody else who understands it Can you help me to understand the following template fragment: I have a problem by understanding why file gets an expressionlist and the other categories not. { use File::Find; use File::Basename; my $exprstatus = $squidguard{'Expressions'} || 'disabled'; our %categories; our @files; sub wanted() { my %tmp; if ( -d $File::Find::name ) { my $domains = $File::Find::name."/domains"; m…

[giacomo (Giacomo Sanchietti)]

I didn’t test the custom expressions since a couple of years, but when you enable it, you should find something like this inside the configuration file:

category custom {
   ...
   expressionlist  "custom/expressions"
 }

It seems UfdbGuard is still supporting it: https://www.urlfilterdb.com/files/downloads/ReferenceManual.pdf

[Vlad]

I checked on the /etc/ufdbguard/ufdbGuard.conf file and while some blacklist categories have the expressionlist line, the categories that I create on the webgui do not have this line, I tried creating a custom template: /etc/e-smith/templates-custom/etc/ufdbguard/ufdbGuard.conf/19categori

category “custom-regex” {
domainlist /var/squidGuard/blacklists/custom/custom-regex/domains
expressionlist /var/squidGuard/blacklists/custom/custom-regex/expressions
}

but nothing was added, I guess there’s more to creating that blacklist entry on the db.

[mrmarkuz (Markus Neuberger)]

You’re right, it seems the list file is always saved as “domains” instead of “expressions” so in ufdbguard.conf it’s handled as domainlist instead of expressionlist.

To workaround it for now, you could rename the domains file to expressions. In this example the custom category is named test:

mv /var/squidGuard/blacklists/custom/test/domains /var/squidGuard/blacklists/custom/test/expressions

And apply the config:

signal-event nethserver-squidguard-update

Now it should work as expected.

I also tried with old server manager without success.

[Vlad]

This didn’t work for me, after signal-event nethserver-squidguard-update or signal-event nethserver-squidguard-save the sites are not blocked and if I do /etc/e-smith/events/actions/system-adjust of course the changes are reverted.

To get it working I used a random blacklist category from Toulo “warez” in this case, and added the regex code to their expressions file, after /etc/e-smith/events/actions/system-adjust every matching site is blocked.

I do want to take a moment to thank you guys for the great work in this project, I use my Neth instance for GW, Proxy (AD joined), VPN, and updating from 7.6 to 7.9 was very easy, thank you for keeping the project updated and the great support!

[giacomo (Giacomo Sanchietti)]

Thank you all for the in-depth analysis, I can confirm we indeed have 2 bugs: a dead feature and a bad UI behavior.

@dnutan

What is the purpose of having these fields linked to Expressions control?

None: this is a bug in the UI. Enable expression matching on URL is not related in any way with Blocked URLs blacklist and Blocked URL whitelist field.

@dnutan

Was expression matching on URLs really implemented on the GUI or do we need to tweak some extra file(s)?

Yes but only for categories coming from blacklist sources, not for custom categories.

The dead feature are the Blocked URLs blacklist and Blocked URL whitelist fields, which are mapped to the following configuration files:

• /var/squidGuard/blacklists/custom/blacklist/urls
• /var/squidGuard/blacklists/custom/whitelist/urls

Both files come from old squidguard. We migrated from squidguard to ufdbGuard more than 5 years ago, and this feature stopped working since than . ufdbGuard doesn’t support URL based blocking. Thank you @Vlad for pointing it out!

So, in the end, I think we need the following changes:

• remove Blocked URLs blacklist and Blocked URL whitelist from Cockpit UI
• remove the corresponding code also from NethGUI
• remove backend code for /var/squidGuard/blacklists/custom/{blacklist,whitelist}/urls

@Vlad I’m sorry to inform you that current implementation doesn’t support your usage scenario and you probably will loose your modification on next update.
So I’m proposing a quick and dirty workaround for you:

• create a file named /etc/e-smith/events/nethserver-squidguard-save/S3custom-expressions, make sure it’s executable then put inside it something like:

  #!/bin/bash
  DIR=/var/squidGuard/blacklists/mycustom
  mkdir -p $DIR
  echo "testexpr" >> $DIR/expressions
  /usr/sbin/ufdbConvertDB $DIR
  chown -R squid:squid $DIR
  

• to apply the config use signal-event nethserver-squidguard-save (do not use /etc/e-smith/events/actions/system-adjust)

With the above procedure, the new category mycustom will have expressions enabled, still I know that writing good expressions for URL filtering is really an hard job!

P.S. Sorry for the late response, we were in the middle of 2-day training course on NS 8

[Vlad]

Thank you for the help.

@giacomo the last workaround didn’t work. the script is executed (after applying chmod +x), while the expressions file is created, I don’t see the new category in the webgui to attach it to a profile, so I used the db command to attach the new category, but still the sites aren’t blocked on the profile.

[giacomo (Giacomo Sanchietti)]

@Vlad

I don’t see the new category in the webgui to attach it to a profile, so I used the db command to attach the new category, but still the sites aren’t blocked on the profile.

Yes, it’s not integrated in the UI in any way.
About if it works or not, you should debug a little ufdbguard, take a look at this: nethserver-squidguard — NethServer 7 documentation