Nethserver-awstats needs testers

sdl
v7

(Stéphane de Labrusse) #1

I did a new module, I need feedbacks. it is a module for awstats to analyse web access from your virtualhosts (nethserver-virtualhosts). Of course I need real statistics.

For now only NS7

yum install http://mirror.de-labrusse.fr/NethDev/nethserver-awstats/nethserver-awstats-0.1.0-1.ns7.sdl.noarch.rpm

Then either configure a virtualhost with a wordpress or whatever web application you want, it should be workable out of the box for previous virtualhosts.

you will find your awstats configurations in /etc/awstats/awstats.*.conf For now it is minimalist, we could sort some options more, I wait your Ideas

for example we could ignore statistics from lan

you must wait some minutes before to get statististics (default is 5 minutes), you have a web panel in nethgui to modify the settings and browse the domain statistics.

The web statistics pages are protected for the admin user (default), think to set a password. You have a panel to put more users


(Alessio Fattorini) #2

Ehi please help this man test his new born :baby: @quality_team do you have some time?
If I remember correcty @GG_jr was interested once


(Giacomo Sanchietti) #3

Excellent! I’m planning to upgrade www.nethserver.org from NS 6 to NS 7.

Great chance to try awstats :wink:


(Stéphane de Labrusse) #4

thank for your interests, you will be surprised by the download statistics. I host a archlinux repository and it is about 5TB (terabyte) for the worse month.

I want to finish this development and I hope to get time to sort a ns6 version.


(Gabriel GHEORGHIU) #5

I’m still interested about AWStats!

And yes, I’m happy because @stephdl did this for us!

I have installed awstats by sdl a few minutes after Stéphane make the announcement.
I have a test server on NS7 with WebTop5, SOGo, Nextcloud and OSTicket installed on it and I will be glad to see statistics for all, if is possible.
Unfortunately, I was very busy and I had no time to “play” with awstats by sdl.

Question: is/will be possible to have statistics also for other web sites/apps which are not hosted on the server on which awstats is installed?

Again, thank you Stéphane! As usually, very good job!

Kind regards,
Gabriel


(Stéphane de Labrusse) #6

Short answer, awstats reads local logs, so if remote logs are copied by scp locally why not.

Concerning other webapp i would develop something based on db property, but for now it is only relied on nethserver-virtualhost


(Stéphane de Labrusse) #7

With the option https://awstats.sourceforge.io/docs/awstats_config.html#OnlyFiles we could create statistics, but with the many possibilities of name/software/alias/virtualhost, maybe we could to think something flexible by properties (with a text area in the panel).

With something like that we will create an awstats configuration easily.

Create separate config files for each of the selections

OnlyFiles "^/mylottosite"
OnlyFiles "^/mypersonalsite"
OnlyFiles "^/myetcsite"

(Stéphane de Labrusse) #8

testing myself my work, I found that awstats has an issue to sort the different domain name, Indeed all domains are logged to tthe same logs.

@davidep what about if we do one custom log per virtualhost, of course with a log rotation.

I meant about this https://www.westhost.com/knowledgebase/pages/viewpage.action?pageId=1115032


(Davide Principi) #9

Seems reasonable…

However the official Apache docs explain why a single log file is good and how to split it by host:

https://httpd.apache.org/docs/2.4/logs.html#virtualhost

IIRC the default CentOS config uses the vhost log format.


(Stéphane de Labrusse) #10

I must inspect, probably if i just do some minor change in the way how to write the log, probably i might solve my issue. With sme server the domain name starts first for each line and awstats works well for the ibays.

It is just a matter to say what to log, thanks for the pointers.

I’m away for three days, I will look for it later


(Stéphane de Labrusse) #11

I speak something like that

[root@smemirror ~]# tailf /var/log/httpd/access_log
archlinux.de-labrusse.fr 141.8.183.13 - - [05/Jun/2017:22:52:58 +0200] "GET /staging/os/i686/redland-storage-virtuoso-1:1.0.17-5-i686.pkg.tar.xz.sig HTTP/1.1" 404 269 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
mirror.de-labrusse.fr 94.21.213.39 - - [05/Jun/2017:22:52:58 +0200] "GET /NethServer/7/repodata/repomd.xml HTTP/1.1" 200 3001 "-" "urlgrabber/3.10 yum/3.4.3"
mirror.de-labrusse.fr 94.21.213.39 - - [05/Jun/2017:22:52:59 +0200] "GET /NethServer/7/repodata/7b7cea297883dc8c1ff5029307951a3d1cd1f99e5f190715157f2b5a783b1978-primary.sqlite.bz2 HTTP/1.1" 200 122908 "-" "urlgrabber/3.10 yum/3.4.3"
archlinux.de-labrusse.fr 141.8.183.13 - - [05/Jun/2017:22:53:01 +0200] "GET /staging/os/i686/redland-storage-mysql-1:1.0.17-5-i686.pkg.tar.xz.sig HTTP/1.1" 404 266 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
smeserver.de-labrusse.fr 51.255.65.55 - - [05/Jun/2017:22:53:03 +0200] "GET /releases/9.2/smeos/i386/SME/xorg-x11-drv-mach64-6.9.5-1.el6.i686.rpm HTTP/1.1" 200 71664 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)"
archlinux.de-labrusse.fr 141.8.183.13 - - [05/Jun/2017:22:53:04 +0200] "GET /staging/os/i686/qt5-quickcontrols-5.9.0-1-i686.pkg.tar.xz.sig HTTP/1.1" 404 259 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
archlinux.de-labrusse.fr 177.96.149.228 - - [05/Jun/2017:22:53:04 +0200] "GET /core/os/x86_64/core.db HTTP/1.1" 200 127082 "-" "curl/7.53.1"
archlinux.de-labrusse.fr 141.8.183.13 - - [05/Jun/2017:22:53:07 +0200] "GET /multilib/os/x86_64/lib32-glibc-2.25-3-x86_64.pkg.tar.xz.sig HTTP/1.1" 200 310 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"

(Davide Principi) #12

It’s false. We should define a custom log format with %v as first column and apply it to virtual hosts.

I’d like a unique additional log for all virtual hosts and %v.

I wouldn’t go with a global modification: it breaks the current log format.


Completing the module AWSTATS
(Stéphane de Labrusse) #13

Well awstats works with a combined log format, for what I saw it doesn’t work well for all the vhosts in one log.

I don’t know why.

For now I created a custom access log for each vhost, it works well, the error log is not changed.


(Stéphane de Labrusse) #14

yum install http://mirror.de-labrusse.fr/NethDev/nethserver-awstats/nethserver-awstats-0.1.2-1.ns7.sdl.noarch.rpm

new panel with awstats settings
access logs are specific per virtualhosts

it should be good, I hope


(Gabriel GHEORGHIU) #15

Nice!

Anyway, besides this:

http://www.awstats.org/docs/awstats_config.html

we need a Legend!

Or, some clues near/below/… every “setting frame”.


(Stéphane de Labrusse) #16

new version, cosmetic changes in the code, thank skitoo

yum install http://mirror.de-labrusse.fr/NethDev/nethserver-awstats/nethserver-awstats-0.1.3-1.ns7.sdl.noarch.rpm


(Stéphane de Labrusse) #17

Released in my repository.

the dnslookup and the GeoIP are available in the panel
elementList for the templates.


(Stéphane de Labrusse) #18

4 posts were split to a new topic: Awstats: access log of virtualhost is empty


(Stéphane de Labrusse) #19