Increase default log retention period

(Davide Principi) #1

I’d like to extend the default log retention period from 4 to 52 weeks.

  • 4 weeks is the default upstream retention period. We usually keep upstream defaults, but it would be acceptable only if I store logs externally IMO (that is typical in cloud infrastructures, but not for home/sme users)
  • 52 weeks is the default retention period I apply to my customers. When I connect to my customers’ systems 4 weeks of past logs are not enough. To investigate a problem I need older information, as this community #support request demostrates: No login in eJabberd, IMAP and SSH

Absolutely true! You’re right and as @stephdl recalls we decided to not force every software to respect the default value.

I want to change only the default log retention period. Period.

I’d like to enforce the new default for new installations starting from 7.6.

What do you think?

Improving Log Retention Policy
(Ralf Jeckel) #2

I had a look at my /var/log. It takes 116MB now. If log retention is encreased from 4 to 52 weeks it should take 1.5 GB. That’s o.k. for me.

(Dan) #3

Can you have logrotate compress the old logs?

(Davide Principi) #4

Thank you for your interest!

I’m happy you’re ok, anyway I want to remark that only new installations will be affected by the change. Existing installations can be adjusted even today with

config setprop logrotate Times 52
signal-event nethserver-base-update

Yes, it should be already possible:

config setprop logrotate Compression enabled
signal-event nethserver-base-update

See also

http://docs.nethserver.org/projects/nethserver-devel/en/v7/nethserver-base.html?highlight=logrotate#log-retention-and-rotation

(Giacomo Sanchietti) #5

I agree.

(Stéphane de Labrusse) #6

Already done on all my servers

(Dan) #7

Hadn’t seen that in the docs–but strange that it isn’t enabled by default.

(Giacomo Sanchietti) #8

Related issue: https://github.com/NethServer/dev/issues/5629

Because it’s the upstream default. IMHO, it’s easier to inspect uncompressed logs.

(Davide Principi) #9

Absolutely!

@danb35 - and if I ran out of disk space I can save a bit of space by enabling the Compression prop… But huge logs are usually the symptom of a problem so…

(Dan) #10

zcat blah | grep whatever doesn’t seem especially difficult…

On the one hand, yes, if your logs are filling up your disk, there’s probably something else wrong. OTOH, I’ve got about 750 MB of logs with the default four-week retention. Increasing retention to a year takes that to nearly 10 GB, which is a fairly significant amount of storage. However, log files are highly compressible. As an example, I took my two oldest messages files, -1014 which was 7.3M, and -1021 which was 6.9M. I ran gzip on -1014, reducing it to 531k–compression ratio of 13.7:1. I ran bzip2 on -1021, reducing it to 329k, a ratio of 21:1.

(Filippo Carletti) #11

zgrep is even easier. :slight_smile:

(Dan) #12

So much the better. See, it isn’t hard! (-:

I tend to think compressed logs should be the default anyway, but certainly if we’re going to go to a year’s retention by default.

(Davide Principi) #13

You both have good points here but we must take the log viewer UI into account. It does not support log compression by now and is really slow. Adding the decompression task would make it even slower!

(Giacomo Sanchietti) #14

As a side node, I forgot to say we have plans to add a simple UI inside cockpit to configure the logrotate behavior.

(Stéphane de Labrusse) #15

did you talk about this ?

Screenshot_2018-11-16%20Syst%C3%A8me%20-%20ns7loc15%20nethservertest%20org
Screenshot_2018-11-16 Système - ns7loc15 nethservertest org.png1721x236 10.7 KB

my really first contribution in cockpit, need probably many other things

(Giacomo Sanchietti) #16

Wow!!!

(Stéphane de Labrusse) #17
(Giacomo Sanchietti) #18

Kudos to @stephdl who did his first contribution to the cockpit package!!

(Stéphane de Labrusse) #19

Cool…what will be the next one…do you have something in mind ?

(Giacomo Sanchietti) #20

I have lot in mind, but nothing written somewhere :frowning:

We have many applications to implement. The main ones are the firewall and mail server.
Also the documentation should be modified to be accessible directly from cockpit.
But I guess is better to discuss such things on another thread.