General Computing
linux compression logging
Updated Wed, 22 Jun 2022 19:20:53 GMT

Log rotation with compression?

I'm running a process on Linux (CentOS 7) which produces a lot of output on stdout. Sending it all to a file will result in a file size greater than 1 TB. But happily the output is quite compressible - it consists of lots of quite repetitive text.

My thought is, can I use some kind of log rotation program like those described here: ( Log rotation of stdout? ), but where once a log is no longer current, it is compressed (and never deleted)? So I end up writing, say, a couple of GB to the file, compress it, and keep writing to a fresh file?


You can use logrotate for this. CentOS 7 is using systemd so you can configure a scheduled logrotate task with something like this:

  • Create a logrotate config file in /etc/systemd/system called mylogrotate.config with the following content:

    /path/to/your/logfile/*.log {
        minsize 1G
        rotate 30

This will tell logrotate to do the following:

  • Old versions of log files are compressed with gzip by default.
  • Truncate the original log file in place after creating a copy, instead of moving the old log file and optionally creating a new one.
  • Postpone compression of the previous log file to the next rotation cycle so you're going to have two uncompressed log files.
  • Log files are rotated when they grow bigger than 1Gbytes.
  • If the log file is missing, go on to the next one without issuing an error message.
  • Don't mail old log files to any address.
  • Log files are rotated 30 times before being removed so you'll have 30 old log files. Anything older than that will be deleted.

Create a systemd service. Create a file in /etc/systemd/system called mylogrotate.service with the following:

Description=Rotate My Log
ExecStart=/usr/sbin/logrotate /etc/systemd/system/mylogrotate.config --state /etc/systemd/system/mylogrotate.state --verbose

Let's schedule this task. Create a third file in /etc/systemd/system called mylogrotate.timer with the following:

Description=Rotate My Log Timer

This will run logrotate every hour.

All has been set up. Now enable the scheduled task.

systemctl enable mylogrotate.timer
systemctl start mylogrotate.timer

Of course do all this as root.

(I'm on Ubuntu but I hope this will work on CentOS without any issues)

Comments (1)

  • +1copytruncate seems useful, but I think it will work if the redirection to the log file is done with >>; only then truncating the file causes the new content to appear from the beginning. Redirection with > keeps its (growing) offset, so any new data after the file is truncated causes the file to grow (with leading zeros) to the old size. — Jul 20, 2018 at 18:32  

Linked Articles

Local articles referenced by this article: