How to Manage Logs with Logrotate

logrotate is designed to ease administration of systems that generate large numbers of log files. It allows automatic rotation, compression, removal, and mailing of log files. Each log file may be handled daily, weekly, monthly, or when it grows too large.

Intensifying this, many application frameworks have their own logging in place. Few manage the deletion or compression of their log files. In cases such as this, you should set up rotating logs and a log backup.

What does Logrotate do?

Logrotate basically helps to manage your log files. Periodically it will read, minimize, back up, creates new log files, and basically anything you may ever want to do with them. This is usually used to to help prevent any single log file from getting unwieldy in size. It is commonly also used to delete old log files so as not to fill your server’s hard drive.

Many application setup logrotate for you by default. For instance, installing Apache in Ubuntu adds the file /etc/logrotate.d/apache2, which is a configuration files used by Logrotate to rotate all apache access and error logs.

Configure Logrotate

The default logrotate file found in  /etc/logrotate.d  and it is going to run once per day. Logs are typically rotated once per day or less (Apache default in Ubuntu is in fact weekly).

Default apache log rotate file is :

/etc/logrotate.d/apache2

/var/log/apache2/*.log {
        weekly
        missingok
        rotate 52
        compress
        delaycompress
        notifempty
        create 640 root adm
        sharedscripts
        postrotate
            /etc/init.d/apache2 reload > /dev/null
        endscript
        prerotate
            if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
                    run-parts /etc/logrotate.d/httpd-prerotate; \
            fi; \
        endscript
}

This will rotate any files in /var/log/apache2 that end in “.log”. This is why, when we create a new apache virtual host, we typically put the logs in /var/log/apache2. Logrotate will automatically manage the log files!

Find these options:

• weekly: Rotate logs once per week. Available options are daily, weekly, monthly, and yearly.
• missingok: If no *.log files are found, don’t freak out
• rotate 52: Keep 52 files before deleting old log files (Thats a default of 52 weeks, or one years worth of logs!)
• compress: Compress (gzip) log files
  • delaycompress: Delays compression until 2nd time around rotating. As a result, you’ll have one current log, one older log which remains uncompressed, and then a series of compressed logs.
  • compresscmd: Set which command to used to compress. Defaults to gzip.
  • uncompresscmd: Set the command to use to uncompress. Defaults to gunzip.
• notifempty: Don’t rotate empty files
• create 640 root adm: Create new log files with set permissions/owner/group, This example creates file with user root and group adm. In many systems, it will be root for owner and group.
• sharedscripts: Run postrotate script after all logs are rotated. If this is not set, it will run postrotate after each matching file is rotated.
• postrotate: Scripts to run after rotating is done. In this case, Apache is reloaded so it writes to the newly created log files. Reloading Apache (gracefully) lets any current connection finish before reloading and setting the new log file to be written to.
• prerotate: Run before log rotating begins.

You will also like this : Real time apache log monitoring for php & apache log

Here’s what I have for a PHP application in production, which uses Mongolog to create log files in addition to the default Apache logger.

/var/www/my-app/application/logs/*.log {
    daily
    missingok
    rotate 7
    compress
    delaycompress
    notifempty
    create 660 appuser www-data
    sharedscripts
    dateext
    dateformat -web01-%Y-%m-%d-%s
    postrotate
        /etc/init.d/apache2 reload > /dev/null
        /usr/bin/s3cmd sync /var/www/my-app/application/logs/*.gz s3://app_logs
    endscript
    prerotate
        if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
            run-parts /etc/logrotate.d/httpd-prerotate; \
        fi; \
    endscript
}

Note that we specify the directory where the log files are written to from the application. The other items to note in the above Logrotate configuration:

• daily: This writes daily, since we expect a decent amount of traffic.
• rotate 7: Keep only the last 7 days of logs in the server.
• create 660 appuser www-data: New log files are owned by appuser, which is and example username of the user used to deploy files to the server. The log files are in group www-data, the same group that Apache typically runs as. The permissions (660) allow both owner and users of the same group to write to the file (appuser and Apache user www-data can then write to and edit these files). This let’s the PHP app write to the log files!
• dateext: Logs by default get a number appended to their filename. This option appends a date instead.
  • dateformat: The format of the date appended to the log filename you want. These logs also add web01, web02 (and so on) to the log file name so we know which webserver the log came from. This is recommended if you are  logging on multiple web servers behind a load balancer and will combine the logs at a later date.
• postrotate: Note that I’m saving log files to Amazon S3 using the s3cmd CLI tool. Using ‘sync’ is similar to rsync – It will overwrite files in S3 with newer files. This is especially why differentiating log file names between web01, web02, etc is necessary. This is some magic right here – Both giving you off-site backups to your log files, and not taking up unnecessary server space.