Log rotation for WebLogic Server (and friends)

If you have a number of WebLogic Server instances running, or applications that write a lot of information into the logs, you might find that your log files, and your stdout files start to eat up a lot of space very quickly.

This can be easily managed with log rolling utilities like logrotate for Linux or logadm for Solaris. These allow you to automate the process of removing old log entries, and they work by moving the log file contents out through a series of files. This means that you will have effectively controlled the amount of space used for logs, plus you have set a time period that logs are kept online before being archived or deleted.

Let’s look at an example to understand how it works:

In this diagram we are looking at the log growth over time, with time being the vertical axis. So at the top we have a single log file – AdminServer.log – and it grows over time, as indicated by the blue bars at the top.

Then, when the first log rotation occurs (the top red dotted line), the content of AdminServer.log is moved into AdminServer.log.0, as indicated by the purple arrow, and AdminServer.log is emptied out, so as WebLogic Server continues to write into this file, we have a much smaller file now (the green one).

This process then repeats. At the next log rotation, the lower red dotted line, we get the contents of AdminServer.log.0 moved into AdminServer.log.1, the contents of AdminServer.log moved into AdminServer.log.0, and AdminServer.log emptied out again.

This continues until we get up to eight files, then the oldest one is deleted and the other seven move one to the right.

Here’s how to set it up:

First, you need to be starting the WebLogic Server instance in a way that the stdout is being appended to a file, (as opposed to just written to a file). To do this, you need to make sure you use the >> shell redirection, not >. To get stdout and stderr in the same file, you would use &>>.

Here’s an example:

/your/fmwhome/user_projects/domains/base_domain/startWebLogicServer.sh &>> /your/logs/adminserver.out &

This assumes that you have WebLogic installed in the Oracle Home /your/fmwhome and that you are storing your stdout/stderr log files in /your/logs.

Note: It is important that you use the append redirection (>>), otherwise your log files will not actually shrink in size after rotation, they will just keep growing, so that defeats the purpose of rotation in the first place.

Note: If you use the Node Manager to start WebLogic Server, it will automatically open the log files in append mode.

Next, you need to set up your log rolling utility – logrotate (on Linux) or logadm (on Solaris). Let’s look at each of these in turn.

logrotate (Linux)

The configuration for logrotate is kept in /etc/logrotate.conf. You need to add a stanza in there for each of the log files you want to rotate. Here is an example for the stdout file from above, and the server log file:

/your/fmwhome/user_projects/domains/base_domain/servers/AdminServer/logs/AdminServer.log {
  copytruncate
  daily
  rotate 8
}
/your/logs/adminserver.out {
  copytruncate
  daily
  rotate 8
}

Let’s explore these. The first line names the log file to rotate. Rotation is essentially going to move that log to a different file and create a new empty log. The line ‘rotate 8‘ tells logrotate to keep up to eight log files. The ‘daily’ means to roll them once a day, and the ‘copytruncate’ tells it which method to use – in this case to copy the file into a new file, then empty it out (as opposed to the other method which is to rename the file and create a new one – this method will not work with WebLogic Server or other JVM applications).

You can also force logrotate to run immediately, which is useful for checking you have everything set up correctly. This is done (as root) by issuing the command:

logrotate -f /etc/logrotate.conf

logrotate has many other options that allow you to specify different time periods, actions to take before and after log rotation, and size limits for when rotation should occur, to name a few. You should take a look at the logrotate documentation to see how to use it to best suit your scenario.

logadm (Solaris)

logadm provides essentially the same capabilities as logrotate, but the configuration is slightly different. To set up the same example as we saw above with logadm, we need to issue the following commands (as root):

logadm -w /your/fmwhome/user_projects/domains/base_domain/servers/AdminServer/logs/AdminServer.log -P 1d -c
logadm -w /your/logs/adminserver.out -P 1d -c

These commands will update the logadm configuration file (/etc/logadm.conf) with the necessary entries – the -w option means ‘write to configuration file’. It is not recommended to edit the file directly, but to use the logadm -w command to update it – to prevent errors.

After the -w, we see the name of the file to rotate, then -P 1d, which means period of one day – i.e. daily – and -c which tells logadm to use the copy and truncate method.

To force logadm to run immediately, you need to do two things. First, you need to tell it to assume the last run was some time in the past, this is done by issuing the same command (as root) with a timestamp on it, e.g.:

logadm -w /your/logs/adminserver.out -P 1d -c -p 'Mon Feb 25 02:00:00 2013'

This tells logadm to assume the last time it ran for this file was on that date, which is more than a day ago.  The last run timestamps are stored in /var/logadm/timestamps if you want to take a look.

You can then issue the following command (as root) to force logadm to run immediately:

logadm

Just like logrotate, logadm has a bunch of other options that let you control how often rotation is done, size limts, pre/post actions, etc. Take a look at the documentation to see what you need for your scenario.

Enjoy!