troubleshooting rsyslog

Having trouble with rsyslog? This page provides some tips on where to look for help and what to do if you need to ask for assistance. This page is continously being expanded.

Useful troubleshooting ressources are:

Malformed Messages and Message Properties

A common trouble source are ill-formed syslog messages, which lead to to all sorts of interesting problems, including malformed hostnames and dates. Read the quoted guide to find relief. A common symptom is that the %HOSTNAME% property is used for generating dynafile names, but some glibberish shows up. This is caused by the malformed syslog messages, so be sure to read the guide if you face that problem. Just let me add that the common work-around is to use %FROMHOST% or %FROMHOST-IP% instead. These do not take the hostname from the message, but rather use the host that sent the message (taken from the socket layer). Of course, this does not work over NAT or relay chains, where the only cure is to make sure senders emit well-formed messages.

Configuration Problems

Rsyslog 3.21.1 and above has been enhanced to support extended configuration checking. It offers a special command line switch (-N1) that puts it into "config verfication mode". In that mode, it interprets and check the configuration file, but does not startup. This mode can be used in parallel to a running instance of rsyslogd.

To enable it, run rsyslog interactively as follows:

/path/to/rsyslogd -f/path/to/config-file -N1

You should also specify other options you usually give (like -c3 and whatever else). Any problems experienced are reported to stderr [aka "your screen" (if not redirected)].

Configuration Graphs

Starting with rsyslog 4.3.1, the "$GenerateConfigGraph" command is supported, a very valuable troubleshooting tool. It permits to generate a graph of how rsyslogd understood its configuration file. It is assumed that many configuration issues can easily be detected just by looking at the configuration graph. Full details of how to generate the graphs, and what to look for can be found in the "$GenerateConfigGraph" manual page.

Asking for Help

If you can't find the answer yourself, you should look at these places for community help.

Debug Log

If you ask for help, there are chances that we need to ask for an rsyslog debug log. The debug log is a detailled report of what rsyslog does during processing. As such, it may even be useful for your very own troubleshooting. People have seen things inside their debug log that enabled them to find problems they did not see before. So having a look at the debug log, even before asking for help, may be useful.

Note that the debug log contains most of those things we consider useful. This is a lot of information, but may still be too few. So it sometimes may happen that you will be asked to run a specific version which has additional debug output. Also, we revise from time to time what is worth putting into the standard debug log. As such, log content may change from version to version. We do not guarantee any specific debug log contents, so do not rely on that. The amount of debug logging can also be controlled via some environment options. Please see debugging support for further details.

In general, it is advisable to run rsyslogd in the foreground to obtain the log. To do so, make sure you know which options are usually used when you start rsyslogd as a background daemon. Let's assume "-c3" is the only option used. Then, do the following:

If you need to submit the logfile, you may want to check if it contains any passwords or other sensitive data. If it does, you can change it to some consistent meaningless value. Do not delete the lines, as this renders the debug log unusable (and makes Rainer quite angry for wasted time, aka significantly reduces the chance he will remain motivated to look at your problem ;)). For the same reason, make sure whatever you change is change consistently. Really!

Debug log file can get quite large. Before submitting them, it is a good idea to zip them. Rainer has handled files of around 1 to 2 GB. If your's is larger ask before submitting. Often, it is sufficient to submit the first 2,000 lines of the log file and around another 1,000 around the area where you see a problem. Also, ask you can submit a file via private mail. Private mail is usually a good way to go for large files or files with sensitive content. However, do NOT send anything sensitive that you do not want the outside to be known. While Rainer so far made effort no to leak any sensitive information, there is no guarantee that doesn't happen. If you need a guarantee, you are probably a candidate for a commercial support contract. Free support comes without any guarantees, include no guarantee on confidentiality [aka "we don't want to be sued for work were are not even paid for ;)]. So if you submit debug logs, do so at your sole risk. By submitting them, you accept this policy.

[manual index] [rsyslog site]

This documentation is part of the rsyslog project.
Copyright © 2008 by Rainer Gerhards and Adiscon. Released under the GNU GPL version 2 or higher.