Unlike many other monitoring tools, Icinga does not include any internal mechanisms for checking the status of hosts and services on your network. Instead, Icinga relies on external programs (called plugins) to do all the dirty work.

Plugins are compiled executables or scripts (Perl scripts, shell scripts, etc.) that can be run from a command line to check the status or a host or service. Icinga uses the results from plugins to determine the current status of hosts and services on your network.

Icinga will execute a plugin whenever there is a need to check the status of a service or host. The plugin does something (notice the very general term) to perform the check and then simply returns the results to Icinga. Icinga will process the results that it receives from the plugin and take any necessary actions (running event handlers, sending out notifications, etc).

Plugins act as an abstraction layer between the monitoring logic present in the Icinga daemon and the actual services and hosts that are being monitored.

The upside of this type of plugin architecture is that you can monitor just about anything you can think of. If you can automate the process of checking something, you can monitor it with Icinga. There are already a lot of plugins that have been created in order to monitor basic resources such as processor load, disk usage, ping rates, etc. If you want to monitor something else, take a look at the documentation on writing plugins and roll your own. It's simple!

The downside to this type of plugin architecture is the fact that Icinga has absolutely no idea what it is that you're monitoring. You could be monitoring network traffic statistics, data error rates, room temperature, CPU voltage, fan speed, processor load, disk space, or the ability of your super-fantastic toaster to properly brown your bread in the morning... Icinga doesn't understand the specifics of what's being monitored - it just tracks changes in the state of those resources. Only the plugins themselves know exactly what they're monitoring and how to perform the actual checks.

There are plugins currently available to monitor many different kinds of devices and services, including:

Plugins are not distributed with Icinga, but you can download the official Nagios plugins and many additional plugins created and maintained by Nagios and Icinga users from the following locations:

After downloading you have to install the plugin(s). Please check the accompanied documentation on how to do that. It may contain important information about the prerequisites like additional packages, (perl) modules and on how to install the plugin together with hints for your distribution.

Sometimes you have to compile the plugin preparing the compile process using "./configure" with or without options. Please check the file config.log for errors regarding missing (devel) packages before issuing the actual compile command (mostly "make" or "make all"). In most cases the plugin is copied to the plugins directory (i.e. /usr/local/icinga/libexec) during "make install".

Sometimes you have to alter the plugin to reflect your environment (i.e. path to "utils.pm", ...). You may create a logical link pointing to the plugin directory instead so you don't have to change the plugin to circumvent this issue easing plugin updates later on doing something like

 $> mkdir /usr/local/nagios
 $> ln -s /usr/local/icinga/libexec /usr/local/nagios/libexec

Switch to the Icinga user as defined in icinga.cfg unless already done and clear the environment

 #> su - icinga
 #> env -i

If you are logged in now then skip to "Extending the environment"

Due to security awareness of the packager / sys admin this might fail because the account is not allowed to login. Please ask your sys admin to change that temporarily or do one of the following

Several checks (like check_oracle_health) depend on various environment variables to be set. Don't put these in .bashrc or similar user dependent files but choose a central location. The default init script sources the file /etc/sysconfig/icinga so that would be an ideal place. Don't use the init script itself because your changes might be lost during updates.

Example of /etc/sysconfig/icinga

 export ORACLE_HOME=/usr/lib/oracle/11.2/client64
 export LD_LIBRARY_PATH=$ORACLE_HOME/lib
 export PATH=$PATH:$ORACLE_HOME

After you logged in these variables are not set but doing so is pretty easy

 $> . /etc/sysconfig/icinga

Please verify the settings

 $> echo $ORACLE_HOME
 $> echo $LD_LIBRARY_PATH
 $> echo $PATH

Nearly all plugins will display basic usage information when you execute them using '-h' or '--help' on the command line. For example, if you want to know how the check_http plugin works or what options it accepts, you should try executing the following command:

 $> ./check_http --help

After the installation of the plugin (see "Obtaining plugins") call it from the command line using the appropriate options. If this works then you can integrate it into Icinga.

Let's imagine you used the following call on the command line

 $> /usr/local/icinga/libexec/sample-plugin.pl -H 192.168.1.2 -a argument1 -p parameter -n 5

The command definition has to contain two directives

 define command{
    command_name check_sample
    command_line $USER1$/sample-plugin.pl -H $HOSTADDRESS$ -a $ARG1$ -p $ARG2$ -n $ARG3$
    }

Then we have to define the check_command directive which is part of the host/service definition starting with the short name followed by the arguments separated by exclamation marks (!):

 check_command check_sample!argument1!parameter!5

As you can see the IP-address is not specified because it is taken from the host definition.

Putting it all together in reverse order shows how Icinga will process the information:

 check_command check_sample!argument1!parameter!5
                                |         |     +-------------------------------------+
                                |         +---------------------------------+         |
                                +---------------------------------+         |         |
                                                                  |         |         |
 Host macro ----------------------------------------+             |         |         |
                                                    |             |         |         |
 User macro --------+                               |             |         |         |
                    |                               |             |         |         |
 command_line      $USER1$/sample-plugin.pl -H $HOSTADDRESS$ -a $ARG1$ -p $ARG2$ -n $ARG3$

results in:

 /usr/local/icinga/libexec/sample-plugin.pl -H 192.168.1.2 -a argument1 -p parameter -n 5

In addition to the macros already mentioned there are a lot of others making life easier. Please note:

NRPE and "dont_blame_nrpe=1"

Using NRPE with arguments requires some attention. Given that you have enabled argument processing on the remote server in nrpe.cfg using "dont_blame_nrpe=1" (or "allow_arguments=1" in nsc.ini) you can pass parameters from the Icinga server to the remote machine. Let us assume some definitions

On the Icinga server

 define command{
    command_name check_nrpe
    command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$ -a $ARG2$
    }

 define service{
    ...
    check_command check_nrpe!check_process!cupsd

on the remote server in the NRPE config file

...
command[check_process]=your_plugin "$ARG1$"

The Icinga process will resolve the definitions as follows

 check_command check_nrpe!check_process!cupsd
                                |         |
                                |         +---------------------------+
                                +---------------------------+         |
                                                            |         |
 Host macro ----------------------------------+             |         |
                                              |             |         |
 User macro --------+                         |             |         |
                    |                         |             |         |
 command_line      $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$ -a $ARG2$

results in:

 /usr/local/icinga/libexec/check_nrpe -H 192.168.1.2 -c check_process -a cupsd

On the remote machine the NRPE process receives a call providing two parameters: "check_process" and "cupsd". The first is stripped to determine the command to be executed so only one argument is passed to the command!

The classic UI allows to inspect the raw command line including values from resource.cfg. Clicking on "ACTIVE" next to "Check type" within host/service check details will give access to this information. If you don't have defined a check yet choose "View Config" from the main menu on the left, then "Command expansion". Please note that the user has to be permitted explicitly using the directive authorized_for_full_command_resolution in cgi.cfg to see the values of variables from resource.cfg. The user running the web server must have read access on this file, too.

If you intend to log the raw command line then change some directives in icinga.cfg to the following values

 #  16 = Host/service checks
 # 256 = Commands
 debug_level=272
 debug_verbosity=2
 max_debug_file_size=1000000000

Some plugins support specifying ranges for the warning and critical values. Please check the documentation if this is the case for the plugin you want to use. The following is an excerpt from the developer guidelines:

A range is defined as a start and end point (inclusive) on a numeric scale (possibly negative or positive infinity).

A threshold is a range with an alert level (either warning or critical).

The theory is that the plugin will do some sort of check which returns back a numerical value, or metric, which is then compared to the warning and critical thresholds.

This is the generalised format for ranges:

[@]start:end

Notes:

Example ranges

Command line examples

Check the configuration using "/etc/init.d/icinga show-errors" and resolve any errors before issuing "/etc/init.d/icinga restart". Wait until the object is checked and look at the status details. There might be errors.

You can find information on the technical aspects of plugins, as well as how to go about creating your own custom plugins here.