Icinga

10.2. NRPE

10.2.1. Introduction
10.2.2. Prerequisites
10.2.3. Download
10.2.4. Optional changes
10.2.5. Compile on the Icinga server
10.2.6. First test
10.2.7. Remote system(s)
10.2.8. Prerequisites on the remote host(s)
10.2.9. Second test
10.2.10. Installation on the remote host
10.2.11. Third test
10.2.12. Troubleshooting
10.2.13. Security
10.2.14. Definition of local checks
10.2.15. Definitions on the Icinga server
10.2.16. More Troubleshooting
10.2.17. Upgrading

10.2.1. Introduction

Nagios Remote Plugin Executor (or NRPE for short) is an addon used to execute plugins to monitor "local" resources on remote (Linux/Unix) systems. Some resources cannot (or should not) be monitored via SNMP or using other agents across the network so you have to check them using programs installed locally on the machines to be monitored and transmit the results back to the Icinga server. In contrast to NSCA this is done actively, i.e. initiated by the Icinga server.

[Note] Note

Using NSClient++ instead of NRPE on the remote host you can execute checks on Windows machines as well.

You can use check_by_ssh to execute plugins on remote machines but there is a drawback to this approach. Setting up an SSH session consumes CPU resources on both the local and the remote machine which may become a performance issue if your are monitoring a lot of hosts and/or services this way. Using NRPE is a bit less secure than SSH but in many cases the performance may outweigh the security difference. SSL can be actived though if you need a more secure connection.

Figure 10.1. NRPE

NRPE

check_nrpe is a plugin executed by the local Icinga server like any other plugin. It calls the NRPE process which is running as a daemon on the remote machine. The daemon itself executes the plugin on the same machine and transmits the information gathered back to the check_nrpe plugin which in turn delivers it to Icinga.

[Note] Note

Depending on the CPU / OS of the remote machine you may have to compile NRPE and the plugins on several platforms.

Using NRPE you will mostly monitor resources located on the same machine like CPU load, disk space, memory usage, processes running etc. but it can also be used to check resources which may not be reachable directly from the monitoring server itself. The machine running the NRPE daemon is acting as a relay in this case.

Figure 10.2. NRPE remote

NRPE remote

The following instructions are partially based on documentation found in the original NRPE package by Ethan Galstad.

10.2.2. Prerequisites

  • Icinga should be up and running on the monitoring server

  • a C-compiler (like gcc) is installed on the local host. If not:

     #> yum install gcc       # RHEL / Fedora / CentOS
     #> apt-get install gcc   # Debian / Ubuntu
     #> zypper install gcc    # SLES / openSuSE (or use YaST instead)
  • openssl is (optionally) installed on both the local host. If not:

     #> yum install openssl openssl-devel       # RHEL / Fedora / CentOS
     #> apt-get install openssl openssl-devel   # Debian / Ubuntu
     #> zypper install openssl openssl-devel    # SLES / openSuSE (or use YaST instead)

10.2.3. Download

[Note] Note

Instead of installing NRPE from scratch you may want to use a package which might be available for your OS.

If you are planning to install from source then please use the official release tarball using something like

 #> wget http://sourceforge.net/projects/nagios/files/nrpe-2.x/nrpe-2.15/nrpe-2.15.tar.gz -O nrpe.tgz
 #> tar xzf nrpe.tgz
[Important] Important

Please don't use git snapshots unless you have an issue which might be solved in the current developer version.

10.2.4. Optional changes

The maximum length of data to be transmitted is set to 2,048 bytes, the maximal length of plugin output is set to 512 bytes. If that is not sufficient then you have to alter the appropriate value in nrpe/include/common.h (and recompile NRPE!)

 #define MAX_INPUT_BUFFER        2048    /* max size of most buffers we use */
 #define MAX_PLUGINOUTPUT_LENGTH 512 

Please keep in mind that you have to recompile the programs if you change these values at a later stage.

Due to the setting of the following define in include/common.h (in Icinga core) the max. value cannot exceed 8,192 bytes.

 #define MAX_EXTERNAL_COMMAND_LENGTH     8192   /* max length of an external command */

10.2.5. Compile on the Icinga server

Change to the newly created directory and call configure and make

 #> cd nrpe-2.15
 #> ./configure 
 #> make all
 #> make install-plugin
[Note] Note

If you want to use SSL at a later stage then you have to use "./configure --enable-ssl" instead. There are other options to specify the location of SSL files if they are not found automatically.

If the user or group running the daemon deviate from "icinga" or the port to be used is not the default 5666 you can use several options to specify different values (--with-nrpe-user=<user>, --with-nrpe-group=<group>, --with-nrpe-port=<port>). For a complete list of the options available call "./configure -h". "make install-plugin" will copy check_nrpe to the plugin directory.

[Note] Note

You may want to check if SSL is included using "ldd src/check_nrpe" and "ldd src/nrpe".

10.2.6. First test

Startup the daemon and call the plugin

 #> /usr/src/nrpe-2.15/src/nrpe -n \
    -c /usr/src/nrpe-2.15/sample-config/nrpe.cfg -d
 #> /usr/local/icinga/libexec/check_nrpe -H 127.0.0.1 -n

This should return the version of NRPE. If you receive the message "CHECK_NRPE: Error receiving data from daemon" the monitoring server was not found in nrpe.cfg (directive allowed_hosts). Multiple IP addresses are separated by commas.

Stop the daemon

 #> kill `ps -ef | grep "sample-config/nrpe.cfg" | grep -v grep | awk '{print $2}'`

10.2.7. Remote system(s)

The configuration and installation on the Icinga server is finished so far. The second part has to be done on the remote host(s) running the NRPE daemon which listens for incoming requests, executing them and returning the results to the Icinga server.

10.2.8. Prerequisites on the remote host(s)

  • Make sure that the necessary plugins are available. Take a look at the quickstart guide if you are unsure on how to install them.

  • You can copy the folder nrpe-2.15 including the sub-directories from the Icinga server. One way would be

     #> cd /usr/src/
     #> scp -pr <Icinga-server>:/$PWD/nrpe-2.15 .

    [Note] Note

    If the architecture of your remote host differs from your Icinga server you will have to recompile the sources. This is true if you are using different CPUs (i386/Itanium/PA-RISC/...) and/or different OS versions (32-Bit/64-Bit). If this is the case then you have to install a C-Compiler and OpenSSL (if you want to use SSL) before you can start to compile.

     #> cd nrpe-2.15
     #> make distclean
     #> ./configure     # please use the same options as on the Icinga server 
     #> make all

    Edit the config file sample-config/nrpe.cfg and change the setting of "allowed_hosts=<IP address>" to the IP address of your Icinga server. Multiple IP addresses are separated by commas.

10.2.9. Second test

Startup the daemon on the remote host

 #> /usr/src/nrpe-2.15/src/nrpe -n \
    -c /usr/src/nrpe-2.15/sample-config/nrpe.cfg -d

and execute the plugin on the Icinga server once more, this time using the IP address of the remote host

 #> /usr/local/icinga/libexec/check_nrpe -H <IP remote host> -n

This should return the version of NRPE. If you receive the message "CHECK_NRPE: Error receiving data from daemon" the specified host was not found in nrpe.cfg (directive allowed_hosts) on the remote host.

Stop the daemon on the remote host

 #> kill `ps -ef | grep "sample-config/nrpe.cfg" | grep -v grep | awk '{print $2}'`

10.2.10. Installation on the remote host

Independent from the method the NRPE process is running on the remote host you need a config file containing the commands to be called. You install it issuing

 #> make install-daemon-config

There are two ways to run the nrpe process, one as a standalone daemon, the other using xinetd (which is recommended).

  • nrpe daemon

    First install the daemon

     #> make install-daemon

    If you choose to use xinetd the daemon will be started automatically. Otherwise you have to start the daemon manually

     #> /usr/local/icinga/bin/nrpe -c /usr/local/icinga/etc/nrpe.cfg
  • inetd/xinetd

    If you want the daemon to be started by (x)inetd you have to extend /etc/services, alter/copy another file and restart (x)inetd. If the package is not installed then please do the following

     #> yum install xinetd       # RHEL / Fedora / CentOS
     #> apt-get install xinetd   # Debian / Ubuntu
     #> zypper install xinetd    # SLES / openSuSE (or use YaST instead)
    [Note] Note

    The setting of "server_port" specified in nrpe.cfg is ignored when you use inetd/xinetd.

     #> echo "nrpe 5666/tcp # nrpe" >> /etc/services

    Depending on the superserver which is installed on your system there are three alternatives

    • inetd WITH tcpwrappers

      Add entries to your /etc/hosts.allow and /etc/hosts.deny file to enable TCP wrapper protection for the nrpe service. This is optional, although highly recommended. Add "nrpe stream tcp nowait <user> /usr/sbin/tcpd <nrpe-binary> -c <nrpe-cfg> --inetd" to /etc/inetd.conf, e.g.

       #> echo "nrpe stream tcp nowait icinga /usr/sbin/tcpd /usr/local/icinga/bin/nrpe \
          -c /usr/local/icinga/etc/nrpe.cfg --inetd" >> /etc/inetd.conf
       #> /etc/init.d/inetd restart
    • inetd WITHOUT tcpwrappers

      Add "nrpe stream tcp nowait <user> <nrpe-binary> -c <nrpe-cfg> --inetd" to /etc/inetd.conf, e.g.

       #> echo "nrpe stream tcp nowait icinga /usr/local/icinga/bin/nrpe \
          -c /usr/local/icinga/etc/nrpe.cfg --inetd" >> /etc/inetd.conf
       #> /etc/init.d/inetd restart
    • xinetd (recommended)

      Consider editing the config file nrpe.xinetd in the sample-config folder and replacing the address following <only_from> by the IP address of the Icinga server (where check_nrpe will be running). Multiple IP addresses are separated by spaces.

      Add entries to your /etc/hosts.allow and /etc/hosts.deny file to enable TCP wrapper protection for the nrpe service. This is optional, although highly recommended. Copy the file to your xinetd folder and restart the xinetd process

       #> make install-xinetd
       #> /etc/init.d/xinetd restart

10.2.11. Third test

Switch to the Icinga server, change to the Icinga user and run another test

 #> su - icinga
 $> /usr/local/icinga/libexec/check_nrpe -H <IP remote server>

This should return the version of NRPE another time. If this test fails then there is no sense in continuing. Instead verify the settings in nrpe.cfg/nrpe.xinet on the remote server. Check for messages in your syslog (e.g. /var/log/messages) on the remote server as well.

10.2.12. Troubleshooting

Check that the nrpe process is running on the remote server

  • when installed as standalone daemon

     #> ps -ef | grep -v grep | grep nrpe

    If not, then

    • start it as mentioned above

    • check that the file /usr/local/icinga/etc/nrpe.cfg is present

    • the allowed_hosts directive in the file /usr/local/icinga/etc/nrpe.cfg contains an entry for the IP address of the Icinga server. Multiple IP addresses are separated by commas.

  • when installed using xinetd

     #> netstat -at | grep -v grep | grep nrpe

    The output should be showing something like

     tcp 0 0 *:nrpe *:* LISTEN

    If not then verify that

    • you added the nrpe entry to your /etc/services file

    • the file /etc/xinetd.d/nrpe is present

    • the only_from directive in the file /etc/xinetd.d/nrpe contains an entry for the IP address of the Icinga server. Multiple IP addresses are separated by spaces.

    • xinetd is installed and started

    • the system log files don't show any errors about xinetd and/or nrpe and fix any problems that are reported

Activate "debug=1" in nrpe.cfg, restart the daemon (if applicable) and look for messages in the syslog / nrpe.log.

10.2.13. Security

Read the SECURITY file for more information on the security risks of running NRPE, along with an explanation of what kind of protection the encryption provides you.

10.2.14. Definition of local checks

Some things have been predefined in etc/nrpe.cfg on the remote host

 # command[<command_name>]=<command_line>
 command[check_users]=/usr/local/icinga/libexec/check_users -w 5 -c 10
 command[check_load]=/usr/local/icinga/libexec/check_load -w 1.5,1.1,0.9 -c 3.0,2.2,1.9
 command[check_hda1]=/usr/local/icinga/libexec/check_disk -w 20% -c 10% -p /dev/hda1
 command[check_zombie_procs]=/usr/local/icinga/libexec/check_procs -w 5 -c 10 -s Z
 command[check_total_procs]=/usr/local/icinga/libexec/check_procs -w 150 -c 200 

The first line shows the general format

String Description
command tag showing that the following is a definition for a command
<command_name> link between the command definition on the Icinga server and the command on the remote host
<command_line> call of the plugin including all necessary arguments

10.2.15. Definitions on the Icinga server

Now we switch over to the Icinga server to create some object definitions. First add a command definition to your configuration (unless you already have it). As usual the name of the config file is up to you but most people have a file called commands.cfg.

 define command{
    command_name        check_nrpe
    command_line        $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$
   }

We assume that you already have a host definition like the following

 define host{
    use                 generic-host     ; Inherit default values from a template
    host_name           remotehost       ; The name we're giving to this server
    alias               Linux Host       ; A longer name for the server
    address             192.168.0.1      ; IP address of the server
    }

These example service definitions will use the sample commands shown above.

The following service will monitor the number of currently logged in users on the remote host

 define service{
    use                 generic-service
    host_name           remotehost
    service_description Current Users
    check_command       check_nrpe!check_users
    }

"check_nrpe" is the link between the service directive "check_command" and the directive "command_name" in the command definition on the Icinga server. The "command_line" in the command definition shows that "check_nrpe" is called. "check_users" is passed as the first argument. The nrpe process on the remote host takes this argument and searches for an appropriate definition in nrpe.cfg. The command is executed and the result is transferred back to the check_nrpe-plugin.

The following service will monitor the CPU load on the remote host

 define service{
    use                 generic-service
    host_name           remotehost
    service_description CPU Load
    check_command       check_nrpe!check_load
    }

The following service will monitor the free drive space on /dev/hda1 on the remote host

 define service{
    use                 generic-service
    host_name           remotehost
    service_description /dev/hda1 Free Space
    check_command       check_nrpe!check_hda1
    }

The following service will monitor the total number of processes on the remote host

 define service{
    use                 generic-service
    host_name           remotehost
    service_description Total Processes
    check_command       check_nrpe!check_total_procs
    }

The following service will monitor the number of zombie processes on the remote host

 define service{
    use                 generic-service
    host_name           remotehost
    service_description Zombie Processes
    check_command       check_nrpe!check_zombie_procs
    }

Restart Icinga to include the definitions in your running configuration

 #> /etc/init.d/icinga restart

After some time your plugins should have been called.

10.2.16. More Troubleshooting

Some errors during the initial setup have been mentioned already. Unfortunately you may encounter others errors. Below you'll find hints for some of the more common errors with the NRPE addon.

  • "NRPE: Command timed out after x seconds"

    The command that was run by the NRPE daemon did not finish executing within the specified time. You can increase the timeout for commands by editing the NRPE configuration file and changing the value of the command_timeout variable. Use the -t command line option to specify a longer timeout for the check_nrpe plugin. The following example will increase the timeout to 30 seconds:

     /usr/local/icinga/libexec/check_nrpe -H localhost -c somecommand -t 30

    If you're running the NRPE daemon as a standalone daemon (and not under inetd or xinetd), you'll need to restart it in order for the new timeout to be recognised.

  • "Connection refused or timed out"

    This error can indicate several things:

    • There is a firewall that is blocking the communication between the monitoring host (which runs the check_nrpe plugin) and the remote host (which runs the NRPE daemon). Verify that the firewall rules (e.g. iptables) that are running on the remote host allow for communication and make sure there isn't a physical firewall that is located between the monitoring host and the remote host.

    • If you are using the standalone daemon: The IP address specified in nrpe.cfg (allowed_hosts=...) on the remote server doesn't match the IP address of the monitoring server. If it does then you might have forgotten to restart the daemon after the last change.

    • If you are using the xinetd version: The IP address specified in /etc/xinetd.d/nrpe (only_from=...) on the remote server doesn't match the IP address of the monitoring server. If it does then you might have forgotten to restart the xinetd process after the last change.

    • The NRPE daemon is not installed or running on the remote host. Verify that the NRPE daemon is running as a standalone daemon or under inetd/xinetd with one of the following commands:

       ps axuw | grep nrpe       # if run as standalone daemon
       netstat -at | grep nrpe   # if run via xinetd
  • "CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for an error message."

    First thing you should do is check the remote server logs for an error message. Seriously. :-) This error could be due to the following problem:

    • The check_nrpe plugin was unable to complete an SSL handshake with the NRPE daemon. An error message in the logs should indicate whether or not this was the case. Check the versions of OpenSSL that are installed on the monitoring host and remote host. If you're running a commercial version of SSL on the remote host, there might be some compatibility problems.

  • "NRPE: Unable to read output"

    This error indicates that the command that was run by the NRPE daemon did not return any character output. This could be an indication of the following problems:

    • The path of the plugin to be run is incorrect on the remote host. If you change the definition in nrpe.cfg, remember to restart the daemon.

    • The plugin that is specified in the command line is malfunctioning. Run the command line manually to make sure the plugin returns some kind of text output. DON'T run the command as root!

  • "NRPE: Command 'x' not defined"

    The command 'x' was not defined in the NRPE configuration file on the remote host. Please add the command definition for x. See the existing command definitions in the NRPE configuration file for more information on doing this. If you're running the NRPE daemon as a standalone daemon (and not under inetd or xinetd), you'll need to restart it in order for the new command to be recognised.

If you still have problems then set "debug=1" in nrpe.cfg on the remote host. Remember to restart the NRPE process if it is running as a standalone daemon. Execute the check on the monitoring server. Afterwards you should see debugging information in the syslog (e.g. /var/log/messages) which might help resolving the problem.

You might as well get help using one of the mailing lists or forums (https://www.icinga.com/support/).

10.2.17. Upgrading

  • Upgrading the Icinga server

    Download the software

     #> cd /usr/src
     #> wget "http://sourceforge.net/projects/nagios/files/nrpe-2.x/nrpe-2.15/nrpe-2.15.tar.gz" -O nrpe.tgz
     #> tar xzf nrpe.tgz

    Then compile the software and install the plugin

     #> cd nrpe-2.15
     #> make distclean
     #> ./configure     # use the same options as during the first run
     #> make all
     #> make install-plugin
  • Upgrading the remote host

    Download the software

     #> cd /usr/src
     #> wget "http://sourceforge.net/projects/nagios/files/nrpe-2.x/nrpe-2.15/nrpe-2.15.tar.gz" -O nrpe.tgz
     #> tar xzf nrpe.tgz

    Then compile the software and install the daemon process

     #> cd nrpe-2.15
     #> make distclean
     #> ./configure     # use the same options as during the first run
     #> make all
     ### kill the process if running as standalone daemon
     #> kill `ps -ef | grep "sample-config/nrpe.cfg" | grep -v grep | awk '{print $2}'`
     #> make install-daemon
     ### start the daemon if running as standalone daemon
     #> /usr/src/nrpe/src/nrpe -n \
        -c /usr/src/inrpe/sample-config/nrpe.cfg -d