Puppet, Zabbix and Elasticsearch - elastizabbix Module

I'm happy to introduce a Puppet module which is about three of five my favorite server products: Elasticsearch, Puppet and Zabbix. Just for your information I'll tell that remaining two are Cassandra with DataStax and Atlassian suite.

elastizabbix module installs elastizabbix Zabbix agent script and Zabbix template, allowing us to monitor Elasticsearch cluster in Zabbix. It is an open source module, published under Apache 2.0 license, and managed at Github (contributors are welcome). Talking about the license, it is important to say that the module itself is published under Apache 2.0 license (as already mentioned), while the original elastizabbix project is published under MIT License. One way or another, you are pretty much free to use the code in a way you want. The module is also published at puppet forge.

Requirements

The module requires presence of puppet/zabbix module on both server and client side. At the server side the module requires that zabbix class is used (implemented), and at the client side it requires that zabbix::agent class is used (implemented).

At the client side python 2 or 3 is also required. But since it usually comes preinstalled with the OS - there's nothing you need to do.

Up and Running

Server Side

The module is very simple to use. At the server side (Zabbix server) you should add the following:

class { 'elastizabbix::server': }

Actually, you don't have to add this class at all. The only thing that the class do is trying to install Zabbix template, so you can accomplish the same effect by manually importing the template through Zabbix interface. The name of the template file is templates_app_elasticsearch.xml and you can find it in files folder of the module (as well as in Github repositories of the original elastizabbix project and elastizabbix puppet module project).

You should know that if you are using zabbix class with manage_resources set to true, elastizabbix::server class will automatically install the template in Zabbix server. Before including this option please check the original puppet/zabbix module documentation, especially part that deals with exported resources. Personally, I think that you should not use this option. If this option is not used, then you'll have to manually import the template through Zabbix interface. In this case elastizabbix::server class will prepare the template for you and store it in /tmp folder on Zabbix server. But again, if you are not using manage_resources option - elastizabbix::server class is not too useful since you can find the same template at several other places mentioned above.

Client Side

At the client side (one node from Elasticsearch cluster) you should add the following:

class { 'elastizabbix': }

Linking the Template

Of course, in Zabbix you should link the template with the host where elastizabbix class is implemented.

That's it. You should be able to monitor Elasticsearch cluster in Zabbix.

Parameters

Usually you'll use both classes (elastizabbix and elastizabbix::server) without specifying any parameters. Nevertheless, here I'll explain available parameters, in case you ever need them.

elastizabbix::server Class' Parameters

elastizabbix Class' Parameters

Troubleshooting

If something does not work as expected, there are few things you should check, in the order described in this section.

1. Template Linking

The first thing to check is if the template is linked to the appropriate host in Zabbix. Again, the template should be linked to the host where elastizabbix class is installed. You can check this in Zabbix by selecting Configuration -> Hosts -> the appropriate host (i.e. ESNODE01). When the node is selected, click "Templates" tab and ensure that "Template App ElasticSearch" is among templates enumerated in "Linked templates" box.

When you've ensured that the template is linked correctly, you can go to Monitoring -> Latest data, and then filter to the host of interest (i.e. ESNODE01). Below you should be able to find "Cluster" group, and "Cluster status" item among other items from the group. If you can see the item it means that the template is linked correctly. "Cluster status" item should have some value (green, yellow, red). If you can see any of these value, it means that everything is set correctly. If the value is blank, it means that the client script haven't sent the value, so you can check configuration at the node itself. Note that by default these values are updated once per minute, so give it enough time.

2. Checking elasticsearch_url

The next thing to check is if provided elasticsearch_url value is valid. To check this login to monitored node and execute something like:

$ curl http://localhost:9200/_cluster/health

If the URL is OK you should get some JSON response like:

{"cluster_name":"IT-ES","status":"green","timed_out":false, ...

If the URL is not OK you'll get something like:

curl: (7) Failed to connect to localhost port 9200: Connection refused

So if URL is not OK you'll either change elasticsearch_url appropriately, or change Elasticsearch cluster configuration.

3. Testing elastizabbix Script

The third thing to check is if the installed script works. Ensure that the script is installed in the first place - check if the following file exists: /etc/zabbix/externalscripts/elastizabbix. If the file is not there, check if you've implemented elastizabbix class for this host, and run puppet agent again.

Before checking if the script works you should delete cache files (if any). The cache files are stored in /tmp directory, and they are named elastizabbix-[something].json, for example /tmp/elastizabbix-health.json. Delete these files. After the cache files are deleted execute the following:

$ /etc/zabbix/externalscripts/elastizabbix health status

If the script works as expected you should get "green" (or some other color). If not - you'll get some error message.

If the script does not work as expected check if python is installed by executing python --version. If python is installed you should get something like Python 2.7.6. If not, please install python version 2 or 3.

4. Nothing Helped

If neither of the above steps helped - give up. It simply isn't meant to be. It's not your destiny to use elastizabbix.  

Just kidding! The next thing you can try is to describe your problem in comments below, and I'll jump in and help.