Like any service or component in your architecture, you’ll want to monitor it to ensure that it’s available and gather performance data to help with tuning.
In this post, we’ll look at how we can monitor Solr, what performance metrics we might want to gather and how we can easily achieve this with Opsview.
A check list for service checksSolr is built on Lucene so follows the same layout, an index contains documents that are comprised of fields. As part of the search service value add over Lucene, Solr provides a number of useful ways of obtaining health status / monitoring metrics:
- Health-check status using the /admin/ping handler
- The admin statistics page /admin/stats.jsp (XML styled with XSL)
- JMX MBeans
Instead the list is divided into the checks that can be performed remotely (without an installed agent on the server) and those that are best performed locally to the Solr server.
Remote (agent-less) checksWhat should we look for over the network?
Firstly we can have a host-level check which may perform a network level ping. Next we can check TCP connectivity to the servlet container port and then make an HTTP GET request to the Solr ‘front page’ and check for a known string (e.g. Welcome to Solr).
Now we’ve made it up to the application layer so can start to perform Solr specific checks. Items to monitor may include (delete as applicable):
- Ping status
- Number of docs
- Number of queries / queries per second
- Average response time
- Number of updates
- Cache hit ratios
- Replication status
- Synthetic queries
Agent-based checksInstalling an Opsview agent on the Solr server means we can run additional checks over NRPE (Nagios Remote Plugin Executor). This could be operating system level checks such as memory/disk utilisation or CPU load, or the following:
- Java servlet container process is running
- JMX checks e.g. heap memory or custom MBeans
- File age
- Log parsing for exceptions
Opsview configurationFor the rest of this article you'll need to have Opsview installed (or the Opsview VMWare appliance) and have completed the Quick Start.
Solr-specific PluginInstall the Solr plugin at https://github.com/rbramley/Opsview-solr-checks into /usr/local/nagios/libexec/
The check_solr plugin was developed using Perl, so that it could be contributed back to Opsview. It requires the CPAN XML::XPath module (sudo cpan -i XML::XPath).
The plugin includes usage instructions, check_solr -h which can also be viewed in Opsview by selecting the ‘Show Plugin Help‘ link beneath the Plugin drop down (see Figure 1). The -u option can be used to specify the URL path for multi-core set-ups.
Service check setupFigure 1 gives an example of a service check configuration.
Figure 2 shows the agentless service check group with plugins and their arguments.
Host configurationFigure 3 shows a simplistic host setup with a ping check.
Figure 4 is an extract from the Monitors tab, where we select the checks we want performed for for the current host.
Viewing outputThe check results shown in Figure 5 are visible by navigating through the host group hierarchy.
If you click on the graph icon of Solr Cache Hit Ratios this will drill down onto the graph shown in Figure 6. Clicking on the graph icon for Solr Avg Response Time - standard will take you to the graphs in Figure 7.
If you shutdown Solr, then the check results will start to turn critical and show in red as per Figure 8.
AlternativesThere are a few other plugins available for monitoring Solr from Opsview, depending on your needs:
- http://code.google.com/p/nagios-plugins-shamil - provides ping, replication status and num docs
- http://code.google.com/p/solr-nagios-check - provides QPS, response time and num docs