Wednesday, April 20, 2011

SolrMeter - Benchmarking and Monitoring for Solr

Recently we started using SolrMeter for simulating production queries on our test Solr servers. We quickly found out that the tool can be used for more than just simulating queries. It gives very interesting view of your Solr health including graphs of caches, query performance, etc. In order to feed it a good set, you need to cleanse your production Solr request logs to get the select statements only. SolrMeter expects the queries in a certain format. More precisely if your query were
http://localhost:8080/solr/select?q=type:document AND name:SomeName
SolrMeter expects type:document AND name:SomeName in the text file.

To grab and cleanse the production logs, I built a simple utility in .NET that did some regex to extract the required text from the log file. It is usually the text between the ? and &? in the Solr request log. I then fed this newly generated cleansed log file to SolrMeter, set the location of the solr instance, set the query throughput rate (in minutes) and voila! You have a stress test tool ready!

1 comment:

  1. Yash, that looks like a really useful tool. Something else you may find interesting is New Relic:

    You can monitor a production instance with no noticeable impact. We use New Relic in my client's environment and have found it to be an invaluable and indispensable tool to diagnosing production issues. The setup is super simple and the amount of information you get from New Relic is phenomenal. I highly recommend checking it out when you get the chance.