Page MenuHomePhabricator

Elasticsearch ganglia monitor broken for Logstash due to lack of stats groups
Closed, ResolvedPublic

Description

The kibana interface doesn't tag searches with stats groups [0] and the ganglia monitoring script assumes otherwise:

[PYTHON] Can't call the metric_init function in the python module [elasticsearch
_monitoring].

Traceback (most recent call last):

File "/usr/lib/ganglia/python_modules/elasticsearch_monitoring.py", line 511,

in metric_init

for group in load(url)['_all']['total']['search']['groups']:

KeyError: 'groups'

When queried on the logstash elasticsearch nodes the relevant section of the _stats/search request looks like this:

"_all" : {
  "primaries" : {
    "search" : {
      "open_contexts" : 0,
      "query_total" : 10317,
      "query_time" : "1.6h",
      "query_time_in_millis" : 5915635,
      "query_current" : 0,
      "fetch_total" : 7731,
      "fetch_time" : "23.5s",
      "fetch_time_in_millis" : 23525,
      "fetch_current" : 0
    }
  },
  "total" : {
    "search" : {
      "open_contexts" : 0,
      "query_total" : 19499,
      "query_time" : "3.1h",
      "query_time_in_millis" : 11201005,
      "query_current" : 0,
      "fetch_total" : 14517,
      "fetch_time" : "42.8s",
      "fetch_time_in_millis" : 42819,
      "fetch_current" : 0
    }
  }
}

[0]: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search.html#stats-groups


Version: wmf-deployment
Severity: normal

Details

Reference
bz61384

Event Timeline

bzimport raised the priority of this task from to High.Nov 22 2014, 2:55 AM
bzimport set Reference to bz61384.

Nik says that it should be fine to just skip these metrics if not present in the data. Should be a pretty simple "if 'groups' in ..." check to add.

Change 113471 had a related patch set uploaded by BryanDavis:
Make elasticsearch ganglia monitor compatible with logstash

https://gerrit.wikimedia.org/r/113471

Change 113471 merged by Ori.livneh:
Make elasticsearch ganglia monitor compatible with logstash

https://gerrit.wikimedia.org/r/113471