Page MenuHomePhabricator

Create puppetized logstash cluster in beta project
Closed, ResolvedPublic

Description

The QA team is starting to use logstash as an error analysis tool. They have a twice-daily cron that sends an email to the QA mailing list when there have been new fatals. That email includes a link to logstash.wmflabs.org.

Currently that logstash instance is a single node in the logstash project that is running the initial hand built prototype. This should be changed so that the beta project has it's own prod-like logstash cluster at logstash.beta.wmflabs.org. That cluster should use the production puppet classes via a custom role.


Version: wmf-deployment
Severity: enhancement

Details

Reference
bz61753

Event Timeline

bzimport raised the priority of this task from to High.Nov 22 2014, 3:01 AM
bzimport set Reference to bz61753.
bzimport added a subscriber: Unknown Object (MLST).

If there is nothing hardcoded in the logstash puppet module, that is indeed just about calling the classes with the proper parameters (ex: vary IP addresses and domain name).

Looking at role::logstash, the elasticsearch class is passed a multicast parameter and AFAIK there is no multicast support in labs. The rest might just work :-]

The beta cluster logstash instances should be created in the eqiad datacenter since pmtpa is being phased out.

Logstash is now deployed to deployment-logstash1.eqiad.wmflabs using puppet and trebuchet. The new instance is accessible at https://logstash-beta.wmflabs.org/