Maniphest T63788

Add kafka log files to logstash
Open, MediumPublicFeature
Actions

Assigned To

None

Authored By

	drdee
	Feb 22 2014, 1:46 AM

Tags

Referenced Files

None

Subscribers

Description

To help reduce the cycle time of debugging kafka it would be useful to have the following log files stored and processed by logstash:

kafka broker log files from an21 and an22
kafka consumer log files from the edge caches
kafka producer log files

Version: wmf-deployment
Severity: enhancement

Details

Reference: bz61788

	Subject	Repo	Branch	Lines +/-
	logstash: ship kafka server logs to ELK	operations/puppet	production	+11 -0

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		fgiunchedi	T205849 Begin the implementation of Q1's Logging Infrastructure design (2018-19 Q2 Goal)
		Resolved		fgiunchedi	T205852 Onboard at least 10 new non-sensitive log producers to the logging pipeline
		Open	Feature	None	T63779 Add system logs to logstash (tracking)
		Open	Feature	None	T63788 Add kafka log files to logstash

Event Timeline

• bzimport raised the priority of this task from to Medium.Nov 22 2014, 3:03 AM

• bzimport added a project: Wikimedia-Logstash.

• bzimport set Reference to bz61788.

• bzimport added a subscriber: Unknown Object (MLST).

drdee created this task.Feb 22 2014, 1:46 AM

Tgr added a parent task: T63779: Add system logs to logstash (tracking).Feb 11 2017, 1:37 AM

greg moved this task from Backlog to Service Integration on the Wikimedia-Logstash board.Feb 14 2017, 6:52 PM

herron added a parent task: T205852: Onboard at least 10 new non-sensitive log producers to the logging pipeline.Nov 30 2018, 10:16 PM

herron mentioned this in T205852: Onboard at least 10 new non-sensitive log producers to the logging pipeline.

Change 476982 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: ship kafka server logs to ELK

https://gerrit.wikimedia.org/r/476982

gerritbot added a project: Patch-For-Review.Nov 30 2018, 10:36 PM

Change 476982 merged by Herron:
[operations/puppet@production] logstash: ship kafka server logs to ELK

https://gerrit.wikimedia.org/r/476982

Ladsgroup removed a project: Patch-For-Review.May 28 2019, 3:37 PM

fgiunchedi added a project: observability.Aug 19 2019, 2:34 PM

fgiunchedi moved this task from Inbox to Radar on the observability board.Jul 20 2020, 1:29 PM

Aklapper changed the subtype of this task from "Task" to "Feature Request".Feb 4 2022, 12:23 PM

Aklapper removed a subscriber: • wikibugs-l-list.