There's two separate use cases here:

1. Someone in -operations !log'ing something to the SAL. like "!log I'm going to shutdown tampa, watch out!"

2. Various scripts/tools on the cluster that log things to the SAL eg scap or git-deploy.

Can we decouple these two use cases? It seems that passing #2 over an IRC server is bad design for all of the reasons stated.

Proposal:

• Use case #1 (logmsgbot) should log to Logstash directly in addition to the SAL (initially).
• Use case #2 (scap, trebuchet, etc) should log to Logstash directly.
• Add support to logmsgbot to announce log entries from Logstash that originated there.
• We get rid of the wikitech SAL and create a nice looking public Logstash view.
• Use case #1 no longer logs to SAL.

No more stupidly large edit histories!

(In reply to Greg Grossmeier from comment #1)

• We get rid of the wikitech SAL and create a nice looking public Logstash

view.

This is probably hard given the lack of ACLs in Logstash.

Potentially we could point Wikitech to the new SAL system ( https://tools.wmflabs.org/sal/ ) as implemented via T63544: Add !log entries from #-operations to logstash.

It is just about changing the wikitech sidebar, or adding a new link.

I have updated wm-bot !sal helpers in the ops and releng channels:

#wikimedia-operations:
<hashar> !sal
<wm-bot> https://labsconsole.wikimedia.org/wiki/Server_Admin_Log https://tools.wmflabs.org/sal/production See it and you will know all you need.

#wikimedia-releng
<+hashar> !sal
<wm-bot> https://tools.wmflabs.org/sal/releng

Don't we want log entries to be backed up to wikitech-static and not just be kept in labs though?

Let's think through the failure modes we're trying to guard against:

Bad things

1. Production wikis all returning blank pages/HHVM everywhere falls over/Varnish dies
2. All of our datacenter uplinks get backhoe'd at the same time
3. what else?

Responses

1. is usually a bad deploy, but either way, toollabs should be up as well (and would be a perfect storm if it wasn't)
2. I don't think we'd look at the SAL for this :)
3. ...

In T46791#1567194, @Krenair wrote:

Don't we want log entries to be backed up to wikitech-static and not just be kept in labs though?

*If* we decide that the https://tools.wmflabs.org/sal/ is the nicest way to look at SAL data, we should certainly add some backup system for the Elasticsearch system that powers it. One relatively easy option would be to use a tool like https://github.com/taskrabbit/elasticsearch-dump to copy the index to a data file on a regular basis and archive that dump somewhere outside of the stashbot Labs project.

Mentioned in SAL [2016-06-09T00:06:53Z] <mutante> meta T46791

In T46791#480052, @greg wrote:

There's two separate use cases here:

1. Someone in -operations !log'ing something to the SAL. like "!log I'm going to shutdown tampa, watch out!"

2. Various scripts/tools on the cluster that log things to the SAL eg scap or git-deploy.

Yes, we should I have a straw man proposal here. I think it'll solve the original task issue and works in both a wikitech and/or logstash scenario.

• logmsgbot as it exists goes away. It's replaced with a service (http/restful, I'm guessing).
  • This service could post to wikitech, or logstash, both, or whatever. This protects clients against future backend changes
• Scap, other command line tools in production would just call this directly. Skip IRC or the IRC log is supplemental.
• The !log functionality for humans can remain, but either as a simpler logmsgbot or (more likely/useful) subsumed by wm-bot. Said bot would just call the same service
• [BONUS] scap [log|sal] could do the same logging as IRC. This is great for A) Avoiding context switching when you're on a machine, and B) Fallback for SAL when IRC is busted

Some related thoughts/explanations on T156079: autolog scap3 deployments in beta.

wm-bot already does too many things IMO, and Stashbot currently owns everything beyond the message getting placed by a human or a bot on irc. Stashbot is in gerrit and currently running in a Kubernetes pod. When we get a prod k8s environment it shouldn't be too hard to move it from tool labs to prod and add an HTTP and/or TCP API to make automated log submissions easier.

In T46791#3023711, @bd808 wrote:

Some related thoughts/explanations on T156079: autolog scap3 deployments in beta.

wm-bot already does too many things IMO, and Stashbot currently owns everything beyond the message getting placed by a human or a bot on irc. Stashbot is in gerrit and currently running in a Kubernetes pod. When we get a prod k8s environment it shouldn't be too hard to move it from tool labs to prod and add an HTTP and/or TCP API to make automated log submissions easier.

So you're right re stashbot vs wm-bot. Stashbot should do this. Couple of notes from our IRC discussion:

• An API from stashbot enables us to easily fix the production issue for l10nupdate, scap and other tools
  • And hides logstash/wikitech questions from callers
  • My scap [log|sal] suggestion as a fallback scenario also easily done as well (actually that can happen now, it's already abstracted enough)
• Stashbot can relay the message back to IRC from entries submitted to API. This allows callers to replicate their messages for IRC consumers while not having to speak IRC themselves (use like a query parameter though, some/all messages will/won't want to go to IRC. ie: caller should decide)

In T46791#3023722, @demon wrote:

• My scap [log|sal] suggestion as a fallback scenario also easily done as well (actually that can happen now, it's already abstracted enough)

This was easy. {D567}

I don't really see why my team was tagged with this.