Page MenuHomePhabricator

Provide Redis feed of recent changes for Wikimedia wikis
Closed, DeclinedPublic

Description

Several tools on Tools and probably some on Labs digest the IRC feed of recent changes for Wikimedia wikis to act on/process/filter/etc.

Recently, the RedisPubSubFeedEngine feed engine was added to MediaWiki. It would be nice to use this to set up a more reliable and supported way to feed recent changes to tools in the Labs/Tools project.

With Redis replication available, one Redis collector would gather all recent changes in the cluster, properly firewalled, and then provide this feed to probably multiple slaves in Labs that can be accessed from any Labs/Tools host. As the bandwidth and CPU cycles already exists for the IRC readers, the infrastructure should be able to handle this.

Apparently, by default RedisPubSubFeedEngine posts all recent changes to a channel "rc". This would need to be prefixed à la "#en.wikipedia".


Version: unspecified
Severity: enhancement

Details

Reference
bz59721

Event Timeline

bzimport raised the priority of this task from to Low.Nov 22 2014, 2:22 AM
bzimport added a project: Cloud-VPS.
bzimport set Reference to bz59721.
bzimport added a subscriber: Unknown Object (MLST).
yuvipanda claimed this task.

RCStream is probably good enough, and do not think we'll have much of a load change from providing a redis proxy.