The search indexer instance attempts to reach the *.beta.wmflabs.org which is pointing to a public IP part of labs. That does not work.
A hacky quick solution would be to rewrite any request sent from the search indexer for the squid public IP (208.80.153.219) to use the internal squid instance private IP.
Version: unspecified
Severity: minor
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | hashar | T47868 [OPS] let instances access *.beta.wmflabs public IP (NAT issue in labs) | |||
| Resolved | None | T51300 Internal cross-wiki api request fails on wikidata beta deployment |
Iptable rule would be:
iptables -t nat -I OUTPUT --dest 208.80.153.219 -j DNAT --to-dest 10.4.0.17
I have no idea how to puppetize the iptables rule mentionned in comment #1. So I have filled RT #4824 that list the mail exchanges on ops mailing list.
same issue happens on deployment-upload.pmtpa.wmflabs which is an internal proxy for thumbnails generation.
The text cache has been migrated out of deployment-squid [10.4.0.17] to a varnish instance deployment-cache-text1 [10.4.1.133]
The iptables command is thus:
iptables -t nat -I OUTPUT --dest 208.80.153.219 -j DNAT --to-dest 10.4.1.133
(In reply to comment #5)
same issue happens on deployment-upload.pmtpa.wmflabs which is an internal
proxy for thumbnails generation.
That was unrelated. The thumb handler points directly to the varnish cache via its private IP.
Rephrasing summary.
Wikidata is hit by the same issue (was bug 49300) when some script attempt to access:
http://en.wikipedia.beta.wmflabs.org/w/api.php?action=query&prop=info&redirects=1&converttitles=1&format=json&titles=Keyboard+Cat
The RT is https://rt.wikimedia.org/Ticket/Display.html?id=4824
The workaround is to use an iptables rule to rewrite networking packet:
iptables -t nat -I OUTPUT --dest 208.80.153.219 -j DNAT --to-dest 10.4.1.133
The iptables command for all the beta public IP:
iptables -t nat -I OUTPUT --dest 208.80.153.219 -j DNAT --to-dest 10.4.1.133
iptables -t nat -I OUTPUT --dest 208.80.153.242 -j DNAT --to-dest 10.4.0.211
iptables -t nat -I OUTPUT --dest 208.80.153.243 -j DNAT --to-dest 10.4.0.51
iptables -t nat -I OUTPUT --dest 208.80.153.244 -j DNAT --to-dest 10.4.0.48
iptables -t nat -I OUTPUT --dest 208.80.153.243 -j DNAT --to-dest 10.4.1.82
https://gerrit.wikimedia.org/r/#/c/101192/ converts the above iptables rules to ferm. They can be applied on instances using the puppet class role::beta::natfixup.
I have applied the class on the following instances:
deployment-apache32
deployment-apache33
deployment-bastion
deployment-jobrunner08
deployment-parsoid2
deployment-video06
Change 101209 had a related patch set uploaded by Hashar:
beta: ferm on appservers must allow port 80
Change 101210 had a related patch set uploaded by Hashar:
role::parsoid::beta must allow port 8080