Page MenuHomePhabricator

Contributions from major UK ISPs being assigned to the same two IP addresses
Closed, ResolvedPublic

Description

Author: matthew.britton

Description:
It seems that all customers of Virgin Media are being assigned the IP address 62.30.249.131 ‎when they edit Wikipedia. Similarly, all customers of Be Unlimited are being assigned the IP address 89.167.221.3.

This produces all the usual problems associated with shared IP addresses on a much exaggerated scale.

I strongly suspect the problem lies with MediaWiki. That the same thing has happened with both these major ISPs at the same time suggests that the error does not lie with their configuration. Additionally, customers of these ISPs are not all on a shared IP address at all. My IP address is currently 87.194.147.203, as far as everyone except Wikimedia is concerned, yet I am affected by autoblocks and direct blocks on 89.167.221.3. Other users are reporting the same thing.


Version: unspecified
Severity: major

Details

Reference
bz16569

Event Timeline

bzimport raised the priority of this task from to Medium.Nov 21 2014, 10:29 PM
bzimport set Reference to bz16569.
bzimport added a subscriber: Unknown Object (MLST).

Someone affected please hit this page:

http://leuksman.com/headers.php

and paste in the complete set of headers to check the IP and XFF headers. Could just be that these ISPs instituted a transparent proxy recently.

matthew.britton wrote:

REMOTE_ADDR: 87.194.147.203
Headers: Array
(

[Host] => leuksman.com
[User-Agent] => Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.4) Gecko/2008102920 Firefox/3.0.4 (.NET CLR 3.5.30729)
[Accept] => text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
[Accept-Language] => en,en-us;q=0.7,en-gb;q=0.3
[Accept-Encoding] => gzip,deflate
[Accept-Charset] => ISO-8859-1,utf-8;q=0.7,*;q=0.7
[Keep-Alive] => 300
[Connection] => keep-alive

)

duncan.hill1 wrote:

Is this what you mean Brion?

REMOTE_ADDR: 86.8.176.36
Headers: Array
(

[User-Agent] => Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.19 (KHTML, like Gecko) Version/3.1.2 Safari/525.21
[Accept-Encoding] => gzip, deflate
[Accept] => text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
[Accept-Language] => en-US
[Connection] => keep-alive
[Host] => leuksman.com

)

matthew.britton wrote:

In case it is useful, here is a link to an edit I just logged out and made, showing 89.167.221.3 rather than my 'real' IP address as posted above:

http://en.wikipedia.org/w/index.php?title=Wikipedia:Sandbox&diff=256074711&oldid=256073746

There appears to be some form of transparent proxy in place in Virgin/NTL's network for all traffic routed through it, including transit customers. Possibly only for destination rr.knams.wikimedia.org. When one subjected network (Virgin transit customer) routed around Virgin, the problem disappeared.

Why there is another, seemingly unrelated network showing similar behaviour, I don't know. It's very unlikely to be a problem on our end, but why this is happening remains unclear.

[Note that this mystery proxy does not appear to be sending us X-Forwarded-For headers, so we can't display the "real" source IP address.]

johannes.aquila wrote:

I am a Virgin Media customer in Leeds and affected by this problem. I have the following additional problem:

Once logged in everything seems to work fine, except I can't edit. After clicking "Save page", nothing happens, until after a timeout my browser tries to download a file "index.php".

Other Virgin Media users do not have this problem, see http://en.wikipedia.org/wiki/User_talk:Stwalkerster#Side-effect_of_your_block_of_User_talk:62.30.249.131

Workaround for all problems: Use the secure server.

matthew.britton wrote:

(In reply to comment #5)

Why there is another, seemingly unrelated network showing similar behaviour, I
don't know. It's very unlikely to be a problem on our end, but why this is
happening remains unclear.

89.167.221.3 seems to be used by customers of Be Unlimited, owned by Telefónica Europe (aka O2), perhaps some of their traffic is being routed through the same network.

If it helps, I've found another IP address -- 212.134.155.210 -- that is exhibiting the same thing (no traffic before today, now lots of unrelated contributions from different people), which is probably part of the same problem.

matthew.britton wrote:

Just adding for completeness: 62.24.251.240 (also Virgin Media)

mike.lifeguard+bugs wrote:

(In reply to comment #7)

I am a Virgin Media customer in Leeds and affected by this problem. I have the
following additional problem:

Once logged in everything seems to work fine, except I can't edit. After
clicking "Save page", nothing happens, until after a timeout my browser tries
to download a file "index.php".

Other Virgin Media users do not have this problem, see
http://en.wikipedia.org/wiki/User_talk:Stwalkerster#Side-effect_of_your_block_of_User_talk:62.30.249.131

Workaround for all problems: Use the secure server.

That doesn't sound like a related issue to me, mostly because other users are not having this problem.

matthew.britton wrote:

From en.wikipedia administrators' noticeboard:

Actually, this is where UK users of certain ISPs (at least UK Online) get proxied trough when a site has been flagged by the UK Internet Watch Foundation http://www.iwf.org.uk At least, i think so, I googled a bit for the IP and found the following postings http://bbs.adslguide.org.uk/showthreaded.php?Cat=&Board=ukonline&Number=3474948&page=0&view=expanded&sb=5&o=0 http://bbs.adslguide.org.uk/showflat.php?Cat=&Board=ukonline&Number=3462429&page=0&view=expanded&sb=5&o=0&fpart=all&vc=1(BIG). It seems wikipedia got flagged for childpornography. The other IP addresses might be the filters of other ISPs. --TheDJ (talk • contribs) 00:31, 6 December 2008 (UTC)

This explains a lot if true; we seem to have multiple providers all simultaneously setting up a transparent proxy on Wikimedia, and only Wikimedia. In a way I hope it's not true because it means a media shitstorm, but... meh. Someone ought to contact, er, whoever the relevant authorities are.

duncan.hill1 wrote:

I have emailed Mike Godwin to inform him of the thread at AN.

duncan.hill1 wrote:

Mike Godwin has told me by email (quoted with permission) that "Wikimedia Foundation hasn't been notified of any attempt to block our content as a result of anything having to do with Internet Watch."

matthew.britton wrote:

(In reply to comment #13)

Mike Godwin has told me by email (quoted with permission) that "Wikimedia
Foundation hasn't been notified of any attempt to block our content as a result
of anything having to do with Internet Watch."

It seems, alas, that they don't notify organizations when they block their content. Nor can I find anywhere on their website where I can report a falsely blocked site, only report inappropriate content.

george wrote:

I have emailed the IWF, and will continue to hound them through various media until they respond.

matthew.britton wrote:

The transparent proxies have been put in place in order to allow the ISPs to selectively censor Wikipedia pages.

Currently the pages [[Virgin Killer]] and [[Image:Virgin Killer.jpg]] are censored for myself and users of the other UK ISPs mentioned. The technical details vary; in my case, I get a fake 404 page.

george wrote:

And I get a blank page or one with only the URL, variously.

matthew.britton wrote:

However, the link http://en.wikipedia.org/w/index.php?title=Virgin_Killer works perfectly. So they've taken a bit of a half-hearted approach to it, proving yet again that censorship does not work.

craig9 wrote:

REMOTE_ADDR: 87.194.239.140
Headers: Array
(

[Host] => leuksman.com
[User-Agent] => Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-GB; rv:1.9.0.4) Gecko/2008102920 Firefox/3.0.4
[Accept] => text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
[Accept-Language] => en-gb,en;q=0.5
[Accept-Encoding] => gzip,deflate
[Accept-Charset] => ISO-8859-1,utf-8;q=0.7,*;q=0.7
[Keep-Alive] => 300
[Connection] => keep-alive

)

majorly.wiki wrote:

There's no image on that page. It's the image that's the problem.

t1cr4m wrote:

Only the rr.knams.wikimedia.org IP is routed through the proxy, so leuksman.com won't see headers coming from the proxy. Would it be possible to set up a header viewing php on http://test.wikipedia.org or something?

cgranade wrote:

The IWF has openly admitted to providing an unnamed URL from Wikipedia to UK ISPs for blacklisting: http://www.iwf.org.uk/media/news.249.htm. There's some ongoing discussion of this over on Hacker News: http://news.ycombinator.com/item?id=388300.

that.man.colin wrote:

(In reply to comment #0)

It seems that all customers of Virgin Media are being assigned the IP address
62.30.249.131 ‎when they edit Wikipedia. Similarly, all customers of Be
Unlimited are being assigned the IP address 89.167.221.3.

This produces all the usual problems associated with shared IP addresses on a
much exaggerated scale.

I strongly suspect the problem lies with MediaWiki. That the same thing has
happened with both these major ISPs at the same time suggests that the error
does not lie with their configuration. Additionally, customers of these ISPs
are not all on a shared IP address at all. My IP address is currently
87.194.147.203, as far as everyone except Wikimedia is concerned, yet I am
affected by autoblocks and direct blocks on 89.167.221.3. Other users are
reporting the same thing.

Hi there, Not a techie but what would happen if millions of outraged users reported say the virgin, O2 or even the IWF home page to the IWF as potentially offensive. Would the response be automatically to redirect to an unnamed Virgin, O2 or IWF URL. Would this then be akin to them mounting their own "denial of service" attack upon themselves?

haza.wiki wrote:

(In reply to comment #16)

The transparent proxies have been put in place in order to allow the ISPs to
selectively censor Wikipedia pages.

Currently the pages [[Virgin Killer]] and [[Image:Virgin Killer.jpg]] are
censored for myself and users of the other UK ISPs mentioned. The technical
details vary; in my case, I get a fake 404 page.

I'm currently connecting via a Demon UK connection, and am getting the following results:

http://en.wikipedia.org/wiki/Virgin_Killer - page OK.

http://en.wikipedia.org/w/index.php?title=Virgin_Killer - page OK.

http://en.wikipedia.org/wiki/Image:Virgin_Killer.jpg - 302 redirect to Demon/IWF "403" error (http://iwfwebfilter.thus.net/error/blocked.html)

http://en.wikipedia.org/w/index.php?title=Image:Virgin_Killer.jpg - page OK.

http://upload.wikimedia.org/wikipedia/en/3/33/Virgin_Killer.jpg - image OK.

(In reply to comment #22)

Only the rr.knams.wikimedia.org IP is routed through the proxy, so leuksman.com
won't see headers coming from the proxy.

Right, we've confirmed this already. :)

Would it be possible to set up a
header viewing php on http://test.wikipedia.org or something?

Currently we're just periodically checking logs. Proxies which are sending correct headers are being tracked and added to our accept list, but many people are behind those which aren't.

Per http://www.iwf.org.uk/media/news.251.htm the 'Virgin Killer' album cover article page is being removed from the IWF's filter list, so the proxy problems should start phasing out as ISPs update their copies of the list.

If we get hit again, at least some portion of the ISP proxies which are configured correctly are now in our lists for handling X-Forwarded-For headers. Hopefully the other ISPs will fix their proxies... :P