With the current multi-pass parser it's difficult to fix nesting problems like
these. No nesting state is preserved between doAllQuotes() and
replaceExternalLinks(). Nevertheless, i'm working on another preg_replace hack
that fixes most of these cases.

Target:
''Something [http://www.cool.com cool''] --> <i>Something</i><a
href="http://www.cool.com"..><i>cool></i></a>.

I've got this test passing now:

!! test
Bug 2702: Mismatched <i>, <b> and <a> tags are invalid
!! input
''[http://example.com text'']
[http://test.com '''text]'''
''Something [http://test.com in italic'']
''Something [http://test.com mixed''''', even bold]'''
'''''Now [http://test.com both''''']
!! result
<p><a href="http://example.com" class='external text' title="http://example.com"
rel="nofollow"><i>text</i></a>
<a href="http://test.com" class='external text' title="http://test.com"
rel="nofollow"><b>text</b></a>
<i>Something </i><a href="http://test.com" class='external text'
title="http://test.com" rel="nofollow"><i>in italic</i></a>
<i>Something </i><a href="http://test.com" class='external text'
title="http://test.com" rel="nofollow"><i>mixed</i><b>, even bold</b></a>
<i><b>Now </b></i><a href="http://test.com" class='external text'
title="http://test.com" rel="nofollow"><i><b>both</b></i></a>
</p>
!! end

Committed (code is active if tidy is disabled), closing therefore.

Change 297930 had a related patch set uploaded (by Tim Starling):
Don't run the non-Tidy "bug 2702" hack unless Tidy is really missing

https://gerrit.wikimedia.org/r/297930