Page MenuHomePhabricator

Icinga has httpauth on (not accessible for public)
Closed, DeclinedPublic

Description

Just noticed that Icinga apparently has httpauth on, so it's not accessible for me (the public).
Is there a security issue or other reason that forced this?

also http://status.wikimedia.org/ reports it as down for several days.


Version: wmf-deployment
Severity: normal
URL: https://icinga.wikimedia.org/icinga
See Also:
https://bugzilla.wikimedia.org/show_bug.cgi?id=54713
https://rt.wikimedia.org/Ticket/Display.html?id=6838

Details

Reference
bz60112

Event Timeline

bzimport raised the priority of this task from to Medium.Nov 22 2014, 2:58 AM
bzimport set Reference to bz60112.
bzimport added a subscriber: Unknown Object (MLST).

Logging in works for me with my a Labs / wikitech.wikimedia.org account, but that might just be because I'm in a specific LDAP group, like bug 54713.

(In reply to comment #1)
yep, logging in with wikitech-acc doesn't work for me.

Basically all I expect as answer here is a information why it currently on and when it is expected to be disabled again.
(icinga is on neon and this has nothing to do with graphite's apparently pending security review, right? bug 54713#c5)

RobH said in #wikimedia-operations that "there are security issues with icinga iirc".

Ok. So we used to have Nagios which anyone could have a look at to see what's wrong. Someone decided to switch to another tool (Icinga). Now it turns out that that tool has security issues and public access got disabled? Way to go.....

It's been so since December. Originally I understood it was a matter of days...

2013-12-20 12.31 < whym> icinga.wikimedia.org now requirs authorization from me. Is this how it's intended to be?
2013-12-20 12.39 < paravoid> whym: there are a couple of security vulnerabilities for icinga in the wild, so we've temporarily locked public access

https://gerrit.wikimedia.org/r/#/c/100989/

(In reply to comment #4)

Ok. So we used to have Nagios which anyone could have a look at to see what's
wrong. Someone decided to switch to another tool (Icinga). Now it turns out
that that tool has security issues and public access got disabled? Way to
go.....

IIRC nagois had security issues as well.

Yes, there are security issues with Icinga that forced us to lock it down temporarily back in December 12th.

These are CVE-2013-7106, CVE-2013-7107 & CVE-2013-7108. They are still unfixed in Ubuntu precise (LTS); Icinga is in the universe section, so the Ubuntu security team deals with them on a "best effort" basis (i.e. they might not even update it, at all).

The vulnerability status per Ubuntu distribution can be tracked at:
http://people.canonical.com/~ubuntu-security/cve/2013/CVE-2013-7106.html
http://people.canonical.com/~ubuntu-security/cve/2013/CVE-2013-7107.html
http://people.canonical.com/~ubuntu-security/cve/2013/CVE-2013-7108.html
respectively. Note how they decided to ignore the first one (a CSRF), which shows IMHO a poor judgement from their part.

I don't think we can take the time to do a major Icinga version upgrade right now, nor to backport the fixes ourselves. Our current strategy is "wait for Ubuntu", but if anyone wants to help the backporting process (and optionally engage with the Ubuntu security team so others can benefit from that) that'd be awesome.

Created attachment 14586
Backport fix for CVE-2013-7106.

Attached:

Created attachment 14587
Backport fix for CVE-2013-7108.

Attached:

Created attachment 14588
Backport fix for CVE-2013-7107.

Attached:

Hey, that's good stuff! Thanks! Would you mind terribly contacting the Ubuntu security team to offer these code backports? Their usual response is "you're on your own", but if you attach code they might treat it differently, who knows :)

No, I don't mind, but I need to test it first at least once :-). I've asked petan for access to the Nagios project on Labs, will set up a new instance there and see if the package I baked works.

(Ceterum censeo Debian packaging esse delendam. I simply love Fedora (and other RPM distros) for its cleanliness; on Debian I'm never sure what patches and files end up in the (source) package.)

Hey Tim, have you contacted the Ubuntu security team? Anything we can do to help?

*argl* Forgot to test it; now I see the bugs have expired. I'll test it Real Soon Now(TM) and get back to you if there's anything unsurmountable.

ricky.wikitech wrote:

Been a few months - any update here? Or anything I (as a community member) can do to help with moving this along? :)

this could get fixed also during the debian migration, I believe icinga jessie packages would do

IIRC, it could also be fixed by upgrading neon (?) to Trusty, and as those upgrades always seem imminent and I think they have more value than shepherding those patches back to Ubuntu Precise, I didn't spend any time on the latter as testing them for upstream is too complex in my setup.

IIRC, it could also be fixed by upgrading neon (?) to Trusty

Yep, neon is the icinga host.

akosiaris closed subtask Restricted Task as Resolved.Nov 2 2016, 1:33 PM

Although we do have a security-support version of Icinga running now, I'm against opening up the access to the public. Icinga/Nagios have a non-ideal security history, the CGIs are written in C and the monitoring hosts have privileged access to our production networks. (I'm fine with group-based access for selected, trusted service owners).

Although we do have a security-support version of Icinga running now, I'm against opening up the access to the public. Icinga/Nagios have a non-ideal security history, the CGIs are written in C and the monitoring hosts have privileged access to our production networks. (I'm fine with group-based access for selected, trusted service owners).

What about putting a reverse proxy in front of it with some strict filtering so that only a small part is viewable without logging in? You could just call that one https://icinga-public.wikimedia.org/

Did something changed here in over two years since icinga is login-only?

Did something changed here in over two years since icinga is login-only?

No, and given @MoritzMuehlenhoff 's answer above I don't think there will be any soon. . I know @Multichill's idea sounds like a good compromise but unfortunately it is not very feasible technically. Most of the icinga interface is practically the output of 1 CGI script (status.cgi) which alters it's behaviour based on the HTTP GET params it receives. That makes that approach technically difficult and most importantly quite error prone. It would also probably mean that the end result would not differ much from https://status.wikimedia.org/ (which already exist and provides status functionality) making that approach kind of a moot point.

My current inclination is, unfortunately, to close this task as declined

Such a shame that lots of log and status hosts are shifting towards a private option without even a limited public panel. Really decreases community accountability and removes the positive effects of "Wikis use community ownership".

Thanks for reminding me of this. Since there's been practically no change since my last commit, I 'll indeed close this as declined. Feel free to reopen.

The limited public panel you refer to, does exist btw. And in fact, there are even more public channels they are no that limited.

First of, there is the (limited indeed) panel at https://status.wikimedia.org/

Then most alerts are logged via IRC on #wikimedia-operations. That channel is also publicly logged and archives for many years back exist at http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-operations/. Alerts for others teams go to their respective channels as well. Configuration for the entirety of the infrastructure exists and is at https://phabricator.wikimedia.org/source/operations-puppet/. In fact the IRC channels used per team are at https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/icinga/manifests/ircbot.pp

So yes, the icinga web interface is private these days for security reasons, but the information is not private and we do a lot to disseminate it.