labs instances ntpd fails to create socket / init interface
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Dzahn
	Jan 17 2014, 12:46 PM

Description

when creating a new labs instance and watching syslog while and after it's being built, you see those errors where ntpd tries to use IPv6 and create a socket and init an interface, but that fails, and then it keeps trying again and so on...

Jan 17 12:39:32 wikistats-mod ntpd[7801]: bind(22) AF_INET6 fe80::f816:3eff:fe27:8094%2#123 flags 0x11 failed: Cannot assign requested address
Jan 17 12:39:32 wikistats-mod ntpd[7801]: unable to create socket on eth0 (10) for fe80::f816:3eff:fe27:8094#123
Jan 17 12:39:32 wikistats-mod ntpd[7801]: failed to init interface for address fe80::f816:3eff:fe27:8094
Jan 17 12:40:02 wikistats-mod ntpd[7801]: bind(22) AF_INET6 fe80::f816:3eff:fe27:8094%2#123 flags 0x11 failed: Cannot assign requested address
Jan 17 12:40:02 wikistats-mod ntpd[7801]: unable to create socket on eth0 (11) for fe80::f816:3eff:fe27:8094#123
Jan 17 12:40:02 wikistats-mod ntpd[7801]: failed to init interface for address fe80::f816:3eff:fe27:8094

do we want it to be able to create and use that, then let's fix it, or not, then disable that it's even trying?

Version: unspecified
Severity: minor

Details

Reference: bz60166

Event Timeline

• bzimport raised the priority of this task from to Lowest.Nov 22 2014, 3:01 AM

• bzimport added projects: Cloud-VPS, Upstream.

• bzimport set Reference to bz60166.

• bzimport added a subscriber: Unknown Object (MLST).

Dzahn created this task.Jan 17 2014, 12:46 PM

Upstream bugs (I think):

https://bugs.launchpad.net/ubuntu/+source/ntp/+bug/569919 ("ntpd constantly complains about being unable to create socket")
https://bugs.launchpad.net/ubuntu/+source/ntp/+bug/697592 ("unable to create socket on eth0 (210) for ::ffff:192.168.1.4#123")
https://bugs.launchpad.net/ubuntu/+source/ntp/+bug/700492 ("ntp complains about ipv6 errors every 5 minutes")

The workaround in the latter ("interface ignore ipv6") silenced the log, but also kept "ntpq -p localhost" in the state:

remote refid st t when poll reach delay offset jitter

==============================================================================

linne.wikimedia .INIT. 16 - - 64 0 0.000 0.000 0.000

dobson.wikimedi .INIT. 16 - - 64 0 0.000 0.000 0.000

Removing it again and restarting ntpd immediately produced:

remote refid st t when poll reach delay offset jitter

==============================================================================

linne.wikimedia 208.75.88.4 3 u 26 64 3 35.767 1.247 0.061

dobson.wikimedi 169.229.70.201 3 u 28 64 3 35.647 -0.450 0.063

Those upstream bugs mean that I can ignore this, right? I see this as well but it doesn't seem to cause any symptoms other than log spam.

AFAIUI, yes, the effects seem to be limited to log spam.

Adding 'interface listen ipv4' instead of 'interface ignore ipv6' to /etc/ntp.conf seems to make the error spam go away while still letting ntp work. Is this something that we could put globally or conditionally into operations/puppet/modules/ntp/templates/ntp-server.erb?

Change 118301 had a related patch set uploaded by Tim Landscheidt:
ntp: Work around Labs network error

https://gerrit.wikimedia.org/r/118301

NB: The patch doesn't fix the underlying issue, but just works around it. So this bug needs to remain open after its merge.

Starting ntpd without work-around:

Mar 12 16:56:51 toolsbeta-puppetmaster3 ntpd[26813]: ntpd 4.2.6p3@1.2290-o Tue Jun 5 20:12:08 UTC 2012 (1)

Mar 12 16:56:51 toolsbeta-puppetmaster3 ntpd[26814]: proto: precision = 0.120 usec

Mar 12 16:56:51 toolsbeta-puppetmaster3 ntpd[26814]: ntp_io: estimated max descriptors: 1024, initial socket boundary: 16

Mar 12 16:56:51 toolsbeta-puppetmaster3 ntpd[26814]: Listen and drop on 0 v4wildcard 0.0.0.0 UDP 123

Mar 12 16:56:51 toolsbeta-puppetmaster3 ntpd[26814]: Listen and drop on 1 v6wildcard :: UDP 123

Mar 12 16:56:51 toolsbeta-puppetmaster3 ntpd[26814]: Listen normally on 2 lo 127.0.0.1 UDP 123

Mar 12 16:56:51 toolsbeta-puppetmaster3 ntpd[26814]: Listen normally on 3 eth0 10.68.16.92 UDP 123

Mar 12 16:56:51 toolsbeta-puppetmaster3 ntpd[26814]: bind(20) AF_INET6 fe80::f816:3eff:fe52:c7a0%2#123 flags 0x11 failed: Cannot assign requested address

Mar 12 16:56:51 toolsbeta-puppetmaster3 ntpd[26814]: unable to create socket on eth0 (4) for fe80::f816:3eff:fe52:c7a0#123

Mar 12 16:56:51 toolsbeta-puppetmaster3 ntpd[26814]: failed to init interface for address fe80::f816:3eff:fe52:c7a0

Mar 12 16:56:51 toolsbeta-puppetmaster3 ntpd[26814]: Listen normally on 5 lo ::1 UDP 123

Mar 12 16:56:51 toolsbeta-puppetmaster3 ntpd[26814]: peers refreshed

Mar 12 16:56:51 toolsbeta-puppetmaster3 ntpd[26814]: Listening on routing socket on fd #21 for interface updates

Mar 12 16:56:52 toolsbeta-puppetmaster3 ntpd[26814]: bind(22) AF_INET6 fe80::f816:3eff:fe52:c7a0%2#123 flags 0x11 failed: Cannot assign requested address

Mar 12 16:56:52 toolsbeta-puppetmaster3 ntpd[26814]: unable to create socket on eth0 (6) for fe80::f816:3eff:fe52:c7a0#123

Mar 12 16:56:52 toolsbeta-puppetmaster3 ntpd[26814]: failed to init interface for address fe80::f816:3eff:fe52:c7a0

Starting ntpd with work-around:

Mar 12 17:10:29 toolsbeta-puppetmaster3 ntpd[27758]: ntpd 4.2.6p3@1.2290-o Tue Jun 5 20:12:08 UTC 2012 (1)

Mar 12 17:10:29 toolsbeta-puppetmaster3 ntpd[27759]: proto: precision = 0.120 usec

Mar 12 17:10:29 toolsbeta-puppetmaster3 ntpd[27759]: ntp_io: estimated max descriptors: 1024, initial socket boundary: 16

Mar 12 17:10:29 toolsbeta-puppetmaster3 ntpd[27759]: Listen normally on 0 v4wildcard 0.0.0.0 UDP 123

Mar 12 17:10:29 toolsbeta-puppetmaster3 ntpd[27759]: Listen and drop on 1 v6wildcard :: UDP 123

Mar 12 17:10:29 toolsbeta-puppetmaster3 ntpd[27759]: Listen normally on 2 lo 127.0.0.1 UDP 123

Mar 12 17:10:29 toolsbeta-puppetmaster3 ntpd[27759]: Listen normally on 3 eth0 10.68.16.92 UDP 123

Mar 12 17:10:29 toolsbeta-puppetmaster3 ntpd[27759]: Listen normally on 4 lo ::1 UDP 123

Mar 12 17:10:29 toolsbeta-puppetmaster3 ntpd[27759]: peers refreshed

Mar 12 17:10:29 toolsbeta-puppetmaster3 ntpd[27759]: Listening on routing socket on fd #21 for interface updates

Change 118301 merged by coren:
ntp: Work around Labs network error

https://gerrit.wikimedia.org/r/118301

Aklapper added a project: Cloud-Services.Apr 29 2015, 12:18 PM

Revisiting this because https://bugs.launchpad.net/ubuntu/+source/ntp/+bug/569919 got closed, the issue is effectively resolved.

labs instances ntpd fails to create socket / init interfaceClosed, ResolvedPublicActions

Description

Details

Event Timeline

labs instances ntpd fails to create socket / init interface
Closed, ResolvedPublic
Actions