Page MenuHomePhabricator
Create Task
Maniphest T64432

tools: "git error: server certificate verification failed" for git.wikimedia.org on tools-login-eqiad
Closed, DeclinedPublic
Actions

Description

Fetching origin
error: server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none while accessing https://git.wikimedia.org/git/mediawiki/core.git/info/refs
fatal: HTTP request failed
error: Could not fetch origin

Version: unspecified
Severity: normal

Details

Reference
bz62432

Event Timeline

• bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 2:53 AM
• bzimport added a project: Toolforge.
• bzimport set Reference to bz62432.
Krinkle created this task.Mar 8 2014, 10:19 AM
scfc added a comment.Mar 8 2014, 3:03 PM

To reproduce:

[tim@passepartout ~]$ for HOST in tools-{dev,login}-eqiad.wmflabs.org; do ssh "$HOST" 'git clone https://git.wikimedia.org/git/pywikibot/compat.git $(mktemp -d)'; done
If you are having access problems, please see: https://wikitech.wikimedia.org/wiki/Access#Accessing_public_and_private_instances
Cloning into '/tmp/tmp.sNGj8sMBod'...
If you are having access problems, please see: https://wikitech.wikimedia.org/wiki/Access#Accessing_public_and_private_instances
Cloning into '/tmp/tmp.e4O2n9lnBB'...
error: server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none while accessing https://git.wikimedia.org/git/pywikibot/compat.git/info/refs
fatal: HTTP request failed
[tim@passepartout ~]$

So tools-dev-eqiad works, tools-login-eqiad fails. But the problem doesn't seem to lie with curl:

[tim@passepartout ~]$ for HOST in tools-{dev,login}-eqiad.wmflabs.org; do ssh "$HOST" 'curl https://git.wikimedia.org/git/pywikibot/compat.git > /dev/null'; done
If you are having access problems, please see: https://wikitech.wikimedia.org/wiki/Access#Accessing_public_and_private_instances
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1389 100 1389 0 0 84243 0 --:--:-- --:--:-- --:--:-- 113k
If you are having access problems, please see: https://wikitech.wikimedia.org/wiki/Access#Accessing_public_and_private_instances
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1389 100 1389 0 0 78147 0 --:--:-- --:--:-- --:--:-- 99214
[tim@passepartout ~]$
scfc added a comment.Mar 9 2014, 5:26 PM

Only difference beneath /etc/ssl is in /etc/ssl/certs/java/cacerts which shouldn't affect git clone:

[tim@passepartout ~]$ for CMD in 'sudo find /etc/ssl -not -type d -print0xargs -0r sudo md5sum'; do diff -u <(ssh tools-login.eqiad.wmflabs "$CMD") <(ssh tools-dev.eqiad.wmflabs "$CMD"); done
If you are having access problems, please see: https://wikitech.wikimedia.org/wiki/Access#Accessing_public_and_private_instances
If you are having access problems, please see: https://wikitech.wikimedia.org/wiki/Access#Accessing_public_and_private_instances
--- /dev/fd/63 2014-03-09 17:25:44.783519345 +0000
+++ /dev/fd/62 2014-03-09 17:25:44.784519334 +0000
@@ -286,7 +286,7 @@
c9048f79a8f1da62f89b3eeb8c493689 /etc/ssl/certs/b42ff584.0
8a2b0f016146ed5f78f8bdd828772803 /etc/ssl/certs/NetLock_Qualified_=Class_QA=_Root.pem
f130d662fbfeb1ddc4c35d2e0c67a357 /etc/ssl/certs/Camerfirma_Global_Chambersign_Root.pem
-750061a18276cd2b4fc8debd90cd947f /etc/ssl/certs/java/cacerts
+321edf0746699c5ac1158632a9ad4ea3 /etc/ssl/certs/java/cacerts
e0a3a4ecbfc76649d2c9f4f0d2773565 /etc/ssl/certs/a2df7ad7.0
47efdfb0853adc341e39d422c96fb36f /etc/ssl/certs/TC_TrustCenterGermanyClass_2_CA.pem
485bce6d706a2c6ef08e0d8cfd51760d /etc/ssl/certs/3c860d51.0
[tim@passepartout ~]$
scfc added a comment.Mar 9 2014, 5:29 PM

No differences in relevant packages:

[tim@passepartout ~]$ for CMD in 'sudo dpkg -l'; do diff <(ssh tools-login.eqiad.wmflabs "$CMD") <(ssh tools-dev.eqiad.wmflabs "$CMD"); done
If you are having access problems, please see: https://wikitech.wikimedia.org/wiki/Access#Accessing_public_and_private_instances
If you are having access problems, please see: https://wikitech.wikimedia.org/wiki/Access#Accessing_public_and_private_instances
28d27
< ii automake 1:1.11.3-1ubuntu2 Tool for generating GNU Standards-compliant Makefiles
75d73
< ii dh-autoreconf 5ubuntu1 debhelper add-on to call autoreconf and clean up after the build
1137a1136
> ii nmap 5.21-1.1ubuntu1 The Network Mapper
1215c1214
< ii python-coverage 3.4-1ubuntu1 code coverage tool for Python
> ii python-coverage 3.6-1 Code coverage measurement for Python
1382a1382
> ii terminatord 1.0.6.0ppa2 Terminator daemon
[tim@passepartout ~]$
scfc added a comment.Mar 10 2014, 2:53 AM

Ran "strace -f", extracted the following list of filenames, and all are identical on tools-{dev,login}-eqiad:

/etc/gai.conf
/etc/gcrypt/fips_enabled
/etc/gitconfig
/etc/gnutls/pkcs11.conf
/etc/host.conf
/etc/hosts
/etc/ld.so.cache
/etc/ld.so.nohwcap
/etc/ld.so.preload
/etc/nsswitch.conf
/etc/pkcs11/modules
/etc/pkcs11/pkcs11.conf
/etc/resolv.conf
/etc/ssl/certs/ca-certificates.crt
/lib/x86_64-linux-gnu/libcom_err.so.2
/lib/x86_64-linux-gnu/libcrypt.so.1
/lib/x86_64-linux-gnu/libc.so.6
/lib/x86_64-linux-gnu/libdl.so.2
/lib/x86_64-linux-gnu/libgcrypt.so.11
/lib/x86_64-linux-gnu/libgpg-error.so.0
/lib/x86_64-linux-gnu/libkeyutils.so.1
/lib/x86_64-linux-gnu/libnss_dns.so.2
/lib/x86_64-linux-gnu/libnss_files.so.2
/lib/x86_64-linux-gnu/libpthread.so.0
/lib/x86_64-linux-gnu/libresolv.so.2
/lib/x86_64-linux-gnu/librt.so.1
/lib/x86_64-linux-gnu/libz.so.1
/proc/sys/crypto/fips_enabled
/usr/bin/git
/usr/lib/git-core/git-remote-https
/usr/lib/locale/locale-archive
/usr/lib/x86_64-linux-gnu/libasn1.so.8
/usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4
/usr/lib/x86_64-linux-gnu/libgnutls.so.26
/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2
/usr/lib/x86_64-linux-gnu/libgssapi.so.3
/usr/lib/x86_64-linux-gnu/libhcrypto.so.4
/usr/lib/x86_64-linux-gnu/libheimbase.so.1
/usr/lib/x86_64-linux-gnu/libheimntlm.so.0
/usr/lib/x86_64-linux-gnu/libhx509.so.5
/usr/lib/x86_64-linux-gnu/libidn.so.11
/usr/lib/x86_64-linux-gnu/libk5crypto.so.3
/usr/lib/x86_64-linux-gnu/libkrb5.so.26
/usr/lib/x86_64-linux-gnu/libkrb5.so.3
/usr/lib/x86_64-linux-gnu/libkrb5support.so.0
/usr/lib/x86_64-linux-gnu/liblber-2.4.so.2
/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2
/usr/lib/x86_64-linux-gnu/libp11-kit.so.0
/usr/lib/x86_64-linux-gnu/libroken.so.18
/usr/lib/x86_64-linux-gnu/librtmp.so.0
/usr/lib/x86_64-linux-gnu/libsasl2.so.2
/usr/lib/x86_64-linux-gnu/libsqlite3.so.0
/usr/lib/x86_64-linux-gnu/libtasn1.so.3
/usr/lib/x86_64-linux-gnu/libwind.so.0
/usr/share/git-core/templates/
/usr/share/git-core/templates/branches
/usr/share/git-core/templates/config
/usr/share/git-core/templates/description
/usr/share/git-core/templates/hooks
/usr/share/git-core/templates/hooks/applypatch-msg.sample
/usr/share/git-core/templates/hooks/commit-msg.sample
/usr/share/git-core/templates/hooks/post-update.sample
/usr/share/git-core/templates/hooks/pre-applypatch.sample
/usr/share/git-core/templates/hooks/pre-commit.sample
/usr/share/git-core/templates/hooks/prepare-commit-msg.sample
/usr/share/git-core/templates/hooks/pre-rebase.sample
/usr/share/git-core/templates/hooks/update.sample
/usr/share/git-core/templates/info
/usr/share/git-core/templates/info/exclude
/usr/share/locale/en/LC_MESSAGES/git.mo
/usr/share/locale/en_US/LC_MESSAGES/git.mo
/usr/share/locale/en_US.utf8/LC_MESSAGES/git.mo
/usr/share/locale/en_US.UTF-8/LC_MESSAGES/git.mo
/usr/share/locale/en.utf8/LC_MESSAGES/git.mo
/usr/share/locale/en.UTF-8/LC_MESSAGES/git.mo
/usr/share/locale-langpack/en/LC_MESSAGES/git.mo
/usr/share/locale-langpack/en_US/LC_MESSAGES/git.mo
/usr/share/locale-langpack/en_US.utf8/LC_MESSAGES/git.mo
/usr/share/locale-langpack/en_US.UTF-8/LC_MESSAGES/git.mo
/usr/share/locale-langpack/en.utf8/LC_MESSAGES/git.mo
/usr/share/locale-langpack/en.UTF-8/LC_MESSAGES/git.mo
/usr/share/locale/locale.alias
/var/run/nscd/socket
• Mattflaschen-WMF added a comment.Mar 27 2014, 4:58 PM

Happened on pronunciationrecording.eqiad.wmflabs too.

scfc added a comment.Sep 9 2014, 8:29 PM

I can no longer reproduce this on tools-login or tools-dev. I'm not aware that someone consciously fixed this, so resolving as WORKSFORME.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL