On integration labs project, the GlusterFS volume for /home is screwed and is marked read-only:
- touch /home/jenkins-deploy/foobar touch: cannot touch `foobar': Read-only file system #
The mount point is:
projectstorage.pmtpa.wmnet:/integration-home on /home type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072)
/var/log/glusterfs/home.log shows a bunch of:
remote operation failed: Transport endpoint is not connected
On Sunday 9 Feb at 6:40 we had:
[2014-02-09 06:41:01.523691] W [socket.c:1512:__socket_proto_state_machine] 0-integration-home-client-0: reading from socket failed. Error (Transport endpoint is not connected), peer (10.0.0.41:24448)
[2014-02-09 06:41:01.541383] I [client.c:2090:client_rpc_notify] 0-integration-home-client-0: disconnected
[2014-02-09 06:41:13.483376] E [socket.c:1715:socket_connect_finish] 0-integration-home-client-0: connection to 10.0.0.41:24448 failed (Connection refused)
[2014-02-09 06:47:17.486672] I [glusterfsd.c:889:reincarnate] 0-glusterfsd: Fetching the volume file from server...
then we get message saying it lacks quorum:
0-integration-home-replicate-0: failing truncate due to lack of quorum
Version: unspecified
Severity: major