Page MenuHomePhabricator

glusterFS could not allocate memory
Closed, DeclinedPublic

Description

2013-04-19 19:32:17.144065] W [client3_1-fops.c:2385:client3_1_rename_cbk] 1-deployment-prep-project-client-0: remote operation failed: Cannot allocate memory
[2013-04-19 19:32:17.144123] W [fuse-bridge.c:1516:fuse_rename_cbk] 0-glusterfs-fuse: 267286479: /apache/common-local/php-master/extensions/.git/modules/GoogleMaps/refs/remotes/origin/master.lock -> /apache/common-local/php-master/extensions/.git/modules/GoogleMaps/refs/remotes/origin/master => -1 (Cannot allocate memory)

That is on deployment-prep project when trying to update a git extension.

$ cd /data/project/apache/common-local/php-master/extensions
$ git submodule update --init GoogleMaps
Submodule 'GoogleMaps' () registered for path 'GoogleMaps'
error: Couldn't set refs/remotes/origin/master
From https://gerrit.wikimedia.org/r/p/mediawiki/extensions/GoogleMaps

! 91a381a..7f6454f  master     -> origin/master  (unable to update local ref)

Unable to fetch in submodule path 'GoogleMaps'
$

Got again the same error:

[2013-04-19 19:35:42.160256] W [client3_1-fops.c:2385:client3_1_rename_cbk] 1-deployment-prep-project-client-0: remote operation failed: Cannot allocate memory
[2013-04-19 19:35:42.160537] W [fuse-bridge.c:1516:fuse_rename_cbk] 0-glusterfs-fuse: 267341371: /apache/common-local/php-master/extensions/.git/modules/GoogleMaps/refs/remotes/origin/master.lock -> /apache/common-local/php-master/extensions/.git/modules/GoogleMaps/refs/remotes/origin/master => -1 (Cannot allocate memory)

I have no idea whether that is client side or server side.


Version: unspecified
Severity: major

Details

Reference
bz47425

Event Timeline

bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 1:20 AM
bzimport added a project: Cloud-VPS.
bzimport set Reference to bz47425.

And /logs/ is dead too:

[fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 9759: LOOKUP() /logs/web.log => -1 (Input/output error)

Reassigning bug to infra.

Impact: that at least break the update of MediaWiki extensions.

See also bug 47479 " GlusterFS deployment-prep-project has too many errors -> beta cluster is down"

apparently the "cannot allocate memory" error no more happens on deployment-bastion. Self fixed?

Still happening when update the mw extensions on deployment-prep project:

[2013-04-22 21:45:05.548595] W [client3_1-fops.c:2385:client3_1_rename_cbk] 0-deployment-prep-project-client-1: remote operation failed: Cannot allocate memory
[2013-04-22 21:46:10.686871] W [client3_1-fops.c:649:client3_1_rmdir_cbk] 0-deployment-prep-project-client-0: remote operation failed: Directory not empty
[2013-04-22 21:46:10.687208] W [client3_1-fops.c:649:client3_1_rmdir_cbk] 0-deployment-prep-project-client-1: remote operation failed: Directory not empty

I got rid of Gluster in favor of role::labsnfs::client so there is no point in keeping this bug around I guess.