Project

General

Profile

Bug #9674

nightly failed multiple_rsync.sh

Added by Greg Farnum almost 5 years ago. Updated over 4 years ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
-
Target version:
-
Start date:
10/06/2014
Due date:
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
Pull request ID:

Description

http://qa-proxy.ceph.com/teuthology/teuthology-2014-10-03_23:04:01-fs-giant-distro-basic-multi/527949/

2014-10-05T23:46:20.523 INFO:tasks.workunit.client.0.burnupi14.stderr:rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1070) [sender=3.0.9]

There's nothing in the teuthology.log indicating failed transfers. There's nothing in the kern.log nor sys.log. I tried grepping the client log, and it has a lot of ENOENT (which I spot-checked as all being checks before moving the rsync'ed copy into place). The ENODATA response on security.ima manipulation is the only thing that I think might be broken, but even that xattr is being removed...

History

#1 Updated by Zheng Yan almost 5 years ago

#define ENFILE 23 /* File table overflow */

maybe we should adjust ulimit

#2 Updated by Greg Farnum almost 5 years ago

rsync return codes aren't standard error codes. The man page says that 23 means

Partial transfer due to error

#3 Updated by Zheng Yan almost 5 years ago

2014-10-05T22:47:55.288 INFO:tasks.workunit.client.0.burnupi14.stderr:rsync: send_files failed to open "/usr/lib/virtualbox/VBoxHeadless": Permission denied (13)
2014-10-05T22:47:55.289 INFO:tasks.workunit.client.0.burnupi14.stdout:lib/virtualbox/VBoxHeadless.so
2014-10-05T22:47:55.394 INFO:tasks.workunit.client.0.burnupi14.stdout:lib/virtualbox/VBoxHostChannel.so
2014-10-05T22:47:55.401 INFO:tasks.workunit.client.0.burnupi14.stdout:lib/virtualbox/VBoxKeyboard.so
2014-10-05T22:47:55.495 INFO:tasks.workunit.client.0.burnupi14.stdout:lib/virtualbox/VBoxManage
2014-10-05T22:47:55.569 INFO:tasks.workunit.client.0.burnupi14.stderr:rsync: send_files failed to open "/usr/lib/virtualbox/VBoxNetAdpCtl": Permission denied (13)
2014-10-05T22:47:55.569 INFO:tasks.workunit.client.0.burnupi14.stderr:rsync: send_files failed to open "/usr/lib/virtualbox/VBoxNetDHCP": Permission denied (13)
2014-10-05T22:47:55.570 INFO:tasks.workunit.client.0.burnupi14.stdout:lib/virtualbox/VBoxNetDHCP.so
2014-10-05T22:47:55.638 INFO:tasks.workunit.client.0.burnupi14.stderr:rsync: send_files failed to open "/usr/lib/virtualbox/VBoxNetNAT": Permission denied (13)
2014-10-05T22:47:55.638 INFO:tasks.workunit.client.0.burnupi14.stdout:lib/virtualbox/VBoxNetNAT.so
2014-10-05T22:47:55.681 INFO:tasks.workunit.client.0.burnupi14.stdout:lib/virtualbox/VBoxOGLhostcrutil.so
2014-10-05T22:47:55.734 INFO:tasks.workunit.client.0.burnupi14.stdout:lib/virtualbox/VBoxOGLhosterrorspu.so
2014-10-05T22:47:55.793 INFO:tasks.workunit.client.0.burnupi14.stdout:lib/virtualbox/VBoxOGLrenderspu.so
2014-10-05T22:47:55.839 INFO:tasks.workunit.client.0.burnupi14.stdout:lib/virtualbox/VBoxPython.so
2014-10-05T22:47:55.938 INFO:tasks.workunit.client.0.burnupi14.stdout:lib/virtualbox/VBoxPython2_7.so
2014-10-05T22:47:55.994 INFO:tasks.workunit.client.0.burnupi14.stdout:lib/virtualbox/VBoxREM.so
2014-10-05T22:47:56.046 INFO:tasks.workunit.client.0.burnupi14.stdout:lib/virtualbox/VBoxRT.so
2014-10-05T22:47:56.147 INFO:tasks.workunit.client.0.burnupi14.stderr:rsync: send_files failed to open "/usr/lib/virtualbox/VBoxSDL": Permission denied (13)
2014-10-05T22:47:56.147 INFO:tasks.workunit.client.0.burnupi14.stdout:lib/virtualbox/VBoxSDL.so
2014-10-05T22:47:56.207 INFO:tasks.workunit.client.0.burnupi14.stdout:lib/virtualbox/VBoxSVC
2014-10-05T22:47:56.335 INFO:tasks.workunit.client.0.burnupi14.stdout:lib/virtualbox/VBoxSharedClipboard.so
2014-10-05T22:47:56.337 INFO:tasks.workunit.client.0.burnupi14.stdout:lib/virtualbox/VBoxSharedCrOpenGL.so
2014-10-05T22:47:56.453 INFO:tasks.workunit.client.0.burnupi14.stdout:lib/virtualbox/VBoxSharedFolders.so
2014-10-05T22:47:56.458 INFO:tasks.workunit.client.0.burnupi14.stdout:lib/virtualbox/VBoxTestOGL
2014-10-05T22:47:56.567 INFO:tasks.workunit.client.0.burnupi14.stdout:lib/virtualbox/VBoxVMM.so
2014-10-05T22:47:56.661 INFO:tasks.workunit.client.0.burnupi14.stderr:rsync: send_files failed to open "/usr/lib/virtualbox/VBoxVolInfo": Permission denied (13)
2014-10-05T22:47:56.664 INFO:tasks.workunit.client.0.burnupi14.stdout:lib/virtualbox/VBoxXPCOM.so
2014-10-05T22:47:56.736 INFO:tasks.workunit.client.0.burnupi14.stdout:lib/virtualbox/VBoxXPCOMC.so
2014-10-05T22:47:56.794 INFO:tasks.workunit.client.0.burnupi14.stdout:lib/virtualbox/VBoxXPCOMIPCD
2014-10-05T22:47:56.794 INFO:tasks.workunit.client.0.burnupi14.stdout:lib/virtualbox/VMMGC.gc
2014-10-05T22:47:57.211 INFO:tasks.workunit.client.0.burnupi14.stdout:lib/virtualbox/VMMR0.r0
2014-10-05T22:47:57.285 INFO:tasks.workunit.client.0.burnupi14.stderr:rsync: send_files failed to open "/usr/lib/virtualbox/VirtualBox": Permission denied (13)

I think these permission denied errors are root cause

#4 Updated by Greg Farnum almost 5 years ago

  • Status changed from New to Verified

Well, that would make sense. How did you find those in the log?

We should probably just run this as sudo or something instead of the ubuntu user?

#5 Updated by Zheng Yan almost 5 years ago

rsync asks us to see previous errors;) yes, I think sudo should work

#6 Updated by Greg Farnum almost 5 years ago

  • Status changed from Verified to Testing

commit:477073aba1da880dfd0b8c82f4792788579f28b9 in master and commit:44ce33c12443909b02c7ee451ad45400f55d53c9 in giant

#7 Updated by Greg Farnum almost 5 years ago

  • Assignee set to Greg Farnum

#8 Updated by Greg Farnum almost 5 years ago

I messed up (didn't set sudo everywhere), newer commits will hopefully make it all good. giant:f66bf31b6743246fb1c88238cf18101238dee3a4 and master:3aa7797741f9cff06053a2f31550fe6929039692

#9 Updated by Greg Farnum over 4 years ago

  • Status changed from Testing to Resolved

I haven't seen this fail since then, hurray.

Also available in: Atom PDF