Bug #15427: misc.pull_directory() appears to hang on large transfers (aka - jobs hang at 'sudo chmod 0666 /tmp/*') - teuthology - Ceph

Actions

Copy link

Bug #15427

closed

misc.pull_directory() appears to hang on large transfers (aka - jobs hang at 'sudo chmod 0666 /tmp/*')

Added by Yuri Weinstein about 8 years ago. Updated almost 8 years ago.

Status:

Resolved

Priority:

Urgent

Assignee:

Zack Cerza

Category:

% Done:

Source:

Q/A

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

upgrade/infernalis-x

Crash signature (v1):

Crash signature (v2):

Description

Run: http://pulpito.ovh.sepia.ceph.com:8081/teuthology-2016-04-07_02:10:01-upgrade:infernalis-x-jewel-distro-basic-openstack/
Jobs: 30510, 30523
Logs: http://teuthology.ovh.sepia.ceph.com/teuthology/teuthology-2016-04-07_02:10:01-upgrade:infernalis-x-jewel-distro-basic-openstack/30523/teuthology.log
http://teuthology.ovh.sepia.ceph.com/teuthology/teuthology-2016-04-07_02:10:01-upgrade:infernalis-x-jewel-distro-basic-openstack/30510/teuthology.log

Both are hanging on lines:

2016-04-07T11:11:04.926 INFO:tasks.ceph:Archiving logs...
2016-04-07T11:11:04.930 DEBUG:teuthology.misc:Transferring archived files from target065157:/var/log/ceph to /home/teuthworker/archive/teuthology-2016-04-07_02:10:01-upgrade:infernalis-x-jewel-distro-basic-openstack/30523/remote/target065157/log
2016-04-07T11:11:04.932 INFO:teuthology.orchestra.run.target065157:Running: "python -c 'import os; import tempfile; import sys;(fd,fname) = tempfile.mkstemp();os.close(fd);sys.stdout.write(fname.rstrip());sys.stdout.flush()'" 
2016-04-07T11:11:05.256 INFO:teuthology.orchestra.run.target065157:Running: 'sudo tar cz -f /tmp/tmpSY4FsZ -C /var/log/ceph -- .'
2016-04-07T11:12:35.497 INFO:teuthology.orchestra.run.target065157:Running: 'sudo chmod 0666 /tmp/tmpSY4FsZ'FO:teuthology.orchestra.run.target064051:Running: 'sudo chmod 0666 /tmp/tmpBeaXdf

Actions

Copy link

Updated by Yuri Weinstein about 8 years ago

Description updated (diff)

Actions

Copy link

Updated by Zack Cerza about 8 years ago

Subject changed from jobs hang at 'sudo chmod 0666 /tmp/*' in upgrade:infernalis-x-jewel-distro-basic-openstack to misc.pull_directory() appears to hang on large transfers

Let's add some logging to misc.pull_directory, maybe, to at least show the size of the file we're about to transfer.

Actions

Copy link

Updated by Zack Cerza about 8 years ago

Assignee set to Zack Cerza

Actions

Copy link

Updated by Yuri Weinstein about 8 years ago

Subject changed from misc.pull_directory() appears to hang on large transfers to misc.pull_directory() appears to hang on large transfers (aka - jobs hang at 'sudo chmod 0666 /tmp/*')

Actions

Copy link