Bug #17984: powercycle: fuse mount fails (0.94.10 integration testing) - Ceph - Ceph

Actions

Copy link

Bug #17984

closed

powercycle: fuse mount fails (0.94.10 integration testing)

Added by Nathan Cutler over 7 years ago. Updated over 7 years ago.

Status:

Resolved

Priority:

Urgent

Assignee:

Nathan Cutler

Category:

Target version:

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

description: powercycle/osd/{clusters/3osd-1per-target.yaml fs/ext4.yaml powercycle/default.yaml tasks/cfuse_workunit_kernel_untar_build.yaml}

Test failed ("dead") twice in a row on smithi:

2016-11-21T15:44:10.000 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients...
2016-11-21T15:44:10.000 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {}
2016-11-21T15:44:10.000 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi102.front.sepia.ceph.com /home/ubuntu/cephtest
/mnt.0...
2016-11-21T15:44:10.000 INFO:teuthology.orchestra.run.smithi102:Running: 'mkdir -- /home/ubuntu/cephtest/mnt.0'
2016-11-21T15:44:10.001 INFO:tasks.thrashosds.thrasher:starting do_thrash
2016-11-21T15:44:10.001 INFO:tasks.thrashosds.thrasher:in_osds:  [0, 1, 2] out_osds:  [] dead_osds:  [] live_osds:  [1, 0, 2]
2016-11-21T15:44:10.002 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0
2016-11-21T15:44:10.002 INFO:tasks.thrashosds.thrasher:Reweighting osd 2 to 0.253337561323
2016-11-21T15:44:10.002 INFO:teuthology.orchestra.run.smithi102:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage c
eph osd reweight 2 0.253337561323'
2016-11-21T15:44:10.015 INFO:teuthology.orchestra.run.smithi102:Running: 'sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connectio
ns'
2016-11-21T15:44:10.103 INFO:teuthology.orchestra.run.smithi102.stderr:mount: /sys/fs/fuse/connections already mounted or /sys/fs/fuse/connecti
ons busy
2016-11-21T15:44:10.103 INFO:teuthology.orchestra.run.smithi102.stderr:mount: according to mtab, none is already mounted on /sys/fs/fuse/connec
tions
2016-11-21T15:44:10.105 INFO:teuthology.orchestra.run.smithi102:Running: 'ls /sys/fs/fuse/connections'
2016-11-21T15:44:10.179 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: []
2016-11-21T15:44:10.179 INFO:teuthology.orchestra.run.smithi102:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/cover
age daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0'
2016-11-21T15:44:10.246 INFO:teuthology.orchestra.run.smithi102:Running: 'sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections'
2016-11-21T15:44:10.268 INFO:teuthology.orchestra.run.smithi102.stderr:mount: /sys/fs/fuse/connections already mounted or /sys/fs/fuse/connections busy
2016-11-21T15:44:10.269 INFO:teuthology.orchestra.run.smithi102.stderr:mount: according to mtab, none is already mounted on /sys/fs/fuse/connections
2016-11-21T15:44:10.271 INFO:teuthology.orchestra.run.smithi102:Running: 'ls /sys/fs/fuse/connections'
2016-11-21T15:44:10.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi102.stdout:ceph-fuse[15058]: starting ceph client
2016-11-21T15:44:10.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi102.stderr:2016-11-21 15:44:10.316967 7f87f3b307c0 -1 init, newargv = 0x399c700 newargc=9
2016-11-21T15:44:10.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi102.stderr:ceph-fuse[15058]: starting fuse
2016-11-21T15:44:10.345 INFO:teuthology.orchestra.run.smithi102.stdout:31
2016-11-21T15:44:10.346 INFO:teuthology.orchestra.run.smithi102:Running: 'sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections'
2016-11-21T15:44:10.433 INFO:teuthology.orchestra.run.smithi102.stderr:mount: /sys/fs/fuse/connections already mounted or /sys/fs/fuse/connections busy
2016-11-21T15:44:10.433 INFO:teuthology.orchestra.run.smithi102.stderr:mount: according to mtab, none is already mounted on /sys/fs/fuse/connections
2016-11-21T15:44:10.435 INFO:teuthology.orchestra.run.smithi102:Running: 'ls /sys/fs/fuse/connections'
2016-11-21T15:44:10.510 INFO:teuthology.orchestra.run.smithi102.stdout:31
2016-11-21T15:44:10.510 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [31]
2016-11-21T15:44:10.510 INFO:teuthology.orchestra.run.smithi102:Running: "stat --file-system '--printf=%T\n' -- /home/ubuntu/cephtest/mnt.0" 
2016-11-21T15:44:10.586 INFO:teuthology.orchestra.run.smithi102.stdout:fuseblk
2016-11-21T15:44:10.587 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0
2016-11-21T15:44:10.587 INFO:teuthology.orchestra.run.smithi102:Running: 'sudo chmod 1777 /home/ubuntu/cephtest/mnt.0'
2016-11-21T16:00:29.846 ERROR:paramiko.transport:Socket exception: No route to host (113)

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by Nathan Cutler over 7 years ago

Description updated (diff)

Actions

Copy link

Updated by Nathan Cutler over 7 years ago

hammer baseline

./virtualenv/bin/teuthology-suite -v -c hammer -k testing -m smithi -s powercycle -p 90 --email ncutler@suse.cz --suite-branch hammer --filter 'powercycle/osd/{clusters/3osd-1per-target.yaml fs/ext4.yaml powercycle/default.yaml tasks/cfuse_workunit_kernel_untar_build.yaml}'

dead (same failure) http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-21_22:44:53-powercycle-hammer-testing-basic-smithi/

Actions

Copy link