Bug #17984
closedpowercycle: fuse mount fails (0.94.10 integration testing)
0%
Description
description: powercycle/osd/{clusters/3osd-1per-target.yaml fs/ext4.yaml powercycle/default.yaml tasks/cfuse_workunit_kernel_untar_build.yaml}
Test failed ("dead") twice in a row on smithi:- http://pulpito.ceph.com/smithfarm-2016-11-18_15:27:08-powercycle-hammer-backports-testing-basic-smithi/560554/
- http://pulpito.ceph.com/smithfarm-2016-11-20_21:53:44-powercycle-hammer-backports-testing-basic-smithi/565543/
2016-11-21T15:44:10.000 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2016-11-21T15:44:10.000 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2016-11-21T15:44:10.000 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi102.front.sepia.ceph.com /home/ubuntu/cephtest /mnt.0... 2016-11-21T15:44:10.000 INFO:teuthology.orchestra.run.smithi102:Running: 'mkdir -- /home/ubuntu/cephtest/mnt.0' 2016-11-21T15:44:10.001 INFO:tasks.thrashosds.thrasher:starting do_thrash 2016-11-21T15:44:10.001 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2] out_osds: [] dead_osds: [] live_osds: [1, 0, 2] 2016-11-21T15:44:10.002 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2016-11-21T15:44:10.002 INFO:tasks.thrashosds.thrasher:Reweighting osd 2 to 0.253337561323 2016-11-21T15:44:10.002 INFO:teuthology.orchestra.run.smithi102:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage c eph osd reweight 2 0.253337561323' 2016-11-21T15:44:10.015 INFO:teuthology.orchestra.run.smithi102:Running: 'sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connectio ns' 2016-11-21T15:44:10.103 INFO:teuthology.orchestra.run.smithi102.stderr:mount: /sys/fs/fuse/connections already mounted or /sys/fs/fuse/connecti ons busy 2016-11-21T15:44:10.103 INFO:teuthology.orchestra.run.smithi102.stderr:mount: according to mtab, none is already mounted on /sys/fs/fuse/connec tions 2016-11-21T15:44:10.105 INFO:teuthology.orchestra.run.smithi102:Running: 'ls /sys/fs/fuse/connections' 2016-11-21T15:44:10.179 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2016-11-21T15:44:10.179 INFO:teuthology.orchestra.run.smithi102:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/cover age daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0' 2016-11-21T15:44:10.246 INFO:teuthology.orchestra.run.smithi102:Running: 'sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections' 2016-11-21T15:44:10.268 INFO:teuthology.orchestra.run.smithi102.stderr:mount: /sys/fs/fuse/connections already mounted or /sys/fs/fuse/connections busy 2016-11-21T15:44:10.269 INFO:teuthology.orchestra.run.smithi102.stderr:mount: according to mtab, none is already mounted on /sys/fs/fuse/connections 2016-11-21T15:44:10.271 INFO:teuthology.orchestra.run.smithi102:Running: 'ls /sys/fs/fuse/connections' 2016-11-21T15:44:10.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi102.stdout:ceph-fuse[15058]: starting ceph client 2016-11-21T15:44:10.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi102.stderr:2016-11-21 15:44:10.316967 7f87f3b307c0 -1 init, newargv = 0x399c700 newargc=9 2016-11-21T15:44:10.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi102.stderr:ceph-fuse[15058]: starting fuse 2016-11-21T15:44:10.345 INFO:teuthology.orchestra.run.smithi102.stdout:31 2016-11-21T15:44:10.346 INFO:teuthology.orchestra.run.smithi102:Running: 'sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections' 2016-11-21T15:44:10.433 INFO:teuthology.orchestra.run.smithi102.stderr:mount: /sys/fs/fuse/connections already mounted or /sys/fs/fuse/connections busy 2016-11-21T15:44:10.433 INFO:teuthology.orchestra.run.smithi102.stderr:mount: according to mtab, none is already mounted on /sys/fs/fuse/connections 2016-11-21T15:44:10.435 INFO:teuthology.orchestra.run.smithi102:Running: 'ls /sys/fs/fuse/connections' 2016-11-21T15:44:10.510 INFO:teuthology.orchestra.run.smithi102.stdout:31 2016-11-21T15:44:10.510 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [31] 2016-11-21T15:44:10.510 INFO:teuthology.orchestra.run.smithi102:Running: "stat --file-system '--printf=%T\n' -- /home/ubuntu/cephtest/mnt.0" 2016-11-21T15:44:10.586 INFO:teuthology.orchestra.run.smithi102.stdout:fuseblk 2016-11-21T15:44:10.587 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2016-11-21T15:44:10.587 INFO:teuthology.orchestra.run.smithi102:Running: 'sudo chmod 1777 /home/ubuntu/cephtest/mnt.0' 2016-11-21T16:00:29.846 ERROR:paramiko.transport:Socket exception: No route to host (113)
Updated by Nathan Cutler over 7 years ago
hammer baseline
./virtualenv/bin/teuthology-suite -v -c hammer -k testing -m smithi -s powercycle -p 90 --email ncutler@suse.cz --suite-branch hammer --filter 'powercycle/osd/{clusters/3osd-1per-target.yaml fs/ext4.yaml powercycle/default.yaml tasks/cfuse_workunit_kernel_untar_build.yaml}'
dead (same failure) http://pulpito.front.sepia.ceph.com:80/smithfarm-2016-11-21_22:44:53-powercycle-hammer-testing-basic-smithi/
Updated by Nathan Cutler over 7 years ago
jewel baseline
dead (same failure) http://pulpito.front.sepia.ceph.com/smithfarm-2016-11-21_22:52:16-powercycle-jewel-testing-basic-smithi/
Specifically, this job: http://pulpito.front.sepia.ceph.com/smithfarm-2016-11-21_22:52:16-powercycle-jewel-testing-basic-smithi/567875/
Updated by Nathan Cutler over 7 years ago
However, the same run with -k distro
succeeds: http://tracker.ceph.com/issues/17151#note-19
Updated by Nathan Cutler over 7 years ago
- Related to Bug #17997: ceph-fuse causing OS crash or hang added
Updated by Nathan Cutler over 7 years ago
- Status changed from New to Resolved
- Assignee set to Nathan Cutler
Fixed by specifying -k distro
instead of -k testing
According to Ilya, the testing kernel was temporarily broken by a kernel regression - here is his message:
"Known 4.9 kernel regression [1], should be fixed by [2]. I've just
re-pushed testing branch with the fix, just in case you want to try it
out.
[1] https://bugzilla.kernel.org/show_bug.cgi?id=177801
[2] https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=0ce267ff95a0302cf6fb2a552833abbfb7861a43"