Actions
Bug #14654
closed"failed: mon_thrash" in upgrade:hammer-hammer-distro-basic-openstack
Status:
Can't reproduce
Priority:
High
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
upgrade/hammer
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Run: http://pulpito.ovh.sepia.ceph.com:8081/teuthology-2016-02-03_20:05:02-upgrade:hammer-hammer-distro-basic-openstack/
Jobs: 16525
Logs: http://teuthology.ovh.sepia.ceph.com/teuthology/teuthology-2016-02-03_20:05:02-upgrade:hammer-hammer-distro-basic-openstack/16525/teuthology.log
2016-02-03T22:02:38.437 INFO:teuthology.orchestra.run.target088173.stdout:successfully deleted pool unique_pool_0
2016-02-03T22:02:38.441 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse
2016-02-03T22:02:38.441 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients...
2016-02-03T22:02:38.442 INFO:teuthology.orchestra.run.target088172:Running: "stat --file-system '--printf=%T\n' -- /home/ubuntu/cephtest/mnt.1"
2016-02-03T22:02:38.644 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1
2016-02-03T22:02:38.644 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@target088172.ovh.sepia.ceph.com...
2016-02-03T22:02:38.645 INFO:teuthology.orchestra.run.target088172:Running: 'sudo fusermount -u /home/ubuntu/cephtest/mnt.1'
2016-02-03T22:02:39.626 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.target088172.stderr:ceph-fuse[15290]: fuse finished with error 0 and tester_r 0
2016-02-03T22:02:39.654 INFO:teuthology.orchestra.run.target088172:Running: "stat --file-system '--printf=%T\n' -- /home/ubuntu/cephtest/mnt.1"
2016-02-03T22:02:39.912 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3'
2016-02-03T22:02:40.109 INFO:teuthology.orchestra.run.target088172:Running: 'rmdir -- /home/ubuntu/cephtest/mnt.1'
2016-02-03T22:02:40.404 INFO:teuthology.orchestra.run.target088172:Running: "stat --file-system '--printf=%T\n' -- /home/ubuntu/cephtest/mnt.0"
2016-02-03T22:02:40.700 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0
2016-02-03T22:02:40.701 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@target088172.ovh.sepia.ceph.com...
2016-02-03T22:02:40.701 INFO:teuthology.orchestra.run.target088172:Running: 'sudo fusermount -u /home/ubuntu/cephtest/mnt.0'
2016-02-03T22:02:41.109 INFO:teuthology.orchestra.run.target088172:Running: "stat --file-system '--printf=%T\n' -- /home/ubuntu/cephtest/mnt.0"
2016-02-03T22:02:41.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.target088172.stderr:ceph-fuse[15351]: fuse finished with error 0 and tester_r 0
2016-02-03T22:02:41.440 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3'
2016-02-03T22:02:41.963 INFO:teuthology.orchestra.run.target088172:Running: 'rmdir -- /home/ubuntu/cephtest/mnt.0'
2016-02-03T22:02:42.222 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash
2016-02-03T22:02:42.222 INFO:tasks.mon_thrash:joining mon_thrasher
2016-02-03T22:02:42.224 ERROR:teuthology.run_tasks:Manager failed: mon_thrash
Traceback (most recent call last):
File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 125, in run_tasks
suppress = manager.__exit__(*exc_info)
File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
self.gen.next()
File "/home/teuthworker/src/ceph-qa-suite_hammer/tasks/mon_thrash.py", line 341, in task
thrash_proc.do_join()
File "/home/teuthworker/src/ceph-qa-suite_hammer/tasks/mon_thrash.py", line 143, in do_join
self.thread.get()
File "/home/teuthworker/src/teuthology_master/virtualenv/local/lib/python2.7/site-packages/gevent/greenlet.py", line 251, in get
raise self._exception
AssertionError
Updated by Sage Weil about 8 years ago
- Status changed from New to Need More Info
I don't see any useful info in the log. mon_thrash is doing it's thing...
2016-02-03T21:35:57.524 INFO:tasks.mon_thrash.ceph_manager:quorum_status is {"election_epoch":142,"quorum":[0,1,2],"quorum_names":["b","a","c"],"quorum_leader_name":"b","monmap":{"epoch":1,"fsid":"da47afcb-e58c-4933-a5e9-c409fdf7fd6c","modified":"2016-02-03 20:38:01.342719","created":"2016-02-03 20:38:01.342719","m
ons":[{"rank":0,"name":"b","addr":"158.69.88.172:6789\/0"},{"rank":1,"name":"a","addr":"158.69.88.173:6789\/0"},{"rank":2,"name":"c","addr":"158.69.88.172:6790\/0"}]}}
2016-02-03T21:35:57.525 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3
gets the quorum size it expects, and then no output until we try to join the thread 30 minutes later on shutdown.
Updated by Sage Weil about 7 years ago
- Status changed from Need More Info to Can't reproduce
Actions