Actions
Bug #11382
closedstuck in "pgs peering" after upgrade to v0.80.6 in upgrade:firefly-firefly-distro-basic-vps run
Status:
Can't reproduce
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
upgrade/firefly
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Run: http://pulpito.ceph.com/teuthology-2015-04-10_17:00:01-upgrade:firefly-firefly-distro-basic-vps/
Jobs: ['843678', '843679', '843680', '843681', '843682', '843683', '843684', '843685', '843686', '843688', '843689']
Logs for one: http://qa-proxy.ceph.com/teuthology/teuthology-2015-04-10_17:00:01-upgrade:firefly-firefly-distro-basic-vps/843689/
2015-04-11T12:37:38.222 INFO:teuthology.task.print:**** done v0.80.6 install.upgrade 2015-04-11T12:37:38.222 INFO:teuthology.task.sequential:In sequential, running task ceph.restart... 2015-04-11T12:37:44.221 INFO:tasks.ceph.osd.0:Stopped 2015-04-11T12:37:44.222 INFO:tasks.ceph.osd.0:Restarting daemon 2015-04-11T12:37:44.222 INFO:teuthology.orchestra.run.vpm052:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f -i 0' 2015-04-11T12:37:44.225 INFO:tasks.ceph.osd.0:Started 2015-04-11T12:37:44.225 INFO:tasks.ceph:Waiting until ceph is healthy... 2015-04-11T12:37:44.225 INFO:teuthology.orchestra.run.vpm052:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd dump --format=json' 2015-04-11T12:37:45.348 INFO:tasks.ceph.osd.0.vpm052.stdout:starting osd.0 at :/0 osd_data /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal 2015-04-11T12:37:45.685 INFO:tasks.ceph.osd.0.vpm052.stderr:2015-04-11 19:37:45.689833 7fe610461800 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2015-04-11T12:37:45.688 DEBUG:teuthology.misc:6 of 6 OSDs are up 2015-04-11T12:37:45.689 INFO:teuthology.orchestra.run.vpm052:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph health' 2015-04-11T12:37:46.039 DEBUG:teuthology.misc:Ceph health: HEALTH_OK 2015-04-11T12:37:46.039 INFO:teuthology.task.sequential:In sequential, running task sleep... 2015-04-11T12:37:46.039 INFO:teuthology.task.sleep:Sleeping for 30 2015-04-11T12:38:16.039 INFO:teuthology.task.sequential:In sequential, running task ceph.restart... 2015-04-11T12:38:22.039 INFO:tasks.ceph.osd.1:Stopped 2015-04-11T12:38:22.039 INFO:tasks.ceph.osd.1:Restarting daemon 2015-04-11T12:38:22.040 INFO:teuthology.orchestra.run.vpm052:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f -i 1' 2015-04-11T12:38:22.043 INFO:tasks.ceph.osd.1:Started 2015-04-11T12:38:22.043 INFO:tasks.ceph:Waiting until ceph is healthy... 2015-04-11T12:38:22.043 INFO:teuthology.orchestra.run.vpm052:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd dump --format=json' 2015-04-11T12:38:22.145 INFO:tasks.ceph.osd.1.vpm052.stdout:starting osd.1 at :/0 osd_data /var/lib/ceph/osd/ceph-1 /var/lib/ceph/osd/ceph-1/journal 2015-04-11T12:38:22.265 DEBUG:teuthology.misc:6 of 6 OSDs are up 2015-04-11T12:38:22.266 INFO:teuthology.orchestra.run.vpm052:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph health' 2015-04-11T12:38:22.672 INFO:tasks.ceph.osd.1.vpm052.stderr:2015-04-11 19:38:22.677549 7fcb0d872800 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2015-04-11T12:38:22.820 DEBUG:teuthology.misc:Ceph health: HEALTH_WARN 28 pgs peering; 28 pgs stuck inactive; 28 pgs stuck unclean 2015-04-11T12:38:29.821 INFO:teuthology.orchestra.run.vpm052:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph health' 2015-04-11T12:38:30.007 DEBUG:teuthology.misc:Ceph health: HEALTH_WARN 37 pgs peering; 37 pgs stuck inactive; 37 pgs stuck unclean 2015-04-11T12:38:37.008 INFO:teuthology.orchestra.run.vpm052:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph health' 2015-04-11T12:38:37.196 DEBUG:teuthology.misc:Ceph health: HEALTH_WARN 39 pgs peering; 37 pgs stuck inactive; 37 pgs stuck unclean 2015-04-11T12:38:44.197 INFO:teuthology.orchestra.run.vpm052:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph health' 2015-04-11T12:38:44.387 DEBUG:teuthology.misc:Ceph health: HEALTH_WARN 39 pgs peering; 37 pgs stuck inactive; 37 pgs stuck unclean 2015-04-11T12:38:51.388 INFO:teuthology.orchestra.run.vpm052:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph health' .................. 015-04-11T12:56:23.558 INFO:teuthology.orchestra.run.vpm052:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph health' 2015-04-11T12:56:23.906 DEBUG:teuthology.misc:Ceph health: HEALTH_WARN 39 pgs peering; 50 pgs stuck inactive; 50 pgs stuck unclean 2015-04-11T12:56:24.907 ERROR:teuthology.parallel:Exception in parallel execution Traceback (most recent call last): File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 82, in __exit__ for result in self: File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 101, in next resurrect_traceback(result) File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 19, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/teuthology_master/teuthology/task/parallel.py", line 50, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 41, in run_one_task return fn(**kwargs) File "/home/teuthworker/src/teuthology_master/teuthology/task/sequential.py", line 48, in task mgr.__enter__() File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/var/lib/teuthworker/src/ceph-qa-suite_firefly/tasks/ceph.py", line 1086, in restart healthy(ctx=ctx, config=None) File "/var/lib/teuthworker/src/ceph-qa-suite_firefly/tasks/ceph.py", line 994, in healthy remote=mon0_remote, File "/home/teuthworker/src/teuthology_master/teuthology/misc.py", line 874, in wait_until_healthy while proceed(): File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) MaxWhileTries: 'wait_until_healthy' reached maximum tries (150) after waiting for 900 seconds
Actions