Project

General

Profile

Actions
Copy link

Bug #6521

closed

ceph health hung after upgrading the cluster from bobtail to dumpling to next

Added by Tamilarasi muthamizhan over 10 years ago. Updated over 10 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Sage Weil
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

ceph health hung after upgrading the cluster from bobtail to dumpling to next

ubuntu@teuthology:/a/teuthology-2013-10-10_22:08:14-upgrade-next-testing-basic-plana/44926

2013-10-11T05:30:05.957 INFO:teuthology.task.ceph:Waiting until ceph is healthy...
2013-10-11T05:30:05.957 DEBUG:teuthology.orchestra.run:Running [10.214.132.34]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd dump --format=json'
2013-10-11T05:30:06.019 INFO:teuthology.task.rgw.client.0.out:[10.214.131.21]: 2013-10-11 05:30:06.018614 7f847a9e0780 -1 WARNING: libcurl doesn't support curl_multi_wait()
2013-10-11T05:30:06.019 INFO:teuthology.task.rgw.client.0.out:[10.214.131.21]: 2013-10-11 05:30:06.018621 7f847a9e0780 -1 WARNING: cross zone / region transfer performance may be affected
2013-10-11T05:30:06.297 DEBUG:teuthology.misc:4 of 4 OSDs are up
2013-10-11T05:30:06.297 DEBUG:teuthology.orchestra.run:Running [10.214.132.34]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph health'
2013-10-11T05:30:06.589 DEBUG:teuthology.misc:Ceph health: HEALTH_WARN 63 pgs stale; pool .log has too few pgs; mds cluster is degraded
2013-10-11T05:30:07.590 DEBUG:teuthology.orchestra.run:Running [10.214.132.34]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph health'
2013-10-11T05:30:07.811 DEBUG:teuthology.misc:Ceph health: HEALTH_WARN pool .log has too few pgs; mds cluster is degraded
2013-10-11T05:30:08.811 DEBUG:teuthology.orchestra.run:Running [10.214.132.34]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph health'
2013-10-11T05:30:09.036 DEBUG:teuthology.misc:Ceph health: HEALTH_WARN pool .log has too few pgs; mds cluster is degraded
2013-10-11T05:30:10.037 DEBUG:teuthology.orchestra.run:Running [10.214.132.34]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph health'
2013-10-11T05:30:10.367 DEBUG:teuthology.misc:Ceph health: HEALTH_WARN pool .log has too few pgs; mds cluster is degraded
2013-10-11T05:30:11.367 DEBUG:teuthology.orchestra.run:Running [10.214.132.34]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph health'
2013-10-11T05:30:11.595 DEBUG:teuthology.misc:Ceph health: HEALTH_WARN pool .log has too few pgs; mds cluster is degraded
2013-10-11T05:30:12.595 DEBUG:teuthology.orchestra.run:Running [10.214.132.34]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph health'
2013-10-11T05:30:12.821 DEBUG:teuthology.misc:Ceph health: HEALTH_WARN pool .log has too few pgs; mds cluster is degraded
2013-10-11T05:30:13.821 DEBUG:teuthology.orchestra.run:Running [10.214.132.34]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph health'
2013-10-11T05:30:14.160 DEBUG:teuthology.misc:Ceph health: HEALTH_WARN pool .log has too few pgs
2013-10-11T05:30:15.161 DEBUG:teuthology.orchestra.run:Running [10.214.132.34]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph health'
2013-10-11T05:30:15.383 DEBUG:teuthology.misc:Ceph health: HEALTH_WARN pool .log has too few pgs
2013-10-11T05:30:16.383 DEBUG:teuthology.orchestra.run:Running [10.214.132.34]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph health'
2013-10-11T05:30:16.607 DEBUG:teuthology.misc:Ceph health: HEALTH_WARN pool .log has too few pgs
2013-10-11T05:30:17.608 DEBUG:teuthology.orchestra.run:Running [10.214.132.34]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph health'
2013-10-11T05:30:17.832 DEBUG:teuthology.misc:Ceph health: HEALTH_WARN pool .log has too few pgs
2013-10-11T05:30:18.832 DEBUG:teuthology.orchestra.run:Running [10.214.132.34]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph health'
2013-10-11T05:30:19.056 DEBUG:teuthology.misc:Ceph health: HEALTH_WARN pool .log has too few pgs
2013-10-11T05:30:20.057 DEBUG:teuthology.orchestra.run:Running [10.214.132.34]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph health'
2013-10-11T05:30:20.278 DEBUG:teuthology.misc:Ceph health: HEALTH_WARN pool .log has too few pgs
2013-10-11T05:30:21.277 DEBUG:teuthology.orchestra.run:Running [10.214.132.34]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph health'
2013-10-11T05:30:21.504 DEBUG:teuthology.misc:Ceph health: HEALTH_WARN pool .log has too few pgs
2013-10-11T05:30:22.504 DEBUG:teuthology.orchestra.run:Running [10.214.132.34]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph health'
2013-10-11T05:30:22.731 DEBUG:teuthology.misc:Ceph health: HEALTH_WARN pool .log has too few pgs
2013-10-11T05:30:23.731 DEBUG:teuthology.orchestra.run:Running [10.214.132.34]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph health'
2013-10-11T05:30:23.957 DEBUG:teuthology.misc:Ceph health: HEALTH_WARN pool .log has too few pgs
2013-10-11T05:30:24.957 DEBUG:teuthology.orchestra.run:Running [10.214.132.34]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph health'
2013-10-11T05:30:25.179 DEBUG:teuthology.misc:Ceph health: HEALTH_WARN pool .log has too few pgs
2013-10-11T05:30:26.180 DEBUG:teuthology.orchestra.run:Running [10.214.132.34]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph health'
2013-10-11T05:30:26.399 DEBUG:teuthology.misc:Ceph health: HEALTH_WARN pool .log has too few pgs
2013-10-11T05:30:27.400 DEBUG:teuthology.orchestra.run:Running [10.214.132.34]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph health'
2013-10-11T05:30:27.626 DEBUG:teuthology.misc:Ceph health: HEALTH_WARN pool .log has too few pgs
2013-10-11T05:30:28.625 DEBUG:teuthology.orchestra.run:Running [10.214.132.34]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph health'
2013-10-11T05:30:28.853 DEBUG:teuthology.misc:Ceph health: HEALTH_WARN pool .log has too few pgs
2013-10-11T05:30:29.853 DEBUG:teuthology.orchestra.run:Running [10.214.132.34]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph health'
2013-10-11T05:30:30.079 DEBUG:teuthology.misc:Ceph health: HEALTH_WARN pool .log has too few pgs
2013-10-11T05:30:31.079 DEBUG:teuthology.orchestra.run:Running [10.214.132.34]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph health'
2013-10-11T05:30:31.302 DEBUG:teuthology.misc:Ceph health: HEALTH_WARN pool .log has too few pgs

there is no core or anything unusual except for these warning messages.

Actions
Copy link
 #1

Updated by Tamilarasi muthamizhan over 10 years ago 

ubuntu@teuthology:/a/teuthology-2013-10-10_22:08:14-upgrade-next-testing-basic-plana/44926$ grep role teuthology.log 
  roles:
2013-10-11T05:09:06.046 INFO:teuthology.task.internal:roles: ubuntu@plana44.front.sepia.ceph.com - ['mon.a', 'mds.a', 'osd.0', 'osd.1']
2013-10-11T05:09:06.046 INFO:teuthology.task.internal:roles: ubuntu@plana21.front.sepia.ceph.com - ['mon.b', 'mon.c', 'osd.2', 'osd.3']
2013-10-11T05:09:06.046 INFO:teuthology.task.internal:roles: ubuntu@plana19.front.sepia.ceph.com - ['client.0']

these machines are left in the current state for anyone interested to take a look.
Actions
Copy link
 #2

Updated by Ian Colle over 10 years ago

  • Assignee set to Sage Weil
Actions
Copy link
 #3

Updated by Sage Weil over 10 years ago

  • Priority changed from High to Urgent
Actions
Copy link
 #4

Updated by Sage Weil over 10 years ago

  • Status changed from New to Resolved
Actions
Copy link

Also available in: Atom PDF