Project

General

Profile

Actions

Bug #15344

closed

pgs stuck causing a timeout in upgrade:firefly-hammer-x-infernalis-distro-basic-openstack

Added by Yuri Weinstein about 8 years ago. Updated about 7 years ago.

Status:
Can't reproduce
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
upgrade/firefly-hammer-x
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Run: http://pulpito.ovh.sepia.ceph.com:8081/teuthology-2016-03-31_11:18:04-upgrade:firefly-hammer-x-infernalis-distro-basic-openstack/
Jobs: 26266, 26271
Logs: http://teuthology.ovh.sepia.ceph.com/teuthology/teuthology-2016-03-31_11:18:04-upgrade:firefly-hammer-x-infernalis-distro-basic-openstack/26266/remote/target075176/log/ceph.log.gz

2312 bytes data, 21104 MB used, 139 GB / 159 GB avail
2016-03-31 12:53:02.499107 osd.0 158.69.75.176:6801/10253 2 : cluster [WRN] 1 slow requests, 1 included below; oldest blocked for > 30.385216 secs
2016-03-31 12:53:02.499161 osd.0 158.69.75.176:6801/10253 3 : cluster [WRN] slow request 30.385216 seconds old, received at 2016-03-31 12:52:32.113803: osd_op(client.44107.0:1 foo [write 0~128] 263.7fc1f406 RETRY=1 ack+ondisk+retry+write+known_if_redirected e1676) currently no flag points reached
2016-03-31 12:53:12.265540 mon.0 158.69.75.174:6789/0 94 : cluster [INF] HEALTH_WARN; 3 pgs stuck inactive; 3 pgs stuck unclean
2016-03-31 12:53:32.504072 osd.0 158.69.75.176:6801/10253 4 : cluster [WRN] 1 slow requests, 1 included below; oldest blocked for > 60.390186 secs
2016-03-31 12:53:32.504102 osd.0 158.69.75.176:6801/10253 5 : cluster [WRN] slow request 60.390186 seconds old, received at 2016-03-31 12:52:32.113803: osd_op(client.44107.0:1 foo [write 0~128] 263.7fc1f406 RETRY=1 ack+ondisk+retry+write+known_if_redirected e1676) currently no flag points reached
2016-03-31 12:54:32.515652 osd.0 158.69.75.176:6801/10253 6 : cluster [WRN] 1 slow requests, 1 included below; oldest blocked for > 120.401713 secs
2016-03-31 12:54:32.515859 osd.0 158.69.75.176:6801/10253 7 : cluster [WRN] slow request 120.401713 seconds old, received at 2016-03-31 12:52:32.113803: osd_op(client.44107.0:1 foo [write 0~128] 263.7fc1f406 RETRY=1 ack+ondisk+retry+write+known_if_redirected e1676) currently no flag points reached
2016-03-31 12:54:37.571914 mon.0 158.69.75.174:6789/0 99 : cluster [INF] pgmap v2631: 80 pgs: 3 creating, 77 active+clean; 2312 bytes data, 21105 MB used, 139 GB / 159 GB avail
2016-03-31 12:54:38.599434 mon.0 158.69.75.174:6789/0 100 : cluster [INF] pgmap v2632: 80 pgs: 3 creating, 77 active+clean; 2312 bytes data, 21106 MB used, 139 GB / 159 GB avail
2016-03-31 12:54:40.698440 mon.0 158.69.75.174:6789/0 101 : cluster [INF] pgmap v2633: 80 pgs: 3 creating, 77 active+clean; 2312 bytes data, 21108 MB used, 139 GB / 159 GB avail
2016-03-31 12:54:42.721234 mon.0 158.69.75.174:6789/0 102 : cluster [INF] pgmap v2634: 80 pgs: 3 creating, 77 active+clean; 2312 bytes data, 21109 MB used, 139 GB / 159 GB avail
2016-03-31 12:55:12.266396 mon.0 158.69.75.174:6789/0 105 : cluster [INF] HEALTH_WARN; 3 pgs stuck inactive; 3 pgs stuck unclean; 1 requests are blocked > 32 sec
2016-03-31 12:56:32.540916 osd.0 158.69.75.176:6801/10253 8 : cluster [WRN] 1 slow requests, 1 included below; oldest blocked for > 240.427032 secs
2016-03-31 12:56:32.540939 osd.0 158.69.75.176:6801/10253 9 : cluster [WRN] slow request 240.427032 seconds old, received at 2016-03-31 12:52:32.113803: osd_op(client.44107.0:1 foo [write 0~128] 263.7fc1f406 RETRY=1 ack+ondisk+retry+write+known_if_redirected e1676) currently no flag points reached
2016-03-31 12:56:37.598358 mon.0 158.69.75.174:6789/0 110 : cluster [INF] pgmap v2635: 80 pgs: 3 creating, 77 active+clean; 2312 bytes data, 21110 MB used, 139 GB / 159 GB avail
2016-03-31 12:56:38.700888 mon.0 158.69.75.174:6789/0 111 : cluster [INF] pgmap v2636: 80 pgs: 3 creating, 77 active+clean; 2312 bytes data, 21111 MB used, 139 GB / 159 GB avail
2016-03-31 12:56:40.723265 mon.0 158.69.75.174:6789/0 112 : cluster [INF] pgmap v2637: 80 pgs: 3 creating, 77 active+clean; 2312 bytes data, 21112 MB used, 139 GB / 159 GB avail
2016-03-31 12:56:42.755486 mon.0 158.69.75.174:6789/0 113 : cluster [INF] pgmap v2638: 80 pgs: 3 creating, 77 active+clean; 2312 bytes data, 21113 MB used, 139 GB / 159 GB avail
2016-03-31 12:58:37.620798 mon.0 158.69.75.174:6789/0 120 : cluster [INF] pgmap v2639: 80 pgs: 3 creating, 77 active+clean; 2312 bytes data, 21114 MB used, 139 GB / 159 GB avail
2016-03-31 12:58:38.627792 mon.0 158.69.75.174:6789/0 121 : cluster [INF] pgmap v2640: 80 pgs: 3 creating, 77 active+clean; 2312 bytes data, 21114 MB used, 139 GB / 159 GB avail
2016-03-31 12:58:40.825738 mon.0 158.69.75.174:6789/0 122 : cluster [INF] pgmap v2641: 80 pgs: 3 creating, 77 active+clean; 2312 bytes data, 21116 MB used, 139 GB / 159 GB avail
2016-03-31 12:58:42.854571 mon.0 158.69.75.174:6789/0 123 : cluster [INF] pgmap v2642: 80 pgs: 3 creating, 77 active+clean; 2312 bytes data, 21117 MB used, 139 GB / 159 GB avail
2016-03-31 12:59:12.268235 mon.0 158.69.75.174:6789/0 125 : cluster [INF] HEALTH_WARN; 3 pgs stuck inactive; 3 pgs stuck unclean
2016-03-31 13:00:00.000408 mon.0 158.69.75.174:6789/0 128 : cluster [INF] HEALTH_WARN; 3 pgs stuck inactive; 3 pgs stuck unclean
2016-03-31 13:00:37.647567 mon.0 158.69.75.174:6789/0 130 : cluster [INF] pgmap v2643: 80 pgs: 3 creating, 77 active+clean; 2312 bytes data, 21118 MB used, 139 GB / 159 GB avail
2016-03-31 13:00:38.676986 mon.0 158.69.75.174:6789/0 131 : cluster [INF] pgmap v2644: 80 pgs: 3 creating, 77 active+clean; 2312 bytes data, 21119 MB used, 139 GB / 159 GB avail
2016-03-31 13:00:40.786910 mon.0 158.69.75.174:6789/0 132 : cluster [INF] pgmap v2645: 80 pgs: 3 creating, 77 active+clean; 2312 bytes data, 21120 MB used, 139 GB / 159 GB avail
2016-03-31 13:00:42.815146 mon.0 158.69.75.174:6789/0 133 : cluster [INF] pgmap v2646: 80 pgs: 3 creating, 77 active+clean; 2312 bytes data, 21122 MB used, 139 GB / 159 GB avail
2016-03-31 13:02:37.670446 mon.0 158.69.75.174:6789/0 140 : cluster [INF] pgmap v2647: 80 pgs: 3 creating, 77 active+clean; 2312 bytes data, 21155 MB used, 139 GB / 159 GB avail
2016-03-31 13:02:38.738453 mon.0 158.69.75.174:6789/0 141 : cluster [INF] pgmap v2648: 80 pgs: 3 creating, 77 active+clean; 2312 bytes data, 21187 MB used, 139 GB / 159 GB avail
2016-03-31 13:02:40.760247 mon.0 158.69.75.174:6789/0 142 : cluster [INF] pgmap v2649: 80 pgs: 3 creating, 77 active+clean; 2312 bytes data, 21189 MB used, 139 GB / 159 GB avail
2016-03-31 13:02:42.818690 mon.0 158.69.75.174:6789/0 143 : cluster [INF] pgmap v2650: 80 pgs: 3 creating, 77 active+clean; 2312 bytes data, 21190 MB used, 139 GB / 159 GB avail
2016-03-31 13:04:37.702294 mon.0 158.69.75.174:6789/0 151 : cluster [INF] pgmap v2651: 80 pgs: 3 creating, 77 active+clean;
Actions #1

Updated by Greg Farnum about 7 years ago

  • Status changed from New to Can't reproduce
Actions

Also available in: Atom PDF