Actions
Bug #9459
closedosd: blocked request
Status:
Can't reproduce
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Development
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
root@mira049:~# ceph daemon osd.8 dump_ops_in_flight { "num_ops": 2, "ops": [ { "description": "osd_op(mds.0.75:106846 10003266950.00000000 [??? 1~0,omap-set-header 0~222,omap-set-vals 0~1204] 1.947916dc ondisk+write+known_if_redirected e361891)", "initiated_at": "2014-09-12 08:40:17.158121", "age": "99137.806665", "duration": "0.000000", "type_data": [ "no flag points reached", [ { "time": "2014-09-12 08:40:17.158121", "event": "initiated"}]]}, { "description": "osd_op(mds.0.75:1495523 1000073339e.00000000 [omap-get-header 0~0,omap-get-vals 0~16] 1.52bd10de ack+read+known_if_redirected e361989)", "initiated_at": "2014-09-12 09:32:39.316369", "age": "95995.648417", "duration": "0.000000", "type_data": [ "no flag points reached", [ { "time": "2014-09-12 09:32:39.316369", "event": "initiated"}]]}]} root@mira049:~# ceph daemon osd.8 version {"version":"0.85-669-gc050afd"}
Updated by Sage Weil over 9 years ago
- Priority changed from High to Urgent
saw something similar on another cluster,
{ "ops": [ ? 191Mb 381Mb 572Mb 763Mb 954Mb { "description": "osd_op(osd.16.207:1 100000056f8.0000021a [copy-from ver 1448] 2.9ad0bed3 snapc 1=[] ondisk+write+ignore_overlay+enforce_snapc+known??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? _if_redirected e208)", ?10.3.64.35 => 10.3.64.32 70.9Kb 2.11Mb 1.13Mb "initiated_at": "2014-09-28 20:47:06.941658", ? <= 193Kb 300Kb 402Kb "age": "91226.642188", ?10.3.64.35 => 10.3.64.34 76.4Kb 69.3Kb 63.6Kb "duration": "0.000000", ? <= 76.1Kb 69.2Kb 69.0Kb "type_data": [ ?10.3.64.35 => 10.3.64.33 56.1Kb 66.0Kb 73.8Kb "no flag points reached", ? <= 56.1Kb 66.0Kb 71.6Kb [ ?10.3.64.35 => 10.3.64.60 3.22Kb 2.00Kb 1.96Kb { "time": "2014-09-28 20:47:06.941658", ? <= 624b 416b 395b "event": "initiated"}]]}, ?10.3.64.35 => 10.3.64.23 0b 166b 42b { "description": "osd_op(osd.16.207:2 10000004f6e.0000010c [copy-from ver 1342] 2.b100ced3 snapc 1=[] ondisk+write+ignore_overlay+enforce_snapc+known? <= 0b 333b 83b _if_redirected e208)", ?255.255.255.255 => 10.3.64.60 0b 0b 0b "initiated_at": "2014-09-28 20:47:06.942289", ? <= 0b 0b 26b "age": "91226.641557", ?10.3.64.255 => 10.3.64.60 0b 0b 0b "duration": "0.000000", ? <= 0b 0b 26b "type_data": [ ? "no flag points reached", ? [ ? { "time": "2014-09-28 20:47:06.942289", ? "event": "initiated"}]]}, ? { "description": "osd_op(osd.16.207:3 100000054e0.00000000 [copy-from ver 1429] 2.8c0c6fd3 snapc 1=[] ondisk+write+ignore_overlay+enforce_snapc+known? _if_redirected e208)", ? "initiated_at": "2014-09-28 20:47:06.942660", ? "age": "91226.641186", ? "duration": "0.000000", ? "type_data": [ ? "no flag points reached", ? [ ? { "time": "2014-09-28 20:47:06.942660", ? "event": "initiated"}]]}, ?
libust[593/593]: Warning: HOME environment variable not set. Disabling LTTng-UST per-user tracing. (in setup_local_apps() at lttng-ust-comm.c:253) ? cluster ad4fcf3a-b077-487d-9ae7-e3dc3d955f53 ?HEALTH_WARN 10 requests are blocked > 32 sec; 3 osds have slow requests; noout === osd.9 === ? health HEALTH_WARN 10 requests are blocked > 32 sec; noout flag(s) set ? flag(s) set Starting Ceph osd.9 on corb...already running ? monmap e2: 4 mons at {cora=10.3.64.32:6789/0,corb=10.3.64.33:6789/0,corc=1?6 ops are blocked > 134218 sec libust[711/711]: Warning: HOME environment variable not set. Disabling LTTng-UST per-user tracing. (in setup_local_apps() at lttng-ust-comm.c:253) ?0.3.64.34:6789/0,cord=10.3.64.35:6789/0}, election epoch 78, quorum 0,1,2,3 cor?4 ops are blocked > 2097.15 sec === osd.2 === ?a,corb,corc,cord ?4 ops are blocked > 134218 sec on osd.2 Starting Ceph osd.2 on corb...already running ? mdsmap e125: 1/1/1 up {0=corc=up:active}, 3 up:standby ?2 ops are blocked > 134218 sec on osd.6 libust[823/823]: Warning: HOME environment variable not set. Disabling LTTng-UST per-user tracing. (in setup_local_apps() at lttng-ust-comm.c:253) ? osdmap e569: 20 osds: 20 up, 20 in ?4 ops are blocked > 2097.15 sec on osd.16 === osd.4 === ? flags noout ?3 osds have slow requests Starting Ceph osd.4 on corb...already running ? pgmap v25869: 512 pgs, 4 pools, 343 GB data, 194 kobjects ?noout flag(s) set [root@corb ~]# logout ? 680 GB used, 18288 GB / 18968 GB avail ?
ceph version 0.85-931-gf8ac224 (f8ac2248af8a7f04094c6d2f8844e928212ff6b0) ? ?
Updated by Sage Weil over 9 years ago
- Status changed from Need More Info to Can't reproduce
Actions