Actions
Bug #9459
closedosd: blocked request
Status:
Can't reproduce
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Development
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
root@mira049:~# ceph daemon osd.8 dump_ops_in_flight
{ "num_ops": 2,
"ops": [
{ "description": "osd_op(mds.0.75:106846 10003266950.00000000 [??? 1~0,omap-set-header 0~222,omap-set-vals 0~1204] 1.947916dc ondisk+write+known_if_redirected e361891)",
"initiated_at": "2014-09-12 08:40:17.158121",
"age": "99137.806665",
"duration": "0.000000",
"type_data": [
"no flag points reached",
[
{ "time": "2014-09-12 08:40:17.158121",
"event": "initiated"}]]},
{ "description": "osd_op(mds.0.75:1495523 1000073339e.00000000 [omap-get-header 0~0,omap-get-vals 0~16] 1.52bd10de ack+read+known_if_redirected e361989)",
"initiated_at": "2014-09-12 09:32:39.316369",
"age": "95995.648417",
"duration": "0.000000",
"type_data": [
"no flag points reached",
[
{ "time": "2014-09-12 09:32:39.316369",
"event": "initiated"}]]}]}
root@mira049:~# ceph daemon osd.8 version
{"version":"0.85-669-gc050afd"}
Updated by Sage Weil over 9 years ago
- Priority changed from High to Urgent
saw something similar on another cluster,
{ "ops": [ ? 191Mb 381Mb 572Mb 763Mb 954Mb
{ "description": "osd_op(osd.16.207:1 100000056f8.0000021a [copy-from ver 1448] 2.9ad0bed3 snapc 1=[] ondisk+write+ignore_overlay+enforce_snapc+known???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
_if_redirected e208)", ?10.3.64.35 => 10.3.64.32 70.9Kb 2.11Mb 1.13Mb
"initiated_at": "2014-09-28 20:47:06.941658", ? <= 193Kb 300Kb 402Kb
"age": "91226.642188", ?10.3.64.35 => 10.3.64.34 76.4Kb 69.3Kb 63.6Kb
"duration": "0.000000", ? <= 76.1Kb 69.2Kb 69.0Kb
"type_data": [ ?10.3.64.35 => 10.3.64.33 56.1Kb 66.0Kb 73.8Kb
"no flag points reached", ? <= 56.1Kb 66.0Kb 71.6Kb
[ ?10.3.64.35 => 10.3.64.60 3.22Kb 2.00Kb 1.96Kb
{ "time": "2014-09-28 20:47:06.941658", ? <= 624b 416b 395b
"event": "initiated"}]]}, ?10.3.64.35 => 10.3.64.23 0b 166b 42b
{ "description": "osd_op(osd.16.207:2 10000004f6e.0000010c [copy-from ver 1342] 2.b100ced3 snapc 1=[] ondisk+write+ignore_overlay+enforce_snapc+known? <= 0b 333b 83b
_if_redirected e208)", ?255.255.255.255 => 10.3.64.60 0b 0b 0b
"initiated_at": "2014-09-28 20:47:06.942289", ? <= 0b 0b 26b
"age": "91226.641557", ?10.3.64.255 => 10.3.64.60 0b 0b 0b
"duration": "0.000000", ? <= 0b 0b 26b
"type_data": [ ?
"no flag points reached", ?
[ ?
{ "time": "2014-09-28 20:47:06.942289", ?
"event": "initiated"}]]}, ?
{ "description": "osd_op(osd.16.207:3 100000054e0.00000000 [copy-from ver 1429] 2.8c0c6fd3 snapc 1=[] ondisk+write+ignore_overlay+enforce_snapc+known?
_if_redirected e208)", ?
"initiated_at": "2014-09-28 20:47:06.942660", ?
"age": "91226.641186", ?
"duration": "0.000000", ?
"type_data": [ ?
"no flag points reached", ?
[ ?
{ "time": "2014-09-28 20:47:06.942660", ?
"event": "initiated"}]]}, ?
libust[593/593]: Warning: HOME environment variable not set. Disabling LTTng-UST per-user tracing. (in setup_local_apps() at lttng-ust-comm.c:253) ? cluster ad4fcf3a-b077-487d-9ae7-e3dc3d955f53 ?HEALTH_WARN 10 requests are blocked > 32 sec; 3 osds have slow requests; noout
=== osd.9 === ? health HEALTH_WARN 10 requests are blocked > 32 sec; noout flag(s) set ? flag(s) set
Starting Ceph osd.9 on corb...already running ? monmap e2: 4 mons at {cora=10.3.64.32:6789/0,corb=10.3.64.33:6789/0,corc=1?6 ops are blocked > 134218 sec
libust[711/711]: Warning: HOME environment variable not set. Disabling LTTng-UST per-user tracing. (in setup_local_apps() at lttng-ust-comm.c:253) ?0.3.64.34:6789/0,cord=10.3.64.35:6789/0}, election epoch 78, quorum 0,1,2,3 cor?4 ops are blocked > 2097.15 sec
=== osd.2 === ?a,corb,corc,cord ?4 ops are blocked > 134218 sec on osd.2
Starting Ceph osd.2 on corb...already running ? mdsmap e125: 1/1/1 up {0=corc=up:active}, 3 up:standby ?2 ops are blocked > 134218 sec on osd.6
libust[823/823]: Warning: HOME environment variable not set. Disabling LTTng-UST per-user tracing. (in setup_local_apps() at lttng-ust-comm.c:253) ? osdmap e569: 20 osds: 20 up, 20 in ?4 ops are blocked > 2097.15 sec on osd.16
=== osd.4 === ? flags noout ?3 osds have slow requests
Starting Ceph osd.4 on corb...already running ? pgmap v25869: 512 pgs, 4 pools, 343 GB data, 194 kobjects ?noout flag(s) set
[root@corb ~]# logout ? 680 GB used, 18288 GB / 18968 GB avail ?
ceph version 0.85-931-gf8ac224 (f8ac2248af8a7f04094c6d2f8844e928212ff6b0) ? ?
Updated by Sage Weil over 9 years ago
- Status changed from Need More Info to Can't reproduce
Actions