Project

General

Profile

Bug #23664

cache-try-flush hits wrlock, busy loops

Added by Sage Weil about 1 year ago. Updated about 1 year ago.

Status:
Resolved
Priority:
High
Assignee:
-
Category:
Tiering
Target version:
-
Start date:
04/11/2018
Due date:
% Done:

0%

Source:
Tags:
Backport:
luminous
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:

Description

...
2018-04-10 21:13:28.536 7fedfa5f6700 10 osd.6 pg_epoch: 106 pg[3.3( v 106'284 (0'0,106'284] local-lis/les=105/106 n=48 ec=19/19 lis/c 105/68 les/c/f 106/69/0 104/105/104) [6,1,5]/[6,1] async=[5] r=0 lpr=105 pi=[68,105)/1 rops=1 crt=106'284 lcod 103'282 mlcod 28'20 active+recovering+undersized+degraded+remapped mbc={255={(2+0)=1,(2+1)=46}} ps=[23~1,26~1,29~1,2d~1,30~1,34~3,3a~1,3c~1,3e~1,42~1]] try_flush_mark_clean waiting on write lock
2018-04-10 21:13:28.536 7fedfa5f6700 20 osd.6 pg_epoch: 106 pg[3.3( v 106'284 (0'0,106'284] local-lis/les=105/106 n=48 ec=19/19 lis/c 105/68 les/c/f 106/69/0 104/105/104) [6,1,5]/[6,1] async=[5] r=0 lpr=105 pi=[68,105)/1 rops=1 crt=106'284 lcod 103'282 mlcod 28'20 active+recovering+undersized+degraded+remapped mbc={255={(2+0)=1,(2+1)=46}} ps=[23~1,26~1,29~1,2d~1,30~1,34~3,3a~1,3c~1,3e~1,42~1]] requeue_op 0x556c91736a00
2018-04-10 21:13:28.536 7fedfa5f6700 20 osd.6 op_wq(3) _enqueue_front OpQueueItem(3.3 PGOpItem(op=osd_op(client.4455.0:3308 3.3 3:f109240a:::smithi02615298-32:45 [cache-try-flush] snapc 0=[] ondisk+read+skiprwlocks+known_if_redirected e104) v9) prio 63 cost 0 e106)
2018-04-10 21:13:28.536 7fee02606700 20 osd.6 op_wq(3) _process 3.3 to_process <> waiting <> waiting_peering {}
2018-04-10 21:13:28.536 7fee02606700 20 osd.6 op_wq(3) _process OpQueueItem(3.3 PGOpItem(op=osd_op(client.4455.0:3308 3.3 3:f109240a:::smithi02615298-32:45 [cache-try-flush] snapc 0=[] ondisk+read+skiprwlocks+known_if_redirected e104) v9) prio 63 cost 0 e106) queued
2018-04-10 21:13:28.536 7fee02606700 20 osd.6 op_wq(3) _process 3.3 to_process <OpQueueItem(3.3 PGOpItem(op=osd_op(client.4455.0:3308 3.3 3:f109240a:::smithi02615298-32:45 [cache-try-flush] snapc 0=[] ondisk+read+skiprwlocks+known_if_redirected e104) v9) prio 63 cost 0 e106)> waiting <> waiting_peering {}
2018-04-10 21:13:28.536 7fee02606700 20 osd.6 op_wq(3) _process OpQueueItem(3.3 PGOpItem(op=osd_op(client.4455.0:3308 3.3 3:f109240a:::smithi02615298-32:45 [cache-try-flush] snapc 0=[] ondisk+read+skiprwlocks+known_if_redirected e104) v9) prio 63 cost 0 e106) pg 0x556c8fb68800
2018-04-10 21:13:28.536 7fee02606700 10 osd.6 106 dequeue_op 0x556c91736a00 prio 63 cost 0 latency 2.051467 osd_op(client.4455.0:3308 3.3 3:f109240a:::smithi02615298-32:45 [cache-try-flush] snapc 0=[] ondisk+read+skiprwlocks+known_if_redirected e104) v9 pg pg[3.3( v 106'284 (0'0,106'284] local-lis/les=105/106 n=48 ec=19/19 lis/c 105/68 les/c/f 106/69/0 104/105/104) [6,1,5]/[6,1] async=[5] r=0 lpr=105 pi=[68,105)/1 rops=1 crt=106'284 lcod 103'282 mlcod 28'20 active+recovering+undersized+degraded+rema
pped mbc={255={(2+0)=1,(2+1)=46}} ps=[23~1,26~1,29~1,2d~1,30~1,34~3,3a~1,3c~1,3e~1,42~1]]
2018-04-10 21:13:28.536 7fee02606700 10 osd.6 pg_epoch: 106 pg[3.3( v 106'284 (0'0,106'284] local-lis/les=105/106 n=48 ec=19/19 lis/c 105/68 les/c/f 106/69/0 104/105/104) [6,1,5]/[6,1] async=[5] r=0 lpr=105 pi=[68,105)/1 rops=1 crt=106'284 lcod 103'282 mlcod 28'20 active+recovering+undersized+degraded+remapped mbc={255={(2+0)=1,(2+1)=46}} ps=[23~1,26~1,29~1,2d~1,30~1,34~3,3a~1,3c~1,3e~1,42~1]] _handle_message: 0x556c91736a00
2018-04-10 21:13:28.536 7fee02606700 20 osd.6 pg_epoch: 106 pg[3.3( v 106'284 (0'0,106'284] local-lis/les=105/106 n=48 ec=19/19 lis/c 105/68 les/c/f 106/69/0 104/105/104) [6,1,5]/[6,1] async=[5] r=0 lpr=105 pi=[68,105)/1 rops=1 crt=106'284 lcod 103'282 mlcod 28'20 active+recovering+undersized+degraded+remapped mbc={255={(2+0)=1,(2+1)=46}} ps=[23~1,26~1,29~1,2d~1,30~1,34~3,3a~1,3c~1,3e~1,42~1]] do_op: op osd_op(client.4455.0:3308 3.3 3:f109240a:::smithi02615298-32:45 [cache-try-flush] snapc 0=[] ondisk+read+skiprwlocks+known_if_redirected e104) v9
2018-04-10 21:13:28.536 7fee02606700 20 osd.6 pg_epoch: 106 pg[3.3( v 106'284 (0'0,106'284] local-lis/les=105/106 n=48 ec=19/19 lis/c 105/68 les/c/f 106/69/0 104/105/104) [6,1,5]/[6,1] async=[5] r=0 lpr=105 pi=[68,105)/1 rops=1 crt=106'284 lcod 103'282 mlcod 28'20 active+recovering+undersized+degraded+remapped mbc={255={(2+0)=1,(2+1)=46}} ps=[23~1,26~1,29~1,2d~1,30~1,34~3,3a~1,3c~1,3e~1,42~1]] op_has_sufficient_caps session=0x556c907cf900 pool=3 (cache ) owner=0 pool_app_metadata={} need_read_cap=0 need_write_cap=0 classes=[] -> yes
2018-04-10 21:13:28.536 7fee02606700 10 osd.6 pg_epoch: 106 pg[3.3( v 106'284 (0'0,106'284] local-lis/les=105/106 n=48 ec=19/19 lis/c 105/68 les/c/f 106/69/0 104/105/104) [6,1,5]/[6,1] async=[5] r=0 lpr=105 pi=[68,105)/1 rops=1 crt=106'284 lcod 103'282 mlcod 28'20 active+recovering+undersized+degraded+remapped mbc={255={(2+0)=1,(2+1)=46}} ps=[23~1,26~1,29~1,2d~1,30~1,34~3,3a~1,3c~1,3e~1,42~1]] do_op osd_op(client.4455.0:3308 3.3 3:f109240a:::smithi02615298-32:45 [cache-try-flush] snapc 0=[] ondisk+read+skiprwlocks+known_if_redirected e104) v9 may_cache -> write-ordered flags ondisk+read+skiprwlocks+known_if_redirected
2018-04-10 21:13:28.536 7fee02606700 10 osd.6 pg_epoch: 106 pg[3.3( v 106'284 (0'0,106'284] local-lis/les=105/106 n=48 ec=19/19 lis/c 105/68 les/c/f 106/69/0 104/105/104) [6,1,5]/[6,1] async=[5] r=0 lpr=105 pi=[68,105)/1 rops=1 crt=106'284 lcod 103'282 mlcod 28'20 active+recovering+undersized+degraded+remapped mbc={255={(2+0)=1,(2+1)=46}} ps=[23~1,26~1,29~1,2d~1,30~1,34~3,3a~1,3c~1,3e~1,42~1]] readable_with_acting: locs:0,1,6
2018-04-10 21:13:28.536 7fee02606700 10 osd.6 pg_epoch: 106 pg[3.3( v 106'284 (0'0,106'284] local-lis/les=105/106 n=48 ec=19/19 lis/c 105/68 les/c/f 106/69/0 104/105/104) [6,1,5]/[6,1] async=[5] r=0 lpr=105 pi=[68,105)/1 rops=1 crt=106'284 lcod 103'282 mlcod 28'20 active+recovering+undersized+degraded+remapped mbc={255={(2+0)=1,(2+1)=46}} ps=[23~1,26~1,29~1,2d~1,30~1,34~3,3a~1,3c~1,3e~1,42~1]] find_object_context 3:f109240a:::smithi02615298-32:45 @45 snapset 0=[]:{}
2018-04-10 21:13:28.536 7fee02606700 10 osd.6 pg_epoch: 106 pg[3.3( v 106'284 (0'0,106'284] local-lis/les=105/106 n=48 ec=19/19 lis/c 105/68 les/c/f 106/69/0 104/105/104) [6,1,5]/[6,1] async=[5] r=0 lpr=105 pi=[68,105)/1 rops=1 crt=106'284 lcod 103'282 mlcod 28'20 active+recovering+undersized+degraded+remapped mbc={255={(2+0)=1,(2+1)=46}} ps=[23~1,26~1,29~1,2d~1,30~1,34~3,3a~1,3c~1,3e~1,42~1]] get_object_context: found obc in cache: 0x556c90f77340
2018-04-10 21:13:28.536 7fee02606700 10 osd.6 pg_epoch: 106 pg[3.3( v 106'284 (0'0,106'284] local-lis/les=105/106 n=48 ec=19/19 lis/c 105/68 les/c/f 106/69/0 104/105/104) [6,1,5]/[6,1] async=[5] r=0 lpr=105 pi=[68,105)/1 rops=1 crt=106'284 lcod 103'282 mlcod 28'20 active+recovering+undersized+degraded+remapped mbc={255={(2+0)=1,(2+1)=46}} ps=[23~1,26~1,29~1,2d~1,30~1,34~3,3a~1,3c~1,3e~1,42~1]] get_object_context: 0x556c90f77340 3:f109240a:::smithi02615298-32:head rwstate(read n=1 w=1) oi: 3:f109240a:::smithi02615298-32:head(28'21 client.4455.0:127 dirty s 2521017 uv 21 alloc_hint [0 0 0]) exists: 1 ssc: 0x556c91512420 snapset: 0=[]:{}
2018-04-10 21:13:28.536 7fee02606700 10 osd.6 pg_epoch: 106 pg[3.3( v 106'284 (0'0,106'284] local-lis/les=105/106 n=48 ec=19/19 lis/c 105/68 les/c/f 106/69/0 104/105/104) [6,1,5]/[6,1] async=[5] r=0 lpr=105 pi=[68,105)/1 rops=1 crt=106'284 lcod 103'282 mlcod 28'20 active+recovering+undersized+degraded+remapped mbc={255={(2+0)=1,(2+1)=46}} ps=[23~1,26~1,29~1,2d~1,30~1,34~3,3a~1,3c~1,3e~1,42~1]] find_object_context 3:f109240a:::smithi02615298-32:head want 45 > snapset seq 0 -- HIT 3:f109240a:::smithi02615298-32:head
2018-04-10 21:13:28.536 7fee02606700 10 osd.6 pg_epoch: 106 pg[3.3( v 106'284 (0'0,106'284] local-lis/les=105/106 n=48 ec=19/19 lis/c 105/68 les/c/f 106/69/0 104/105/104) [6,1,5]/[6,1] async=[5] r=0 lpr=105 pi=[68,105)/1 rops=1 crt=106'284 lcod 103'282 mlcod 28'20 active+recovering+undersized+degraded+remapped mbc={255={(2+0)=1,(2+1)=46}} ps=[23~1,26~1,29~1,2d~1,30~1,34~3,3a~1,3c~1,3e~1,42~1]] readable_with_acting: locs:0,1,6
2018-04-10 21:13:28.540 7fee02606700 10 osd.6 pg_epoch: 106 pg[3.3( v 106'284 (0'0,106'284] local-lis/les=105/106 n=48 ec=19/19 lis/c 105/68 les/c/f 106/69/0 104/105/104) [6,1,5]/[6,1] async=[5] r=0 lpr=105 pi=[68,105)/1 rops=1 crt=106'284 lcod 103'282 mlcod 28'20 active+recovering+undersized+degraded+remapped mbc={255={(2+0)=1,(2+1)=46}} ps=[23~1,26~1,29~1,2d~1,30~1,34~3,3a~1,3c~1,3e~1,42~1]] agent_choose_mode flush_mode: idle evict_mode: idle num_objects: 48 num_bytes: 131415847 num_objects_dirty: 23 num_objects_omap: 0 num_dirty: 23 num_user_objects: 48 num_user_bytes: 131430247 num_overhead_bytes: 14400 pool.info.target_max_bytes: 0 pool.info.target_max_objects: 250
2018-04-10 21:13:28.540 7fee02606700 20 osd.6 pg_epoch: 106 pg[3.3( v 106'284 (0'0,106'284] local-lis/les=105/106 n=48 ec=19/19 lis/c 105/68 les/c/f 106/69/0 104/105/104) [6,1,5]/[6,1] async=[5] r=0 lpr=105 pi=[68,105)/1 rops=1 crt=106'284 lcod 103'282 mlcod 28'20 active+recovering+undersized+degraded+remapped mbc={255={(2+0)=1,(2+1)=46}} ps=[23~1,26~1,29~1,2d~1,30~1,34~3,3a~1,3c~1,3e~1,42~1]] agent_choose_mode dirty 0.370967 full 0.774193
2018-04-10 21:13:28.540 7fee02606700 25 osd.6 pg_epoch: 106 pg[3.3( v 106'284 (0'0,106'284] local-lis/les=105/106 n=48 ec=19/19 lis/c 105/68 les/c/f 106/69/0 104/105/104) [6,1,5]/[6,1] async=[5] r=0 lpr=105 pi=[68,105)/1 rops=1 crt=106'284 lcod 103'282 mlcod 28'20 active+recovering+undersized+degraded+remapped mbc={255={(2+0)=1,(2+1)=46}} ps=[23~1,26~1,29~1,2d~1,30~1,34~3,3a~1,3c~1,3e~1,42~1]] maybe_handle_cache_detail 3:f109240a:::smithi02615298-32:head(28'21 client.4455.0:127 dirty s 2521017 uv 21 alloc_hint [0 0 0]) exists missing_oid MIN must_promote 0 in_hit_set 1
2018-04-10 21:13:28.540 7fee02606700 25 osd.6 pg_epoch: 106 pg[3.3( v 106'284 (0'0,106'284] local-lis/les=105/106 n=48 ec=19/19 lis/c 105/68 les/c/f 106/69/0 104/105/104) [6,1,5]/[6,1] async=[5] r=0 lpr=105 pi=[68,105)/1 rops=1 crt=106'284 lcod 103'282 mlcod 28'20 active+recovering+undersized+degraded+remapped mbc={255={(2+0)=1,(2+1)=46}} ps=[23~1,26~1,29~1,2d~1,30~1,34~3,3a~1,3c~1,3e~1,42~1]] do_op oi 3:f109240a:::smithi02615298-32:head(28'21 client.4455.0:127 dirty s 2521017 uv 21 alloc_hint [0 0 0])
2018-04-10 21:13:28.540 7fee02606700 20 osd.6 pg_epoch: 106 pg[3.3( v 106'284 (0'0,106'284] local-lis/les=105/106 n=48 ec=19/19 lis/c 105/68 les/c/f 106/69/0 104/105/104) [6,1,5]/[6,1] async=[5] r=0 lpr=105 pi=[68,105)/1 rops=1 crt=106'284 lcod 103'282 mlcod 28'20 active+recovering+undersized+degraded+remapped mbc={255={(2+0)=1,(2+1)=46}} ps=[23~1,26~1,29~1,2d~1,30~1,34~3,3a~1,3c~1,3e~1,42~1]] do_op: skipping rw locks
2018-04-10 21:13:28.540 7fee02606700 20 osd.6 pg_epoch: 106 pg[3.3( v 106'284 (0'0,106'284] local-lis/les=105/106 n=48 ec=19/19 lis/c 105/68 les/c/f 106/69/0 104/105/104) [6,1,5]/[6,1] async=[5] r=0 lpr=105 pi=[68,105)/1 rops=1 crt=106'284 lcod 103'282 mlcod 28'20 active+recovering+undersized+degraded+remapped mbc={255={(2+0)=1,(2+1)=46}} ps=[23~1,26~1,29~1,2d~1,30~1,34~3,3a~1,3c~1,3e~1,42~1]] do_op obc obc(3:f109240a:::smithi02615298-32:head rwstate(read n=1 w=1))
2018-04-10 21:13:28.540 7fee02606700 10 osd.6 pg_epoch: 106 pg[3.3( v 106'284 (0'0,106'284] local-lis/les=105/106 n=48 ec=19/19 lis/c 105/68 les/c/f 106/69/0 104/105/104) [6,1,5]/[6,1] async=[5] r=0 lpr=105 pi=[68,105)/1 rops=1 crt=106'284 lcod 103'282 mlcod 28'20 active+recovering+undersized+degraded+remapped mbc={255={(2+0)=1,(2+1)=46}} ps=[23~1,26~1,29~1,2d~1,30~1,34~3,3a~1,3c~1,3e~1,42~1]] execute_ctx 0x556c8fe9b800
2018-04-10 21:13:28.540 7fee02606700 10 osd.6 pg_epoch: 106 pg[3.3( v 106'284 (0'0,106'284] local-lis/les=105/106 n=48 ec=19/19 lis/c 105/68 les/c/f 106/69/0 104/105/104) [6,1,5]/[6,1] async=[5] r=0 lpr=105 pi=[68,105)/1 rops=1 crt=106'284 lcod 103'282 mlcod 28'20 active+recovering+undersized+degraded+remapped mbc={255={(2+0)=1,(2+1)=46}} ps=[23~1,26~1,29~1,2d~1,30~1,34~3,3a~1,3c~1,3e~1,42~1]] execute_ctx 3:f109240a:::smithi02615298-32:head [cache-try-flush] ov 28'21 av 106'285 snapc 0=[] snapset 0=[]:{}
2018-04-10 21:13:28.540 7fee02606700 10 osd.6 pg_epoch: 106 pg[3.3( v 106'284 (0'0,106'284] local-lis/les=105/106 n=48 ec=19/19 lis/c 105/68 les/c/f 106/69/0 104/105/104) [6,1,5]/[6,1] async=[5] r=0 lpr=105 pi=[68,105)/1 rops=1 crt=106'284 lcod 103'282 mlcod 28'20 active+recovering+undersized+degraded+remapped mbc={255={(2+0)=1,(2+1)=46}} ps=[23~1,26~1,29~1,2d~1,30~1,34~3,3a~1,3c~1,3e~1,42~1]] do_osd_op 3:f109240a:::smithi02615298-32:head [cache-try-flush]
2018-04-10 21:13:28.540 7fee02606700 10 osd.6 pg_epoch: 106 pg[3.3( v 106'284 (0'0,106'284] local-lis/les=105/106 n=48 ec=19/19 lis/c 105/68 les/c/f 106/69/0 104/105/104) [6,1,5]/[6,1] async=[5] r=0 lpr=105 pi=[68,105)/1 rops=1 crt=106'284 lcod 103'282 mlcod 28'20 active+recovering+undersized+degraded+remapped mbc={255={(2+0)=1,(2+1)=46}} ps=[23~1,26~1,29~1,2d~1,30~1,34~3,3a~1,3c~1,3e~1,42~1]] do_osd_op  cache-try-flush
2018-04-10 21:13:28.540 7fee02606700 10 osd.6 pg_epoch: 106 pg[3.3( v 106'284 (0'0,106'284] local-lis/les=105/106 n=48 ec=19/19 lis/c 105/68 les/c/f 106/69/0 104/105/104) [6,1,5]/[6,1] async=[5] r=0 lpr=105 pi=[68,105)/1 rops=1 crt=106'284 lcod 103'282 mlcod 28'20 active+recovering+undersized+degraded+remapped mbc={255={(2+0)=1,(2+1)=46}} ps=[23~1,26~1,29~1,2d~1,30~1,34~3,3a~1,3c~1,3e~1,42~1]] start_flush 3:f109240a:::smithi02615298-32:head v28'21 uv21 non-blocking/best-effort
2018-04-10 21:13:28.540 7fee02606700 20 osd.6 pg_epoch: 106 pg[3.3( v 106'284 (0'0,106'284] local-lis/les=105/106 n=48 ec=19/19 lis/c 105/68 les/c/f 106/69/0 104/105/104) [6,1,5]/[6,1] async=[5] r=0 lpr=105 pi=[68,105)/1 rops=1 crt=106'284 lcod 103'282 mlcod 28'20 active+recovering+undersized+degraded+remapped mbc={255={(2+0)=1,(2+1)=46}} ps=[23~1,26~1,29~1,2d~1,30~1,34~3,3a~1,3c~1,3e~1,42~1]]  snapset 0=[]:{}
2018-04-10 21:13:28.540 7fee02606700 20 osd.6 pg_epoch: 106 pg[3.3( v 106'284 (0'0,106'284] local-lis/les=105/106 n=48 ec=19/19 lis/c 105/68 les/c/f 106/69/0 104/105/104) [6,1,5]/[6,1] async=[5] r=0 lpr=105 pi=[68,105)/1 rops=1 crt=106'284 lcod 103'282 mlcod 28'20 active+recovering+undersized+degraded+remapped mbc={255={(2+0)=1,(2+1)=46}} ps=[23~1,26~1,29~1,2d~1,30~1,34~3,3a~1,3c~1,3e~1,42~1]] start_flush no older clones
2018-04-10 21:13:28.540 7fee02606700 10 osd.6 pg_epoch: 106 pg[3.3( v 106'284 (0'0,106'284] local-lis/les=105/106 n=48 ec=19/19 lis/c 105/68 les/c/f 106/69/0 104/105/104) [6,1,5]/[6,1] async=[5] r=0 lpr=105 pi=[68,105)/1 rops=1 crt=106'284 lcod 103'282 mlcod 28'20 active+recovering+undersized+degraded+remapped mbc={255={(2+0)=1,(2+1)=46}} ps=[23~1,26~1,29~1,2d~1,30~1,34~3,3a~1,3c~1,3e~1,42~1]] try_flush_mark_clean clearing DIRTY flag for 3:f109240a:::smithi02615298-32:head
2018-04-10 21:13:28.540 7fee02606700 20 osd.6 pg_epoch: 106 pg[3.3( v 106'284 (0'0,106'284] local-lis/les=105/106 n=48 ec=19/19 lis/c 105/68 les/c/f 106/69/0 104/105/104) [6,1,5]/[6,1] async=[5] r=0 lpr=105 pi=[68,105)/1 rops=1 crt=106'284 lcod 103'282 mlcod 28'20 active+recovering+undersized+degraded+remapped mbc={255={(2+0)=1,(2+1)=46}} ps=[23~1,26~1,29~1,2d~1,30~1,34~3,3a~1,3c~1,3e~1,42~1]] simple_opc_create 3:f109240a:::smithi02615298-32:head
2018-04-10 21:13:28.540 7fee02606700 10 osd.6 pg_epoch: 106 pg[3.3( v 106'284 (0'0,106'284] local-lis/les=105/106 n=48 ec=19/19 lis/c 105/68 les/c/f 106/69/0 104/105/104) [6,1,5]/[6,1] async=[5] r=0 lpr=105 pi=[68,105)/1 rops=1 crt=106'284 lcod 103'282 mlcod 28'20 active+recovering+undersized+degraded+remapped mbc={255={(2+0)=1,(2+1)=46}} ps=[23~1,26~1,29~1,2d~1,30~1,34~3,3a~1,3c~1,3e~1,42~1]] try_flush_mark_clean waiting on write lock
2018-04-10 21:13:28.540 7fee02606700 20 osd.6 pg_epoch: 106 pg[3.3( v 106'284 (0'0,106'284] local-lis/les=105/106 n=48 ec=19/19 lis/c 105/68 les/c/f 106/69/0 104/105/104) [6,1,5]/[6,1] async=[5] r=0 lpr=105 pi=[68,105)/1 rops=1 crt=106'284 lcod 103'282 mlcod 28'20 active+recovering+undersized+degraded+remapped mbc={255={(2+0)=1,(2+1)=46}} ps=[23~1,26~1,29~1,2d~1,30~1,34~3,3a~1,3c~1,3e~1,42~1]] requeue_op 0x556c91736a00
...

/a/sage-2018-04-10_19:17:57-rados-wip-sage-testing-2018-04-10-1032-distro-basic-smithi/2380733


Related issues

Related to RADOS - Bug #23847: osd stuck recovery Duplicate 04/24/2018
Copied to RADOS - Backport #23914: luminous: cache-try-flush hits wrlock, busy loops Resolved

History

#1 Updated by Josh Durgin about 1 year ago

  • Category set to Tiering
  • Priority changed from High to Normal

#2 Updated by Sage Weil about 1 year ago

#3 Updated by Sage Weil about 1 year ago

  • Priority changed from Normal to High

reproducing this semi-frequently, see #23847

This should fix it: https://github.com/ceph/ceph/pull/21653

#4 Updated by Sage Weil about 1 year ago

  • Status changed from Verified to Need Review

#5 Updated by Sage Weil about 1 year ago

  • Status changed from Need Review to Pending Backport

#6 Updated by Nathan Cutler about 1 year ago

  • Backport set to luminous

#7 Updated by Nathan Cutler about 1 year ago

  • Copied to Backport #23914: luminous: cache-try-flush hits wrlock, busy loops added

#8 Updated by Nathan Cutler about 1 year ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF