Project

General

Profile

Bug #48917

no reply for copy-get

Added by Neha Ojha about 3 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2021-01-06T08:38:20.276+0000 7f65cdccf700 10 osd.7 pg_epoch: 163 pg[4.0( v 163'10637 (89'7635,163'10637] local-lis/les=25/26 n=1014 ec=25/25 lis/c=25/25 les/c/f=26/26/0 sis=25) [7,3,0] r=0 lpr=25 crt=163'10637 lcod 163'10636 mlcod 163'10636 active+clean] start_copy 4:021d04d8:::smithi15521682-4449:head from 3:4090608d:::smithi15521682-4211:head @3 v0 flags 0
2021-01-06T08:38:20.276+0000 7f65cdccf700  1 -- 172.21.15.64:0/16344 --> [v2:172.21.15.64:6816/16346,v1:172.21.15.64:6818/16346] -- osd_op(unknown.0.13:5378 4.2 4:4090608d:::smithi15521682-4211:head [copy-get max 8388608 in=29b] snapc 0=[] ondisk+read+known_if_redirected e164) v8 -- 0x56505e16de00 con 0x5650510c0800
2021-01-06T08:38:20.276+0000 7f65cdccf700  5 --2- 172.21.15.64:0/16344 >> [v2:172.21.15.64:6816/16346,v1:172.21.15.64:6818/16346] conn(0x5650510c0800 0x5650511aa500 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).send_message enqueueing message m=0x56505e16de00 type=42 osd_op(osd.7.13:5378 4.2 4:4090608d:::smithi15521682-4211:head [copy-get max 8388608 in=29b] snapc 0=[] ondisk+read+known_if_redirected e164) v8
2021-01-06T08:38:20.276+0000 7f65ed784700 20 --2- 172.21.15.64:0/16344 >> [v2:172.21.15.64:6816/16346,v1:172.21.15.64:6818/16346] conn(0x5650510c0800 0x5650511aa500 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).prepare_send_message m=osd_op(osd.7.13:5378 4.2 4:4090608d:::smithi15521682-4211:head [copy-get max 8388608 in=29b] snapc 0=[] ondisk+read+known_if_redirected e164) v8
2021-01-06T08:38:20.276+0000 7f65ed784700 20 --2- 172.21.15.64:0/16344 >> [v2:172.21.15.64:6816/16346,v1:172.21.15.64:6818/16346] conn(0x5650510c0800 0x5650511aa500 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).prepare_send_message encoding features 4540138297136906239 0x56505e16de00 osd_op(osd.7.13:5378 4.2 4:4090608d:::smithi15521682-4211:head [copy-get max 8388608 in=29b] snapc 0=[] ondisk+read+known_if_redirected e164) v8
2021-01-06T08:38:20.276+0000 7f65ed784700  5 --2- 172.21.15.64:0/16344 >> [v2:172.21.15.64:6816/16346,v1:172.21.15.64:6818/16346] conn(0x5650510c0800 0x5650511aa500 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).write_message sending message m=0x56505e16de00 seq=35 osd_op(osd.7.13:5378 4.2 4:4090608d:::smithi15521682-4211:head [copy-get max 8388608 in=29b] snapc 0=[] ondisk+read+known_if_redirected e164) v8
2021-01-06T08:38:21.296+0000 7f65d5cdf700  1 -- 172.21.15.64:0/16344 --> [v2:172.21.15.64:6816/16346,v1:172.21.15.64:6818/16346] -- osd_op(unknown.0.13:5378 4.2 4:4090608d:::smithi15521682-4211:head [copy-get max 8388608 in=29b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e165) v8 -- 0x5650547e3a40 con 0x5650510c0800
2021-01-06T08:38:21.296+0000 7f65d5cdf700  5 --2- 172.21.15.64:0/16344 >> [v2:172.21.15.64:6816/16346,v1:172.21.15.64:6818/16346] conn(0x5650510c0800 0x5650511aa500 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).send_message enqueueing message m=0x5650547e3a40 type=42 osd_op(osd.7.13:5378 4.2 4:4090608d:::smithi15521682-4211:head [copy-get max 8388608 in=29b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e165) v8
2021-01-06T08:38:21.296+0000 7f65ed784700 20 --2- 172.21.15.64:0/16344 >> [v2:172.21.15.64:6816/16346,v1:172.21.15.64:6818/16346] conn(0x5650510c0800 0x5650511aa500 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).prepare_send_message m=osd_op(osd.7.13:5378 4.2 4:4090608d:::smithi15521682-4211:head [copy-get max 8388608 in=29b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e165) v8
2021-01-06T08:38:21.296+0000 7f65ed784700 20 --2- 172.21.15.64:0/16344 >> [v2:172.21.15.64:6816/16346,v1:172.21.15.64:6818/16346] conn(0x5650510c0800 0x5650511aa500 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).prepare_send_message encoding features 4540138297136906239 0x5650547e3a40 osd_op(osd.7.13:5378 4.2 4:4090608d:::smithi15521682-4211:head [copy-get max 8388608 in=29b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e165) v8
2021-01-06T08:38:21.296+0000 7f65ed784700  5 --2- 172.21.15.64:0/16344 >> [v2:172.21.15.64:6816/16346,v1:172.21.15.64:6818/16346] conn(0x5650510c0800 0x5650511aa500 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).write_message sending message m=0x5650547e3a40 seq=37 osd_op(osd.7.13:5378 4.2 4:4090608d:::smithi15521682-4211:head [copy-get max 8388608 in=29b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e165) v8
2021-01-06T08:42:41.893+0000 7f65d5cdf700  1 -- 172.21.15.64:0/16344 --> [v2:172.21.15.64:6816/16346,v1:172.21.15.64:6818/16346] -- osd_op(unknown.0.13:5378 4.2 4:4090608d:::smithi15521682-4211:head [copy-get max 8388608 in=29b] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e215) v8 -- 0x5650547e4d00 con 0x56505c104000
2021-01-06T08:42:41.893+0000 7f65d5cdf700  5 --2- 172.21.15.64:0/16344 >> [v2:172.21.15.64:6816/16346,v1:172.21.15.64:6818/16346] conn(0x56505c104000 0x56505081e300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).send_message enqueueing message m=0x5650547e4d00 type=42 osd_op(osd.7.13:5378 4.2 4:4090608d:::smithi15521682-4211:head [copy-get max 8388608 in=29b] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected e215) v8
.
.
2021-01-06T08:38:20.276+0000 7f65d1cd7700 10 osd.7 pg_epoch: 163 pg[4.0( v 163'10637 (89'7635,163'10637] local-lis/les=25/26 n=1014 ec=25/25 lis/c=25/25 les/c/f=26/26/0 sis=25) [7,3,0] r=0 lpr=25 crt=163'10637 lcod 163'10636 mlcod 163'10636 active+clean] wait_for_blocked_object 4:021d04d8:::smithi15521682-4449:head 0x565056ede680
.
.
2021-01-06T08:39:15.096+0000 7f65cdccf700 10 osd.7 pg_epoch: 185 pg[4.0( v 165'10666 (89'7665,165'10666] local-lis/les=25/26 n=1019 ec=25/25 lis/c=25/25 les/c/f=26/26/0 sis=25) [7,3,0] r=0 lpr=25 crt=165'10666 lcod 165'10665 mlcod 165'10665 active+clean+scrubbing [ 4.0:  ] ] _range_available_for_scrub: scrub delayed, 4:021d04d8:::smithi15521682-4449:head is blocked

the copy-get operation before scrub never received a response, what is weird though is start_copy 4:021d04d8:::smithi15521682-4449:head from 3:4090608d:::smithi15521682-4211:head @3 tells us the object is on pool 3 while the messenger level logging shows the message was sent to PG 4.2.

/a/kchai-2021-01-06_02:57:51-rados-wip-kefu-testing-2021-01-05-2058-distro-basic-smithi/5758565

Also available in: Atom PDF