Project

General

Profile

Backport #20780

jewel: ceph-osd: PGs getting stuck in scrub state, stalling RBD

Added by David Zafman over 2 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
High
Assignee:
Target version:
Release:
jewel
Crash signature:


Related issues

Copied from RADOS - Bug #20041: ceph-osd: PGs getting stuck in scrub state, stalling RBD Resolved 05/22/2017

History

#1 Updated by David Zafman over 2 years ago

  • Copied from Bug #20041: ceph-osd: PGs getting stuck in scrub state, stalling RBD added

#2 Updated by David Zafman over 2 years ago

https://github.com/ceph/ceph/pull/16405

The master version is going through a test run, but I'm confident it won't find any issues. On the original reporting cluster the fix seems to be working properly.

#3 Updated by David Zafman over 2 years ago

  • Status changed from Pending Backport to In Progress

#4 Updated by David Zafman over 2 years ago

  • Priority changed from Urgent to High

#5 Updated by Nathan Cutler over 2 years ago

  • Tracker changed from Bug to Backport
  • Subject changed from JEWEL: ceph-osd: PGs getting stuck in scrub state, stalling RBD to jewel: ceph-osd: PGs getting stuck in scrub state, stalling RBD
  • Description updated (diff)
  • Release set to jewel

description

See the attached logs for the remove op against rbd_data.21aafa6b8b4567.0000000000000aaa

2017-05-17 20:12:06.943616 7fea1a7ff700 10 osd.23 pg_epoch: 181936 pg[2.aa( v 181936'11193583 (181936'11190568,181936'11193583] local-les=171705 n=4036 ec=114 les/c/f 171705/171705/0 171703/171704/171649) [23,41,9] r=0 lpr=171704 crt=181936'11193581 lcod 181936'11193582 mlcod 181936'11193582 active+clean+scrubbing snaptrimq=[...7b05~1,27b07~1,27b09~1,27b0b~1,27b0d~1,27b0f~1,27b11~2,27b15~2,27b18~1,27b1a~1,27b1c~2,27b22~1,27b24~1,27b29~2,27b2c~1,27b2e~1,27b30~1,27b32~1,27b34~1,27b36~1,27b38~1,27b3a~1,27b3f~1,27b41~1,27b43~1,27b48~1,27b4a~1,27b4c~1,27b4e~1,27b53~4,27b58~1,27b5b~1,27b5d~1,27b61~2,27b64~1,27b67~2,27b6b~1,27b6d~2,27b71~2,27b76~1,27b7d~2,27b83~1,27b86~1,27b88~1,27b8a~1,27b8c~2,27b8f~1,27b96~2,27b99~1,27b9b~1,27b9d~2,27ba0~1,27ba3~1,27ba6~1,27ba9~1,27bab~2,27bae~2,27bb2~2,27bb5~1,27bb8~2,27bbb~1,27bbe~2,27bc1~1,27bc4~1,27bc6~1,27bc8~1,27bcb~2,27bce~1,27bd0~1,27bd2~2,27bd6~2,27bd9~1,27bdc~1,27be0~1,27be3~2,27be9~1,27beb~2,27bef~1,27bf1~2,27bf6~1,27bf8~2,27c02~3,27c06~1,27c08~1,27c0a~1,27c0c~1,27c0e~1,27c10~2,27c13~2,27c16~1,27c19~2,27c1c~3,27c21~1,27c23~1,27c28~1,27c2c~1,27c2e~1,27c32~1,27c37~2,27c3d~1,27c42~1,27c45~1,27c48~3,27c52~1,27c54~1,27c56~1,27c5b~1,27c60~2,27c64~1,27c68~1,27c76~2,27c79~2,27c7c~1,27c7e~1,27c80~1,27c82~1,27c8b~2,27c8f~2,27c92~1,27c94~3,27c98~1,27c9a~1,27c9c~1,27c9f~1,27ca1~1,27ca4~3,27ca9~1,27cab~2,27cb2~3,27cb7~1,27cbb~1,27cbe~2,27cc2~2,27cc8~1,27cca~1,27ccf~1,27cd4~2,27cd8~1,27cdb~2,27cde~1,27ce0~3,27ce5~2,27ce8~2,27cec~1,27cf3~3,27cf8~1,27cfb~2,27cfe~1,27d00~1,27d06~1,27d08~1,27d0f~2,27d12~1,27d14~1,27d17~2,27d1a~1,27d20~2,27d26~2,27d2a~1,27d2e~2,27d32~1,27d34~3,27d3b~3,27d44~1,27d48~2,27d4c~2,27d50~2,27d55~f7b,28cd1~4,28cd7~1,28cd9~1,28cdb~1,28cdd~1,28cdf~1,28ce1~1,28ce3~2,28ce6~3,28ceb~1,28ced~1,28cef~2,28cf2~1,28cf5~1,28cf7~1,28cfa~1,28cfc~7,28d04~2,28d08~2,28d0c~1,28d0e~1,28d10~1,28d12~2,28d16~2,28d19~3,28d1d~1,28d1f~1,28d21~4,28d26~4,28d2b~1,28d2d~1,28d2f~1,28d31~1,28d33~1,28d35~1,28d37~4,28d3d~1,28d3f~1,28d41~1,28d43~1,28d45~1,28d47~1,28d4a~1,28d4c~1,28d4e~1,28d50~1,28d52~1,28d55~4,28d5a~1,28d5d~1,28d5f~1,28d63~1,28d65~3,28d69~1,28d6b~1,28d6d~1,28d6f~1,28d71~1,28d73~1,28d75~1,28d77~3,28d7b~1,28d7d~1,28d7f~1,28d81~1,28d83~1,28d86~1,28d88~4,28d8d~4,28d92~1,28d94~1,28d96~1,28d98~1,28d9a~1,28d9d~1,28d9f~1,28da1~1,28da3~1,28da5~1,28da7~1,28da9~1,28dad~1,28daf~1,28db1~4,28db7~1,28db9~1,28dbb~1,28dbd~2,28dc1~2,28dc5~1,28dc7~1,28dc9~1,28dcb~1,28dcd~2,28dd0~2,28dd3~1,28dd5~1,28dd7~1,28dd9~1,28ddb~4,28de0~3,28de5~3,28de9~3,28ded~1,28def~4,28df4~4,28df9~4,28dfe~4,28e03~1,28e06~1,28e08~1,28e0b~1,28e0d~1,28e0f~1,28e11~1,28e15~1,28e17~4,28e1c~1,28e1e~1,28e20~1,28e22~1,28e24~1,28e26~1,28e28~1,28e2a~1,28e2c~1,28e2e~1,28e30~1,28e32~1,28e34~1,28e36~4,28e3b~1,28e3d~1,28e3f~1,28e41~4,28e46~1,28e48~1,28e4a~1,28e4c~1,28e4e~1,28e50~3,28e54~1,28e56~1,28e58~1,28e5a~4,28e5f~1,28e61~3,28e65~1,28e68~1,28e6a~4,28e70~1,28e72~4,28e77~1,28e79~4,28e7f~1,28e81~1,28e83~1,28e85~1,28e87~1,28e8a~1,28e8c~1,28e8e~1,28e90~1,28e92~4,28e97~1,28e99~1,28e9b~1,28e9d~1,28e9f~1,28ea1~1,28ea3~1,28ea5~1,28ea7~1,28ea9~4,28eae~1,28eb0~1,28eb2~1,28eb4~1,28eb6~1,28eb9~1,28ebb~1,28ebd~1,28ec0~1,28ec5~2,28ec9~1,28ecb~1,28ece~2,28ed1~1,28ed3~2,28ed6~2,28ed9~1,28edb~1,28ede~1,28ee0~1,28ee2~1,28ee4~1,28ee6~1,28ee8~1,28eea~1,28eed~2,28ef1~2,28ef5~3,28ef9~3,28efe~1,28f00~2,28f04~2,28f07~4,28f0d~2,28f11~1,28f16~1,28f18~1,28f1b~2,28f1f~1,28f21~1,28f23~1,28f25~1,28f27~1,28f29~1,28f2b~1,28f2e~2,28f32~1,28f34~2,28f38~2,28f3c~2,28f40~2,28f44~1,28f46~1,28f48~4,28f4e~1,28f50~1,28f52~1,28f54~1,28f57~1,28f5a~1,28f5c~1,28f5e~1,28f61~2,28f64~1,28f67~3,28f6b~2,28f6e~3]] do_op osd_op(client.29745941.0:76103 2.cebed0aa rbd_data.21aafa6b8b4567.0000000000000aaa [delete] snapc 28a43=[] ack+ondisk+write+known_if_redirected e181936) v7 may_write -> write-ordered flags ack+ondisk+write+known_if_redirected

2017-05-17 20:12:06.943670 7fea1a7ff700 20 osd.23 pg_epoch: 181936 pg[2.aa( v 181936'11193583 (181936'11190568,181936'11193583] local-les=171705 n=4036 ec=114 les/c/f 171705/171705/0 171703/171704/171649) [23,41,9] r=0 lpr=171704 crt=181936'11193581 lcod 181936'11193582 mlcod 181936'11193582 active+clean+scrubbing snaptrimq=[...db~1,27add~1,27adf~1,27ae1~2,27ae6~1,27ae8~1,27aea~1,27aec~1,27aee~1,27af0~1,27af2~1,27af4~1,27af6~1,27af8~1,27afa~1,27afc~1,27afe~1,27b00~1,27b02~2,27b05~1,27b07~1,27b09~1,27b0b~1,27b0d~1,27b0f~1,27b11~2,27b15~2,27b18~1,27b1a~1,27b1c~2,27b22~1,27b24~1,27b29~2,27b2c~1,27b2e~1,27b30~1,27b32~1,27b34~1,27b36~1,27b38~1,27b3a~1,27b3f~1,27b41~1,27b43~1,27b48~1,27b4a~1,27b4c~1,27b4e~1,27b53~4,27b58~1,27b5b~1,27b5d~1,27b61~2,27b64~1,27b67~2,27b6b~1,27b6d~2,27b71~2,27b76~1,27b7d~2,27b83~1,27b86~1,27b88~1,27b8a~1,27b8c~2,27b8f~1,27b96~2,27b99~1,27b9b~1,27b9d~2,27ba0~1,27ba3~1,27ba6~1,27ba9~1,27bab~2,27bae~2,27bb2~2,27bb5~1,27bb8~2,27bbb~1,27bbe~2,27bc1~1,27bc4~1,27bc6~1,27bc8~1,27bcb~2,27bce~1,27bd0~1,27bd2~2,27bd6~2,27bd9~1,27bdc~1,27be0~1,27be3~2,27be9~1,27beb~2,27bef~1,27bf1~2,27bf6~1,27bf8~2,27c02~3,27c06~1,27c08~1,27c0a~1,27c0c~1,27c0e~1,27c10~2,27c13~2,27c16~1,27c19~2,27c1c~3,27c21~1,27c23~1,27c28~1,27c2c~1,27c2e~1,27c32~1,27c37~2,27c3d~1,27c42~1,27c45~1,27c48~3,27c52~1,27c54~1,27c56~1,27c5b~1,27c60~2,27c64~1,27c68~1,27c76~2,27c79~2,27c7c~1,27c7e~1,27c80~1,27c82~1,27c8b~2,27c8f~2,27c92~1,27c94~3,27c98~1,27c9a~1,27c9c~1,27c9f~1,27ca1~1,27ca4~3,27ca9~1,27cab~2,27cb2~3,27cb7~1,27cbb~1,27cbe~2,27cc2~2,27cc8~1,27cca~1,27ccf~1,27cd4~2,27cd8~1,27cdb~2,27cde~1,27ce0~3,27ce5~2,27ce8~2,27cec~1,27cf3~3,27cf8~1,27cfb~2,27cfe~1,27d00~1,27d06~1,27d08~1,27d0f~2,27d12~1,27d14~1,27d17~2,27d1a~1,27d20~2,27d26~2,27d2a~1,27d2e~2,27d32~1,27d34~3,27d3b~3,27d44~1,27d48~2,27d4c~2,27d50~2,27d55~f7b,28cd1~4,28cd7~1,28cd9~1,28cdb~1,28cdd~1,28cdf~1,28ce1~1,28ce3~2,28ce6~3,28ceb~1,28ced~1,28cef~2,28cf2~1,28cf5~1,28cf7~1,28cfa~1,28cfc~7,28d04~2,28d08~2,28d0c~1,28d0e~1,28d10~1,28d12~2,28d16~2,28d19~3,28d1d~1,28d1f~1,28d21~4,28d26~4,28d2b~1,28d2d~1,28d2f~1,28d31~1,28d33~1,28d35~1,28d37~4,28d3d~1,28d3f~1,28d41~1,28d43~1,28d45~1,28d47~1,28d4a~1,28d4c~1,28d4e~1,28d50~1,28d52~1,28d55~4,28d5a~1,28d5d~1,28d5f~1,28d63~1,28d65~3,28d69~1,28d6b~1,28d6d~1,28d6f~1,28d71~1,28d73~1,28d75~1,28d77~3,28d7b~1,28d7d~1,28d7f~1,28d81~1,28d83~1,28d86~1,28d88~4,28d8d~4,28d92~1,28d94~1,28d96~1,28d98~1,28d9a~1,28d9d~1,28d9f~1,28da1~1,28da3~1,28da5~1,28da7~1,28da9~1,28dad~1,28daf~1,28db1~4,28db7~1,28db9~1,28dbb~1,28dbd~2,28dc1~2,28dc5~1,28dc7~1,28dc9~1,28dcb~1,28dcd~2,28dd0~2,28dd3~1,28dd5~1,28dd7~1,28dd9~1,28ddb~4,28de0~3,28de5~3,28de9~3,28ded~1,28def~4,28df4~4,28df9~4,28dfe~4,28e03~1,28e06~1,28e08~1,28e0b~1,28e0d~1,28e0f~1,28e11~1,28e15~1,28e17~4,28e1c~1,28e1e~1,28e20~1,28e22~1,28e24~1,28e26~1,28e28~1,28e2a~1,28e2c~1,28e2e~1,28e30~1,28e32~1,28e34~1,28e36~4,28e3b~1,28e3d~1,28e3f~1,28e41~4,28e46~1,28e48~1,28e4a~1,28e4c~1,28e4e~1,28e50~3,28e54~1,28e56~1,28e58~1,28e5a~4,28e5f~1,28e61~3,28e65~1,28e68~1,28e6a~4,28e70~1,28e72~4,28e77~1,28e79~4,28e7f~1,28e81~1,28e83~1,28e85~1,28e87~1,28e8a~1,28e8c~1,28e8e~1,28e90~1,28e92~4,28e97~1,28e99~1,28e9b~1,28e9d~1,28e9f~1,28ea1~1,28ea3~1,28ea5~1,28ea7~1,28ea9~4,28eae~1,28eb0~1,28eb2~1,28eb4~1,28eb6~1,28eb9~1,28ebb~1,28ebd~1,28ec0~1,28ec5~2,28ec9~1,28ecb~1,28ece~2,28ed1~1,28ed3~2,28ed6~2,28ed9~1,28edb~1,28ede~1,28ee0~1,28ee2~1,28ee4~1,28ee6~1,28ee8~1,28eea~1,28eed~2,28ef1~2,28ef5~3,28ef9~3,28efe~1,28f00~2,28f04~2,28f07~4,28f0d~2,28f11~1,28f16~1,28f18~1,28f1b~2,28f1f~1,28f21~1,28f23~1,28f25~1,28f27~1,28f29~1,28f2b~1,28f2e~2,28f32~1,28f34~2,28f38~2,28f3c~2,28f40~2,28f44~1,28f46~1,28f48~4,28f4e~1,28f50~1,28f52~1,28f54~1,28f57~1,28f5a~1,28f5c~1,28f5e~1,28f61~2,28f64~1,28f67~3,28f6b~2,28f6e~3]] do_op: waiting for scrub

#6 Updated by Nathan Cutler over 2 years ago

  • Description updated (diff)

#7 Updated by Nathan Cutler over 2 years ago

  • File deleted (stuck-scrub-snap.txt)

#8 Updated by David Zafman over 2 years ago

  • Status changed from In Progress to Fix Under Review

#9 Updated by Nathan Cutler over 2 years ago

  • Status changed from Fix Under Review to Resolved
  • Target version set to v10.2.10

Also available in: Atom PDF