Project

General

Profile

Bug #48868

Background recovery request fall into infinite recursive loop

Added by Xuehan Xu 4 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Now BackgroundRecovery requests doesn't take those objects whose recovery is triggered by UrgentRecovery requests. This may lead to an infinite start_primary_recovery_ops loop if BackgroundRecovery request is left with only UrgetRecovery triggered object recoveries.

INFO  2021-01-13 04:46:43,448 [shard 0] osd - start_primary_recovery_ops recovering 1 in pg  pg_epoch 40 pg[3.0( v 38'465 lc 38'464 (0'0,38'465] local-lis/les=39/40 n=0 ec=14/14 lis/c=39/14 les/c/f=40/15/0 sis=39) [2,3,1] r=0 lpr=39 pi=[14,39)/1 crt=38'465 mlcod 0'0 active+recovering+degraded , missing missing(1 may_include_deletes = 1)
INFO  2021-01-13 04:46:43,448 [shard 0] osd - start_primary_recovery_ops 3:02e47717:::smithi08331910-525:head item.need 38'465  (missing)  (missing head)  (recovering)  (recovering head)
INFO  2021-01-13 04:46:43,448 [shard 0] osd - start_primary_recovery_ops started 0 skipped 0
DEBUG 2021-01-13 04:46:43,448 [shard 0] osd -  pg_epoch 40 pg[3.0( v 38'465 lc 38'464 (0'0,38'465] local-lis/les=39/40 n=0 ec=14/14 lis/c=39/14 les/c/f=40/15/0 sis=39) [2,3,1] r=0 lpr=39 pi=[14,39)/1 crt=38'465 mlcod 0'0 active+recovering+degraded needs_recovery primary has 1 missing
INFO  2021-01-13 04:46:43,448 [shard 0] osd - start_primary_recovery_ops recovering 1 in pg  pg_epoch 40 pg[3.0( v 38'465 lc 38'464 (0'0,38'465] local-lis/les=39/40 n=0 ec=14/14 lis/c=39/14 les/c/f=40/15/0 sis=39) [2,3,1] r=0 lpr=39 pi=[14,39)/1 crt=38'465 mlcod 0'0 active+recovering+degraded , missing missing(1 may_include_deletes = 1)
INFO  2021-01-13 04:46:43,448 [shard 0] osd - start_primary_recovery_ops 3:02e47717:::smithi08331910-525:head item.need 38'465  (missing)  (missing head)  (recovering)  (recovering head)
INFO  2021-01-13 04:46:43,448 [shard 0] osd - start_primary_recovery_ops started 0 skipped 0
DEBUG 2021-01-13 04:46:43,448 [shard 0] osd -  pg_epoch 40 pg[3.0( v 38'465 lc 38'464 (0'0,38'465] local-lis/les=39/40 n=0 ec=14/14 lis/c=39/14 les/c/f=40/15/0 sis=39) [2,3,1] r=0 lpr=39 pi=[14,39)/1 crt=38'465 mlcod 0'0 active+recovering+degraded needs_recovery primary has 1 missing
INFO  2021-01-13 04:46:43,448 [shard 0] osd - start_primary_recovery_ops recovering 1 in pg  pg_epoch 40 pg[3.0( v 38'465 lc 38'464 (0'0,38'465] local-lis/les=39/40 n=0 ec=14/14 lis/c=39/14 les/c/f=40/15/0 sis=39) [2,3,1] r=0 lpr=39 pi=[14,39)/1 crt=38'465 mlcod 0'0 active+recovering+degraded , missing missing(1 may_include_deletes = 1)
INFO  2021-01-13 04:46:43,448 [shard 0] osd - start_primary_recovery_ops 3:02e47717:::smithi08331910-525:head item.need 38'465  (missing)  (missing head)  (recovering)  (recovering head)
INFO  2021-01-13 04:46:43,448 [shard 0] osd - start_primary_recovery_ops started 0 skipped 0
DEBUG 2021-01-13 04:46:43,448 [shard 0] osd -  pg_epoch 40 pg[3.0( v 38'465 lc 38'464 (0'0,38'465] local-lis/les=39/40 n=0 ec=14/14 lis/c=39/14 les/c/f=40/15/0 sis=39) [2,3,1] r=0 lpr=39 pi=[14,39)/1 crt=38'465 mlcod 0'0 active+recovering+degraded needs_recovery primary has 1 missing
INFO  2021-01-13 04:46:43,448 [shard 0] osd - start_primary_recovery_ops recovering 1 in pg  pg_epoch 40 pg[3.0( v 38'465 lc 38'464 (0'0,38'465] local-lis/les=39/40 n=0 ec=14/14 lis/c=39/14 les/c/f=40/15/0 sis=39) [2,3,1] r=0 lpr=39 pi=[14,39)/1 crt=38'465 mlcod 0'0 active+recovering+degraded , missing missing(1 may_include_deletes = 1)
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops 3:02e47717:::smithi08331910-525:head item.need 38'465  (missing)  (missing head)  (recovering)  (recovering head)
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops started 0 skipped 0
DEBUG 2021-01-13 04:46:43,449 [shard 0] osd -  pg_epoch 40 pg[3.0( v 38'465 lc 38'464 (0'0,38'465] local-lis/les=39/40 n=0 ec=14/14 lis/c=39/14 les/c/f=40/15/0 sis=39) [2,3,1] r=0 lpr=39 pi=[14,39)/1 crt=38'465 mlcod 0'0 active+recovering+degraded needs_recovery primary has 1 missing
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops recovering 1 in pg  pg_epoch 40 pg[3.0( v 38'465 lc 38'464 (0'0,38'465] local-lis/les=39/40 n=0 ec=14/14 lis/c=39/14 les/c/f=40/15/0 sis=39) [2,3,1] r=0 lpr=39 pi=[14,39)/1 crt=38'465 mlcod 0'0 active+recovering+degraded , missing missing(1 may_include_deletes = 1)
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops 3:02e47717:::smithi08331910-525:head item.need 38'465  (missing)  (missing head)  (recovering)  (recovering head)
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops started 0 skipped 0
DEBUG 2021-01-13 04:46:43,449 [shard 0] osd -  pg_epoch 40 pg[3.0( v 38'465 lc 38'464 (0'0,38'465] local-lis/les=39/40 n=0 ec=14/14 lis/c=39/14 les/c/f=40/15/0 sis=39) [2,3,1] r=0 lpr=39 pi=[14,39)/1 crt=38'465 mlcod 0'0 active+recovering+degraded needs_recovery primary has 1 missing
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops recovering 1 in pg  pg_epoch 40 pg[3.0( v 38'465 lc 38'464 (0'0,38'465] local-lis/les=39/40 n=0 ec=14/14 lis/c=39/14 les/c/f=40/15/0 sis=39) [2,3,1] r=0 lpr=39 pi=[14,39)/1 crt=38'465 mlcod 0'0 active+recovering+degraded , missing missing(1 may_include_deletes = 1)
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops 3:02e47717:::smithi08331910-525:head item.need 38'465  (missing)  (missing head)  (recovering)  (recovering head)
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops started 0 skipped 0
DEBUG 2021-01-13 04:46:43,449 [shard 0] osd -  pg_epoch 40 pg[3.0( v 38'465 lc 38'464 (0'0,38'465] local-lis/les=39/40 n=0 ec=14/14 lis/c=39/14 les/c/f=40/15/0 sis=39) [2,3,1] r=0 lpr=39 pi=[14,39)/1 crt=38'465 mlcod 0'0 active+recovering+degraded needs_recovery primary has 1 missing
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops recovering 1 in pg  pg_epoch 40 pg[3.0( v 38'465 lc 38'464 (0'0,38'465] local-lis/les=39/40 n=0 ec=14/14 lis/c=39/14 les/c/f=40/15/0 sis=39) [2,3,1] r=0 lpr=39 pi=[14,39)/1 crt=38'465 mlcod 0'0 active+recovering+degraded , missing missing(1 may_include_deletes = 1)
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops 3:02e47717:::smithi08331910-525:head item.need 38'465  (missing)  (missing head)  (recovering)  (recovering head)
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops started 0 skipped 0
DEBUG 2021-01-13 04:46:43,449 [shard 0] osd -  pg_epoch 40 pg[3.0( v 38'465 lc 38'464 (0'0,38'465] local-lis/les=39/40 n=0 ec=14/14 lis/c=39/14 les/c/f=40/15/0 sis=39) [2,3,1] r=0 lpr=39 pi=[14,39)/1 crt=38'465 mlcod 0'0 active+recovering+degraded needs_recovery primary has 1 missing
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops recovering 1 in pg  pg_epoch 40 pg[3.0( v 38'465 lc 38'464 (0'0,38'465] local-lis/les=39/40 n=0 ec=14/14 lis/c=39/14 les/c/f=40/15/0 sis=39) [2,3,1] r=0 lpr=39 pi=[14,39)/1 crt=38'465 mlcod 0'0 active+recovering+degraded , missing missing(1 may_include_deletes = 1)
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops 3:02e47717:::smithi08331910-525:head item.need 38'465  (missing)  (missing head)  (recovering)  (recovering head)
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops started 0 skipped 0
DEBUG 2021-01-13 04:46:43,449 [shard 0] osd -  pg_epoch 40 pg[3.0( v 38'465 lc 38'464 (0'0,38'465] local-lis/les=39/40 n=0 ec=14/14 lis/c=39/14 les/c/f=40/15/0 sis=39) [2,3,1] r=0 lpr=39 pi=[14,39)/1 crt=38'465 mlcod 0'0 active+recovering+degraded needs_recovery primary has 1 missing
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops recovering 1 in pg  pg_epoch 40 pg[3.0( v 38'465 lc 38'464 (0'0,38'465] local-lis/les=39/40 n=0 ec=14/14 lis/c=39/14 les/c/f=40/15/0 sis=39) [2,3,1] r=0 lpr=39 pi=[14,39)/1 crt=38'465 mlcod 0'0 active+recovering+degraded , missing missing(1 may_include_deletes = 1)
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops 3:02e47717:::smithi08331910-525:head item.need 38'465  (missing)  (missing head)  (recovering)  (recovering head)
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops started 0 skipped 0
DEBUG 2021-01-13 04:46:43,449 [shard 0] osd -  pg_epoch 40 pg[3.0( v 38'465 lc 38'464 (0'0,38'465] local-lis/les=39/40 n=0 ec=14/14 lis/c=39/14 les/c/f=40/15/0 sis=39) [2,3,1] r=0 lpr=39 pi=[14,39)/1 crt=38'465 mlcod 0'0 active+recovering+degraded needs_recovery primary has 1 missing
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops recovering 1 in pg  pg_epoch 40 pg[3.0( v 38'465 lc 38'464 (0'0,38'465] local-lis/les=39/40 n=0 ec=14/14 lis/c=39/14 les/c/f=40/15/0 sis=39) [2,3,1] r=0 lpr=39 pi=[14,39)/1 crt=38'465 mlcod 0'0 active+recovering+degraded , missing missing(1 may_include_deletes = 1)
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops 3:02e47717:::smithi08331910-525:head item.need 38'465  (missing)  (missing head)  (recovering)  (recovering head)
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops started 0 skipped 0
DEBUG 2021-01-13 04:46:43,449 [shard 0] osd -  pg_epoch 40 pg[3.0( v 38'465 lc 38'464 (0'0,38'465] local-lis/les=39/40 n=0 ec=14/14 lis/c=39/14 les/c/f=40/15/0 sis=39) [2,3,1] r=0 lpr=39 pi=[14,39)/1 crt=38'465 mlcod 0'0 active+recovering+degraded needs_recovery primary has 1 missing
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops recovering 1 in pg  pg_epoch 40 pg[3.0( v 38'465 lc 38'464 (0'0,38'465] local-lis/les=39/40 n=0 ec=14/14 lis/c=39/14 les/c/f=40/15/0 sis=39) [2,3,1] r=0 lpr=39 pi=[14,39)/1 crt=38'465 mlcod 0'0 active+recovering+degraded , missing missing(1 may_include_deletes = 1)
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops 3:02e47717:::smithi08331910-525:head item.need 38'465  (missing)  (missing head)  (recovering)  (recovering head)
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops started 0 skipped 0
DEBUG 2021-01-13 04:46:43,449 [shard 0] osd -  pg_epoch 40 pg[3.0( v 38'465 lc 38'464 (0'0,38'465] local-lis/les=39/40 n=0 ec=14/14 lis/c=39/14 les/c/f=40/15/0 sis=39) [2,3,1] r=0 lpr=39 pi=[14,39)/1 crt=38'465 mlcod 0'0 active+recovering+degraded needs_recovery primary has 1 missing
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops recovering 1 in pg  pg_epoch 40 pg[3.0( v 38'465 lc 38'464 (0'0,38'465] local-lis/les=39/40 n=0 ec=14/14 lis/c=39/14 les/c/f=40/15/0 sis=39) [2,3,1] r=0 lpr=39 pi=[14,39)/1 crt=38'465 mlcod 0'0 active+recovering+degraded , missing missing(1 may_include_deletes = 1)
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops 3:02e47717:::smithi08331910-525:head item.need 38'465  (missing)  (missing head)  (recovering)  (recovering head)
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops started 0 skipped 0
DEBUG 2021-01-13 04:46:43,449 [shard 0] osd -  pg_epoch 40 pg[3.0( v 38'465 lc 38'464 (0'0,38'465] local-lis/les=39/40 n=0 ec=14/14 lis/c=39/14 les/c/f=40/15/0 sis=39) [2,3,1] r=0 lpr=39 pi=[14,39)/1 crt=38'465 mlcod 0'0 active+recovering+degraded needs_recovery primary has 1 missing
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops recovering 1 in pg  pg_epoch 40 pg[3.0( v 38'465 lc 38'464 (0'0,38'465] local-lis/les=39/40 n=0 ec=14/14 lis/c=39/14 les/c/f=40/15/0 sis=39) [2,3,1] r=0 lpr=39 pi=[14,39)/1 crt=38'465 mlcod 0'0 active+recovering+degraded , missing missing(1 may_include_deletes = 1)
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops 3:02e47717:::smithi08331910-525:head item.need 38'465  (missing)  (missing head)  (recovering)  (recovering head)
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops started 0 skipped 0
DEBUG 2021-01-13 04:46:43,449 [shard 0] osd -  pg_epoch 40 pg[3.0( v 38'465 lc 38'464 (0'0,38'465] local-lis/les=39/40 n=0 ec=14/14 lis/c=39/14 les/c/f=40/15/0 sis=39) [2,3,1] r=0 lpr=39 pi=[14,39)/1 crt=38'465 mlcod 0'0 active+recovering+degraded needs_recovery primary has 1 missing
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops recovering 1 in pg  pg_epoch 40 pg[3.0( v 38'465 lc 38'464 (0'0,38'465] local-lis/les=39/40 n=0 ec=14/14 lis/c=39/14 les/c/f=40/15/0 sis=39) [2,3,1] r=0 lpr=39 pi=[14,39)/1 crt=38'465 mlcod 0'0 active+recovering+degraded , missing missing(1 may_include_deletes = 1)
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops 3:02e47717:::smithi08331910-525:head item.need 38'465  (missing)  (missing head)  (recovering)  (recovering head)
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops started 0 skipped 0
DEBUG 2021-01-13 04:46:43,449 [shard 0] osd -  pg_epoch 40 pg[3.0( v 38'465 lc 38'464 (0'0,38'465] local-lis/les=39/40 n=0 ec=14/14 lis/c=39/14 les/c/f=40/15/0 sis=39) [2,3,1] r=0 lpr=39 pi=[14,39)/1 crt=38'465 mlcod 0'0 active+recovering+degraded needs_recovery primary has 1 missing
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops recovering 1 in pg  pg_epoch 40 pg[3.0( v 38'465 lc 38'464 (0'0,38'465] local-lis/les=39/40 n=0 ec=14/14 lis/c=39/14 les/c/f=40/15/0 sis=39) [2,3,1] r=0 lpr=39 pi=[14,39)/1 crt=38'465 mlcod 0'0 active+recovering+degraded , missing missing(1 may_include_deletes = 1)
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops 3:02e47717:::smithi08331910-525:head item.need 38'465  (missing)  (missing head)  (recovering)  (recovering head)
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops started 0 skipped 0
DEBUG 2021-01-13 04:46:43,449 [shard 0] osd -  pg_epoch 40 pg[3.0( v 38'465 lc 38'464 (0'0,38'465] local-lis/les=39/40 n=0 ec=14/14 lis/c=39/14 les/c/f=40/15/0 sis=39) [2,3,1] r=0 lpr=39 pi=[14,39)/1 crt=38'465 mlcod 0'0 active+recovering+degraded needs_recovery primary has 1 missing
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops recovering 1 in pg  pg_epoch 40 pg[3.0( v 38'465 lc 38'464 (0'0,38'465] local-lis/les=39/40 n=0 ec=14/14 lis/c=39/14 les/c/f=40/15/0 sis=39) [2,3,1] r=0 lpr=39 pi=[14,39)/1 crt=38'465 mlcod 0'0 active+recovering+degraded , missing missing(1 may_include_deletes = 1)
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops 3:02e47717:::smithi08331910-525:head item.need 38'465  (missing)  (missing head)  (recovering)  (recovering head)
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops started 0 skipped 0
DEBUG 2021-01-13 04:46:43,449 [shard 0] osd -  pg_epoch 40 pg[3.0( v 38'465 lc 38'464 (0'0,38'465] local-lis/les=39/40 n=0 ec=14/14 lis/c=39/14 les/c/f=40/15/0 sis=39) [2,3,1] r=0 lpr=39 pi=[14,39)/1 crt=38'465 mlcod 0'0 active+recovering+degraded needs_recovery primary has 1 missing
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops recovering 1 in pg  pg_epoch 40 pg[3.0( v 38'465 lc 38'464 (0'0,38'465] local-lis/les=39/40 n=0 ec=14/14 lis/c=39/14 les/c/f=40/15/0 sis=39) [2,3,1] r=0 lpr=39 pi=[14,39)/1 crt=38'465 mlcod 0'0 active+recovering+degraded , missing missing(1 may_include_deletes = 1)
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops 3:02e47717:::smithi08331910-525:head item.need 38'465  (missing)  (missing head)  (recovering)  (recovering head)
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops started 0 skipped 0
DEBUG 2021-01-13 04:46:43,449 [shard 0] osd -  pg_epoch 40 pg[3.0( v 38'465 lc 38'464 (0'0,38'465] local-lis/les=39/40 n=0 ec=14/14 lis/c=39/14 les/c/f=40/15/0 sis=39) [2,3,1] r=0 lpr=39 pi=[14,39)/1 crt=38'465 mlcod 0'0 active+recovering+degraded needs_recovery primary has 1 missing
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops recovering 1 in pg  pg_epoch 40 pg[3.0( v 38'465 lc 38'464 (0'0,38'465] local-lis/les=39/40 n=0 ec=14/14 lis/c=39/14 les/c/f=40/15/0 sis=39) [2,3,1] r=0 lpr=39 pi=[14,39)/1 crt=38'465 mlcod 0'0 active+recovering+degraded , missing missing(1 may_include_deletes = 1)
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops 3:02e47717:::smithi08331910-525:head item.need 38'465  (missing)  (missing head)  (recovering)  (recovering head)
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops started 0 skipped 0
DEBUG 2021-01-13 04:46:43,449 [shard 0] osd -  pg_epoch 40 pg[3.0( v 38'465 lc 38'464 (0'0,38'465] local-lis/les=39/40 n=0 ec=14/14 lis/c=39/14 les/c/f=40/15/0 sis=39) [2,3,1] r=0 lpr=39 pi=[14,39)/1 crt=38'465 mlcod 0'0 active+recovering+degraded needs_recovery primary has 1 missing
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops recovering 1 in pg  pg_epoch 40 pg[3.0( v 38'465 lc 38'464 (0'0,38'465] local-lis/les=39/40 n=0 ec=14/14 lis/c=39/14 les/c/f=40/15/0 sis=39) [2,3,1] r=0 lpr=39 pi=[14,39)/1 crt=38'465 mlcod 0'0 active+recovering+degraded , missing missing(1 may_include_deletes = 1)
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops 3:02e47717:::smithi08331910-525:head item.need 38'465  (missing)  (missing head)  (recovering)  (recovering head)
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops started 0 skipped 0
DEBUG 2021-01-13 04:46:43,449 [shard 0] osd -  pg_epoch 40 pg[3.0( v 38'465 lc 38'464 (0'0,38'465] local-lis/les=39/40 n=0 ec=14/14 lis/c=39/14 les/c/f=40/15/0 sis=39) [2,3,1] r=0 lpr=39 pi=[14,39)/1 crt=38'465 mlcod 0'0 active+recovering+degraded needs_recovery primary has 1 missing
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops recovering 1 in pg  pg_epoch 40 pg[3.0( v 38'465 lc 38'464 (0'0,38'465] local-lis/les=39/40 n=0 ec=14/14 lis/c=39/14 les/c/f=40/15/0 sis=39) [2,3,1] r=0 lpr=39 pi=[14,39)/1 crt=38'465 mlcod 0'0 active+recovering+degraded , missing missing(1 may_include_deletes = 1)
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops 3:02e47717:::smithi08331910-525:head item.need 38'465  (missing)  (missing head)  (recovering)  (recovering head)
INFO  2021-01-13 04:46:43,449 [shard 0] osd - start_primary_recovery_ops started 0 skipped 0
DEBUG 2021-01-13 04:46:43,449 [shard 0] osd -  pg_epoch 40 pg[3.0( v 38'465 lc 38'464 (0'0,38'465] local-lis/les=39/40 n=0 ec=14/14 lis/c=39/14 les/c/f=40/15/0 sis=39) [2,3,1] r=0 lpr=39 pi=[14,39)/1 crt=38'465 mlcod 0'0 active+recovering+degraded needs_recovery primary has 1 missing
Segmentation fault on shard 0.
Backtrace:
  0x00000000013ff55c
  0x00000000013c2610
  0x00000000013c2912
  0x00000000013c2982
  /lib64/libpthread.so.0+0x0000000000012dbf
  /lib64/libstdc++.so.6+0x0000000000097d9f
  /lib64/libstdc++.so.6+0x0000000000094e29
  /lib64/libstdc++.so.6+0x000000000010d413
  /lib64/libstdc++.so.6+0x00000000000ff6e7
  /lib64/libstdc++.so.6+0x00000000000ffb73
  0x0000000000c76357
  0x00000000009464a2
  0x0000000000797b7b
  0x00000000007e64c0
  0x00000000007e6d7e
  0x00000000006e3a1d
  0x00000000006e4c72
  0x00000000006e609e
  0x00000000008a0fc0
  0x00000000014bc072
  0x0000000000893b3e
  0x0000000000897b98
  0x000000000088271e
  0x000000000087e7d4
  0x000000000087f0fe
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253
  0x000000000087f253

Also available in: Atom PDF