Project

General

Profile

Actions
Copy link

Bug #55559

closed

osd-backfill-stats.sh fails in TEST_backfill_ec_prim_out

Added by Laura Flores almost 2 years ago. Updated almost 2 years ago.

Status:
Duplicate
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
quincy
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

/a/yuriw-2022-04-28_14:23:18-rados-wip-yuri-testing-2022-04-27-1456-quincy-distro-default-smithi/6811107

2022-04-29T07:37:28.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-stats.sh:89: check:  '[' erasure = erasure ']'
2022-04-29T07:37:28.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-stats.sh:91: check:  addp=p
2022-04-29T07:37:28.009 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-stats.sh:94: check:  eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"'
2022-04-29T07:37:28.010 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-stats.sh:94: check:  grep 'pg[[]1.0.*backfilling.*update_calc_stats '
2022-04-29T07:37:28.010 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-stats.sh:94: check:  tail -1
2022-04-29T07:37:28.010 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-stats.sh:94: check:  sed 's/.*[)] \([[][^ p]*\).*$/\1/'
2022-04-29T07:37:28.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-stats.sh:94: check:  UPACT=
2022-04-29T07:37:28.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-stats.sh:95: check:  '[' -n '' ']'
2022-04-29T07:37:28.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-stats.sh:95: check:  return 1
2022-04-29T07:37:28.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-stats.sh:608: TEST_backfill_ec_prim_out:  return 1
2022-04-29T07:37:28.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-stats.sh:39: run:  return 1

Related issues 1 (1 open — 0 closed)

Is duplicate of RADOS - Bug #47026: osd-backfill-stats.sh fails in TEST_backfill_ec_down_all_outNew

Actions
Actions
Copy link
 #1

Updated by Laura Flores almost 2 years ago

  • Related to Bug #47026: osd-backfill-stats.sh fails in TEST_backfill_ec_down_all_out added
Actions
Copy link
 #2

Updated by Radoslaw Zarzynski almost 2 years ago

Hello Laura! Is there a thing that makes you think this isn't a duplicate of #47026?

Actions
Copy link
 #3

Updated by Laura Flores almost 2 years ago

I opened a new issue since a different test failed this time. The failure does look the same though, so maybe the one tracker suffices.

Actions
Copy link
 #4

Updated by Laura Flores almost 2 years ago

  • Related to deleted (Bug #47026: osd-backfill-stats.sh fails in TEST_backfill_ec_down_all_out)
Actions
Copy link
 #5

Updated by Laura Flores almost 2 years ago

  • Is duplicate of Bug #47026: osd-backfill-stats.sh fails in TEST_backfill_ec_down_all_out added
Actions
Copy link
 #6

Updated by Laura Flores almost 2 years ago

  • Status changed from New to Duplicate
Actions
Copy link
 #7

Updated by Radoslaw Zarzynski almost 2 years ago

The common theme between these failures (this one and #47026) is check() function of qa/standalone/osd-backfill/osd-backfill-stats.sh script.

FIND_UPACT='grep "pg[[]${PG}.*backfilling.*update_calc_stats " $log | tail -1 | sed "s/.*[)] \([[][^ p]*\).*$/\1/"'

...

function check() {
    local dir=$1
    local PG=$2
    local primary=$3
    local type=$4
    local degraded_start=$5
    local degraded_end=$6
    local misplaced_start=$7
    local misplaced_end=$8
    local primary_start=${9:-}
    local primary_end=${10:-}
    local check_setup=${11:-true}

    local log=$(grep -l +backfilling $dir/osd.$primary.log)

    ...

    UPACT=$(eval $FIND_UPACT)
    [ -n "$UPACT" ] || return 1

Quite likely it's low priority thing taking into account it has replicated 2 times over a year but let's observe.

Actions
Copy link
 #8

Updated by Nitzan Mordechai almost 2 years ago

Looks like we don't have the correct primary (was osd.1, changed to osd.4, and after the wait_for_clean was back to osd.1 but we didn't check that)

when we are trying to grep and set the log:

2022-04-29T07:37:27.997 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-stats.sh:77: check:  grep -l +backfilling td/osd-backfill-stats/osd.4.log
2022-04-29T07:37:27.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-stats.sh:77: check:  local log=

it will be empty, then we greping from the empty log and fail..

2022-04-29T07:37:25.635+0000 7fbcd2f85700 1 osd.1 pg_epoch: 38 pg[1.0s0( v 29'200 (29'100,29'200] local-lis/les=34/36 n=200 ec=27/27 lis/c=34/34 les/c/f=36/37/0 sis=38 pruub=11.558121681s) [4,0,2]p4(0) r=-1 lpr=38 pi=[34,38)/1 crt=29'200 lcod 29'199 mlcod 0'0 active pruub 369.137115479s@ mbc={0={},1={},2={}}] start_peering_interval up [4,0,2] -> [4,0,2], acting [1,0,2] -> [4,0,2], acting_primary 1(0) -> 4, up_primary 4(0) -> 4, role 0 -> -1, features acting 4540138303579357183 upacting 4540138303579357183

same as #47026

Actions
Copy link

Also available in: Atom PDF