Project

General

Profile

Actions
Copy link

Bug #4254

closed

osd: failure to recover before timeout on rados bench and thrashing; negative stats

Added by Sage Weil about 11 years ago. Updated about 9 years ago.

Status:
Resolved
Priority:
High
Assignee:
Guang Yang
Category:
OSD
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

snippet:

2013-02-23T10:50:34.510 INFO:teuthology.task.thrashosds.ceph_manager:   health HEALTH_WARN 1 pgs backfilling; 1 pgs stuck unclean; recovery -706/10675 degraded (-6.614%);  recovering 0 o/s, 1947KB/s
   monmap e1: 3 mons at {a=10.214.132.28:6789/0,b=10.214.132.27:6789/0,c=10.214.132.28:6790/0}, election epoch 6, quorum 0,1,2 a,b,c
   osdmap e170: 6 osds: 6 up, 5 in
    pgmap v2442: 72 pgs: 71 active+clean, 1 active+remapped+backfilling; 20592 MB data, 43484 MB used, 2746 GB / 2794 GB avail; 0B/s wr, 4op/s; -706/10675 degraded (-6.614%);  recovering 0 o/s, 1947KB/s
   mdsmap e5: 1/1/1 up {0=a=up:active}

job was
ubuntu@teuthology:/a/sage-2013-02-23_08:44:35-regression-master-testing-basic/10262$ cat orig.config.yaml 
kernel:
  kdb: true
  sha1: 92a49fb0f79f3300e6e50ddf56238e70678e4202
nuke-on-error: true
overrides:
  ceph:
    conf:
      global:
        ms inject socket failures: 5000
      osd:
        osd op thread timeout: 60
    fs: btrfs
    log-whitelist:
    - slow request
    sha1: 704db850131643b26bafe6594946cacce483c171
  s3tests:
    branch: master
  workunit:
    sha1: 704db850131643b26bafe6594946cacce483c171
roles:
- - mon.a
  - mon.c
  - osd.0
  - osd.1
  - osd.2
- - mon.b
  - mds.a
  - osd.3
  - osd.4
  - osd.5
  - client.0
tasks:
- chef: null
- clock: null
- install: null
- ceph:
    log-whitelist:
    - wrongly marked me down
    - objects unfound and apparently lost
- thrashosds:
    timeout: 1200
- radosbench:
    clients:
    - client.0
    time: 1800

Related issues 2 (0 open — 2 closed)

Related to Ceph - Bug #4109: incorrect degraded countDuplicateSamuel Just02/12/2013

Actions
Related to Ceph - Bug #3720: Ceph Reporting Negative Number of Degraded objectsDuplicateSamuel Just01/03/2013

Actions
Actions
Copy link
 #1

Updated by Ian Colle about 11 years ago

  • Assignee set to Samuel Just
Actions
Copy link
 #2

Updated by Tamilarasi muthamizhan about 11 years ago

recent log:
ubuntu@teuthology:/a/teuthology-2013-02-25_01:00:05-regression-master-testing-gcov/11488
ubuntu@teuthology:/a/teuthology-2013-02-25_01:00:05-regression-master-testing-gcov/11494

Actions
Copy link
 #3

Updated by Tamilarasi muthamizhan about 11 years ago

recent log:
ubuntu@teuthology:/a/teuthology-2013-03-01_20:00:05-regression-last-master-basic/14898

Actions
Copy link
 #4

Updated by Tamilarasi muthamizhan about 11 years ago

recent logs:
ubuntu@teuthology:/a/teuthology-2013-03-10_01:00:05-regression-master-testing-gcov/20626
ubuntu@teuthology:/a/teuthology-2013-03-10_01:00:05-regression-master-testing-gcov/20638

Actions
Copy link
 #5

Updated by Ian Colle about 11 years ago

  • Status changed from 12 to Need More Info
Actions
Copy link
 #6

Updated by Ian Colle about 11 years ago

  • Target version set to v0.61 - Cuttlefish
Actions
Copy link
 #7

Updated by Samuel Just about 11 years ago

This could easily have been caused by #4627 d7b7acefc8e106f2563771a721944c57e10d54fb. I suggest we mark it resolved.

Actions
Copy link
 #8

Updated by Samuel Just about 11 years ago

  • Status changed from Need More Info to Resolved

This hasn't been seen recently, and could have been fixed by d7b7acefc8e106f2563771a721944c57e10d54fb. Marking it resolved until it comes up again.

Actions
Copy link
 #9

Updated by Noah Watkins over 10 years ago

I think I'm seeing the negative stats again on mira103

`sudo ceph f json-pretty pg dump pools | grep "\"`

Actions
Copy link
 #10

Updated by Zhi Zhang over 9 years ago

I am seeing this issue again on v0.80.4. I stopped 3 osd processes and marked them as out to trigger data migration (~2TB x 3 = ~6TB data). During the migration, there are also write/read traffic to this cluster.

After migration finished, only 4 pgs are not active+clean for a long time and status shows negative number, see below status:

-bash-4.1$ sudo ceph -s
cluster cd149cf7-7392-4de6-96f0-9a73e90413c6
health HEALTH_WARN 4 pgs backfilling; 4 pgs stuck unclean; 6 requests are blocked > 32 sec; recovery -18427/1626921117 objects degraded (-0.001%); noscrub,nodeep-scrub flag(s) set
monmap e1: 1 mons at {osd123=x.x.x.x:6789/0}, election epoch 2, quorum 0 osd123
osdmap e9549: 175 osds: 172 up, 172 in
flags noscrub,nodeep-scrub
pgmap v1402813: 7872 pgs, 11 pools, 267 TB data, 141 Mobjects
380 TB used, 243 TB / 623 TB avail
-18427/1626921117 objects degraded (-0.001%)
7868 active+clean
4 active+remapped+backfilling
client io 42395 kB/s rd, 100 MB/s wr, 3039 op/s

Both the negative number and slow requests won't go away even I stop my load. I have to restart multiple osd processes to make pg active+clean and slow request gone.

What I am worried about is that if this problem might cause data loss because we have such large number of object now, it is hard to check every object,

Actions
Copy link
 #11

Updated by Guang Yang about 9 years ago

  • Status changed from Resolved to New
  • Priority changed from Urgent to High
  • Target version deleted (v0.61 - Cuttlefish)

This come agains in our cluster with Firefly deployed:

-bash-4.1$ sudo ceph -s
    cluster 1ecef041-c643-4b95-91ee-fbf4aebba765
     health HEALTH_WARN 35 pgs recovering; 272 pgs recovery_wait; 307 pgs stuck unclean; recovery -4938/9544594349 objects degraded (-0.000%); nobackfill,norecover flag(s) set
     monmap e2: 3 mons at {mon01c002=10.214.143.208:6789/0,mon02c002=10.214.145.208:6789/0,mon03c002=10.214.145.144:6789/0}, election epoch 54, quorum 0,1,2 mon01c002,mon03c002,mon02c002
     osdmap e8795: 540 osds: 540 up, 540 in
            flags nobackfill,norecover
      pgmap v9550650: 11424 pgs, 9 pools, 1392 TB data, 827 Mobjects
            2004 TB used, 936 TB / 2941 TB avail
            -4938/9544594349 objects degraded (-0.000%)
                 272 active+recovery_wait
               11096 active+clean
                  35 active+recovering
                  21 active+clean+scrubbing+deep
  client io 66648 kB/s rd, 226 kB/s wr, 198 op/s

-bash-4.1$ ceph -v
ceph version 0.80.4 (7c241cfaa6c8c068bc9da8578ca00b9f4fc7567f)

-bash-4.1$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.5 (Santiago)

Actions
Copy link
 #12

Updated by Guang Yang about 9 years ago

I found an issue when calculating the number of degraded objects for EC pool - https://github.com/ceph/ceph/pull/3726

Actions
Copy link
 #13

Updated by Guang Yang about 9 years ago

Hi Sam,
When calculating the number of degraded objects, is that for sure the num_objects in primary is always larger than its peers?
It seems not always true..

-bash-4.1$ sudo ceph pg 3.1d61 query | grep num_objects\" 
              "num_objects": 105659,              // this should be the primary?
                  "num_objects": 105660,
                  "num_objects": 105660,
                  "num_objects": 105660,
                  "num_objects": 105660,
                  "num_objects": 105660,
                  "num_objects": 105660,
                  "num_objects": 105660,
                  "num_objects": 105659,
                  "num_objects": 105660,
                  "num_objects": 105660,
Actions
Copy link
 #14

Updated by Samuel Just about 9 years ago

  • Assignee changed from Samuel Just to Guang Yang
Actions
Copy link
 #15

Updated by Sage Weil about 9 years ago

  • Status changed from New to Resolved

latest iteration of this is fixed now, and haven't seen old version recently either.

Actions
Copy link
 #16

Updated by Guang Yang about 9 years ago

Hi Sage,
I think the patch in comment #12 does not address the problem we saw on our cluster, what we saw is more like comment #13 revealed, the primary has less objects than some of its replicas, so that the number of degraded objects is negative. It is more like http://tracker.ceph.com/issues/7737.

Actions
Copy link

Also available in: Atom PDF