Project

General

Profile

Actions

Bug #15781

closed

No scrub information available for pg 0.8

Added by Sage Weil about 8 years ago. Updated 11 months ago.

Status:
Can't reproduce
Priority:
Urgent
Assignee:
David Zafman
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2016-05-07T13:26:46.986 INFO:teuthology.orchestra.run.smithi026.stderr:instructing pg 0.8 on osd.3 to deep-scrub
2016-05-07T13:26:50.188 INFO:tasks.ceph.osd.3.smithi035.stderr:2016-05-07 20:26:50.188964 7fc92f4ad700 -1 log_channel(cluster) log [ERR] : 0.8 shard 3: soid 0:1c3a8a5d:::benchmark_data_smithi026_8762_object1000:head data_digest 0xa00e5e99 != known data_digest 0x4e33d242 from auth shard 5, size 4097 != known size 40
96
2016-05-07T13:26:50.212 INFO:tasks.ceph.osd.3.smithi035.stderr:2016-05-07 20:26:50.213144 7fc92f4ad700 -1 log_channel(cluster) log [ERR] : 0.8 deep-scrub 0 missing, 1 inconsistent objects
2016-05-07T13:26:50.213 INFO:tasks.ceph.osd.3.smithi035.stderr:2016-05-07 20:26:50.213151 7fc92f4ad700 -1 log_channel(cluster) log [ERR] : 0.8 deep-scrub 1 errors
2016-05-07T13:27:46.999 INFO:teuthology.orchestra.run.smithi026:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph pg dump --format=json'
2016-05-07T13:27:47.359 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all in format json
2016-05-07T13:27:47.371 INFO:teuthology.orchestra.run.smithi026:Running: 'rados list-inconsistent-pg rbd --format=json'
2016-05-07T13:27:47.482 INFO:teuthology.orchestra.run.smithi026:Running: 'rados list-inconsistent-obj 0.8 --format=json'
2016-05-07T13:27:47.569 INFO:teuthology.orchestra.run.smithi026.stderr:No scrub information available for pg 0.8
2016-05-07T13:27:47.569 INFO:teuthology.orchestra.run.smithi026.stderr:error 2: (2) No such file or directory


/a/yuriw-2016-05-07_09:11:04-rados-wip-yuri-testing---basic-smithi/176828
Actions #1

Updated by David Zafman about 8 years ago

  • Assignee set to David Zafman
Actions #2

Updated by Samuel Just over 7 years ago

  • Status changed from New to Can't reproduce
Actions #3

Updated by Matthew Roy over 7 years ago

I might have just hit this bug on a real cluster. A disk was gently failing, generating read errors which resulted in inconsistent PGs. I told them to repair, but then thought better of it and decided to mark the osd as out (osd.0). The pgs then ended up in this state after the cluster completed recovery:

root@atropos:~# ceph health detail
HEALTH_ERR 8 pgs inconsistent; 14 scrub errors; mds0: Client smilebox:smilebox-cephfs failing to respond to cache pressure
pg 1.3f is active+clean+inconsistent, acting [14,5,6]
pg 1.10 is active+clean+inconsistent, acting [5,2,3]
pg 0.23 is active+clean+inconsistent, acting [10,8,12]
pg 1.45 is active+clean+inconsistent, acting [14,3,10]
pg 1.5c is active+clean+inconsistent, acting [7,11,12]
pg 1.66 is active+clean+inconsistent, acting [1,2,9]
pg 1.6f is active+clean+inconsistent, acting [1,2,6]
pg 1.7d is active+clean+inconsistent, acting [10,2,12]
14 scrub errors
mds0: Client smilebox:smilebox-cephfs failing to respond to cache pressure(client_id: 275976)
root@atropos:~# rados list-inconsistent-obj 0.23 --format=json-pretty
No scrub information available for pg 0.23
error 2: (2) No such file or directory
Actions #4

Updated by Kenneth Waegeman about 7 years ago

We've just found the same issue on our cluster too:

ceph health detail
HEALTH_ERR 1 pgs inconsistent; 6 scrub errors
pg 5.d7 is active+clean+inconsistent, acting [53,42,120,14,59,33,68,20,128,142,149]
6 scrub errors

tried to repair, but did not work.
I tried to run list-inconsistent-obj , but got:

rados list-inconsistent-obj 5.d7
No scrub information available for pg 5.d7
error 2: (2) No such file or directory

Actions #5

Updated by Patrick Fruh over 6 years ago

Just had the same happen on our cluster.
I updated our hosts and after rebooting, the xfs filesystem on 2 OSDs on one host was dead

[  991.046980] XFS (sdg1): Internal error XFS_WANT_CORRUPTED_GOTO at line 3524 of file fs/xfs/libxfs/xfs_btree.c.  Caller xfs_free_ag_extent+0x4c1/0x7a0 [xfs]
[  991.047464] CPU: 26 PID: 9281 Comm: tp_fstore_op Not tainted 4.13.1-1 #1
[  991.047466] Hardware name: Dell Inc. PowerEdge R730xd/0WCJNT, BIOS 2.4.3 01/17/2017
[  991.047468] Call Trace:
[  991.047478]  dump_stack+0x63/0x89
[  991.047530]  xfs_error_report+0x3b/0x40 [xfs]
[  991.047567]  ? xfs_free_ag_extent+0x4c1/0x7a0 [xfs]
[  991.047611]  xfs_btree_insert+0x1b0/0x1c0 [xfs]
[  991.047646]  xfs_free_ag_extent+0x4c1/0x7a0 [xfs]
[  991.047688]  xfs_free_extent+0xbb/0x150 [xfs]
[  991.047752]  xfs_trans_free_extent+0x4f/0x110 [xfs]
[  991.047799]  ? xfs_trans_add_item+0x5d/0x90 [xfs]
[  991.047844]  xfs_extent_free_finish_item+0x26/0x40 [xfs]
[  991.047882]  xfs_defer_finish+0x149/0x410 [xfs]
[  991.047931]  xfs_itruncate_extents+0x124/0x2d0 [xfs]
[  991.047971]  xfs_inactive_truncate+0xad/0x100 [xfs]
[  991.048015]  xfs_inactive+0xec/0x110 [xfs]
[  991.048055]  xfs_fs_destroy_inode+0xa4/0x210 [xfs]
[  991.048060]  destroy_inode+0x3b/0x60
[  991.048064]  evict+0x142/0x1a0
[  991.048068]  iput+0x17d/0x1d0
[  991.048074]  do_unlinkat+0x194/0x2b0
[  991.048078]  SyS_unlink+0x16/0x20
[  991.048085]  do_syscall_64+0x67/0x150
[  991.048091]  entry_SYSCALL64_slow_path+0x25/0x25
[  991.048094] RIP: 0033:0x7f01d8585367
[  991.048096] RSP: 002b:00007f01c6bfb618 EFLAGS: 00000246 ORIG_RAX: 0000000000000057
[  991.048099] RAX: ffffffffffffffda RBX: 000055929efff400 RCX: 00007f01d8585367
[  991.048100] RDX: 00007f01d907c3f8 RSI: 00007f01c6bfb5e0 RDI: 00005592bc09f4e8
[  991.048101] RBP: 000055929efff400 R08: 00007f01d84e3938 R09: 0000000000000001
[  991.048102] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f01c6bfb980
[  991.048104] R13: 00007f01c6bfc340 R14: 00007f01c6bfb8c0 R15: 00007f01c6bfc340
[  991.048141] XFS (sdg1): xfs_do_force_shutdown(0x8) called from line 236 of file fs/xfs/libxfs/xfs_defer.c.  Return address = 0xffffffffc068c1c7
[  991.206593] XFS (sdg1): Corruption of in-memory data detected.  Shutting down filesystem
[  991.206862] XFS (sdg1): Please umount the filesystem and rectify the problem(s)

so I repaired them with xfs_repair -L and put them back online and after the recovery of the degraded PGs everything went green again.

Now after about 10 hours (with our daily scrubbing window in between) I'm getting:

ceph-admin@admin:[~]: ceph health detail
HEALTH_ERR 1 pgs inconsistent; 2 scrub errors
pg 8.313 is active+clean+inconsistent, acting [36,9,24]
2 scrub errors

OSD 24 is the one with the xfs filesystem error from before.

Repair doesn't seem to fix it, it recovers a few things but shows the same inconsistent pg at the end of it, also "rados list-inconsisten-obj" can't find the scrub information:

ceph-admin@admin:[~]: rados list-inconsistent-obj 8.313
No scrub information available for pg 8.313
error 2: (2) No such file or directory
Actions #6

Updated by Patrick Fruh over 6 years ago

Here's the query of the pg, current time being 2017-09-12 12:51:00

ceph-admin@admin:[~]: ceph pg 8.313 query
{
    "state": "active+clean+inconsistent",
    "snap_trimq": "[]",
    "epoch": 16328,
    "up": [
        36,
        9,
        24
    ],
    "acting": [
        36,
        9,
        24
    ],
    "actingbackfill": [
        "9",
        "24",
        "36" 
    ],
    "info": {
        "pgid": "8.313",
        "last_update": "16328'4138539",
        "last_complete": "16328'4138539",
        "log_tail": "16316'4135499",
        "last_user_version": 4138539,
        "last_backfill": "MAX",
        "last_backfill_bitwise": 1,
        "purged_snaps": "[1~9,34~2,37~1,52~1,55~2,59~a,64~1,6a~1,6c~5,72~1]",
        "history": {
            "epoch_created": 2548,
            "last_epoch_started": 16327,
            "last_epoch_clean": 16327,
            "last_epoch_split": 0,
            "last_epoch_marked_full": 0,
            "same_up_since": 16324,
            "same_interval_since": 16325,
            "same_primary_since": 16325,
            "last_scrub": "16321'4138121",
            "last_scrub_stamp": "2017-09-12 12:17:16.862962",
            "last_deep_scrub": "16321'4138121",
            "last_deep_scrub_stamp": "2017-09-12 12:17:16.862962",
            "last_clean_scrub_stamp": "2017-09-11 00:38:37.136245" 
        },
        "stats": {
            "version": "16328'4138539",
            "reported_seq": "4117781",
            "reported_epoch": "16328",
            "state": "active+clean+inconsistent",
            "last_fresh": "2017-09-12 12:50:40.879013",
            "last_change": "2017-09-12 12:19:40.864289",
            "last_active": "2017-09-12 12:50:40.879013",
            "last_peered": "2017-09-12 12:50:40.879013",
            "last_clean": "2017-09-12 12:50:40.879013",
            "last_became_active": "2017-09-12 12:18:12.322524",
            "last_became_peered": "2017-09-12 12:18:12.322524",
            "last_unstale": "2017-09-12 12:50:40.879013",
            "last_undegraded": "2017-09-12 12:50:40.879013",
            "last_fullsized": "2017-09-12 12:50:40.879013",
            "mapping_epoch": 16324,
            "log_start": "16316'4135499",
            "ondisk_log_start": "16316'4135499",
            "created": 2548,
            "last_epoch_clean": 16327,
            "parent": "0.0",
            "parent_split_bits": 10,
            "last_scrub": "16321'4138121",
            "last_scrub_stamp": "2017-09-12 12:17:16.862962",
            "last_deep_scrub": "16321'4138121",
            "last_deep_scrub_stamp": "2017-09-12 12:17:16.862962",
            "last_clean_scrub_stamp": "2017-09-11 00:38:37.136245",
            "log_size": 3040,
            "ondisk_log_size": 3040,
            "stats_invalid": false,
            "dirty_stats_invalid": false,
            "omap_stats_invalid": false,
            "hitset_stats_invalid": false,
            "hitset_bytes_stats_invalid": false,
            "pin_stats_invalid": false,
            "stat_sum": {
                "num_bytes": 13725943296,
                "num_objects": 3451,
                "num_object_clones": 575,
                "num_object_copies": 10353,
                "num_objects_missing_on_primary": 0,
                "num_objects_missing": 0,
                "num_objects_degraded": 0,
                "num_objects_misplaced": 0,
                "num_objects_unfound": 0,
                "num_objects_dirty": 3451,
                "num_whiteouts": 0,
                "num_read": 1547019,
                "num_read_kb": 79518748,
                "num_write": 2917643,
                "num_write_kb": 76792573,
                "num_scrub_errors": 2,
                "num_shallow_scrub_errors": 0,
                "num_deep_scrub_errors": 2,
                "num_objects_recovered": 7195,
                "num_bytes_recovered": 29916246528,
                "num_keys_recovered": 0,
                "num_objects_omap": 0,
                "num_objects_hit_set_archive": 0,
                "num_bytes_hit_set_archive": 0,
                "num_flush": 0,
                "num_flush_kb": 0,
                "num_evict": 0,
                "num_evict_kb": 0,
                "num_promote": 0,
                "num_flush_mode_high": 0,
                "num_flush_mode_low": 0,
                "num_evict_mode_some": 0,
                "num_evict_mode_full": 0,
                "num_objects_pinned": 0
            },
            "up": [
                36,
                9,
                24
            ],
            "acting": [
                36,
                9,
                24
            ],
            "blocked_by": [],
            "up_primary": 36,
            "acting_primary": 36
        },
        "empty": 0,
        "dne": 0,
        "incomplete": 0,
        "last_epoch_started": 16327,
        "hit_set_history": {
            "current_last_update": "0'0",
            "history": []
        }
    },
    "peer_info": [
        {
            "peer": "9",
            "pgid": "8.313",
            "last_update": "16328'4138539",
            "last_complete": "16328'4138539",
            "log_tail": "16316'4135099",
            "last_user_version": 4138132,
            "last_backfill": "MAX",
            "last_backfill_bitwise": 1,
            "purged_snaps": "[1~9,34~2,37~1,52~1,55~2,59~a,64~1,6a~1,6c~5,72~1]",
            "history": {
                "epoch_created": 2548,
                "last_epoch_started": 16327,
                "last_epoch_clean": 16327,
                "last_epoch_split": 0,
                "last_epoch_marked_full": 0,
                "same_up_since": 16324,
                "same_interval_since": 16325,
                "same_primary_since": 16325,
                "last_scrub": "16321'4138121",
                "last_scrub_stamp": "2017-09-12 12:17:16.862962",
                "last_deep_scrub": "16321'4138121",
                "last_deep_scrub_stamp": "2017-09-12 12:17:16.862962",
                "last_clean_scrub_stamp": "2017-09-11 00:38:37.136245" 
            },
            "stats": {
                "version": "16323'4138132",
                "reported_seq": "4117369",
                "reported_epoch": "16326",
                "state": "remapped+inconsistent+peering",
                "last_fresh": "2017-09-12 12:18:04.434843",
                "last_change": "2017-09-12 12:18:04.412687",
                "last_active": "2017-09-12 12:17:52.681445",
                "last_peered": "2017-09-12 12:17:52.681445",
                "last_clean": "2017-09-12 12:17:14.504126",
                "last_became_active": "2017-09-12 12:17:36.831098",
                "last_became_peered": "2017-09-12 12:17:36.831098",
                "last_unstale": "2017-09-12 12:18:04.434843",
                "last_undegraded": "2017-09-12 12:18:04.434843",
                "last_fullsized": "2017-09-12 12:18:04.434843",
                "mapping_epoch": 16324,
                "log_start": "16316'4135099",
                "ondisk_log_start": "16316'4135099",
                "created": 2548,
                "last_epoch_clean": 16323,
                "parent": "0.0",
                "parent_split_bits": 10,
                "last_scrub": "16321'4138121",
                "last_scrub_stamp": "2017-09-12 12:17:16.862962",
                "last_deep_scrub": "16321'4138121",
                "last_deep_scrub_stamp": "2017-09-12 12:17:16.862962",
                "last_clean_scrub_stamp": "2017-09-11 00:38:37.136245",
                "log_size": 3033,
                "ondisk_log_size": 3033,
                "stats_invalid": false,
                "dirty_stats_invalid": false,
                "omap_stats_invalid": false,
                "hitset_stats_invalid": false,
                "hitset_bytes_stats_invalid": false,
                "pin_stats_invalid": false,
                "stat_sum": {
                    "num_bytes": 13725943296,
                    "num_objects": 3451,
                    "num_object_clones": 575,
                    "num_object_copies": 10353,
                    "num_objects_missing_on_primary": 0,
                    "num_objects_missing": 0,
                    "num_objects_degraded": 0,
                    "num_objects_misplaced": 0,
                    "num_objects_unfound": 0,
                    "num_objects_dirty": 3451,
                    "num_whiteouts": 0,
                    "num_read": 1546881,
                    "num_read_kb": 79514552,
                    "num_write": 2917236,
                    "num_write_kb": 76790980,
                    "num_scrub_errors": 2,
                    "num_shallow_scrub_errors": 0,
                    "num_deep_scrub_errors": 2,
                    "num_objects_recovered": 7192,
                    "num_bytes_recovered": 29903663616,
                    "num_keys_recovered": 0,
                    "num_objects_omap": 0,
                    "num_objects_hit_set_archive": 0,
                    "num_bytes_hit_set_archive": 0,
                    "num_flush": 0,
                    "num_flush_kb": 0,
                    "num_evict": 0,
                    "num_evict_kb": 0,
                    "num_promote": 0,
                    "num_flush_mode_high": 0,
                    "num_flush_mode_low": 0,
                    "num_evict_mode_some": 0,
                    "num_evict_mode_full": 0,
                    "num_objects_pinned": 0
                },
                "up": [
                    36,
                    9,
                    24
                ],
                "acting": [
                    36,
                    9,
                    24
                ],
                "blocked_by": [],
                "up_primary": 36,
                "acting_primary": 36
            },
            "empty": 0,
            "dne": 0,
            "incomplete": 0,
            "last_epoch_started": 16327,
            "hit_set_history": {
                "current_last_update": "0'0",
                "history": []
            }
        },
        {
            "peer": "24",
            "pgid": "8.313",
            "last_update": "16328'4138539",
            "last_complete": "16328'4138539",
            "log_tail": "16316'4135099",
            "last_user_version": 4138132,
            "last_backfill": "MAX",
            "last_backfill_bitwise": 1,
            "purged_snaps": "[1~9,34~2,37~1,52~1,55~2,59~a,64~1,6a~1,6c~5,72~1]",
            "history": {
                "epoch_created": 2548,
                "last_epoch_started": 16327,
                "last_epoch_clean": 16327,
                "last_epoch_split": 0,
                "last_epoch_marked_full": 0,
                "same_up_since": 16324,
                "same_interval_since": 16325,
                "same_primary_since": 16325,
                "last_scrub": "16321'4138121",
                "last_scrub_stamp": "2017-09-12 12:17:16.862962",
                "last_deep_scrub": "16321'4138121",
                "last_deep_scrub_stamp": "2017-09-12 12:17:16.862962",
                "last_clean_scrub_stamp": "2017-09-11 00:38:37.136245" 
            },
            "stats": {
                "version": "16323'4138131",
                "reported_seq": "4117377",
                "reported_epoch": "16323",
                "state": "active+undersized+degraded+inconsistent",
                "last_fresh": "2017-09-12 12:17:42.690845",
                "last_change": "2017-09-12 12:17:36.831098",
                "last_active": "2017-09-12 12:17:42.690845",
                "last_peered": "2017-09-12 12:17:42.690845",
                "last_clean": "2017-09-12 12:17:14.504126",
                "last_became_active": "2017-09-12 12:17:36.831098",
                "last_became_peered": "2017-09-12 12:17:36.831098",
                "last_unstale": "2017-09-12 12:17:42.690845",
                "last_undegraded": "2017-09-12 12:17:35.813915",
                "last_fullsized": "2017-09-12 12:17:35.813915",
                "mapping_epoch": 16324,
                "log_start": "16316'4135099",
                "ondisk_log_start": "16316'4135099",
                "created": 2548,
                "last_epoch_clean": 16323,
                "parent": "0.0",
                "parent_split_bits": 10,
                "last_scrub": "16321'4138121",
                "last_scrub_stamp": "2017-09-12 12:17:16.862962",
                "last_deep_scrub": "16321'4138121",
                "last_deep_scrub_stamp": "2017-09-12 12:17:16.862962",
                "last_clean_scrub_stamp": "2017-09-11 00:38:37.136245",
                "log_size": 3032,
                "ondisk_log_size": 3032,
                "stats_invalid": false,
                "dirty_stats_invalid": false,
                "omap_stats_invalid": false,
                "hitset_stats_invalid": false,
                "hitset_bytes_stats_invalid": false,
                "pin_stats_invalid": false,
                "stat_sum": {
                    "num_bytes": 13725943296,
                    "num_objects": 3451,
                    "num_object_clones": 575,
                    "num_object_copies": 10353,
                    "num_objects_missing_on_primary": 0,
                    "num_objects_missing": 0,
                    "num_objects_degraded": 3451,
                    "num_objects_misplaced": 0,
                    "num_objects_unfound": 0,
                    "num_objects_dirty": 3451,
                    "num_whiteouts": 0,
                    "num_read": 1546881,
                    "num_read_kb": 79514552,
                    "num_write": 2917236,
                    "num_write_kb": 76790980,
                    "num_scrub_errors": 2,
                    "num_shallow_scrub_errors": 0,
                    "num_deep_scrub_errors": 2,
                    "num_objects_recovered": 7192,
                    "num_bytes_recovered": 29903663616,
                    "num_keys_recovered": 0,
                    "num_objects_omap": 0,
                    "num_objects_hit_set_archive": 0,
                    "num_bytes_hit_set_archive": 0,
                    "num_flush": 0,
                    "num_flush_kb": 0,
                    "num_evict": 0,
                    "num_evict_kb": 0,
                    "num_promote": 0,
                    "num_flush_mode_high": 0,
                    "num_flush_mode_low": 0,
                    "num_evict_mode_some": 0,
                    "num_evict_mode_full": 0,
                    "num_objects_pinned": 0
                },
                "up": [
                    36,
                    9,
                    24
                ],
                "acting": [
                    36,
                    9,
                    24
                ],
                "blocked_by": [],
                "up_primary": 36,
                "acting_primary": 36
            },
            "empty": 0,
            "dne": 0,
            "incomplete": 0,
            "last_epoch_started": 16327,
            "hit_set_history": {
                "current_last_update": "0'0",
                "history": []
            }
        }
    ],
    "recovery_state": [
        {
            "name": "Started\/Primary\/Active",
            "enter_time": "2017-09-12 12:18:07.920607",
            "might_have_unfound": [
                {
                    "osd": "9",
                    "status": "already probed" 
                },
                {
                    "osd": "24",
                    "status": "already probed" 
                }
            ],
            "recovery_progress": {
                "backfill_targets": [],
                "waiting_on_backfill": [],
                "last_backfill_started": "MIN",
                "backfill_info": {
                    "begin": "MIN",
                    "end": "MIN",
                    "objects": []
                },
                "peer_backfill_info": [],
                "backfills_in_flight": [],
                "recovering": [],
                "pg_backend": {
                    "pull_from_peer": [],
                    "pushing": []
                }
            },
            "scrub": {
                "scrubber.epoch_start": "0",
                "scrubber.active": 0,
                "scrubber.state": "INACTIVE",
                "scrubber.start": "MIN",
                "scrubber.end": "MIN",
                "scrubber.subset_last_update": "0'0",
                "scrubber.deep": false,
                "scrubber.seed": 0,
                "scrubber.waiting_on": 0,
                "scrubber.waiting_on_whom": []
            }
        },
        {
            "name": "Started",
            "enter_time": "2017-09-12 12:18:06.130121" 
        }
    ],
    "agent_state": {}
}
ceph-admin@admin:[~]: ceph pg 8.313 query
{
    "state": "active+clean+inconsistent",
    "snap_trimq": "[]",
    "epoch": 16328,
    "up": [
        36,
        9,
        24
    ],
    "acting": [
        36,
        9,
        24
    ],
    "actingbackfill": [
        "9",
        "24",
        "36" 
    ],
    "info": {
        "pgid": "8.313",
        "last_update": "16328'4138545",
        "last_complete": "16328'4138545",
        "log_tail": "16316'4135499",
        "last_user_version": 4138545,
        "last_backfill": "MAX",
        "last_backfill_bitwise": 1,
        "purged_snaps": "[1~9,34~2,37~1,52~1,55~2,59~a,64~1,6a~1,6c~5,72~1]",
        "history": {
            "epoch_created": 2548,
            "last_epoch_started": 16327,
            "last_epoch_clean": 16327,
            "last_epoch_split": 0,
            "last_epoch_marked_full": 0,
            "same_up_since": 16324,
            "same_interval_since": 16325,
            "same_primary_since": 16325,
            "last_scrub": "16321'4138121",
            "last_scrub_stamp": "2017-09-12 12:17:16.862962",
            "last_deep_scrub": "16321'4138121",
            "last_deep_scrub_stamp": "2017-09-12 12:17:16.862962",
            "last_clean_scrub_stamp": "2017-09-11 00:38:37.136245" 
        },
        "stats": {
            "version": "16328'4138545",
            "reported_seq": "4117787",
            "reported_epoch": "16328",
            "state": "active+clean+inconsistent",
            "last_fresh": "2017-09-12 12:51:15.143107",
            "last_change": "2017-09-12 12:19:40.864289",
            "last_active": "2017-09-12 12:51:15.143107",
            "last_peered": "2017-09-12 12:51:15.143107",
            "last_clean": "2017-09-12 12:51:15.143107",
            "last_became_active": "2017-09-12 12:18:12.322524",
            "last_became_peered": "2017-09-12 12:18:12.322524",
            "last_unstale": "2017-09-12 12:51:15.143107",
            "last_undegraded": "2017-09-12 12:51:15.143107",
            "last_fullsized": "2017-09-12 12:51:15.143107",
            "mapping_epoch": 16324,
            "log_start": "16316'4135499",
            "ondisk_log_start": "16316'4135499",
            "created": 2548,
            "last_epoch_clean": 16327,
            "parent": "0.0",
            "parent_split_bits": 10,
            "last_scrub": "16321'4138121",
            "last_scrub_stamp": "2017-09-12 12:17:16.862962",
            "last_deep_scrub": "16321'4138121",
            "last_deep_scrub_stamp": "2017-09-12 12:17:16.862962",
            "last_clean_scrub_stamp": "2017-09-11 00:38:37.136245",
            "log_size": 3046,
            "ondisk_log_size": 3046,
            "stats_invalid": false,
            "dirty_stats_invalid": false,
            "omap_stats_invalid": false,
            "hitset_stats_invalid": false,
            "hitset_bytes_stats_invalid": false,
            "pin_stats_invalid": false,
            "stat_sum": {
                "num_bytes": 13725943296,
                "num_objects": 3451,
                "num_object_clones": 575,
                "num_object_copies": 10353,
                "num_objects_missing_on_primary": 0,
                "num_objects_missing": 0,
                "num_objects_degraded": 0,
                "num_objects_misplaced": 0,
                "num_objects_unfound": 0,
                "num_objects_dirty": 3451,
                "num_whiteouts": 0,
                "num_read": 1547019,
                "num_read_kb": 79518748,
                "num_write": 2917649,
                "num_write_kb": 76792604,
                "num_scrub_errors": 2,
                "num_shallow_scrub_errors": 0,
                "num_deep_scrub_errors": 2,
                "num_objects_recovered": 7195,
                "num_bytes_recovered": 29916246528,
                "num_keys_recovered": 0,
                "num_objects_omap": 0,
                "num_objects_hit_set_archive": 0,
                "num_bytes_hit_set_archive": 0,
                "num_flush": 0,
                "num_flush_kb": 0,
                "num_evict": 0,
                "num_evict_kb": 0,
                "num_promote": 0,
                "num_flush_mode_high": 0,
                "num_flush_mode_low": 0,
                "num_evict_mode_some": 0,
                "num_evict_mode_full": 0,
                "num_objects_pinned": 0
            },
            "up": [
                36,
                9,
                24
            ],
            "acting": [
                36,
                9,
                24
            ],
            "blocked_by": [],
            "up_primary": 36,
            "acting_primary": 36
        },
        "empty": 0,
        "dne": 0,
        "incomplete": 0,
        "last_epoch_started": 16327,
        "hit_set_history": {
            "current_last_update": "0'0",
            "history": []
        }
    },
    "peer_info": [
        {
            "peer": "9",
            "pgid": "8.313",
            "last_update": "16328'4138545",
            "last_complete": "16328'4138545",
            "log_tail": "16316'4135099",
            "last_user_version": 4138132,
            "last_backfill": "MAX",
            "last_backfill_bitwise": 1,
            "purged_snaps": "[1~9,34~2,37~1,52~1,55~2,59~a,64~1,6a~1,6c~5,72~1]",
            "history": {
                "epoch_created": 2548,
                "last_epoch_started": 16327,
                "last_epoch_clean": 16327,
                "last_epoch_split": 0,
                "last_epoch_marked_full": 0,
                "same_up_since": 16324,
                "same_interval_since": 16325,
                "same_primary_since": 16325,
                "last_scrub": "16321'4138121",
                "last_scrub_stamp": "2017-09-12 12:17:16.862962",
                "last_deep_scrub": "16321'4138121",
                "last_deep_scrub_stamp": "2017-09-12 12:17:16.862962",
                "last_clean_scrub_stamp": "2017-09-11 00:38:37.136245" 
            },
            "stats": {
                "version": "16323'4138132",
                "reported_seq": "4117369",
                "reported_epoch": "16326",
                "state": "remapped+inconsistent+peering",
                "last_fresh": "2017-09-12 12:18:04.434843",
                "last_change": "2017-09-12 12:18:04.412687",
                "last_active": "2017-09-12 12:17:52.681445",
                "last_peered": "2017-09-12 12:17:52.681445",
                "last_clean": "2017-09-12 12:17:14.504126",
                "last_became_active": "2017-09-12 12:17:36.831098",
                "last_became_peered": "2017-09-12 12:17:36.831098",
                "last_unstale": "2017-09-12 12:18:04.434843",
                "last_undegraded": "2017-09-12 12:18:04.434843",
                "last_fullsized": "2017-09-12 12:18:04.434843",
                "mapping_epoch": 16324,
                "log_start": "16316'4135099",
                "ondisk_log_start": "16316'4135099",
                "created": 2548,
                "last_epoch_clean": 16323,
                "parent": "0.0",
                "parent_split_bits": 10,
                "last_scrub": "16321'4138121",
                "last_scrub_stamp": "2017-09-12 12:17:16.862962",
                "last_deep_scrub": "16321'4138121",
                "last_deep_scrub_stamp": "2017-09-12 12:17:16.862962",
                "last_clean_scrub_stamp": "2017-09-11 00:38:37.136245",
                "log_size": 3033,
                "ondisk_log_size": 3033,
                "stats_invalid": false,
                "dirty_stats_invalid": false,
                "omap_stats_invalid": false,
                "hitset_stats_invalid": false,
                "hitset_bytes_stats_invalid": false,
                "pin_stats_invalid": false,
                "stat_sum": {
                    "num_bytes": 13725943296,
                    "num_objects": 3451,
                    "num_object_clones": 575,
                    "num_object_copies": 10353,
                    "num_objects_missing_on_primary": 0,
                    "num_objects_missing": 0,
                    "num_objects_degraded": 0,
                    "num_objects_misplaced": 0,
                    "num_objects_unfound": 0,
                    "num_objects_dirty": 3451,
                    "num_whiteouts": 0,
                    "num_read": 1546881,
                    "num_read_kb": 79514552,
                    "num_write": 2917236,
                    "num_write_kb": 76790980,
                    "num_scrub_errors": 2,
                    "num_shallow_scrub_errors": 0,
                    "num_deep_scrub_errors": 2,
                    "num_objects_recovered": 7192,
                    "num_bytes_recovered": 29903663616,
                    "num_keys_recovered": 0,
                    "num_objects_omap": 0,
                    "num_objects_hit_set_archive": 0,
                    "num_bytes_hit_set_archive": 0,
                    "num_flush": 0,
                    "num_flush_kb": 0,
                    "num_evict": 0,
                    "num_evict_kb": 0,
                    "num_promote": 0,
                    "num_flush_mode_high": 0,
                    "num_flush_mode_low": 0,
                    "num_evict_mode_some": 0,
                    "num_evict_mode_full": 0,
                    "num_objects_pinned": 0
                },
                "up": [
                    36,
                    9,
                    24
                ],
                "acting": [
                    36,
                    9,
                    24
                ],
                "blocked_by": [],
                "up_primary": 36,
                "acting_primary": 36
            },
            "empty": 0,
            "dne": 0,
            "incomplete": 0,
            "last_epoch_started": 16327,
            "hit_set_history": {
                "current_last_update": "0'0",
                "history": []
            }
        },
        {
            "peer": "24",
            "pgid": "8.313",
            "last_update": "16328'4138545",
            "last_complete": "16328'4138545",
            "log_tail": "16316'4135099",
            "last_user_version": 4138132,
            "last_backfill": "MAX",
            "last_backfill_bitwise": 1,
            "purged_snaps": "[1~9,34~2,37~1,52~1,55~2,59~a,64~1,6a~1,6c~5,72~1]",
            "history": {
                "epoch_created": 2548,
                "last_epoch_started": 16327,
                "last_epoch_clean": 16327,
                "last_epoch_split": 0,
                "last_epoch_marked_full": 0,
                "same_up_since": 16324,
                "same_interval_since": 16325,
                "same_primary_since": 16325,
                "last_scrub": "16321'4138121",
                "last_scrub_stamp": "2017-09-12 12:17:16.862962",
                "last_deep_scrub": "16321'4138121",
                "last_deep_scrub_stamp": "2017-09-12 12:17:16.862962",
                "last_clean_scrub_stamp": "2017-09-11 00:38:37.136245" 
            },
            "stats": {
                "version": "16323'4138131",
                "reported_seq": "4117377",
                "reported_epoch": "16323",
                "state": "active+undersized+degraded+inconsistent",
                "last_fresh": "2017-09-12 12:17:42.690845",
                "last_change": "2017-09-12 12:17:36.831098",
                "last_active": "2017-09-12 12:17:42.690845",
                "last_peered": "2017-09-12 12:17:42.690845",
                "last_clean": "2017-09-12 12:17:14.504126",
                "last_became_active": "2017-09-12 12:17:36.831098",
                "last_became_peered": "2017-09-12 12:17:36.831098",
                "last_unstale": "2017-09-12 12:17:42.690845",
                "last_undegraded": "2017-09-12 12:17:35.813915",
                "last_fullsized": "2017-09-12 12:17:35.813915",
                "mapping_epoch": 16324,
                "log_start": "16316'4135099",
                "ondisk_log_start": "16316'4135099",
                "created": 2548,
                "last_epoch_clean": 16323,
                "parent": "0.0",
                "parent_split_bits": 10,
                "last_scrub": "16321'4138121",
                "last_scrub_stamp": "2017-09-12 12:17:16.862962",
                "last_deep_scrub": "16321'4138121",
                "last_deep_scrub_stamp": "2017-09-12 12:17:16.862962",
                "last_clean_scrub_stamp": "2017-09-11 00:38:37.136245",
                "log_size": 3032,
                "ondisk_log_size": 3032,
                "stats_invalid": false,
                "dirty_stats_invalid": false,
                "omap_stats_invalid": false,
                "hitset_stats_invalid": false,
                "hitset_bytes_stats_invalid": false,
                "pin_stats_invalid": false,
                "stat_sum": {
                    "num_bytes": 13725943296,
                    "num_objects": 3451,
                    "num_object_clones": 575,
                    "num_object_copies": 10353,
                    "num_objects_missing_on_primary": 0,
                    "num_objects_missing": 0,
                    "num_objects_degraded": 3451,
                    "num_objects_misplaced": 0,
                    "num_objects_unfound": 0,
                    "num_objects_dirty": 3451,
                    "num_whiteouts": 0,
                    "num_read": 1546881,
                    "num_read_kb": 79514552,
                    "num_write": 2917236,
                    "num_write_kb": 76790980,
                    "num_scrub_errors": 2,
                    "num_shallow_scrub_errors": 0,
                    "num_deep_scrub_errors": 2,
                    "num_objects_recovered": 7192,
                    "num_bytes_recovered": 29903663616,
                    "num_keys_recovered": 0,
                    "num_objects_omap": 0,
                    "num_objects_hit_set_archive": 0,
                    "num_bytes_hit_set_archive": 0,
                    "num_flush": 0,
                    "num_flush_kb": 0,
                    "num_evict": 0,
                    "num_evict_kb": 0,
                    "num_promote": 0,
                    "num_flush_mode_high": 0,
                    "num_flush_mode_low": 0,
                    "num_evict_mode_some": 0,
                    "num_evict_mode_full": 0,
                    "num_objects_pinned": 0
                },
                "up": [
                    36,
                    9,
                    24
                ],
                "acting": [
                    36,
                    9,
                    24
                ],
                "blocked_by": [],
                "up_primary": 36,
                "acting_primary": 36
            },
            "empty": 0,
            "dne": 0,
            "incomplete": 0,
            "last_epoch_started": 16327,
            "hit_set_history": {
                "current_last_update": "0'0",
                "history": []
            }
        }
    ],
    "recovery_state": [
        {
            "name": "Started\/Primary\/Active",
            "enter_time": "2017-09-12 12:18:07.920607",
            "might_have_unfound": [
                {
                    "osd": "9",
                    "status": "already probed" 
                },
                {
                    "osd": "24",
                    "status": "already probed" 
                }
            ],
            "recovery_progress": {
                "backfill_targets": [],
                "waiting_on_backfill": [],
                "last_backfill_started": "MIN",
                "backfill_info": {
                    "begin": "MIN",
                    "end": "MIN",
                    "objects": []
                },
                "peer_backfill_info": [],
                "backfills_in_flight": [],
                "recovering": [],
                "pg_backend": {
                    "pull_from_peer": [],
                    "pushing": []
                }
            },
            "scrub": {
                "scrubber.epoch_start": "0",
                "scrubber.active": 0,
                "scrubber.state": "INACTIVE",
                "scrubber.start": "MIN",
                "scrubber.end": "MIN",
                "scrubber.subset_last_update": "0'0",
                "scrubber.deep": false,
                "scrubber.seed": 0,
                "scrubber.waiting_on": 0,
                "scrubber.waiting_on_whom": []
            }
        },
        {
            "name": "Started",
            "enter_time": "2017-09-12 12:18:06.130121" 
        }
    ],
    "agent_state": {}
}
Actions #7

Updated by Jan-Willem Michels over 6 years ago

Just wanted everybody to know we have the same problem.
This is on a cluster that had no real problem apart from this error.
The problem started on kraken release, and is still there after luminous upgrade
All disks are bluestore
pgrepair does not work- do anything, deep scrub will start but find no real problem.

rados list-inconsistent-pg default.rgw.buckets.data
["15.720"]

rados list-inconsistent-obj 15.720 --format=json-pretty
No scrub information available for pg 15.720
error 2: (2) No such file or directory

if somebody want's we can provide extra info.

Actions #8

Updated by Jake Grimmett almost 6 years ago

Just seen what might be the same issue on a cluster upgraded from Luminous to Mimic (13.2.0)
  1. rados list-inconsistent-pg compressed_ecpool
    ["4.ff","4.2e2","4.551","4.61c","4.8cd","4.a20","4.e19"]
  1. rados list-inconsistent-obj 4.ff --format=json-pretty
    No scrub information available for pg 4.ff
    error 2: (2) No such file or directory

The following commands appear to run OK, but nothing appears in the
osd log to indicate that the commands are running. The OSD's are
otherwise working & logging OK.

  1. ceph pg scrub 4.e19
    instructing pg 4.e19s0 on osd.246 to scrub
  1. ceph pg repair 4.e19
    instructing pg 4.e19s0 on osd.246 to repair
  1. ceph osd scrub 246
    instructed osd(s) 246 to scrub
  1. ceph osd repair 246
    instructed osd(s) 246 to repair

It does not matter which osd or pg the repair is initiated on.

From the OSD logs, and 'ceph -s' I can see that the OSD's are still
doing automatic background pg scrubs, just not the ones I have asked
them to do, at the time of my request they are not currently scrubbing.

Could it be that my commands are not being sent to the OSD's?

Any idea on how to debug this?
Further info:

Output of 'ceph pg 4.e19 query' is here:
http://p.ip.fi/9x5v

Output of 'ceph daemon osd.246 config show' is here
http://p.ip.fi/RAuk

Cluster has 10 nodes, 128GB RAM, dual Xeon
450 Bluestore SATA OSD, EC 8:2
4 NVME OSD, replicated
used for cephfs (2.3PB), daily snapshots only

  1. ceph health detail
    HEALTH_ERR 9500031/5149746146 objects misplaced (0.184%); 80 scrub
    errors; Possible data damage: 7 pgs inconsistent
    OBJECT_MISPLACED 9500031/5149746146 objects misplaced (0.184%)
    OSD_SCRUB_ERRORS 80 scrub errors
    PG_DAMAGED Possible data damage: 7 pgs inconsistent
    pg 4.ff is active+clean+inconsistent, acting
    [318,403,150,13,225,261,382,175,282,324]
    pg 4.2e2 is active+clean+inconsistent, acting
    [352,59,328,451,195,119,42,66,158,150]
    pg 4.551 is active+clean+inconsistent, acting
    [391,105,124,150,205,22,269,184,293,91]
    pg 4.61c is active+clean+inconsistent, acting
    [382,131,84,35,282,214,236,366,309,150]
    pg 4.8cd is active+clean+inconsistent, acting
    [353,58,5,252,187,183,323,150,387,32]
    pg 4.a20 is active+clean+inconsistent, acting
    [346,104,398,282,225,133,150,70,165,17]
    pg 4.e19 is active+clean+inconsistent, acting
    [246,447,245,98,170,348,111,155,150,295]
Actions #9

Updated by lin zhou over 5 years ago

just report the same error.
And I try to find the object,three replicas look the same.

first, search ceph.log, and I got:
388.c29 shard 295: soid 388:9430fef2:::c2e226a9-b855-45c5-a17f-b1c697755072.1813469.4__multipart_dumbo%2f180888654%2f20181221%2fxtrabackup_full_x19_30044_20181221025000%2fx19.xbstream.2~ntwW9vwutbmOJ4bDZYehERT2AokbtAi.3595:head candidate had a read error

then I search three osd contain this ps:
\73281ed56c92a56da078b1ae52e888e0 DIR_9/DIR_2/DIR_C/DIR_0/DIR_F/c2e226a9-b855-45c5-a17f-b1c697755072.1813469.4\\u\\umultipart\\udumbo\\s180888654\\s20181221\\sxtrabackup\\ufull\\ux19\\u30044\\u20181221025000\\sx19.xbstream.2~ntwW9vwutbmOJ4bDZYehERT2AokbtAi.3595__head_4F7F0C29__184

the md5 value is the same.
stat info is different:
root@cld-osd3-48:/home/ceph/var/lib/osd/ceph-33/current/388.c29_head# stat XXXX
Size: 4194304 Blocks: 8200 IO Block: 4096 regular file
Device: 891h/2193d Inode: 4300403471 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 999/ ceph) Gid: ( 999/ ceph)
Access: 2018-12-21 14:17:12.945132144 +0800
Modify: 2018-12-21 14:17:12.965132073 +0800
Change: 2018-12-21 14:17:13.761129235 +0800

root@cld-osd24-48:/home/ceph/var/lib/osd/ceph-279/current/388.c29_head# stat XXXX
Size: 4194304 Blocks: 8200 IO Block: 4096 regular file
Device: 831h/2097d Inode: 8646464869 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 999/ ceph) Gid: ( 999/ ceph)
Access: 2019-01-07 10:54:23.010293026 +0800
Modify: 2019-01-07 10:54:23.010293026 +0800
Change: 2019-01-07 10:54:23.014293004 +0800

root@cld-osd31-48:/home/ceph/var/lib/osd/ceph-363/current/388.c29_head# stat XXX
Size: 4194304 Blocks: 8200 IO Block: 4096 regular file
Device: 831h/2097d Inode: 13141445890 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 999/ ceph) Gid: ( 999/ ceph)
Access: 2018-12-21 14:17:12.946862160 +0800
Modify: 2018-12-21 14:17:12.966862262 +0800
Change: 2018-12-21 14:17:13.762866312 +0800

Actions #10

Updated by lin zhou over 5 years ago

lin zhou wrote:

just report the same error.
And I try to find the object,three replicas look the same.

first, search ceph.log, and I got:
388.c29 shard 295: soid 388:9430fef2:::c2e226a9-b855-45c5-a17f-b1c697755072.1813469.4__multipart_dumbo%2f180888654%2f20181221%2fxtrabackup_full_x19_30044_20181221025000%2fx19.xbstream.2~ntwW9vwutbmOJ4bDZYehERT2AokbtAi.3595:head candidate had a read error

then I search three osd contain this ps:
\73281ed56c92a56da078b1ae52e888e0 DIR_9/DIR_2/DIR_C/DIR_0/DIR_F/c2e226a9-b855-45c5-a17f-b1c697755072.1813469.4\\u\\umultipart\\udumbo\\s180888654\\s20181221\\sxtrabackup\\ufull\\ux19\\u30044\\u20181221025000\\sx19.xbstream.2~ntwW9vwutbmOJ4bDZYehERT2AokbtAi.3595__head_4F7F0C29__184

the md5 value is the same.
stat info is different:
root@cld-osd3-48:/home/ceph/var/lib/osd/ceph-33/current/388.c29_head# stat XXXX
Size: 4194304 Blocks: 8200 IO Block: 4096 regular file
Device: 891h/2193d Inode: 4300403471 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 999/ ceph) Gid: ( 999/ ceph)
Access: 2018-12-21 14:17:12.945132144 +0800
Modify: 2018-12-21 14:17:12.965132073 +0800
Change: 2018-12-21 14:17:13.761129235 +0800

root@cld-osd24-48:/home/ceph/var/lib/osd/ceph-279/current/388.c29_head# stat XXXX
Size: 4194304 Blocks: 8200 IO Block: 4096 regular file
Device: 831h/2097d Inode: 8646464869 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 999/ ceph) Gid: ( 999/ ceph)
Access: 2019-01-07 10:54:23.010293026 +0800
Modify: 2019-01-07 10:54:23.010293026 +0800
Change: 2019-01-07 10:54:23.014293004 +0800

root@cld-osd31-48:/home/ceph/var/lib/osd/ceph-363/current/388.c29_head# stat XXX
Size: 4194304 Blocks: 8200 IO Block: 4096 regular file
Device: 831h/2097d Inode: 13141445890 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 999/ ceph) Gid: ( 999/ ceph)
Access: 2018-12-21 14:17:12.946862160 +0800
Modify: 2018-12-21 14:17:12.966862262 +0800
Change: 2018-12-21 14:17:13.762866312 +0800

my ceph version is 10.2.11

Actions #11

Updated by Niklas Hambuechen 11 months ago

Can we reopen this bug?

I also hit it with Ceph 16.2.7:

https://tracker.ceph.com/issues/61739

Actions #12

Updated by can zhu 11 months ago

I alose hit it with ceph 16.2.10

root@node02:/# ceph version
ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable)

root@node02:/# ceph health detail
HEALTH_ERR 1 OSDs or CRUSH {nodes, device-classes} have {NOUP,NODOWN,NOIN,NOOUT} flags set; 6 scrub errors; Possible data damage: 1 pg inconsistent; Degraded data redundancy: 354469/570197792 objects degraded (0.062%), 5 pgs degraded, 6 pgs undersized
[WRN] OSD_FLAGS: 1 OSDs or CRUSH {nodes, device-classes} have {NOUP,NODOWN,NOIN,NOOUT} flags set
host node05 has flags noout
[ERR] OSD_SCRUB_ERRORS: 6 scrub errors
[ERR] PG_DAMAGED: Possible data damage: 1 pg inconsistent
pg 36.2f4 is active+clean+inconsistent, acting [104,99,1]
[WRN] PG_DEGRADED: Degraded data redundancy: 354469/570197792 objects degraded (0.062%), 5 pgs degraded, 6 pgs undersized
pg 34.8 is stuck undersized for 23h, current state active+undersized+degraded+remapped+backfill_wait, last acting [2147483647,12,65,18,17,48,81,79]
pg 34.10 is stuck undersized for 23h, current state active+undersized+degraded+remapped+backfill_wait, last acting [102,29,17,6,63,12,92,2147483647]
pg 34.c6 is stuck undersized for 23h, current state active+undersized+remapped+backfilling, last acting [74,97,103,2147483647,63,29,46,12]
pg 34.1a6 is stuck undersized for 23h, current state active+undersized+degraded+remapped+backfilling, last acting [74,2147483647,67,98,37,99,95,63]
pg 34.3cb is stuck undersized for 23h, current state active+undersized+degraded+remapped+backfill_wait, last acting [92,95,94,2147483647,103,79,68,15]
pg 36.3f is stuck undersized for 23h, current state active+undersized+degraded+remapped+backfilling, last acting [102,67]
root@node02:/# rados list-inconsistent-obj 36.2f4 --forma=json-pretty
No scrub information available for pg 36.2f4
error 2: (2) No such file or directory

Actions

Also available in: Atom PDF