Bug #15781
closedNo scrub information available for pg 0.8
0%
Description
2016-05-07T13:26:46.986 INFO:teuthology.orchestra.run.smithi026.stderr:instructing pg 0.8 on osd.3 to deep-scrub 2016-05-07T13:26:50.188 INFO:tasks.ceph.osd.3.smithi035.stderr:2016-05-07 20:26:50.188964 7fc92f4ad700 -1 log_channel(cluster) log [ERR] : 0.8 shard 3: soid 0:1c3a8a5d:::benchmark_data_smithi026_8762_object1000:head data_digest 0xa00e5e99 != known data_digest 0x4e33d242 from auth shard 5, size 4097 != known size 40 96 2016-05-07T13:26:50.212 INFO:tasks.ceph.osd.3.smithi035.stderr:2016-05-07 20:26:50.213144 7fc92f4ad700 -1 log_channel(cluster) log [ERR] : 0.8 deep-scrub 0 missing, 1 inconsistent objects 2016-05-07T13:26:50.213 INFO:tasks.ceph.osd.3.smithi035.stderr:2016-05-07 20:26:50.213151 7fc92f4ad700 -1 log_channel(cluster) log [ERR] : 0.8 deep-scrub 1 errors 2016-05-07T13:27:46.999 INFO:teuthology.orchestra.run.smithi026:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph pg dump --format=json' 2016-05-07T13:27:47.359 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all in format json 2016-05-07T13:27:47.371 INFO:teuthology.orchestra.run.smithi026:Running: 'rados list-inconsistent-pg rbd --format=json' 2016-05-07T13:27:47.482 INFO:teuthology.orchestra.run.smithi026:Running: 'rados list-inconsistent-obj 0.8 --format=json' 2016-05-07T13:27:47.569 INFO:teuthology.orchestra.run.smithi026.stderr:No scrub information available for pg 0.8 2016-05-07T13:27:47.569 INFO:teuthology.orchestra.run.smithi026.stderr:error 2: (2) No such file or directory
/a/yuriw-2016-05-07_09:11:04-rados-wip-yuri-testing---basic-smithi/176828
Updated by Samuel Just over 7 years ago
- Status changed from New to Can't reproduce
Updated by Matthew Roy about 7 years ago
I might have just hit this bug on a real cluster. A disk was gently failing, generating read errors which resulted in inconsistent PGs. I told them to repair, but then thought better of it and decided to mark the osd as out (osd.0). The pgs then ended up in this state after the cluster completed recovery:
root@atropos:~# ceph health detail HEALTH_ERR 8 pgs inconsistent; 14 scrub errors; mds0: Client smilebox:smilebox-cephfs failing to respond to cache pressure pg 1.3f is active+clean+inconsistent, acting [14,5,6] pg 1.10 is active+clean+inconsistent, acting [5,2,3] pg 0.23 is active+clean+inconsistent, acting [10,8,12] pg 1.45 is active+clean+inconsistent, acting [14,3,10] pg 1.5c is active+clean+inconsistent, acting [7,11,12] pg 1.66 is active+clean+inconsistent, acting [1,2,9] pg 1.6f is active+clean+inconsistent, acting [1,2,6] pg 1.7d is active+clean+inconsistent, acting [10,2,12] 14 scrub errors mds0: Client smilebox:smilebox-cephfs failing to respond to cache pressure(client_id: 275976) root@atropos:~# rados list-inconsistent-obj 0.23 --format=json-pretty No scrub information available for pg 0.23 error 2: (2) No such file or directory
Updated by Kenneth Waegeman about 7 years ago
We've just found the same issue on our cluster too:
ceph health detail
HEALTH_ERR 1 pgs inconsistent; 6 scrub errors
pg 5.d7 is active+clean+inconsistent, acting [53,42,120,14,59,33,68,20,128,142,149]
6 scrub errors
tried to repair, but did not work.
I tried to run list-inconsistent-obj , but got:
rados list-inconsistent-obj 5.d7
No scrub information available for pg 5.d7
error 2: (2) No such file or directory
Updated by Patrick Fruh over 6 years ago
Just had the same happen on our cluster.
I updated our hosts and after rebooting, the xfs filesystem on 2 OSDs on one host was dead
[ 991.046980] XFS (sdg1): Internal error XFS_WANT_CORRUPTED_GOTO at line 3524 of file fs/xfs/libxfs/xfs_btree.c. Caller xfs_free_ag_extent+0x4c1/0x7a0 [xfs] [ 991.047464] CPU: 26 PID: 9281 Comm: tp_fstore_op Not tainted 4.13.1-1 #1 [ 991.047466] Hardware name: Dell Inc. PowerEdge R730xd/0WCJNT, BIOS 2.4.3 01/17/2017 [ 991.047468] Call Trace: [ 991.047478] dump_stack+0x63/0x89 [ 991.047530] xfs_error_report+0x3b/0x40 [xfs] [ 991.047567] ? xfs_free_ag_extent+0x4c1/0x7a0 [xfs] [ 991.047611] xfs_btree_insert+0x1b0/0x1c0 [xfs] [ 991.047646] xfs_free_ag_extent+0x4c1/0x7a0 [xfs] [ 991.047688] xfs_free_extent+0xbb/0x150 [xfs] [ 991.047752] xfs_trans_free_extent+0x4f/0x110 [xfs] [ 991.047799] ? xfs_trans_add_item+0x5d/0x90 [xfs] [ 991.047844] xfs_extent_free_finish_item+0x26/0x40 [xfs] [ 991.047882] xfs_defer_finish+0x149/0x410 [xfs] [ 991.047931] xfs_itruncate_extents+0x124/0x2d0 [xfs] [ 991.047971] xfs_inactive_truncate+0xad/0x100 [xfs] [ 991.048015] xfs_inactive+0xec/0x110 [xfs] [ 991.048055] xfs_fs_destroy_inode+0xa4/0x210 [xfs] [ 991.048060] destroy_inode+0x3b/0x60 [ 991.048064] evict+0x142/0x1a0 [ 991.048068] iput+0x17d/0x1d0 [ 991.048074] do_unlinkat+0x194/0x2b0 [ 991.048078] SyS_unlink+0x16/0x20 [ 991.048085] do_syscall_64+0x67/0x150 [ 991.048091] entry_SYSCALL64_slow_path+0x25/0x25 [ 991.048094] RIP: 0033:0x7f01d8585367 [ 991.048096] RSP: 002b:00007f01c6bfb618 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 991.048099] RAX: ffffffffffffffda RBX: 000055929efff400 RCX: 00007f01d8585367 [ 991.048100] RDX: 00007f01d907c3f8 RSI: 00007f01c6bfb5e0 RDI: 00005592bc09f4e8 [ 991.048101] RBP: 000055929efff400 R08: 00007f01d84e3938 R09: 0000000000000001 [ 991.048102] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f01c6bfb980 [ 991.048104] R13: 00007f01c6bfc340 R14: 00007f01c6bfb8c0 R15: 00007f01c6bfc340 [ 991.048141] XFS (sdg1): xfs_do_force_shutdown(0x8) called from line 236 of file fs/xfs/libxfs/xfs_defer.c. Return address = 0xffffffffc068c1c7 [ 991.206593] XFS (sdg1): Corruption of in-memory data detected. Shutting down filesystem [ 991.206862] XFS (sdg1): Please umount the filesystem and rectify the problem(s)
so I repaired them with xfs_repair -L and put them back online and after the recovery of the degraded PGs everything went green again.
Now after about 10 hours (with our daily scrubbing window in between) I'm getting:
ceph-admin@admin:[~]: ceph health detail HEALTH_ERR 1 pgs inconsistent; 2 scrub errors pg 8.313 is active+clean+inconsistent, acting [36,9,24] 2 scrub errors
OSD 24 is the one with the xfs filesystem error from before.
Repair doesn't seem to fix it, it recovers a few things but shows the same inconsistent pg at the end of it, also "rados list-inconsisten-obj" can't find the scrub information:
ceph-admin@admin:[~]: rados list-inconsistent-obj 8.313 No scrub information available for pg 8.313 error 2: (2) No such file or directory
Updated by Patrick Fruh over 6 years ago
Here's the query of the pg, current time being 2017-09-12 12:51:00
ceph-admin@admin:[~]: ceph pg 8.313 query
{
"state": "active+clean+inconsistent",
"snap_trimq": "[]",
"epoch": 16328,
"up": [
36,
9,
24
],
"acting": [
36,
9,
24
],
"actingbackfill": [
"9",
"24",
"36"
],
"info": {
"pgid": "8.313",
"last_update": "16328'4138539",
"last_complete": "16328'4138539",
"log_tail": "16316'4135499",
"last_user_version": 4138539,
"last_backfill": "MAX",
"last_backfill_bitwise": 1,
"purged_snaps": "[1~9,34~2,37~1,52~1,55~2,59~a,64~1,6a~1,6c~5,72~1]",
"history": {
"epoch_created": 2548,
"last_epoch_started": 16327,
"last_epoch_clean": 16327,
"last_epoch_split": 0,
"last_epoch_marked_full": 0,
"same_up_since": 16324,
"same_interval_since": 16325,
"same_primary_since": 16325,
"last_scrub": "16321'4138121",
"last_scrub_stamp": "2017-09-12 12:17:16.862962",
"last_deep_scrub": "16321'4138121",
"last_deep_scrub_stamp": "2017-09-12 12:17:16.862962",
"last_clean_scrub_stamp": "2017-09-11 00:38:37.136245"
},
"stats": {
"version": "16328'4138539",
"reported_seq": "4117781",
"reported_epoch": "16328",
"state": "active+clean+inconsistent",
"last_fresh": "2017-09-12 12:50:40.879013",
"last_change": "2017-09-12 12:19:40.864289",
"last_active": "2017-09-12 12:50:40.879013",
"last_peered": "2017-09-12 12:50:40.879013",
"last_clean": "2017-09-12 12:50:40.879013",
"last_became_active": "2017-09-12 12:18:12.322524",
"last_became_peered": "2017-09-12 12:18:12.322524",
"last_unstale": "2017-09-12 12:50:40.879013",
"last_undegraded": "2017-09-12 12:50:40.879013",
"last_fullsized": "2017-09-12 12:50:40.879013",
"mapping_epoch": 16324,
"log_start": "16316'4135499",
"ondisk_log_start": "16316'4135499",
"created": 2548,
"last_epoch_clean": 16327,
"parent": "0.0",
"parent_split_bits": 10,
"last_scrub": "16321'4138121",
"last_scrub_stamp": "2017-09-12 12:17:16.862962",
"last_deep_scrub": "16321'4138121",
"last_deep_scrub_stamp": "2017-09-12 12:17:16.862962",
"last_clean_scrub_stamp": "2017-09-11 00:38:37.136245",
"log_size": 3040,
"ondisk_log_size": 3040,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"stat_sum": {
"num_bytes": 13725943296,
"num_objects": 3451,
"num_object_clones": 575,
"num_object_copies": 10353,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 3451,
"num_whiteouts": 0,
"num_read": 1547019,
"num_read_kb": 79518748,
"num_write": 2917643,
"num_write_kb": 76792573,
"num_scrub_errors": 2,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 2,
"num_objects_recovered": 7195,
"num_bytes_recovered": 29916246528,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0
},
"up": [
36,
9,
24
],
"acting": [
36,
9,
24
],
"blocked_by": [],
"up_primary": 36,
"acting_primary": 36
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 16327,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
"peer_info": [
{
"peer": "9",
"pgid": "8.313",
"last_update": "16328'4138539",
"last_complete": "16328'4138539",
"log_tail": "16316'4135099",
"last_user_version": 4138132,
"last_backfill": "MAX",
"last_backfill_bitwise": 1,
"purged_snaps": "[1~9,34~2,37~1,52~1,55~2,59~a,64~1,6a~1,6c~5,72~1]",
"history": {
"epoch_created": 2548,
"last_epoch_started": 16327,
"last_epoch_clean": 16327,
"last_epoch_split": 0,
"last_epoch_marked_full": 0,
"same_up_since": 16324,
"same_interval_since": 16325,
"same_primary_since": 16325,
"last_scrub": "16321'4138121",
"last_scrub_stamp": "2017-09-12 12:17:16.862962",
"last_deep_scrub": "16321'4138121",
"last_deep_scrub_stamp": "2017-09-12 12:17:16.862962",
"last_clean_scrub_stamp": "2017-09-11 00:38:37.136245"
},
"stats": {
"version": "16323'4138132",
"reported_seq": "4117369",
"reported_epoch": "16326",
"state": "remapped+inconsistent+peering",
"last_fresh": "2017-09-12 12:18:04.434843",
"last_change": "2017-09-12 12:18:04.412687",
"last_active": "2017-09-12 12:17:52.681445",
"last_peered": "2017-09-12 12:17:52.681445",
"last_clean": "2017-09-12 12:17:14.504126",
"last_became_active": "2017-09-12 12:17:36.831098",
"last_became_peered": "2017-09-12 12:17:36.831098",
"last_unstale": "2017-09-12 12:18:04.434843",
"last_undegraded": "2017-09-12 12:18:04.434843",
"last_fullsized": "2017-09-12 12:18:04.434843",
"mapping_epoch": 16324,
"log_start": "16316'4135099",
"ondisk_log_start": "16316'4135099",
"created": 2548,
"last_epoch_clean": 16323,
"parent": "0.0",
"parent_split_bits": 10,
"last_scrub": "16321'4138121",
"last_scrub_stamp": "2017-09-12 12:17:16.862962",
"last_deep_scrub": "16321'4138121",
"last_deep_scrub_stamp": "2017-09-12 12:17:16.862962",
"last_clean_scrub_stamp": "2017-09-11 00:38:37.136245",
"log_size": 3033,
"ondisk_log_size": 3033,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"stat_sum": {
"num_bytes": 13725943296,
"num_objects": 3451,
"num_object_clones": 575,
"num_object_copies": 10353,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 3451,
"num_whiteouts": 0,
"num_read": 1546881,
"num_read_kb": 79514552,
"num_write": 2917236,
"num_write_kb": 76790980,
"num_scrub_errors": 2,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 2,
"num_objects_recovered": 7192,
"num_bytes_recovered": 29903663616,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0
},
"up": [
36,
9,
24
],
"acting": [
36,
9,
24
],
"blocked_by": [],
"up_primary": 36,
"acting_primary": 36
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 16327,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "24",
"pgid": "8.313",
"last_update": "16328'4138539",
"last_complete": "16328'4138539",
"log_tail": "16316'4135099",
"last_user_version": 4138132,
"last_backfill": "MAX",
"last_backfill_bitwise": 1,
"purged_snaps": "[1~9,34~2,37~1,52~1,55~2,59~a,64~1,6a~1,6c~5,72~1]",
"history": {
"epoch_created": 2548,
"last_epoch_started": 16327,
"last_epoch_clean": 16327,
"last_epoch_split": 0,
"last_epoch_marked_full": 0,
"same_up_since": 16324,
"same_interval_since": 16325,
"same_primary_since": 16325,
"last_scrub": "16321'4138121",
"last_scrub_stamp": "2017-09-12 12:17:16.862962",
"last_deep_scrub": "16321'4138121",
"last_deep_scrub_stamp": "2017-09-12 12:17:16.862962",
"last_clean_scrub_stamp": "2017-09-11 00:38:37.136245"
},
"stats": {
"version": "16323'4138131",
"reported_seq": "4117377",
"reported_epoch": "16323",
"state": "active+undersized+degraded+inconsistent",
"last_fresh": "2017-09-12 12:17:42.690845",
"last_change": "2017-09-12 12:17:36.831098",
"last_active": "2017-09-12 12:17:42.690845",
"last_peered": "2017-09-12 12:17:42.690845",
"last_clean": "2017-09-12 12:17:14.504126",
"last_became_active": "2017-09-12 12:17:36.831098",
"last_became_peered": "2017-09-12 12:17:36.831098",
"last_unstale": "2017-09-12 12:17:42.690845",
"last_undegraded": "2017-09-12 12:17:35.813915",
"last_fullsized": "2017-09-12 12:17:35.813915",
"mapping_epoch": 16324,
"log_start": "16316'4135099",
"ondisk_log_start": "16316'4135099",
"created": 2548,
"last_epoch_clean": 16323,
"parent": "0.0",
"parent_split_bits": 10,
"last_scrub": "16321'4138121",
"last_scrub_stamp": "2017-09-12 12:17:16.862962",
"last_deep_scrub": "16321'4138121",
"last_deep_scrub_stamp": "2017-09-12 12:17:16.862962",
"last_clean_scrub_stamp": "2017-09-11 00:38:37.136245",
"log_size": 3032,
"ondisk_log_size": 3032,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"stat_sum": {
"num_bytes": 13725943296,
"num_objects": 3451,
"num_object_clones": 575,
"num_object_copies": 10353,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 3451,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 3451,
"num_whiteouts": 0,
"num_read": 1546881,
"num_read_kb": 79514552,
"num_write": 2917236,
"num_write_kb": 76790980,
"num_scrub_errors": 2,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 2,
"num_objects_recovered": 7192,
"num_bytes_recovered": 29903663616,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0
},
"up": [
36,
9,
24
],
"acting": [
36,
9,
24
],
"blocked_by": [],
"up_primary": 36,
"acting_primary": 36
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 16327,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
}
],
"recovery_state": [
{
"name": "Started\/Primary\/Active",
"enter_time": "2017-09-12 12:18:07.920607",
"might_have_unfound": [
{
"osd": "9",
"status": "already probed"
},
{
"osd": "24",
"status": "already probed"
}
],
"recovery_progress": {
"backfill_targets": [],
"waiting_on_backfill": [],
"last_backfill_started": "MIN",
"backfill_info": {
"begin": "MIN",
"end": "MIN",
"objects": []
},
"peer_backfill_info": [],
"backfills_in_flight": [],
"recovering": [],
"pg_backend": {
"pull_from_peer": [],
"pushing": []
}
},
"scrub": {
"scrubber.epoch_start": "0",
"scrubber.active": 0,
"scrubber.state": "INACTIVE",
"scrubber.start": "MIN",
"scrubber.end": "MIN",
"scrubber.subset_last_update": "0'0",
"scrubber.deep": false,
"scrubber.seed": 0,
"scrubber.waiting_on": 0,
"scrubber.waiting_on_whom": []
}
},
{
"name": "Started",
"enter_time": "2017-09-12 12:18:06.130121"
}
],
"agent_state": {}
}
ceph-admin@admin:[~]: ceph pg 8.313 query
{
"state": "active+clean+inconsistent",
"snap_trimq": "[]",
"epoch": 16328,
"up": [
36,
9,
24
],
"acting": [
36,
9,
24
],
"actingbackfill": [
"9",
"24",
"36"
],
"info": {
"pgid": "8.313",
"last_update": "16328'4138545",
"last_complete": "16328'4138545",
"log_tail": "16316'4135499",
"last_user_version": 4138545,
"last_backfill": "MAX",
"last_backfill_bitwise": 1,
"purged_snaps": "[1~9,34~2,37~1,52~1,55~2,59~a,64~1,6a~1,6c~5,72~1]",
"history": {
"epoch_created": 2548,
"last_epoch_started": 16327,
"last_epoch_clean": 16327,
"last_epoch_split": 0,
"last_epoch_marked_full": 0,
"same_up_since": 16324,
"same_interval_since": 16325,
"same_primary_since": 16325,
"last_scrub": "16321'4138121",
"last_scrub_stamp": "2017-09-12 12:17:16.862962",
"last_deep_scrub": "16321'4138121",
"last_deep_scrub_stamp": "2017-09-12 12:17:16.862962",
"last_clean_scrub_stamp": "2017-09-11 00:38:37.136245"
},
"stats": {
"version": "16328'4138545",
"reported_seq": "4117787",
"reported_epoch": "16328",
"state": "active+clean+inconsistent",
"last_fresh": "2017-09-12 12:51:15.143107",
"last_change": "2017-09-12 12:19:40.864289",
"last_active": "2017-09-12 12:51:15.143107",
"last_peered": "2017-09-12 12:51:15.143107",
"last_clean": "2017-09-12 12:51:15.143107",
"last_became_active": "2017-09-12 12:18:12.322524",
"last_became_peered": "2017-09-12 12:18:12.322524",
"last_unstale": "2017-09-12 12:51:15.143107",
"last_undegraded": "2017-09-12 12:51:15.143107",
"last_fullsized": "2017-09-12 12:51:15.143107",
"mapping_epoch": 16324,
"log_start": "16316'4135499",
"ondisk_log_start": "16316'4135499",
"created": 2548,
"last_epoch_clean": 16327,
"parent": "0.0",
"parent_split_bits": 10,
"last_scrub": "16321'4138121",
"last_scrub_stamp": "2017-09-12 12:17:16.862962",
"last_deep_scrub": "16321'4138121",
"last_deep_scrub_stamp": "2017-09-12 12:17:16.862962",
"last_clean_scrub_stamp": "2017-09-11 00:38:37.136245",
"log_size": 3046,
"ondisk_log_size": 3046,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"stat_sum": {
"num_bytes": 13725943296,
"num_objects": 3451,
"num_object_clones": 575,
"num_object_copies": 10353,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 3451,
"num_whiteouts": 0,
"num_read": 1547019,
"num_read_kb": 79518748,
"num_write": 2917649,
"num_write_kb": 76792604,
"num_scrub_errors": 2,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 2,
"num_objects_recovered": 7195,
"num_bytes_recovered": 29916246528,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0
},
"up": [
36,
9,
24
],
"acting": [
36,
9,
24
],
"blocked_by": [],
"up_primary": 36,
"acting_primary": 36
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 16327,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
"peer_info": [
{
"peer": "9",
"pgid": "8.313",
"last_update": "16328'4138545",
"last_complete": "16328'4138545",
"log_tail": "16316'4135099",
"last_user_version": 4138132,
"last_backfill": "MAX",
"last_backfill_bitwise": 1,
"purged_snaps": "[1~9,34~2,37~1,52~1,55~2,59~a,64~1,6a~1,6c~5,72~1]",
"history": {
"epoch_created": 2548,
"last_epoch_started": 16327,
"last_epoch_clean": 16327,
"last_epoch_split": 0,
"last_epoch_marked_full": 0,
"same_up_since": 16324,
"same_interval_since": 16325,
"same_primary_since": 16325,
"last_scrub": "16321'4138121",
"last_scrub_stamp": "2017-09-12 12:17:16.862962",
"last_deep_scrub": "16321'4138121",
"last_deep_scrub_stamp": "2017-09-12 12:17:16.862962",
"last_clean_scrub_stamp": "2017-09-11 00:38:37.136245"
},
"stats": {
"version": "16323'4138132",
"reported_seq": "4117369",
"reported_epoch": "16326",
"state": "remapped+inconsistent+peering",
"last_fresh": "2017-09-12 12:18:04.434843",
"last_change": "2017-09-12 12:18:04.412687",
"last_active": "2017-09-12 12:17:52.681445",
"last_peered": "2017-09-12 12:17:52.681445",
"last_clean": "2017-09-12 12:17:14.504126",
"last_became_active": "2017-09-12 12:17:36.831098",
"last_became_peered": "2017-09-12 12:17:36.831098",
"last_unstale": "2017-09-12 12:18:04.434843",
"last_undegraded": "2017-09-12 12:18:04.434843",
"last_fullsized": "2017-09-12 12:18:04.434843",
"mapping_epoch": 16324,
"log_start": "16316'4135099",
"ondisk_log_start": "16316'4135099",
"created": 2548,
"last_epoch_clean": 16323,
"parent": "0.0",
"parent_split_bits": 10,
"last_scrub": "16321'4138121",
"last_scrub_stamp": "2017-09-12 12:17:16.862962",
"last_deep_scrub": "16321'4138121",
"last_deep_scrub_stamp": "2017-09-12 12:17:16.862962",
"last_clean_scrub_stamp": "2017-09-11 00:38:37.136245",
"log_size": 3033,
"ondisk_log_size": 3033,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"stat_sum": {
"num_bytes": 13725943296,
"num_objects": 3451,
"num_object_clones": 575,
"num_object_copies": 10353,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 3451,
"num_whiteouts": 0,
"num_read": 1546881,
"num_read_kb": 79514552,
"num_write": 2917236,
"num_write_kb": 76790980,
"num_scrub_errors": 2,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 2,
"num_objects_recovered": 7192,
"num_bytes_recovered": 29903663616,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0
},
"up": [
36,
9,
24
],
"acting": [
36,
9,
24
],
"blocked_by": [],
"up_primary": 36,
"acting_primary": 36
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 16327,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "24",
"pgid": "8.313",
"last_update": "16328'4138545",
"last_complete": "16328'4138545",
"log_tail": "16316'4135099",
"last_user_version": 4138132,
"last_backfill": "MAX",
"last_backfill_bitwise": 1,
"purged_snaps": "[1~9,34~2,37~1,52~1,55~2,59~a,64~1,6a~1,6c~5,72~1]",
"history": {
"epoch_created": 2548,
"last_epoch_started": 16327,
"last_epoch_clean": 16327,
"last_epoch_split": 0,
"last_epoch_marked_full": 0,
"same_up_since": 16324,
"same_interval_since": 16325,
"same_primary_since": 16325,
"last_scrub": "16321'4138121",
"last_scrub_stamp": "2017-09-12 12:17:16.862962",
"last_deep_scrub": "16321'4138121",
"last_deep_scrub_stamp": "2017-09-12 12:17:16.862962",
"last_clean_scrub_stamp": "2017-09-11 00:38:37.136245"
},
"stats": {
"version": "16323'4138131",
"reported_seq": "4117377",
"reported_epoch": "16323",
"state": "active+undersized+degraded+inconsistent",
"last_fresh": "2017-09-12 12:17:42.690845",
"last_change": "2017-09-12 12:17:36.831098",
"last_active": "2017-09-12 12:17:42.690845",
"last_peered": "2017-09-12 12:17:42.690845",
"last_clean": "2017-09-12 12:17:14.504126",
"last_became_active": "2017-09-12 12:17:36.831098",
"last_became_peered": "2017-09-12 12:17:36.831098",
"last_unstale": "2017-09-12 12:17:42.690845",
"last_undegraded": "2017-09-12 12:17:35.813915",
"last_fullsized": "2017-09-12 12:17:35.813915",
"mapping_epoch": 16324,
"log_start": "16316'4135099",
"ondisk_log_start": "16316'4135099",
"created": 2548,
"last_epoch_clean": 16323,
"parent": "0.0",
"parent_split_bits": 10,
"last_scrub": "16321'4138121",
"last_scrub_stamp": "2017-09-12 12:17:16.862962",
"last_deep_scrub": "16321'4138121",
"last_deep_scrub_stamp": "2017-09-12 12:17:16.862962",
"last_clean_scrub_stamp": "2017-09-11 00:38:37.136245",
"log_size": 3032,
"ondisk_log_size": 3032,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"stat_sum": {
"num_bytes": 13725943296,
"num_objects": 3451,
"num_object_clones": 575,
"num_object_copies": 10353,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 3451,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 3451,
"num_whiteouts": 0,
"num_read": 1546881,
"num_read_kb": 79514552,
"num_write": 2917236,
"num_write_kb": 76790980,
"num_scrub_errors": 2,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 2,
"num_objects_recovered": 7192,
"num_bytes_recovered": 29903663616,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0
},
"up": [
36,
9,
24
],
"acting": [
36,
9,
24
],
"blocked_by": [],
"up_primary": 36,
"acting_primary": 36
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 16327,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
}
],
"recovery_state": [
{
"name": "Started\/Primary\/Active",
"enter_time": "2017-09-12 12:18:07.920607",
"might_have_unfound": [
{
"osd": "9",
"status": "already probed"
},
{
"osd": "24",
"status": "already probed"
}
],
"recovery_progress": {
"backfill_targets": [],
"waiting_on_backfill": [],
"last_backfill_started": "MIN",
"backfill_info": {
"begin": "MIN",
"end": "MIN",
"objects": []
},
"peer_backfill_info": [],
"backfills_in_flight": [],
"recovering": [],
"pg_backend": {
"pull_from_peer": [],
"pushing": []
}
},
"scrub": {
"scrubber.epoch_start": "0",
"scrubber.active": 0,
"scrubber.state": "INACTIVE",
"scrubber.start": "MIN",
"scrubber.end": "MIN",
"scrubber.subset_last_update": "0'0",
"scrubber.deep": false,
"scrubber.seed": 0,
"scrubber.waiting_on": 0,
"scrubber.waiting_on_whom": []
}
},
{
"name": "Started",
"enter_time": "2017-09-12 12:18:06.130121"
}
],
"agent_state": {}
}
Updated by Jan-Willem Michels over 6 years ago
Just wanted everybody to know we have the same problem.
This is on a cluster that had no real problem apart from this error.
The problem started on kraken release, and is still there after luminous upgrade
All disks are bluestore
pgrepair does not work- do anything, deep scrub will start but find no real problem.
rados list-inconsistent-pg default.rgw.buckets.data
["15.720"]
rados list-inconsistent-obj 15.720 --format=json-pretty
No scrub information available for pg 15.720
error 2: (2) No such file or directory
if somebody want's we can provide extra info.
Updated by Jake Grimmett almost 6 years ago
- rados list-inconsistent-pg compressed_ecpool
["4.ff","4.2e2","4.551","4.61c","4.8cd","4.a20","4.e19"]
- rados list-inconsistent-obj 4.ff --format=json-pretty
No scrub information available for pg 4.ff
error 2: (2) No such file or directory
The following commands appear to run OK, but nothing appears in the
osd log to indicate that the commands are running. The OSD's are
otherwise working & logging OK.
- ceph pg scrub 4.e19
instructing pg 4.e19s0 on osd.246 to scrub
- ceph pg repair 4.e19
instructing pg 4.e19s0 on osd.246 to repair
- ceph osd scrub 246
instructed osd(s) 246 to scrub
- ceph osd repair 246
instructed osd(s) 246 to repair
It does not matter which osd or pg the repair is initiated on.
From the OSD logs, and 'ceph -s' I can see that the OSD's are still
doing automatic background pg scrubs, just not the ones I have asked
them to do, at the time of my request they are not currently scrubbing.
Could it be that my commands are not being sent to the OSD's?
Any idea on how to debug this?
Further info:
Output of 'ceph pg 4.e19 query' is here:
http://p.ip.fi/9x5v
Output of 'ceph daemon osd.246 config show' is here
http://p.ip.fi/RAuk
Cluster has 10 nodes, 128GB RAM, dual Xeon
450 Bluestore SATA OSD, EC 8:2
4 NVME OSD, replicated
used for cephfs (2.3PB), daily snapshots only
- ceph health detail
HEALTH_ERR 9500031/5149746146 objects misplaced (0.184%); 80 scrub
errors; Possible data damage: 7 pgs inconsistent
OBJECT_MISPLACED 9500031/5149746146 objects misplaced (0.184%)
OSD_SCRUB_ERRORS 80 scrub errors
PG_DAMAGED Possible data damage: 7 pgs inconsistent
pg 4.ff is active+clean+inconsistent, acting
[318,403,150,13,225,261,382,175,282,324]
pg 4.2e2 is active+clean+inconsistent, acting
[352,59,328,451,195,119,42,66,158,150]
pg 4.551 is active+clean+inconsistent, acting
[391,105,124,150,205,22,269,184,293,91]
pg 4.61c is active+clean+inconsistent, acting
[382,131,84,35,282,214,236,366,309,150]
pg 4.8cd is active+clean+inconsistent, acting
[353,58,5,252,187,183,323,150,387,32]
pg 4.a20 is active+clean+inconsistent, acting
[346,104,398,282,225,133,150,70,165,17]
pg 4.e19 is active+clean+inconsistent, acting
[246,447,245,98,170,348,111,155,150,295]
Updated by lin zhou over 5 years ago
just report the same error.
And I try to find the object,three replicas look the same.
first, search ceph.log, and I got:
388.c29 shard 295: soid 388:9430fef2:::c2e226a9-b855-45c5-a17f-b1c697755072.1813469.4__multipart_dumbo%2f180888654%2f20181221%2fxtrabackup_full_x19_30044_20181221025000%2fx19.xbstream.2~ntwW9vwutbmOJ4bDZYehERT2AokbtAi.3595:head candidate had a read error
then I search three osd contain this ps:
\73281ed56c92a56da078b1ae52e888e0 DIR_9/DIR_2/DIR_C/DIR_0/DIR_F/c2e226a9-b855-45c5-a17f-b1c697755072.1813469.4\\u\\umultipart\\udumbo\\s180888654\\s20181221\\sxtrabackup\\ufull\\ux19\\u30044\\u20181221025000\\sx19.xbstream.2~ntwW9vwutbmOJ4bDZYehERT2AokbtAi.3595__head_4F7F0C29__184
the md5 value is the same.
stat info is different:
root@cld-osd3-48:/home/ceph/var/lib/osd/ceph-33/current/388.c29_head# stat XXXX
Size: 4194304 Blocks: 8200 IO Block: 4096 regular file
Device: 891h/2193d Inode: 4300403471 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 999/ ceph) Gid: ( 999/ ceph)
Access: 2018-12-21 14:17:12.945132144 +0800
Modify: 2018-12-21 14:17:12.965132073 +0800
Change: 2018-12-21 14:17:13.761129235 +0800
root@cld-osd24-48:/home/ceph/var/lib/osd/ceph-279/current/388.c29_head# stat XXXX
Size: 4194304 Blocks: 8200 IO Block: 4096 regular file
Device: 831h/2097d Inode: 8646464869 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 999/ ceph) Gid: ( 999/ ceph)
Access: 2019-01-07 10:54:23.010293026 +0800
Modify: 2019-01-07 10:54:23.010293026 +0800
Change: 2019-01-07 10:54:23.014293004 +0800
root@cld-osd31-48:/home/ceph/var/lib/osd/ceph-363/current/388.c29_head# stat XXX
Size: 4194304 Blocks: 8200 IO Block: 4096 regular file
Device: 831h/2097d Inode: 13141445890 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 999/ ceph) Gid: ( 999/ ceph)
Access: 2018-12-21 14:17:12.946862160 +0800
Modify: 2018-12-21 14:17:12.966862262 +0800
Change: 2018-12-21 14:17:13.762866312 +0800
Updated by lin zhou over 5 years ago
lin zhou wrote:
just report the same error.
And I try to find the object,three replicas look the same.first, search ceph.log, and I got:
388.c29 shard 295: soid 388:9430fef2:::c2e226a9-b855-45c5-a17f-b1c697755072.1813469.4__multipart_dumbo%2f180888654%2f20181221%2fxtrabackup_full_x19_30044_20181221025000%2fx19.xbstream.2~ntwW9vwutbmOJ4bDZYehERT2AokbtAi.3595:head candidate had a read errorthen I search three osd contain this ps:
\73281ed56c92a56da078b1ae52e888e0 DIR_9/DIR_2/DIR_C/DIR_0/DIR_F/c2e226a9-b855-45c5-a17f-b1c697755072.1813469.4\\u\\umultipart\\udumbo\\s180888654\\s20181221\\sxtrabackup\\ufull\\ux19\\u30044\\u20181221025000\\sx19.xbstream.2~ntwW9vwutbmOJ4bDZYehERT2AokbtAi.3595__head_4F7F0C29__184the md5 value is the same.
stat info is different:
root@cld-osd3-48:/home/ceph/var/lib/osd/ceph-33/current/388.c29_head# stat XXXX
Size: 4194304 Blocks: 8200 IO Block: 4096 regular file
Device: 891h/2193d Inode: 4300403471 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 999/ ceph) Gid: ( 999/ ceph)
Access: 2018-12-21 14:17:12.945132144 +0800
Modify: 2018-12-21 14:17:12.965132073 +0800
Change: 2018-12-21 14:17:13.761129235 +0800root@cld-osd24-48:/home/ceph/var/lib/osd/ceph-279/current/388.c29_head# stat XXXX
Size: 4194304 Blocks: 8200 IO Block: 4096 regular file
Device: 831h/2097d Inode: 8646464869 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 999/ ceph) Gid: ( 999/ ceph)
Access: 2019-01-07 10:54:23.010293026 +0800
Modify: 2019-01-07 10:54:23.010293026 +0800
Change: 2019-01-07 10:54:23.014293004 +0800root@cld-osd31-48:/home/ceph/var/lib/osd/ceph-363/current/388.c29_head# stat XXX
Size: 4194304 Blocks: 8200 IO Block: 4096 regular file
Device: 831h/2097d Inode: 13141445890 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 999/ ceph) Gid: ( 999/ ceph)
Access: 2018-12-21 14:17:12.946862160 +0800
Modify: 2018-12-21 14:17:12.966862262 +0800
Change: 2018-12-21 14:17:13.762866312 +0800
my ceph version is 10.2.11
Updated by Niklas Hambuechen 10 months ago
Updated by can zhu 10 months ago
I alose hit it with ceph 16.2.10
root@node02:/# ceph version
ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable)
root@node02:/# ceph health detail
HEALTH_ERR 1 OSDs or CRUSH {nodes, device-classes} have {NOUP,NODOWN,NOIN,NOOUT} flags set; 6 scrub errors; Possible data damage: 1 pg inconsistent; Degraded data redundancy: 354469/570197792 objects degraded (0.062%), 5 pgs degraded, 6 pgs undersized
[WRN] OSD_FLAGS: 1 OSDs or CRUSH {nodes, device-classes} have {NOUP,NODOWN,NOIN,NOOUT} flags set
host node05 has flags noout
[ERR] OSD_SCRUB_ERRORS: 6 scrub errors
[ERR] PG_DAMAGED: Possible data damage: 1 pg inconsistent
pg 36.2f4 is active+clean+inconsistent, acting [104,99,1]
[WRN] PG_DEGRADED: Degraded data redundancy: 354469/570197792 objects degraded (0.062%), 5 pgs degraded, 6 pgs undersized
pg 34.8 is stuck undersized for 23h, current state active+undersized+degraded+remapped+backfill_wait, last acting [2147483647,12,65,18,17,48,81,79]
pg 34.10 is stuck undersized for 23h, current state active+undersized+degraded+remapped+backfill_wait, last acting [102,29,17,6,63,12,92,2147483647]
pg 34.c6 is stuck undersized for 23h, current state active+undersized+remapped+backfilling, last acting [74,97,103,2147483647,63,29,46,12]
pg 34.1a6 is stuck undersized for 23h, current state active+undersized+degraded+remapped+backfilling, last acting [74,2147483647,67,98,37,99,95,63]
pg 34.3cb is stuck undersized for 23h, current state active+undersized+degraded+remapped+backfill_wait, last acting [92,95,94,2147483647,103,79,68,15]
pg 36.3f is stuck undersized for 23h, current state active+undersized+degraded+remapped+backfilling, last acting [102,67]
root@node02:/# rados list-inconsistent-obj 36.2f4 --forma=json-pretty
No scrub information available for pg 36.2f4
error 2: (2) No such file or directory