Project

General

Profile

Actions
Copy link

Bug #54966

open

osd/ec: some pg status stuck at active+recovery_unfound+degraded+remapped

Added by jianwei zhang about 2 years ago. Updated almost 2 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
OSD
Target version:
-
% Done:

0%

Source:
Community (user)
Tags:
v15.2.13
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
v15.2.8
ceph-qa-suite:
rados
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

reproduce:
1. 5 hosts
2. cephfs (replica pool + ec pool (allow_ec_overwrite))
3. 3 hosts create ec(2+1) data pool and 3 replicas meta pool for base_cluster
4. running vdbench test file case
5. expand remain 2 hosts to cluster, data pool and meta pool, tips: osd one by one join up&in to cluster
6. all osds is up && in
7. backfill and recovery over

result:
ceph -s
11 active+recovery_unfound+degraded+remapped

analyse:
I found a reason, and I need everyone to discuss it together

Through the ceph log、 pg and object information on the disk,
The 3 ec shard data of the desired eversion of ec(2+1) are storaged on the disks, and are not lost.
The problem lies in the missing object after the peering is completed. In the process of adding the source osd required for recovery, it will be skipped the source osd that because the named serial number of the object in the pg is greater than the last_backfill(MIN), and it is not added to the source osd set, source osds not enough ec(2) for recover, resulting in recovery_unfound.

problem code:

MissingLoc::add_source_info {
for ()
...
if (p->first >= oinfo.last_backfill) {
// FIXME: this is probably true, although it could conceivably
// be in the undefined region! Hmm!
ldout(cct, 10) << "search_for_missing " << soid << " " << need << " also missing on osd." << fromosd
<< " (past last_backfill " << oinfo.last_backfill << ")" << dendl;
continue;
}
...
}

commit b99e135848ca5666308344cf5ecc9c7f95f30137
Author: Sage Weil <>
Date: Mon Dec 5 17:25:21 2011 -0800

osd: make backfill (basically) work again
Still need to handle concurrent updates, log recovery vs backfill, etc.
Signed-off-by: Sage Weil &lt;&gt;

evidence :
(Take an recovery_unfound object as an example) : 101000000000133.0000001d

  1. ceph pg 3.2f2 list_unfound {
    "oid": {
    "oid": "101000000000133.0000001d",
    "key": "",
    "snapid": -2,
    "hash": 1021279986,
    "max": 0,
    "pool": 3,
    "namespace": ""
    },
    "need": "374'872",
    "have": "0'0",
    "flags": "none",
    "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1",
    "locations": [
    "36(0)"
    ]
    }
  1. /etc/ceph/ceph.conf debug_osd = 30/30
  2. filter add source osd for missing object after peering
    [root@node124 ceph]# grep -e Primary -e 101000000000133.0000001d ceph-osd.57.log|grep -E "3\.2f2.*transitioning to Primary|search_for_missing.*101000000000133.0000001d"
    2022-03-19T14:04:42.223+0800 7f82f2028700 1 osd.57 pg_epoch: 5030 pg[3.2f2s0( v 785'2826 lc 0'0 (503'2242,785'2826] local-lis/les=5003/5004 n=230 ec=198/198 lis/c=4985/282 les/c/f=4986/283/0 sis=5030) [76,59,41]/[57,41,37]p57(0) r=0 lpr=5030 pi=[282,5030)/9 crt=785'2826 mlcod 0'0 remapped m=18 mbc={}] state<Start>: transitioning to Primary
    2022-03-19T14:04:43.303+0800 7f82f2028700 10 osd.57 pg_epoch: 5031 pg[3.2f2s0( v 785'2826 lc 0'0 (503'2242,785'2826] local-lis/les=5030/5031 n=230 ec=198/198 lis/c=4985/282 les/c/f=4986/283/0 sis=5030) [76,59,41]/[57,41,37]p57(0) backfill=[41(2),59(1),76(0)] r=0 lpr=5030 pi=[282,5030)/9 crt=785'2826 mlcod 0'0 remapped m=18 u=18 mbc={}] search_for_missing 3:4f7efb3c:::101000000000133.0000001d:head 374'872 also missing on osd.57(0)
    2022-03-19T14:04:43.303+0800 7f82f2028700 10 osd.57 pg_epoch: 5031 pg[3.2f2s0( v 785'2826 lc 0'0 (503'2242,785'2826] local-lis/les=5030/5031 n=230 ec=198/198 lis/c=4985/282 les/c/f=4986/283/0 sis=5030) [76,59,41]/[57,41,37]p57(0) backfill=[41(2),59(1),76(0)] r=0 lpr=5030 pi=[282,5030)/9 crt=785'2826 mlcod 0'0 remapped m=18 u=18 mbc={}] search_for_missing 3:4f7efb3c:::101000000000133.0000001d:head 374'872 also missing on osd.37(2)
    2022-03-19T14:04:43.304+0800 7f82f2028700 10 osd.57 pg_epoch: 5031 pg[3.2f2s0( v 785'2826 lc 0'0 (503'2242,785'2826] local-lis/les=5030/5031 n=230 ec=198/198 lis/c=4985/282 les/c/f=4986/283/0 sis=5030) [76,59,41]/[57,41,37]p57(0) backfill=[41(2),59(1),76(0)] r=0 lpr=5030 pi=[282,5030)/9 crt=785'2826 mlcod 0'0 remapped m=18 u=18 mbc={}] search_for_missing 3:4f7efb3c:::101000000000133.0000001d:head 374'872 also missing on osd.41(1)
    2022-03-19T14:04:43.305+0800 7f82f2028700 10 osd.57 pg_epoch: 5031 pg[3.2f2s0( v 785'2826 lc 0'0 (503'2242,785'2826] local-lis/les=5030/5031 n=230 ec=198/198 lis/c=4985/282 les/c/f=4986/283/0 sis=5030) [76,59,41]/[57,41,37]p57(0) backfill=[41(2),59(1),76(0)] r=0 lpr=5030 pi=[282,5030)/9 crt=785'2826 mlcod 0'0 remapped m=18 u=18 mbc={}] search_for_missing 3:4f7efb3c:::101000000000133.0000001d:head 374'872 also missing on osd.41(2) (past last_backfill MIN) ///3.2f2s2.101000000000133.0000001d.osd41.object_info_t(on disk object_info) //However, because the serial number of the object is greater than the last_backfill of osd.41(2), it was skipped and not added to the source osd

2022-03-19T14:04:43.306+0800 7f82f2028700 10 osd.57 pg_epoch: 5031 pg[3.2f2s0( v 785'2826 lc 0'0 (503'2242,785'2826] local-lis/les=5030/5031 n=230 ec=198/198 lis/c=4985/282 les/c/f=4986/283/0 sis=5030) [76,59,41]/[57,41,37]p57(0) backfill=[41(2),59(1),76(0)] r=0 lpr=5030 pi=[282,5030)/9 crt=785'2826 mlcod 0'0 remapped m=18 u=18 mbc={}] search_for_missing 3:4f7efb3c:::101000000000133.0000001d:head 374'872 also missing on osd.59(1) (past last_backfill MIN) ///3.2f2s1.101000000000133.0000001d.osd59.object_info_t(on disk object_info) ///However, because the serial number of the object is greater than the last_backfill of osd.59(1), it was skipped and not added to the source osd

2022-03-19T14:04:43.307+0800 7f82f2028700 10 osd.57 pg_epoch: 5031 pg[3.2f2s0( v 785'2826 lc 0'0 (503'2242,785'2826] local-lis/les=5030/5031 n=230 ec=198/198 lis/c=4985/282 les/c/f=4986/283/0 sis=5030) [76,59,41]/[57,41,37]p57(0) backfill=[41(2),59(1),76(0)] r=0 lpr=5030 pi=[282,5030)/9 crt=785'2826 mlcod 0'0 remapped m=18 u=18 mbc={}] search_for_missing 3:4f7efb3c:::101000000000133.0000001d:head 374'872 also missing on osd.76(0) (past last_backfill MIN)
2022-03-19T14:04:43.344+0800 7f82f2028700 10 osd.57 pg_epoch: 5031 pg[3.2f2s0( v 785'2826 lc 0'0 (503'2242,785'2826] local-lis/les=5030/5031 n=230 ec=198/198 lis/c=4985/282 les/c/f=4986/283/0 sis=5030) [76,59,41]/[57,41,37]p57(0) backfill=[41(2),59(1),76(0)] r=0 lpr=5030 pi=[282,5030)/9 crt=785'2826 mlcod 0'0 activating+degraded+remapped m=18 u=18 mbc={}] search_for_missing 3:4f7efb3c:::101000000000133.0000001d:head 374'872 also missing on osd.44(1) (last_update 346'424 < needed 374'872)
2022-03-19T14:04:43.348+0800 7f82f2028700 10 osd.57 pg_epoch: 5031 pg[3.2f2s0( v 785'2826 lc 0'0 (503'2242,785'2826] local-lis/les=5030/5031 n=230 ec=198/198 lis/c=4985/282 les/c/f=4986/283/0 sis=5030) [76,59,41]/[57,41,37]p57(0) backfill=[41(2),59(1),76(0)] r=0 lpr=5030 pi=[282,5030)/9 crt=785'2826 mlcod 0'0 activating+degraded+remapped m=18 u=18 mbc={0={(0+1)=17},1={(0+0)=17},2={(0+0)=17}}] search_for_missing 3:4f7efb3c:::101000000000133.0000001d:head 374'872 is on osd.36(0) ///3.2f2s0.101000000000133.0000001d.osd36.object_info_t(on disk object_info) //Only osd.36(0) meets all the conditions and is added to the source osd

2022-03-19T14:04:43.362+0800 7f82f2028700 10 osd.57 pg_epoch: 5031 pg[3.2f2s0( v 785'2826 lc 0'0 (503'2242,785'2826] local-lis/les=5030/5031 n=230 ec=198/198 lis/c=4985/282 les/c/f=4986/283/0 sis=5030) [76,59,41]/[57,41,37]p57(0) backfill=[41(2),59(1),76(0)] r=0 lpr=5030 pi=[282,5030)/9 crt=785'2826 mlcod 0'0 activating+degraded+remapped m=18 u=18 mbc={0={(0+1)=18},1={(0+0)=18},2={(0+0)=18}}] search_for_missing 3:4f7efb3c:::101000000000133.0000001d:head 374'872 also missing on osd.57(1) (last_update 362'664 < needed 374'872)
2022-03-19T14:04:43.365+0800 7f82ef823700 10 osd.57 pg_epoch: 5031 pg[3.2f2s0( v 785'2826 lc 0'0 (503'2242,785'2826] local-lis/les=5030/5031 n=230 ec=198/198 lis/c=4985/282 les/c/f=4986/283/0 sis=5030) [76,59,41]/[57,41,37]p57(0) backfill=[41(2),59(1),76(0)] r=0 lpr=5030 pi=[282,5030)/9 crt=785'2826 mlcod 0'0 activating+degraded+remapped m=18 u=18 mbc={0={(0+1)=18},1={(0+0)=18},2={(0+0)=18}}] search_for_missing 3:4f7efb3c:::101000000000133.0000001d:head 374'872 also missing on osd.75(0) (past last_backfill MIN)
2022-03-19T14:04:43.378+0800 7f82f2028700 10 osd.57 pg_epoch: 5031 pg[3.2f2s0( v 785'2826 lc 0'0 (503'2242,785'2826] local-lis/les=5030/5031 n=230 ec=198/198 lis/c=4985/282 les/c/f=4986/283/0 sis=5030) [76,59,41]/[57,41,37]p57(0) backfill=[41(2),59(1),76(0)] r=0 lpr=5030 pi=[282,5030)/9 crt=785'2826 mlcod 0'0 activating+degraded+remapped m=18 u=18 mbc={0={(0+1)=18},1={(0+0)=18},2={(0+0)=18}}] search_for_missing 3:4f7efb3c:::101000000000133.0000001d:head 374'872 also missing on osd.39(2)
2022-03-19T14:04:43.384+0800 7f82f482d700 10 osd.57 pg_epoch: 5031 pg[3.2f2s0( v 785'2826 lc 0'0 (503'2242,785'2826] local-lis/les=5030/5031 n=230 ec=198/198 lis/c=4985/282 les/c/f=4986/283/0 sis=5030) [76,59,41]/[57,41,37]p57(0) backfill=[41(2),59(1),76(0)] r=0 lpr=5030 pi=[282,5030)/9 crt=785'2826 mlcod 0'0 activating+degraded+remapped m=18 u=18 mbc={0={(0+1)=18},1={(0+0)=18},2={(0+0)=18}}] search_for_missing 3:4f7efb3c:::101000000000133.0000001d:head 374'872 also missing on osd.48(1) (last_update 351'468 < needed 374'872)
2022-03-19T14:04:43.387+0800 7f82f2028700 10 osd.57 pg_epoch: 5031 pg[3.2f2s0( v 785'2826 lc 0'0 (503'2242,785'2826] local-lis/les=5030/5031 n=230 ec=198/198 lis/c=4985/282 les/c/f=4986/283/0 sis=5030) [76,59,41]/[57,41,37]p57(0) backfill=[41(2),59(1),76(0)] r=0 lpr=5030 pi=[282,5030)/9 crt=785'2826 mlcod 0'0 activating+degraded+remapped m=18 u=18 mbc={0={(0+1)=18},1={(0+0)=18},2={(0+0)=18}}] search_for_missing 3:4f7efb3c:::101000000000133.0000001d:head 374'872 also missing on osd.73(0) (past last_backfill MIN)

ceph-object-tool: (object_info_t and check bluestore) {
"oid": {
"oid": "101000000000133.0000001d",
"key": "",
"snapid": -2,
"hash": 1021279986,
"max": 0,
"pool": 3,
"namespace": ""
},
"version": "374'872",
"prior_version": "374'871",
"last_reqid": "client.133562.0:359556",
"user_version": 872,
"size": 4194304,
"mtime": "2022-03-18T11:41:39.433611+0800",
"local_mtime": "2022-03-18T11:41:39.436761+0800",
"lost": 0,
"flags": [
"dirty"
],
"truncate_seq": 0,
"truncate_size": 0,
"data_digest": "0xffffffff",
"omap_digest": "0xffffffff",
"expected_object_size": 0,
"expected_write_size": 0,
"alloc_hint_flags": 0,
"manifest": {
"type": 0
},
"watchers": {}
}

3.2f2]# for i in $(ls -tlhr|grep 101000000000133.0000001d|grep -v -e '0 Mar 19'|grep -e osd|awk '{print $NF}'); do echo $i; ceph-dencoder import $i type object_info_t decode dump_json;done
3.2f2s0.101000000000133.0000001d.osd36.object_info_t

3.2f2s2.101000000000133.0000001d.osd41.object_info_t {
"oid": {
"oid": "101000000000133.0000001d",
"key": "",
"snapid": -2,
"hash": 1021279986,
"max": 0,
"pool": 3,
"namespace": ""
},
"version": "374'872",
"prior_version": "374'871",
"last_reqid": "client.133562.0:359556",
"user_version": 872,
"size": 4194304,
"mtime": "2022-03-18T11:41:39.433611+0800",
"local_mtime": "2022-03-18T11:41:39.436761+0800",
"lost": 0,
"flags": [
"dirty"
],
"truncate_seq": 0,
"truncate_size": 0,
"data_digest": "0xffffffff",
"omap_digest": "0xffffffff",
"expected_object_size": 0,
"expected_write_size": 0,
"alloc_hint_flags": 0,
"manifest": {
"type": 0
},
"watchers": {}
}

3.2f2s1.101000000000133.0000001d.osd59.object_info_t {
"oid": {
"oid": "101000000000133.0000001d",
"key": "",
"snapid": -2,
"hash": 1021279986,
"max": 0,
"pool": 3,
"namespace": ""
},
"version": "374'872",
"prior_version": "374'871",
"last_reqid": "client.133562.0:359556",
"user_version": 872,
"size": 4194304,
"mtime": "2022-03-18T11:41:39.433611+0800",
"local_mtime": "2022-03-18T11:41:39.436761+0800",
"lost": 0,
"flags": [
"dirty"
],
"truncate_seq": 0,
"truncate_size": 0,
"data_digest": "0xffffffff",
"omap_digest": "0xffffffff",
"expected_object_size": 0,
"expected_write_size": 0,
"alloc_hint_flags": 0,
"manifest": {
"type": 0
},
"watchers": {}
}

node121 ceph-objectstore-tool --op fuse --data-path /var/lib/ceph/osd/ceph-36/ --mountpoint /mnt/fuse-osd36 &
node122 ceph-objectstore-tool --op fuse --data-path /var/lib/ceph/osd/ceph-41/ --mountpoint /mnt/fuse-osd41 &
node124 ceph-objectstore-tool --op fuse --data-path /var/lib/ceph/osd/ceph-59/ --mountpoint /mnt/fuse-osd59 &

[root@node121 0#3:4f7efb3c:::101000000000133.0000001d:head#]# md5sum /mnt/fuse-osd36/3.2f2s0_head/all/0#3:4f7efb3c:::101000000000133.0000001d:head#/data
ea003c1e3b13d63d6ed67d0595e8ff1b /mnt/fuse-osd36/3.2f2s0_head/all/0#3:4f7efb3c:::101000000000133.0000001d:head#/data

[root@node122 2#3:4f7efb3c:::101000000000133.0000001d:head#]# md5sum /mnt/fuse-osd41/3.2f2s2_head/all/2#3:4f7efb3c:::101000000000133.0000001d:head#/data
52ed9a463dd46982113bf748e7f0c4fe /mnt/fuse-osd41/3.2f2s2_head/all/2#3:4f7efb3c:::101000000000133.0000001d:head#/data

[root@node124 1#3:4f7efb3c:::101000000000133.0000001d:head#]# md5sum /mnt/fuse-osd59/3.2f2s1_head/all/1#3:4f7efb3c:::101000000000133.0000001d:head#/data
f339660aa3a182d89be8110d67024a04 /mnt/fuse-osd59/3.2f2s1_head/all/1#3:4f7efb3c:::101000000000133.0000001d:head#/data

ceph-object-tool: dump pg info and pglog, then parse

[root@node121 3.2f2]# python3 parse_data.py 101000000000133.0000001d *.log
['3.2f2s0.osd36.log', '3.2f2s0.osd57.log', '3.2f2s0.osd73.log', '3.2f2s0.osd75.log', '3.2f2s0.osd76.log', '3.2f2s1.osd41.log', '3.2f2s1.osd44.log', '3.2f2s1.osd48.log', '3.2f2s1.osd57.log', '3.2f2s1.osd59.log', '3.2f2s2.osd37.log', '3.2f2s2.osd39.log', '3.2f2s2.osd41.log']
========================missing object===========================
3.2f2s0.osd36.log
3.2f2s0.osd57.log {'object': '3:4f7efb3c:::101000000000133.0000001d:head', 'need': "374'872", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1'}
3.2f2s0.osd73.log
3.2f2s0.osd75.log
3.2f2s0.osd76.log
3.2f2s1.osd41.log {'object': '3:4f7efb3c:::101000000000133.0000001d:head', 'need': "374'872", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1'}
3.2f2s1.osd44.log
3.2f2s1.osd48.log
3.2f2s1.osd57.log
3.2f2s1.osd59.log
3.2f2s2.osd37.log {'object': '3:4f7efb3c:::101000000000133.0000001d:head', 'need': "374'872", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1'}
3.2f2s2.osd39.log {'object': '3:4f7efb3c:::101000000000133.0000001d:head', 'need': "374'872", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1'}
3.2f2s2.osd41.log
========================pg log object==========================
3.2f2s0.osd36.log [0'0, 375'884] {'op': 'modify', 'object': '3:4f7efb3c:::101000000000133.0000001d:head', 'version': "374'833", 'prior_version': "0'0", 'reqid': 'client.133721.0:366904', 'extra_reqids': [], 'mtime': '2022-03-18T11:41:03.843215+0800', 'return_code': 0, 'mod_desc': {'object_mod_desc': {'can_local_rollback': True, 'rollback_info_completed': True, 'ops': [{'code': 'CREATE'}]}}, 'clean_regions': {'object_clean_regions': {'clean_offsets': '[1048576~18446744073708503039]', 'clean_omap': True, 'new_object': False}}} {'op': 'modify', 'object': '3:4f7efb3c:::101000000000133.0000001d:head', 'version': "374'834", 'prior_version': "374'833", 'reqid': 'client.133721.0:366972', 'extra_reqids': [], 'mtime': '2022-03-18T11:41:03.967153+0800', 'return_code': 0, 'mod_desc': {'object_mod_desc': {'can_local_rollback': True, 'rollback_info_completed': False, 'ops': [{'code': 'SETATTRS', 'attrs': ['_', 'hinfo_key', 'snapset']}, {'code': 'APPEND', 'old_size': 1048576}]}}, 'clean_regions': {'object_clean_regions': {'clean_offsets': '[0~1048576,2097152~18446744073707454463]', 'clean_omap': True, 'new_object': False}}} {'op': 'modify', 'object': '3:4f7efb3c:::101000000000133.0000001d:head', 'version': "374'835", 'prior_version': "374'834", 'reqid': 'client.133721.0:367041', 'extra_reqids': [], 'mtime': '2022-03-18T11:41:04.090498+0800', 'return_code': 0, 'mod_desc': {'object_mod_desc': {'can_local_rollback': True, 'rollback_info_completed': False, 'ops': [{'code': 'SETATTRS', 'attrs': ['_', 'hinfo_key', 'snapset']}, {'code': 'APPEND', 'old_size': 2097152}]}}, 'clean_regions': {'object_clean_regions': {'clean_offsets': '[0~2097152,3145728~18446744073706405887]', 'clean_omap': True, 'new_object': False}}} {'op': 'modify', 'object': '3:4f7efb3c:::101000000000133.0000001d:head', 'version': "374'836", 'prior_version': "374'835", 'reqid': 'client.133721.0:367114', 'extra_reqids': [], 'mtime': '2022-03-18T11:41:04.210199+0800', 'return_code': 0, 'mod_desc': {'object_mod_desc': {'can_local_rollback': True, 'rollback_info_completed': False, 'ops': [{'code': 'SETATTRS', 'attrs': ['_', 'hinfo_key', 'snapset']}, {'code': 'APPEND', 'old_size': 3145728}]}}, 'clean_regions': {'object_clean_regions': {'clean_offsets': '[0~3145728,4194304~18446744073705357311]', 'clean_omap': True, 'new_object': False}}} {'op': 'modify', 'object': '3:4f7efb3c:::101000000000133.0000001d:head', 'version': "374'853", 'prior_version': "374'836", 'reqid': 'client.114246.0:360904', 'extra_reqids': [], 'mtime': '2022-03-18T11:41:25.208243+0800', 'return_code': 0, 'mod_desc': {'object_mod_desc': {'can_local_rollback': True, 'rollback_info_completed': False, 'ops': [{'code': 'SETATTRS', 'attrs': ['_', 'hinfo_key', 'snapset']}, {'code': 'ROLLBACK_EXTENTS', 'gen': 853, 'snaps': '[0,524288]'}]}}, 'clean_regions': {'object_clean_regions': {'clean_offsets': '[1048576~18446744073708503039]', 'clean_omap': True, 'new_object': False}}} {'op': 'modify', 'object': '3:4f7efb3c:::101000000000133.0000001d:head', 'version': "374'854", 'prior_version': "374'853", 'reqid': 'client.114246.0:360926', 'extra_reqids': [], 'mtime': '2022-03-18T11:41:25.256403+0800', 'return_code': 0, 'mod_desc': {'object_mod_desc': {'can_local_rollback': True, 'rollback_info_completed': False, 'ops': [{'code': 'SETATTRS', 'attrs': ['_', 'hinfo_key', 'snapset']}, {'code': 'ROLLBACK_EXTENTS', 'gen': 854, 'snaps': '[524288,524288]'}]}}, 'clean_regions': {'object_clean_regions': {'clean_offsets': '[0~1048576,2097152~18446744073707454463]', 'clean_omap': True, 'new_object': False}}} {'op': 'modify', 'object': '3:4f7efb3c:::101000000000133.0000001d:head', 'version': "374'855", 'prior_version': "374'854", 'reqid': 'client.114246.0:360958', 'extra_reqids': [], 'mtime': '2022-03-18T11:41:25.295928+0800', 'return_code': 0, 'mod_desc': {'object_mod_desc': {'can_local_rollback': True, 'rollback_info_completed': False, 'ops': [{'code': 'SETATTRS', 'attrs': ['_', 'hinfo_key', 'snapset']}, {'code': 'ROLLBACK_EXTENTS', 'gen': 855, 'snaps': '[1048576,524288]'}]}}, 'clean_regions': {'object_clean_regions': {'clean_offsets': '[0~2097152,3145728~18446744073706405887]', 'clean_omap': True, 'new_object': False}}} {'op': 'modify', 'object': '3:4f7efb3c:::101000000000133.0000001d:head', 'version': "374'856", 'prior_version': "374'855", 'reqid': 'client.114246.0:360973', 'extra_reqids': [], 'mtime': '2022-03-18T11:41:25.323967+0800', 'return_code': 0, 'mod_desc': {'object_mod_desc': {'can_local_rollback': True, 'rollback_info_completed': False, 'ops': [{'code': 'SETATTRS', 'attrs': ['_', 'hinfo_key', 'snapset']}, {'code': 'ROLLBACK_EXTENTS', 'gen': 856, 'snaps': '[1572864,524288]'}]}}, 'clean_regions': {'object_clean_regions': {'clean_offsets': '[0~3145728,4194304~18446744073705357311]', 'clean_omap': True, 'new_object': False}}} {'op': 'modify', 'object': '3:4f7efb3c:::101000000000133.0000001d:head', 'version': "374'869", 'prior_version': "374'856", 'reqid': 'client.133562.0:359426', 'extra_reqids': [], 'mtime': '2022-03-18T11:41:39.277007+0800', 'return_code': 0, 'mod_desc': {'object_mod_desc': {'can_local_rollback': True, 'rollback_info_completed': False, 'ops': [{'code': 'SETATTRS', 'attrs': ['_', 'hinfo_key', 'snapset']}, {'code': 'ROLLBACK_EXTENTS', 'gen': 869, 'snaps': '[0,524288]'}]}}, 'clean_regions': {'object_clean_regions': {'clean_offsets': '[1048576~18446744073708503039]', 'clean_omap': True, 'new_object': False}}} {'op': 'modify', 'object': '3:4f7efb3c:::101000000000133.0000001d:head', 'version': "374'870", 'prior_version': "374'869", 'reqid': 'client.133562.0:359461', 'extra_reqids': [], 'mtime': '2022-03-18T11:41:39.334504+0800', 'return_code': 0, 'mod_desc': {'object_mod_desc': {'can_local_rollback': True, 'rollback_info_completed': False, 'ops': [{'code': 'SETATTRS', 'attrs': ['_', 'hinfo_key', 'snapset']}, {'code': 'ROLLBACK_EXTENTS', 'gen': 870, 'snaps': '[524288,524288]'}]}}, 'clean_regions': {'object_clean_regions': {'clean_offsets': '[0~1048576,2097152~18446744073707454463]', 'clean_omap': True, 'new_object': False}}} {'op': 'modify', 'object': '3:4f7efb3c:::101000000000133.0000001d:head', 'version': "374'871", 'prior_version': "374'870", 'reqid': 'client.133562.0:359504', 'extra_reqids': [], 'mtime': '2022-03-18T11:41:39.385649+0800', 'return_code': 0, 'mod_desc': {'object_mod_desc': {'can_local_rollback': True, 'rollback_info_completed': False, 'ops': [{'code': 'SETATTRS', 'attrs': ['_', 'hinfo_key', 'snapset']}, {'code': 'ROLLBACK_EXTENTS', 'gen': 871, 'snaps': '[1048576,524288]'}]}}, 'clean_regions': {'object_clean_regions': {'clean_offsets': '[0~2097152,3145728~18446744073706405887]', 'clean_omap': True, 'new_object': False}}} {'op': 'modify', 'object': '3:4f7efb3c:::101000000000133.0000001d:head', 'version': "374'872", 'prior_version': "374'871", 'reqid': 'client.133562.0:359556', 'extra_reqids': [], 'mtime': '2022-03-18T11:41:39.433611+0800', 'return_code': 0, 'mod_desc': {'object_mod_desc': {'can_local_rollback': True, 'rollback_info_completed': False, 'ops': [{'code': 'SETATTRS', 'attrs': ['_', 'hinfo_key', 'snapset']}, {'code': 'ROLLBACK_EXTENTS', 'gen': 872, 'snaps': '[1572864,524288]'}]}}, 'clean_regions': {'object_clean_regions': {'clean_offsets': '[0~3145728,4194304~18446744073705357311]', 'clean_omap': True, 'new_object': False}}}
3.2f2s0.osd57.log [503'2242, 785'2826]
3.2f2s0.osd73.log [466'1540, 488'2076]
3.2f2s0.osd75.log [466'1540, 491'2080]
3.2f2s0.osd76.log [503'2242, 785'2826]
3.2f2s1.osd41.log [503'2242, 785'2826]
3.2f2s1.osd44.log [0'0, 346'424]
3.2f2s1.osd48.log [0'0, 351'468]
3.2f2s1.osd57.log [0'0, 362'664]
3.2f2s1.osd59.log [503'2242, 785'2826]
3.2f2s2.osd37.log [503'2242, 785'2826]
3.2f2s2.osd39.log [291'200, 386'1120] {'op': 'modify', 'object': '3:4f7efb3c:::101000000000133.0000001d:head', 'version': "374'833", 'prior_version': "0'0", 'reqid': 'client.133721.0:366904', 'extra_reqids': [], 'mtime': '2022-03-18T11:41:03.843215+0800', 'return_code': 0, 'mod_desc': {'object_mod_desc': {'can_local_rollback': True, 'rollback_info_completed': True, 'ops': [{'code': 'CREATE'}]}}, 'clean_regions': {'object_clean_regions': {'clean_offsets': '[1048576~18446744073708503039]', 'clean_omap': True, 'new_object': False}}} {'op': 'modify', 'object': '3:4f7efb3c:::101000000000133.0000001d:head', 'version': "374'834", 'prior_version': "374'833", 'reqid': 'client.133721.0:366972', 'extra_reqids': [], 'mtime': '2022-03-18T11:41:03.967153+0800', 'return_code': 0, 'mod_desc': {'object_mod_desc': {'can_local_rollback': True, 'rollback_info_completed': False, 'ops': [{'code': 'SETATTRS', 'attrs': ['_', 'hinfo_key', 'snapset']}, {'code': 'APPEND', 'old_size': 1048576}]}}, 'clean_regions': {'object_clean_regions': {'clean_offsets': '[0~1048576,2097152~18446744073707454463]', 'clean_omap': True, 'new_object': False}}} {'op': 'modify', 'object': '3:4f7efb3c:::101000000000133.0000001d:head', 'version': "374'835", 'prior_version': "374'834", 'reqid': 'client.133721.0:367041', 'extra_reqids': [], 'mtime': '2022-03-18T11:41:04.090498+0800', 'return_code': 0, 'mod_desc': {'object_mod_desc': {'can_local_rollback': True, 'rollback_info_completed': False, 'ops': [{'code': 'SETATTRS', 'attrs': ['_', 'hinfo_key', 'snapset']}, {'code': 'APPEND', 'old_size': 2097152}]}}, 'clean_regions': {'object_clean_regions': {'clean_offsets': '[0~2097152,3145728~18446744073706405887]', 'clean_omap': True, 'new_object': False}}} {'op': 'modify', 'object': '3:4f7efb3c:::101000000000133.0000001d:head', 'version': "374'836", 'prior_version': "374'835", 'reqid': 'client.133721.0:367114', 'extra_reqids': [], 'mtime': '2022-03-18T11:41:04.210199+0800', 'return_code': 0, 'mod_desc': {'object_mod_desc': {'can_local_rollback': True, 'rollback_info_completed': False, 'ops': [{'code': 'SETATTRS', 'attrs': ['_', 'hinfo_key', 'snapset']}, {'code': 'APPEND', 'old_size': 3145728}]}}, 'clean_regions': {'object_clean_regions': {'clean_offsets': '[0~3145728,4194304~18446744073705357311]', 'clean_omap': True, 'new_object': False}}} {'op': 'modify', 'object': '3:4f7efb3c:::101000000000133.0000001d:head', 'version': "374'853", 'prior_version': "374'836", 'reqid': 'client.114246.0:360904', 'extra_reqids': [], 'mtime': '2022-03-18T11:41:25.208243+0800', 'return_code': 0, 'mod_desc': {'object_mod_desc': {'can_local_rollback': True, 'rollback_info_completed': False, 'ops': [{'code': 'SETATTRS', 'attrs': ['_', 'hinfo_key', 'snapset']}, {'code': 'ROLLBACK_EXTENTS', 'gen': 853, 'snaps': '[0,524288]'}]}}, 'clean_regions': {'object_clean_regions': {'clean_offsets': '[1048576~18446744073708503039]', 'clean_omap': True, 'new_object': False}}} {'op': 'modify', 'object': '3:4f7efb3c:::101000000000133.0000001d:head', 'version': "374'854", 'prior_version': "374'853", 'reqid': 'client.114246.0:360926', 'extra_reqids': [], 'mtime': '2022-03-18T11:41:25.256403+0800', 'return_code': 0, 'mod_desc': {'object_mod_desc': {'can_local_rollback': True, 'rollback_info_completed': False, 'ops': [{'code': 'SETATTRS', 'attrs': ['_', 'hinfo_key', 'snapset']}, {'code': 'ROLLBACK_EXTENTS', 'gen': 854, 'snaps': '[524288,524288]'}]}}, 'clean_regions': {'object_clean_regions': {'clean_offsets': '[0~1048576,2097152~18446744073707454463]', 'clean_omap': True, 'new_object': False}}} {'op': 'modify', 'object': '3:4f7efb3c:::101000000000133.0000001d:head', 'version': "374'855", 'prior_version': "374'854", 'reqid': 'client.114246.0:360958', 'extra_reqids': [], 'mtime': '2022-03-18T11:41:25.295928+0800', 'return_code': 0, 'mod_desc': {'object_mod_desc': {'can_local_rollback': True, 'rollback_info_completed': False, 'ops': [{'code': 'SETATTRS', 'attrs': ['_', 'hinfo_key', 'snapset']}, {'code': 'ROLLBACK_EXTENTS', 'gen': 855, 'snaps': '[1048576,524288]'}]}}, 'clean_regions': {'object_clean_regions': {'clean_offsets': '[0~2097152,3145728~18446744073706405887]', 'clean_omap': True, 'new_object': False}}} {'op': 'modify', 'object': '3:4f7efb3c:::101000000000133.0000001d:head', 'version': "374'856", 'prior_version': "374'855", 'reqid': 'client.114246.0:360973', 'extra_reqids': [], 'mtime': '2022-03-18T11:41:25.323967+0800', 'return_code': 0, 'mod_desc': {'object_mod_desc': {'can_local_rollback': True, 'rollback_info_completed': False, 'ops': [{'code': 'SETATTRS', 'attrs': ['_', 'hinfo_key', 'snapset']}, {'code': 'ROLLBACK_EXTENTS', 'gen': 856, 'snaps': '[1572864,524288]'}]}}, 'clean_regions': {'object_clean_regions': {'clean_offsets': '[0~3145728,4194304~18446744073705357311]', 'clean_omap': True, 'new_object': False}}} {'op': 'modify', 'object': '3:4f7efb3c:::101000000000133.0000001d:head', 'version': "374'869", 'prior_version': "374'856", 'reqid': 'client.133562.0:359426', 'extra_reqids': [], 'mtime': '2022-03-18T11:41:39.277007+0800', 'return_code': 0, 'mod_desc': {'object_mod_desc': {'can_local_rollback': True, 'rollback_info_completed': False, 'ops': [{'code': 'SETATTRS', 'attrs': ['_', 'hinfo_key', 'snapset']}, {'code': 'ROLLBACK_EXTENTS', 'gen': 869, 'snaps': '[0,524288]'}]}}, 'clean_regions': {'object_clean_regions': {'clean_offsets': '[1048576~18446744073708503039]', 'clean_omap': True, 'new_object': False}}} {'op': 'modify', 'object': '3:4f7efb3c:::101000000000133.0000001d:head', 'version': "374'870", 'prior_version': "374'869", 'reqid': 'client.133562.0:359461', 'extra_reqids': [], 'mtime': '2022-03-18T11:41:39.334504+0800', 'return_code': 0, 'mod_desc': {'object_mod_desc': {'can_local_rollback': True, 'rollback_info_completed': False, 'ops': [{'code': 'SETATTRS', 'attrs': ['_', 'hinfo_key', 'snapset']}, {'code': 'ROLLBACK_EXTENTS', 'gen': 870, 'snaps': '[524288,524288]'}]}}, 'clean_regions': {'object_clean_regions': {'clean_offsets': '[0~1048576,2097152~18446744073707454463]', 'clean_omap': True, 'new_object': False}}} {'op': 'modify', 'object': '3:4f7efb3c:::101000000000133.0000001d:head', 'version': "374'871", 'prior_version': "374'870", 'reqid': 'client.133562.0:359504', 'extra_reqids': [], 'mtime': '2022-03-18T11:41:39.385649+0800', 'return_code': 0, 'mod_desc': {'object_mod_desc': {'can_local_rollback': True, 'rollback_info_completed': False, 'ops': [{'code': 'SETATTRS', 'attrs': ['_', 'hinfo_key', 'snapset']}, {'code': 'ROLLBACK_EXTENTS', 'gen': 871, 'snaps': '[1048576,524288]'}]}}, 'clean_regions': {'object_clean_regions': {'clean_offsets': '[0~2097152,3145728~18446744073706405887]', 'clean_omap': True, 'new_object': False}}} {'op': 'modify', 'object': '3:4f7efb3c:::101000000000133.0000001d:head', 'version': "374'872", 'prior_version': "374'871", 'reqid': 'client.133562.0:359556', 'extra_reqids': [], 'mtime': '2022-03-18T11:41:39.433611+0800', 'return_code': 0, 'mod_desc': {'object_mod_desc': {'can_local_rollback': True, 'rollback_info_completed': False, 'ops': [{'code': 'SETATTRS', 'attrs': ['_', 'hinfo_key', 'snapset']}, {'code': 'ROLLBACK_EXTENTS', 'gen': 872, 'snaps': '[1572864,524288]'}]}}, 'clean_regions': {'object_clean_regions': {'clean_offsets': '[0~3145728,4194304~18446744073705357311]', 'clean_omap': True, 'new_object': False}}}
3.2f2s2.osd41.log [503'2242, 785'2826]

Files

txt-type-description.txt (29.9 KB) txt-type-description.txt jianwei zhang, 03/19/2022 12:58 PM
Actions
Copy link

Also available in: Atom PDF