Ceph » RADOS

btw down pg is 1.1735.

Starting OSD 381 crashes 65, 133 and 118. Stoping 65 enables to start remaining OSDs, starting it crash 118 and 133.

To decrease debug points I removed 1.1735s0 from OSD 118 (it was copied from 133 there).

added debug_osd 20/20 to config, started OSD 133 and after it booted, started OSD 65. Both of them crashed, logs posted.
OSD 133: c396e54f-f96e-4312-9a18-77ea5a6ed5c2
OSD 65: a76c4057-8acf-46c2-aa02-39528b98dd27

I had to delete affected pool to reclaim occupied space so I am unable to verify any fixes

we hit this bug too, RBD+EC

@{
"op": "modify",
"object": "1:4ec1cf44:::xbd_data.100119495cff.0000000000002897:head",
"version": "1624'39707",
"prior_version": "1624'39706",
"reqid": "client.77068.0:18102704",
"extra_reqids": [],
"mtime": "2017-11-12 03:44:55.752943",
"return_code": 0,
"mod_desc": {
"object_mod_desc": {
"can_local_rollback": true,
"rollback_info_completed": false,
"ops": [ {
"code": "SETATTRS",
"attrs": [
"_",
"hinfo_key",
"snapset"
]
}, {
"code": "ROLLBACK_EXTENTS",
"gen": 39707,
"snaps": "[536576,4096]"
}
]
}
}
}, {
"op": "modify",
"object": "1:4ec1cf44:::xbd_data.100119495cff.0000000000002897:head",
"version": "1624'39709",
"prior_version": "1624'39708",
"reqid": "client.86630.0:19763476",
"extra_reqids": [],
"mtime": "2017-11-12 03:44:55.774969",
"return_code": 0,
"mod_desc": {
"object_mod_desc": {
"can_local_rollback": true,
"rollback_info_completed": false,
"ops": [ {
"code": "SETATTRS",
"attrs": [
"_",
"hinfo_key",
"snapset"
]
}, {
"code": "ROLLBACK_EXTENTS",
"gen": 39709,
"snaps": "[540672,4096]"
}
]
}
@
the last two updates, it's divergent entries, version 39708 disappear. i guess op has applied disorderly

liu mingxin wrote:

@{
"op": "modify",
"object": "1:4ec1cf44:::rbd_data.100119495cff.0000000000002897:head",
"version": "1624'39707",
"prior_version": "1624'39706",
"reqid": "client.77068.0:18102704",
"extra_reqids": [],
"mtime": "2017-11-12 03:44:55.752943",
"return_code": 0,
"mod_desc": {
"object_mod_desc": {
"can_local_rollback": true,
"rollback_info_completed": false,
"ops": [ {
"code": "SETATTRS",
"attrs": [
"_",
"hinfo_key",
"snapset"
]
}, {
"code": "ROLLBACK_EXTENTS",
"gen": 39707,
"snaps": "[536576,4096]"
}
]
}
}
}, {
"op": "modify",
"object": "1:4ec1cf44:::rbd_data.100119495cff.0000000000002897:head",
"version": "1624'39709",
"prior_version": "1624'39708",
"reqid": "client.86630.0:19763476",
"extra_reqids": [],
"mtime": "2017-11-12 03:44:55.774969",
"return_code": 0,
"mod_desc": {
"object_mod_desc": {
"can_local_rollback": true,
"rollback_info_completed": false,
"ops": [ {
"code": "SETATTRS",
"attrs": [
"_",
"hinfo_key",
"snapset"
]
}, {
"code": "ROLLBACK_EXTENTS",
"gen": 39709,
"snaps": "[540672,4096]"
}
]
}
@
the last two updates, it's divergent entries, version 39708 disappear. i guess op has applied disorderly

we hit this bug too in ec pool 2+1, i find one peer did not receive one piece of op message sended from primary osd, but the former and latter had been handled and added to the log, which may cause this assert.

see https://github.com/ceph/ceph/pull/16675

Chang Liu wrote:

see https://github.com/ceph/ceph/pull/16675

seems that patch was already applied since the "FAILED assert(i->prior_version == last || i->is_error())"