Bug #44715: common/TrackedOp.cc: 163: FAILED ceph_assert((sharded_in_flight_list.back())->ops_in_flight_sharded.empty())mon: - RADOS - Ceph

Actions

Copy link

#1

Updated by Josh Durgin about 4 years ago

Related to Bug #42347: nautilus assert during osd shutdown: FAILED ceph_assert((sharded_in_flight_list.back())->ops_in_flight_sharded.empty()) added

Actions

Copy link

#2

Updated by Sebastian Wagner about 4 years ago

Related to Bug #45001: mon+cephadm: ceph_assert((sharded_in_flight_list.back())->ops_in_flight_sharded.empty()) added

Actions

Copy link

#3

Updated by Sebastian Wagner about 4 years ago

http://pulpito.ceph.com/swagner-2020-04-08_10:27:55-rados-wip-swagner2-testing-2020-04-08-0014-distro-basic-smithi/4935237/

Actions

Copy link

#4

Updated by Sebastian Wagner about 4 years ago

Backport set to octopus

Actions

Copy link

#5

Updated by Sebastian Wagner about 4 years ago

http://pulpito.ceph.com/swagner-2020-04-09_21:46:02-rados-wip-swagner2-testing-2020-04-09-1541-distro-basic-smithi/

Actions

Copy link

#6

Updated by Sebastian Wagner about 4 years ago

Related to Bug #45008: [osd crash]The ceph-osd assert with rbd bench io added

Actions

Copy link

#7

Updated by Sebastian Wagner about 4 years ago

http://pulpito.ceph.com/swagner-2020-04-15_09:10:55-rados-wip-swagner2-testing-2020-04-14-1813-distro-basic-smithi/

Actions

Copy link

#8

Updated by Sebastian Wagner about 4 years ago

Status changed from New to Fix Under Review
Pull request ID set to 34624

Actions

Copy link

#9

Updated by Brad Hubbard almost 4 years ago

/a/teuthology-2020-04-26_02:30:03-rados-octopus-distro-basic-smithi/4984693
/a/teuthology-2020-04-26_02:30:03-rados-octopus-distro-basic-smithi/4985044
/a/teuthology-2020-04-26_02:30:03-rados-octopus-distro-basic-smithi/4984948
/a/teuthology-2020-04-26_02:30:03-rados-octopus-distro-basic-smithi/4984523
/a/teuthology-2020-04-26_02:30:03-rados-octopus-distro-basic-smithi/4984620
/a/teuthology-2020-04-26_02:30:03-rados-octopus-distro-basic-smithi/4985009
/a/teuthology-2020-04-26_02:30:03-rados-octopus-distro-basic-smithi/4984842

Actions

Copy link

#10

Updated by Sebastian Wagner almost 4 years ago

http://pulpito.ceph.com/swagner-2020-04-30_14:11:16-rados-wip-swagner2-testing-2020-04-29-1247-distro-basic-smithi/5002368/

Actions

Copy link

#11

Updated by Neha Ojha almost 4 years ago

Priority changed from High to Urgent

Actions

Copy link

#12

Updated by Sebastian Wagner almost 4 years ago

http://pulpito.ceph.com/swagner-2020-05-07_09:50:39-rados-wip-swagner3-testing-2020-05-06-1727-distro-basic-smithi/5030240

Actions

Copy link

#13

Updated by Neha Ojha almost 4 years ago

Status changed from Fix Under Review to Need More Info

I am not able to reproduce this failure on octopus or on master:

http://pulpito.ceph.com/nojha-2020-05-13_17:20:44-rados:cephadm-octopus-distro-basic-smithi/
http://pulpito.ceph.com/nojha-2020-05-12_16:52:34-rados:cephadm-master-distro-basic-smithi/

Sebastian, have you seen this recently?

Actions

Copy link

#14

Updated by Neha Ojha almost 4 years ago

Priority changed from Urgent to Normal

Lowering severity since we haven't seen it in two weeks.

Actions

Copy link

#15

Updated by Neha Ojha over 2 years ago

Has duplicate Bug #51872: crash: OpTracker::~OpTracker(): assert((sharded_in_flight_list.back())->ops_in_flight_sharded.empty()) added

Actions

Copy link

#16

Updated by Neha Ojha over 2 years ago

Affected versions according to telemetry

Ceph - v15.1.0, Ceph - v15.1.1, Ceph - v15.2.1, Ceph - v15.2.10, Ceph - v15.2.11, Ceph - v15.2.13, Ceph - v15.2.3, Ceph - v15.2.4, Ceph - v15.2.5, Ceph - v15.2.6, Ceph - v15.2.7, Ceph - v15.2.8, Ceph - v15.2.9

Actions

Copy link

#17

Updated by Telemetry Bot over 2 years ago

Crash signature (v1) updated (diff)
Crash signature (v2) updated (diff)
Affected Versions v15.1.0, v15.1.1, v15.2.1, v15.2.10, v15.2.11, v15.2.13, v15.2.3, v15.2.4, v15.2.5, v15.2.6, v15.2.7, v15.2.8, v15.2.9 added

http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=58697a1c8d484e18346c670afb3d304b05e74afae83c2153c1fd42774dd2335a

Assert condition: (sharded_in_flight_list.back())->ops_in_flight_sharded.empty()
Assert function: OpTracker::~OpTracker()

Sanitized backtrace:

    pthread_getname_np()
    ceph::logging::Log::dump_recent()
    OpTracker::~OpTracker()
    Monitor::~Monitor()
    Monitor::~Monitor()
    main()
    __libc_start_main()
    _start()

Crash dump sample:

{
    "archived": "2021-08-04 07:29:34.379387",
    "assert_condition": "(sharded_in_flight_list.back())->ops_in_flight_sharded.empty()",
    "assert_file": "common/TrackedOp.cc",
    "assert_func": "OpTracker::~OpTracker()",
    "assert_line": 163,
    "assert_msg": "common/TrackedOp.cc: In function 'OpTracker::~OpTracker()' thread 7fc1be4616c0 time 2021-08-04T07:08:17.896542+0000\ncommon/TrackedOp.cc: 163: FAILED ceph_assert((sharded_in_flight_list.back())->ops_in_flight_sharded.empty())",
    "assert_thread_name": "ceph-mon",
    "backtrace": [
        "(()+0x12b20) [0x7fc1b313fb20]",
        "(pthread_getname_np()+0x48) [0x7fc1b3140d98]",
        "(ceph::logging::Log::dump_recent()+0x428) [0x7fc1b5999978]",
        "(()+0x4be2db) [0x5645bad8b2db]",
        "(()+0x12b20) [0x7fc1b313fb20]",
        "(gsignal()+0x10f) [0x7fc1b1da77ff]",
        "(abort()+0x127) [0x7fc1b1d91c35]",
        "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7fc1b5649d61]",
        "(()+0x27af2a) [0x7fc1b5649f2a]",
        "(OpTracker::~OpTracker()+0x39) [0x7fc1b572a029]",
        "(Monitor::~Monitor()+0x324) [0x5645bab5a054]",
        "(Monitor::~Monitor()+0xd) [0x5645bab5ac7d]",
        "(main()+0x565e) [0x5645baaed4ee]",
        "(__libc_start_main()+0xf3) [0x7fc1b1d937b3]",
        "(_start()+0x2e) [0x5645bab16d8e]" 
    ],
    "ceph_version": "15.2.13",
    "crash_id": "2021-08-04T07:08:17.914171Z_29954452-36f0-4f07-accc-981946a051ba",
    "entity_name": "mon.756e25a057bfef9c6032d1a2c5a6693436335ca0",
    "os_id": "centos",
    "os_name": "CentOS Linux",
    "os_version": "8",
    "os_version_id": "8",
    "process_name": "ceph-mon",
    "stack_sig": "f49fedb392abd22f17e98f0c0417a3b5df912f359a2a9a4683adab96078c818e",
    "timestamp": "2021-08-04T07:08:17.914171Z",
    "utsname_machine": "x86_64",
    "utsname_release": "4.18.0-305.10.2.el8_4.x86_64",
    "utsname_sysname": "Linux",
    "utsname_version": "#1 SMP Tue Jul 20 17:25:16 UTC 2021" 
}

Actions

Copy link

#18

Updated by Neha Ojha over 2 years ago

Status changed from Need More Info to New
Crash signature (v1) updated (diff)

Actions

Copy link

#19

Updated by Neha Ojha over 2 years ago

Has duplicate Bug #52218: crash: OpTracker::~OpTracker(): assert((sharded_in_flight_list.back())->ops_in_flight_sharded.empty()) added

Actions

Copy link

#20

Updated by Neha Ojha over 2 years ago

Has duplicate Bug #52217: crash: OpTracker::~OpTracker(): assert((sharded_in_flight_list.back())->ops_in_flight_sharded.empty()) added

Actions

Copy link

#21

Updated by Neha Ojha over 2 years ago

Has duplicate Bug #52214: crash: OpTracker::~OpTracker(): assert((sharded_in_flight_list.back())->ops_in_flight_sharded.empty()) added

Actions

Copy link

#22

Updated by Neha Ojha over 2 years ago

Has duplicate Bug #52213: crash: OpTracker::~OpTracker(): assert((sharded_in_flight_list.back())->ops_in_flight_sharded.empty()) added

Actions

Copy link

#23

Updated by Neha Ojha over 2 years ago

Status changed from New to Fix Under Review

Actions

Copy link

#24

Updated by Neha Ojha over 2 years ago

Has duplicate Bug #52140: crash: OpTracker::~OpTracker(): assert((sharded_in_flight_list.back())->ops_in_flight_sharded.empty()) added

Actions

Copy link

#25

Updated by Yuri Weinstein over 2 years ago

https://github.com/ceph/ceph/pull/34624 merged

Actions

Copy link

#26

Updated by Neha Ojha over 2 years ago

Status changed from Fix Under Review to Pending Backport
Backport changed from octopus to octopus, pacific

Actions

Copy link

#27

Updated by Backport Bot over 2 years ago

Copied to Backport #52791: pacific: common/TrackedOp.cc: 163: FAILED ceph_assert((sharded_in_flight_list.back())->ops_in_flight_sharded.empty())mon: added

Actions

Copy link

#28

Updated by Backport Bot over 2 years ago

Copied to Backport #52792: octopus: common/TrackedOp.cc: 163: FAILED ceph_assert((sharded_in_flight_list.back())->ops_in_flight_sharded.empty())mon: added

Actions

Copy link

#29

Updated by Telemetry Bot about 2 years ago

Crash signature (v1) updated (diff)
Crash signature (v2) updated (diff)
Affected Versions v13.2.6, v14.2.0, v14.2.1, v14.2.10, v14.2.11, v14.2.15, v14.2.16, v14.2.18, v14.2.19, v14.2.2, v14.2.20, v14.2.22, v14.2.6, v14.2.8, v14.2.9, v15.2.14, v15.2.15 added