Project

General

Profile

Actions
Copy link

Bug #20742

closed

"ObjectCacher.cc: 1196: FAILED assert(ob->last_commit_tid < tid)" in powercycle

Added by Yuri Weinstein almost 7 years ago. Updated over 4 years ago.

Status:
Can't reproduce
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
powercycle
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Run: http://pulpito.ceph.com/teuthology-2017-07-20_20:39:57-powercycle-master-testing-basic-mira/
Job: 1425568
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2017-07-20_20:39:57-powercycle-master-testing-basic-mira/1425568/teuthology.log

2017-07-21T07:10:06.619 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.mira041.stderr:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.1.1-230-g27e8d75/rpm/el7/BUILD/ceph-12.1.1-230-g27e8d75/src/osdc/ObjectCacher.cc: 1196: FAILED assert(ob->last_commit_tid < tid)
2017-07-21T07:10:06.620 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.mira041.stderr: ceph version 12.1.1-230-g27e8d75 (27e8d75f61f2343dcfae4077dd5ccdbcbffd8def) luminous (rc)
2017-07-21T07:10:06.620 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.mira041.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x2c62ed0400]
2017-07-21T07:10:06.621 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.mira041.stderr: 2: (ObjectCacher::bh_write_commit(long, sobject_t, std::vector<std::pair<long, unsigned long>, std::allocator<std::pair<long, unsigned long> > >&, unsigned long, int)+0xf34) [0x2c631da4b4]
2017-07-21T07:10:06.621 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.mira041.stderr: 3: (ObjectCacher::C_WriteCommit::finish(int)+0x58) [0x2c631e7e78]
2017-07-21T07:10:06.622 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.mira041.stderr: 4: (Context::complete(int)+0x9) [0x2c62e6c899]
2017-07-21T07:10:06.622 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.mira041.stderr: 5: (C_Lock::finish(int)+0x29) [0x2c62e6d1e9]
2017-07-21T07:10:06.622 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.mira041.stderr: 6: (Context::complete(int)+0x9) [0x2c62e6c899]
2017-07-21T07:10:06.622 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.mira041.stderr: 7: (Finisher::finisher_thread_entry()+0x198) [0x2c62ecf398]
2017-07-21T07:10:06.622 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.mira041.stderr: 8: (()+0x7dc5) [0x7fdbc0aa2dc5]
2017-07-21T07:10:06.623 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.mira041.stderr: 9: (clone()+0x6d) [0x7fdbbf983ced]
2017-07-21T07:10:06.623 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.mira041.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Related issues 1 (0 open — 1 closed)

Related to RADOS - Bug #23827: osd sends op_reply out of orderResolvedNeha Ojha

Actions
Actions
Copy link
 #2

Updated by Yuri Weinstein about 6 years ago

also in http://qa-proxy.ceph.com/teuthology/yuriw-2018-02-16_22:39:45-powercycle-wip-yuriw-master_2.16.18-distro-basic-smithi/2195068/teuthology.log

2018-02-17T04:17:17.790 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi183.stderr:/build/ceph-13.0.1-2014-gdf684a4/src/osdc/ObjectCacher.cc: In function 'void ObjectCacher::bh_write_commit(int64_t, sobject_t, std::vector<std::pair<long int, long unsigned int> >&, ceph_tid_t, int)' thread 7f13ab4d4700 time 2018-02-17 04:17:17.790988
2018-02-17T04:17:17.790 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi183.stderr:/build/ceph-13.0.1-2014-gdf684a4/src/osdc/ObjectCacher.cc: 1199: FAILED assert(ob->last_commit_tid < tid)
2018-02-17T04:17:17.790 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi183.stderr: ceph version 13.0.1-2014-gdf684a4 (df684a4e8ce3d07dc36a47017391a91f3f378084) mimic (dev)
2018-02-17T04:17:17.790 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi183.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0xf5) [0x7f13b3a9ae95]
2018-02-17T04:17:17.791 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi183.stderr: 2: (ObjectCacher::bh_write_commit(long, sobject_t, std::vector<std::pair<long, unsigned long>, std::allocator<std::pair<long, unsigned long> > >&, unsigned long, int)+0xde5) [0x561a4d7a6c15]
2018-02-17T04:17:17.791 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi183.stderr: 3: (ObjectCacher::C_WriteCommit::finish(int)+0x6a) [0x561a4d7b176a]
2018-02-17T04:17:17.791 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi183.stderr: 4: (Context::complete(int)+0x9) [0x561a4d75bdb9]
2018-02-17T04:17:17.791 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi183.stderr: 5: (C_Lock::finish(int)+0x29) [0x561a4d75cae9]
2018-02-17T04:17:17.792 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi183.stderr: 6: (Context::complete(int)+0x9) [0x561a4d75bdb9]
2018-02-17T04:17:17.792 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi183.stderr: 7: (Finisher::finisher_thread_entry()+0x31f) [0x7f13b3a994af]
2018-02-17T04:17:17.792 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi183.stderr: 8: (()+0x76ba) [0x7f13b35cc6ba]
2018-02-17T04:17:17.792 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi183.stderr: 9: (clone()+0x6d) [0x7f13b2df53dd]
2018-02-17T04:17:17.792 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi183.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Actions
Copy link
 #3

Updated by Yuri Weinstein about 6 years ago

  • Priority changed from Normal to Urgent
Actions
Copy link
 #4

Updated by Sage Weil about 6 years ago

  • Related to Bug #23827: osd sends op_reply out of order added
Actions
Copy link
 #5

Updated by Yuri Weinstein almost 6 years ago

Also in http://pulpito.ceph.com/yuriw-2018-06-12_21:34:02-powercycle-mimic-distro-basic-smithi/
jobs: 2659987

Assertion: /build/ceph-13.2.0-38-g5681597/src/osdc/ObjectCacher.cc: 1216: FAILED assert(ob->last_commit_tid < tid)
ceph version 13.2.0-38-g5681597 (5681597d48256ffb8d81ccacfbe157770e42fb4d) mimic (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7fbf10c325e2]
 2: (()+0x26b7a7) [0x7fbf10c327a7]
 3: (ObjectCacher::bh_write_commit(long, sobject_t, std::vector<std::pair<long, unsigned long>, std::allocator<std::pair<long, unsigned long> > >&, unsigned long, int)+0xc5d) [0x562bdc5d4d7d]
 4: (ObjectCacher::C_WriteCommit::finish(int)+0x6a) [0x562bdc5e072a]
 5: (Context::complete(int)+0x9) [0x562bdc58fa39]
 6: (C_Lock::finish(int)+0x29) [0x562bdc590709]
 7: (Context::complete(int)+0x9) [0x562bdc58fa39]
 8: (Finisher::finisher_thread_entry()+0x12e) [0x7fbf10c30b7e]
 9: (()+0x76db) [0x7fbf107af6db]
 10: (clone()+0x3f) [0x7fbf0f99088f]
Actions
Copy link
 #6

Updated by Sage Weil over 4 years ago

  • Status changed from New to Can't reproduce
Actions
Copy link

Also available in: Atom PDF