Project

General

Profile

Actions

Bug #9585

closed

ceph assertion using rocksdb store in master branch

Added by Tamilarasi muthamizhan over 9 years ago. Updated about 9 years ago.

Status:
Can't reproduce
Priority:
High
Assignee:
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

ceph version 0.85-980-gc5906ec (c5906eca2ffa837891ba7d84775ece7b91f6c5c8)

ceph assertion when rocksdb is used for osd objectstore with powercycling on.

also, please note that this happens only with powercycling on, otherwise it works fine.


2014-09-23 21:39:47.528273 7f31d7a67700 -1 os/KeyValueStore.cc: In function 'unsigned int KeyValueStore::
_do_transaction(ObjectStore::Transaction&, KeyValueStore::BufferTransaction&, ThreadPool::TPHandle*)' thr
ead 7f31d7a67700 time 2014-09-23 21:39:47.526554
os/KeyValueStore.cc: 1544: FAILED assert(0 == "unexpected error")

 ceph version 0.85-917-g5fab7db (5fab7db3db060de4b634297e99ebb66bf1c4fa48)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xb77595]
 2: (KeyValueStore::_do_transaction(ObjectStore::Transaction&, KeyValueStore::BufferTransaction&, ThreadP
ool::TPHandle*)+0x76f) [0x92f17f]
 3: (KeyValueStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Tra
nsaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x13f) [0x931fef]
 4: (KeyValueStore::_do_op(KeyValueStore::OpSequencer*, ThreadPool::TPHandle&)+0xac) [0x93220c]
 5: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa66) [0xb682a6]
 6: (ThreadPool::WorkThread::entry()+0x10) [0xb69330]
 7: (()+0x7df3) [0x7f31e13afdf3]
 8: (clone()+0x6d) [0x7f31dfe923dd]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---

config file:
interactive-on-error: true
overrides:
  ceph:
    conf:
      osd:
        keyvaluestore_backend: rocksdb
        osd_objectstore: keyvaluestore-dev
roles:
- - mon.a
  - mds.0
  - osd.0
  - osd.1
  - osd.2
  - client.0
targets:
  ubuntu@mira042.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDD1KCsGjFVs1+kFGQTAuMq0s3y1IgnQ/AYqQZDCrsUKzdVWm+eA2TQtrH0ije7qb43ReD8c5o4ceUn9aPr22p3IIP/qWlPwocqXsB7fue3p+pwOU8B3yfaVYlgZkAV39O6wdvzfRvtoPHLGpvI6Ey+S5HNsSR1Jek2bnJ1mtC+e8fMqhyY3kbEYu7+cPKTFoLcSG2X3BH0ptWo3kDjHlzu7bgSnzmr9qKIwQThZ8T+VYOYGmijzWd+MmUWg8eo/0de4Dz02wOZLYM8wuqvVy52wq/JXfN7a7Zj5a1PopBbXMpnMSMRAYxbTNPjmQSROvEQLTzEkn3f/2vQat6RdyE5
tasks:
- chef: null
- install:
    branch: master
- ceph: null
- thrashosds:
    chance_down: 1.0
    powercycle: true
    timeout: 600
- ceph-fuse: null
- workunit:
    clients:
      client.0:
      - suites/blogbench.sh
- interactive: null

logs are copied to :/home/ubuntu/rocksdb_bug

Actions #1

Updated by Haomai Wang over 9 years ago

It looks like that powercycle will make header's bitmap inconsistence with actual data keys.

Actions #2

Updated by Samuel Just over 9 years ago

  • Priority changed from Urgent to High
Actions #3

Updated by Tamilarasi muthamizhan over 9 years ago

  • Assignee changed from Josh Durgin to Haomai Wang
Actions #4

Updated by Haomai Wang over 9 years ago

Hi Tamilarasi, it's still broken for the master branch? Give a link to the corresponding job for pulpito.ceph.com?

Actions #5

Updated by Samuel Just about 9 years ago

  • Status changed from New to Can't reproduce
Actions

Also available in: Atom PDF