Project

General

Profile

Actions
Copy link

Bug #11317

closed

wip-newstore: Directory not empty not handled on operation

Added by Mark Nelson about 9 years ago. Updated almost 9 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Sage Weil
Category:
OSD
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Hit this when testing the wip-newstore branch using the filestore. Sam thinks it's likely from the wip-temp refactor. May not happen consistently as this test has previously been run with wip-newstore and passed.

    -3> 2015-04-02 17:00:42.242461 7f1b68ce7700  0 filestore(/tmp/cbt/mnt/osd-device-28-data)  error (39) Directory not empty not handled on operation 0x6aa5fe8 (43457.0.1, or op 1, co
unting from 0)
    -2> 2015-04-02 17:00:42.242511 7f1b68ce7700  0 filestore(/tmp/cbt/mnt/osd-device-28-data) ENOTEMPTY suggests garbage data in osd data dir
    -1> 2015-04-02 17:00:42.242516 7f1b68ce7700  0 filestore(/tmp/cbt/mnt/osd-device-28-data)  transaction dump:
{
    "ops": [
        {
            "op_num": 0,
            "op_name": "remove",
            "collection": "5.2fc_head",
            "oid": "5\/2fc\/\/head" 
        },
        {
            "op_num": 1,
            "op_name": "rmcoll",
            "collection": "5.2fc_head" 
        }
    ]
}

     0> 2015-04-02 17:00:42.245076 7f1b68ce7700 -1 os/FileStore.cc: In function 'unsigned int FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int, ThreadPool::TPHandle*
)' thread 7f1b68ce7700 time 2015-04-02 17:00:42.242643
os/FileStore.cc: 2939: FAILED assert(0 == "unexpected error")

 ceph version 0.93-986-g4b8259f (4b8259fc6a7c3ac7d2964fb0cf8522e01a8253ff)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xcaca65]
 2: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0xbba) [0x98d80a]
 3: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x64) [0x99ba44]
 4: (FileStore::_do_op(FileStore::OpSequencer*, ThreadPool::TPHandle&)+0x17e) [0x99bbde]
 5: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa76) [0xc9d8e6]
 6: (ThreadPool::WorkThread::entry()+0x10) [0xc9e970]
 7: (()+0x7ee5) [0x7f1b75e91ee5]
 8: (clone()+0x6d) [0x7f1b7496fb8d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 0 lockdep
   0/ 0 context
   0/ 0 crush
   0/ 0 mds
   0/ 0 mds_balancer
   0/ 0 mds_locker
   0/ 0 mds_log
   0/ 0 mds_log_expire
   0/ 0 mds_migrator
   0/ 0 buffer
   0/ 0 timer
   0/ 0 filer
   0/ 1 striper
   0/ 0 objecter
   0/ 0 rados
   0/ 0 rbd
   0/ 5 rbd_replay
   0/ 0 journaler
   0/ 0 objectcacher
   0/ 0 client
   0/ 0 osd
   0/ 0 optracker
   0/ 0 objclass
   0/ 0 filestore
   1/ 3 keyvaluestore
   0/ 0 journal
   0/ 0 ms
   0/ 0 mon
   0/ 0 monc
   0/ 0 paxos
   0/ 0 tp
   0/ 0 auth
   1/ 5 crypto
   0/ 0 finisher
   0/ 0 heartbeatmap
   0/ 0 perfcounter
   0/ 0 rgw
   1/10 civetweb
   1/ 5 javaclient
   0/ 0 asok
   0/ 0 throttle
   0/ 0 refs
   1/ 5 xio
   5/ 5 newstore
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /tmp/cbt/ceph/log/osd.28.log
--- end dump of recent events ---
2015-04-02 17:00:42.248811 7f1b68ce7700 -1 *** Caught signal (Aborted) **
 in thread 7f1b68ce7700

 ceph version 0.93-986-g4b8259f (4b8259fc6a7c3ac7d2964fb0cf8522e01a8253ff)
 1: /usr/bin/ceph-osd() [0xbaca12]
 2: (()+0xf6d0) [0x7f1b75e996d0]
 3: (gsignal()+0x37) [0x7f1b748b0877]
 4: (abort()+0x148) [0x7f1b748b1f68]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7f1b751b6dd5]
 6: (()+0x5ed46) [0x7f1b751b4d46]
 7: (()+0x5ed73) [0x7f1b751b4d73]
 8: (()+0x5ef93) [0x7f1b751b4f93]
 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x27a) [0xcacc5a]
 10: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0xbba) [0x98d80a]
 11: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x64) [0x99ba44]
 12: (FileStore::_do_op(FileStore::OpSequencer*, ThreadPool::TPHandle&)+0x17e) [0x99bbde]
 13: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa76) [0xc9d8e6]
 14: (ThreadPool::WorkThread::entry()+0x10) [0xc9e970]
 15: (()+0x7ee5) [0x7f1b75e91ee5]
 16: (clone()+0x6d) [0x7f1b7496fb8d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
     0> 2015-04-02 17:00:42.248811 7f1b68ce7700 -1 *** Caught signal (Aborted) **
 in thread 7f1b68ce7700

 ceph version 0.93-986-g4b8259f (4b8259fc6a7c3ac7d2964fb0cf8522e01a8253ff)
 1: /usr/bin/ceph-osd() [0xbaca12]
 2: (()+0xf6d0) [0x7f1b75e996d0]
 3: (gsignal()+0x37) [0x7f1b748b0877]
 4: (abort()+0x148) [0x7f1b748b1f68]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7f1b751b6dd5]
 6: (()+0x5ed46) [0x7f1b751b4d46]
 7: (()+0x5ed73) [0x7f1b751b4d73]
 8: (()+0x5ef93) [0x7f1b751b4f93]
 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x27a) [0xcacc5a]
 10: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0xbba) [0x98d80a]
 11: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x64) [0x99ba44]
 12: (FileStore::_do_op(FileStore::OpSequencer*, ThreadPool::TPHandle&)+0x17e) [0x99bbde]
 13: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa76) [0xc9d8e6]
 14: (ThreadPool::WorkThread::entry()+0x10) [0xc9e970]
 15: (()+0x7ee5) [0x7f1b75e91ee5]
 16: (clone()+0x6d) [0x7f1b7496fb8d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 0 lockdep
   0/ 0 context
   0/ 0 crush
   0/ 0 mds
   0/ 0 mds_balancer
   0/ 0 mds_locker
   0/ 0 mds_log
   0/ 0 mds_log_expire
   0/ 0 mds_migrator
   0/ 0 buffer
   0/ 0 timer
   0/ 0 filer
   0/ 1 striper
   0/ 0 objecter
   0/ 0 rados
   0/ 0 rbd
   0/ 5 rbd_replay
   0/ 0 journaler
   0/ 0 objectcacher
   0/ 0 client
   0/ 0 osd
   0/ 0 optracker
   0/ 0 objclass
   0/ 0 filestore
   1/ 3 keyvaluestore
   0/ 0 journal
   0/ 0 ms
   0/ 0 mon
   0/ 0 monc
   0/ 0 paxos
   0/ 0 tp
   0/ 0 auth
   1/ 5 crypto
   0/ 0 finisher
   0/ 0 heartbeatmap
   0/ 0 perfcounter
   0/ 0 rgw
   1/10 civetweb
   1/ 5 javaclient
   0/ 0 asok
   0/ 0 throttle
   0/ 0 refs
   1/ 5 xio
   5/ 5 newstore
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /tmp/cbt/ceph/log/osd.28.log
--- end dump of recent events ---
Actions
Copy link
 #1

Updated by Mark Nelson about 9 years ago

Hit this again during a repeat test though not in quite the same place. Appeared both times after a pool had been deleted and was in the process of being recreated.

Actions
Copy link
 #2

Updated by Sage Weil almost 9 years ago

  • Status changed from New to Resolved
Actions
Copy link

Also available in: Atom PDF