Bug #1949: osd: ENOTEMPTY on collection removal from snaptrimmer - Ceph - Ceph

Actions

Copy link

Bug #1949

closed

osd: ENOTEMPTY on collection removal from snaptrimmer

Added by Sage Weil over 12 years ago. Updated about 12 years ago.

Status:

Resolved

Priority:

Normal

Assignee:

Sage Weil

Category:

OSD

Target version:

v0.42

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description


os/FileStore.cc: In function 'unsigned int FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t)', in thread '7f9a84c42700'
os/FileStore.cc: 2424: FAILED assert(0 == "ENOTEMPTY suggests garbage data in osd data dir")
 ceph version 0.39-476-g94f55f4 (commit:94f55f4811eafadb6e8a0f353b8e24c3a8b0baaf)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x76) [0x8d81e2]
 2: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long)+0x1f3b) [0xa62ecd]
 3: (FileStore::do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long)+0x109) [0xa60479]
 4: (FileStore::_do_op(FileStore::OpSequencer*)+0x1dd) [0xa5f1c3]
 5: (FileStore::OpWQ::_process(FileStore::OpSequencer*)+0x27) [0xa7654d]
 6: (ThreadPool::WorkQueue<FileStore::OpSequencer>::_void_process(void*)+0x2e) [0xa82d28]
 7: (ThreadPool::worker()+0x4af) [0x8d4ff7]
 8: (ThreadPool::WorkThread::entry()+0x1c) [0x85dbda]
 9: (Thread::_entry_func(void*)+0x23) [0x8ae0c5]
 10: (()+0x7971) [0x7f9a8d8c5971]
 11: (clone()+0x6d) [0x7f9a8bf5092d]
 ceph version 0.39-476-g94f55f4 (commit:94f55f4811eafadb6e8a0f353b8e24c3a8b0baaf)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x76) [0x8d81e2]
 2: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long)+0x1f3b) [0xa62ecd]
 3: (FileStore::do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long)+0x109) [0xa60479]
 4: (FileStore::_do_op(FileStore::OpSequencer*)+0x1dd) [0xa5f1c3]
 5: (FileStore::OpWQ::_process(FileStore::OpSequencer*)+0x27) [0xa7654d]
 6: (ThreadPool::WorkQueue<FileStore::OpSequencer>::_void_process(void*)+0x2e) [0xa82d28]
 7: (ThreadPool::worker()+0x4af) [0x8d4ff7]
 8: (ThreadPool::WorkThread::entry()+0x1c) [0x85dbda]
 9: (Thread::_entry_func(void*)+0x23) [0x8ae0c5]
 10: (()+0x7971) [0x7f9a8d8c5971]
 11: (clone()+0x6d) [0x7f9a8bf5092d]

log is at metropolis:/home/sage/osd.snaptrimmer.log

unfortunately teuthology had already removed the osd data, so i can't see what was there.

Actions

Copy link

Updated by Sage Weil about 12 years ago

i have a full log (osd 20 filestore 20) leading up to this at metropolis:/home/sage/osd.enotempty.log

Actions

Copy link

Updated by Sage Weil about 12 years ago

Target version set to v0.43

another log, with filestore debugging, and the contents of the fs. There was

ubuntu@sepia72:/tmp/cephtest$ ls -al d/current/0.17_79/
total 608
drwxr-xr-x 1 ubuntu ubuntu      54 2012-02-09 16:29 .
drwxr-xr-x 1 ubuntu ubuntu     962 2012-02-09 16:29 ..
-rw-r--r-- 3 ubuntu ubuntu 1412859 2012-02-09 16:29 sepia721569-71__89_4A3C96B7

in the dir and it looks like it was written by an OSDOp right during the TrimmingObjects state. Seems like a pretty simple race that we're not handling..

Actions

Copy link