Project

General

Profile

Bug #1949

osd: ENOTEMPTY on collection removal from snaptrimmer

Added by Sage Weil almost 12 years ago. Updated almost 12 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Sage Weil
Category:
OSD
Target version:
v0.42
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description


os/FileStore.cc: In function 'unsigned int FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t)', in thread '7f9a84c42700'
os/FileStore.cc: 2424: FAILED assert(0 == "ENOTEMPTY suggests garbage data in osd data dir")
 ceph version 0.39-476-g94f55f4 (commit:94f55f4811eafadb6e8a0f353b8e24c3a8b0baaf)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x76) [0x8d81e2]
 2: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long)+0x1f3b) [0xa62ecd]
 3: (FileStore::do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long)+0x109) [0xa60479]
 4: (FileStore::_do_op(FileStore::OpSequencer*)+0x1dd) [0xa5f1c3]
 5: (FileStore::OpWQ::_process(FileStore::OpSequencer*)+0x27) [0xa7654d]
 6: (ThreadPool::WorkQueue<FileStore::OpSequencer>::_void_process(void*)+0x2e) [0xa82d28]
 7: (ThreadPool::worker()+0x4af) [0x8d4ff7]
 8: (ThreadPool::WorkThread::entry()+0x1c) [0x85dbda]
 9: (Thread::_entry_func(void*)+0x23) [0x8ae0c5]
 10: (()+0x7971) [0x7f9a8d8c5971]
 11: (clone()+0x6d) [0x7f9a8bf5092d]
 ceph version 0.39-476-g94f55f4 (commit:94f55f4811eafadb6e8a0f353b8e24c3a8b0baaf)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x76) [0x8d81e2]
 2: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long)+0x1f3b) [0xa62ecd]
 3: (FileStore::do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long)+0x109) [0xa60479]
 4: (FileStore::_do_op(FileStore::OpSequencer*)+0x1dd) [0xa5f1c3]
 5: (FileStore::OpWQ::_process(FileStore::OpSequencer*)+0x27) [0xa7654d]
 6: (ThreadPool::WorkQueue<FileStore::OpSequencer>::_void_process(void*)+0x2e) [0xa82d28]
 7: (ThreadPool::worker()+0x4af) [0x8d4ff7]
 8: (ThreadPool::WorkThread::entry()+0x1c) [0x85dbda]
 9: (Thread::_entry_func(void*)+0x23) [0x8ae0c5]
 10: (()+0x7971) [0x7f9a8d8c5971]
 11: (clone()+0x6d) [0x7f9a8bf5092d]

log is at metropolis:/home/sage/osd.snaptrimmer.log

unfortunately teuthology had already removed the osd data, so i can't see what was there.

Associated revisions

Revision 7c6dff48 (diff)
Added by Sage Weil almost 12 years ago

osd: filter trimming|purged snaps out of op SnapContext

We can receive an op with an old SnapContext that includes snaps that we've
already trimmed or are in the process of trimming. Filter them out!
Otherwise we will recreate and add links into collections we've already
marked as removed, and we'll get things like ENOTEMPTY when we try to
remove them. Or just leave them laying around.

Fixes: #1949
Signed-off-by: Sage Weil <>

History

#1 Updated by Sage Weil almost 12 years ago

i have a full log (osd 20 filestore 20) leading up to this at metropolis:/home/sage/osd.enotempty.log

#2 Updated by Sage Weil almost 12 years ago

  • Target version set to v0.43

another log, with filestore debugging, and the contents of the fs. There was

ubuntu@sepia72:/tmp/cephtest$ ls -al d/current/0.17_79/
total 608
drwxr-xr-x 1 ubuntu ubuntu      54 2012-02-09 16:29 .
drwxr-xr-x 1 ubuntu ubuntu     962 2012-02-09 16:29 ..
-rw-r--r-- 3 ubuntu ubuntu 1412859 2012-02-09 16:29 sepia721569-71__89_4A3C96B7

in the dir and it looks like it was written by an OSDOp right during the TrimmingObjects state. Seems like a pretty simple race that we're not handling..

#3 Updated by Sage Weil almost 12 years ago

  • Status changed from New to 7
  • Assignee set to Sage Weil

#4 Updated by Sage Weil almost 12 years ago

  • Status changed from 7 to Resolved
  • Target version changed from v0.43 to v0.42

Also available in: Atom PDF