Project

General

Profile

Actions
Copy link

Bug #9105

open

~ObjectCacher behaves poorly on EBLACKLISTED

Added by John Spray over 9 years ago. Updated over 4 years ago.

Status:
New
Priority:
Low
Assignee:
-
Category:
Code Hygiene
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
osdc
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

In ceph master 78dc4df

http://qa-proxy.ceph.com/teuthology/teuthology-2014-08-11_23:00:01-rbd-master-testing-basic-multi/418034/teuthology.log

2014-08-12T19:50:41.401 INFO:tasks.workunit.client.0.plana38.stderr:osdc/ObjectCacher.cc: In function 'ObjectCacher::~ObjectCacher()' thread 7f16aa1f7740 time 2014-08-12 19:50:41.370364
2014-08-12T19:50:41.402 INFO:tasks.workunit.client.0.plana38.stderr:osdc/ObjectCacher.cc: 529: FAILED assert(i->empty())
2014-08-12T19:50:41.402 INFO:tasks.workunit.client.0.plana38.stderr: ceph version 0.83-738-ge84a9d4 (e84a9d468471e322a5c1a36e8187a9fcd1624832)
2014-08-12T19:50:41.402 INFO:tasks.workunit.client.0.plana38.stderr: 1: (ObjectCacher::~ObjectCacher()+0x182) [0x7f169c310a12]
2014-08-12T19:50:41.402 INFO:tasks.workunit.client.0.plana38.stderr: 2: (librbd::ImageCtx::~ImageCtx()+0x2a) [0x7f169c2bd7da]
2014-08-12T19:50:41.402 INFO:tasks.workunit.client.0.plana38.stderr: 3: (librbd::close_image(librbd::ImageCtx*)+0x92) [0x7f169c2cf932]
2014-08-12T19:50:41.402 INFO:tasks.workunit.client.0.plana38.stderr: 4: (rbd_close()+0x9) [0x7f169c2b3f19]
2014-08-12T19:50:41.402 INFO:tasks.workunit.client.0.plana38.stderr: 5: (ffi_call_unix64()+0x4c) [0x7f16a89f2adc]
2014-08-12T19:50:41.402 INFO:tasks.workunit.client.0.plana38.stderr: 6: (ffi_call()+0x1fc) [0x7f16a89f240c]
2014-08-12T19:50:41.403 INFO:tasks.workunit.client.0.plana38.stderr: 7: (_ctypes_callproc()+0x48e) [0x7f16a8c095fe]
2014-08-12T19:50:41.403 INFO:tasks.workunit.client.0.plana38.stderr: 8: (()+0x15f9e) [0x7f16a8c0af9e]
2014-08-12T19:50:41.403 INFO:tasks.workunit.client.0.plana38.stderr: 9: (PyEval_EvalFrameEx()+0x970) [0x52cc20]
2014-08-12T19:50:41.403 INFO:tasks.workunit.client.0.plana38.stderr: 10: (PyEval_EvalFrameEx()+0xc82) [0x52cf32]
2014-08-12T19:50:41.403 INFO:tasks.workunit.client.0.plana38.stderr: 11: python() [0x56d0aa]
2014-08-12T19:50:41.403 INFO:tasks.workunit.client.0.plana38.stderr: 12: python() [0x4d9854]
2014-08-12T19:50:41.403 INFO:tasks.workunit.client.0.plana38.stderr: 13: (PyObject_CallFunctionObjArgs()+0x128) [0x54a328]
2014-08-12T19:50:41.403 INFO:tasks.workunit.client.0.plana38.stderr: 14: (PyEval_EvalFrameEx()+0x25f3) [0x52e8a3]
2014-08-12T19:50:41.404 INFO:tasks.workunit.client.0.plana38.stderr: 15: (PyEval_EvalCodeEx()+0x2a4) [0x55c594]
2014-08-12T19:50:41.404 INFO:tasks.workunit.client.0.plana38.stderr: 16: (PyEval_EvalCode()+0x32) [0x5b7392]
2014-08-12T19:50:41.404 INFO:tasks.workunit.client.0.plana38.stderr: 17: python() [0x469663]
2014-08-12T19:50:41.404 INFO:tasks.workunit.client.0.plana38.stderr: 18: (PyRun_FileExFlags()+0x92) [0x4699e3]
2014-08-12T19:50:41.404 INFO:tasks.workunit.client.0.plana38.stderr: 19: (PyRun_SimpleFileExFlags()+0x2ee) [0x469f1c]
2014-08-12T19:50:41.404 INFO:tasks.workunit.client.0.plana38.stderr: 20: (Py_Main()+0xb5e) [0x46ab81]
2014-08-12T19:50:41.404 INFO:tasks.workunit.client.0.plana38.stderr: 21: (__libc_start_main()+0xf5) [0x7f16a9a26ec5]
2014-08-12T19:50:41.405 INFO:tasks.workunit.client.0.plana38.stderr: 22: python() [0x57497e]
Actions
Copy link
 #1

Updated by John Spray over 9 years ago

Started failing in 061c8e93f76dc4fd6290d6d15723d76e73267444 where rbd_cache and rbd_cache_writethrough_until_flush were enabled by default.

Actions
Copy link
 #2

Updated by John Spray over 9 years ago

This is happening when the librbd-using client is blacklisted, ObjectCacher fails to flush when requested, and ImageCtx tries to delete it anyway, hitting the assertion that it should be empty when deleted.

Actions
Copy link
 #3

Updated by Sage Weil over 9 years ago

John Spray wrote:

This is happening when the librbd-using client is blacklisted, ObjectCacher fails to flush when requested, and ImageCtx tries to delete it anyway, hitting the assertion that it should be empty when deleted.

when objectcacher gets EBLACKLISTED it should probably throw away all buffers or something... or mark them with an error state so that the caller knows the problem but we can still shut down.

Actions
Copy link
 #4

Updated by John Spray over 9 years ago

  • Status changed from New to Fix Under Review
Actions
Copy link
 #5

Updated by John Spray over 9 years ago

Punting on a general purpose fix for ObjectCacher for the time being, and just fixing this in librbd teardown.

Actions
Copy link
 #6

Updated by Sage Weil over 9 years ago

  • Project changed from Ceph to rbd
  • Priority changed from Normal to Urgent
  • Source changed from other to Q/A
Actions
Copy link
 #7

Updated by John Spray over 9 years ago

NB the handing for this case in rbd landed with wip-objecter, keep this ticket open for general purpose ObjectCacher solution?

Actions
Copy link
 #8

Updated by Sage Weil over 9 years ago

  • Project changed from rbd to CephFS
  • Subject changed from librbd hitting assertion in ~ObjectCacher to ~ObjectCacher behaves poorly on EBLACKLISTED
  • Status changed from Fix Under Review to 12
  • Priority changed from Urgent to Low
Actions
Copy link
 #9

Updated by Greg Farnum almost 8 years ago

  • Category set to Code Hygiene
  • Component(FS) osdc added
Actions
Copy link
 #10

Updated by Patrick Donnelly over 4 years ago

  • Status changed from 12 to New
Actions
Copy link

Also available in: Atom PDF