Ceph

Bug #9592

Updated by Loïc Dachary over 9 years ago

I find this issue while i was trying to run a 1GB Write Cosbench Workload using librados.(My 1MB write & read run was successful) 
 The Run hang permanently and nothing written to cluster, however I observed client is sending the data (n/w bandwidth usage was 5-10Gbits). 
 AFAIK, COSbench uses java binding for Librados. I debugged further with python binding. 
 After installing ceph-python , I executed following program:- 
 <pre> 

 ######################## 
 import rados 
 try: 
 cluster = rados.Rados(conffile='/etc/ceph/ceph.conf') 
 except TypeError as e: 
 print 'Argument validation error: ', e 
 raise e 
 print "Created cluster handle." 
 try: 
 cluster.connect() 
 except Exception as e: 
 print "connection error: ", e 
 raise e 
 finally: 
 print "Connected to the cluster." 
 print "\n\nI/O Context and Object Operations" 
 print "=================================" 
 print "\nCreating a context for the 'data' pool" 
 if not cluster.pool_exists('data'): 
 raise RuntimeError('No data pool exists') 
 ioctx = cluster.open_ioctx('data') 
 print "\nWriting object 'bm' with contents 'Bonjour tout le monde!' to pool 'data'." 
 ioctx.write("bm", "Bonjour tout le monde!") 
 print "Writing XATTR 'lang' with value 'fr_FR' to object 'bm'" 
 ioctx.set_xattr("bm", "lang", "fr_FR") 
 print "\nWriting object 'bm1' with contents 'Bonjour tout le monde!' to pool 'data'." 
 ioctx.write("bm1", "Bonjour tout le monde!"*10000) 
 print "Writing XATTR 'lang' with value 'fr_FR' to object 'bm1'" 
 ioctx.set_xattr("bm1", "lang", "fr_FR") 
 print "\nWriting object 'hw' with contents 'Hello World!' to pool 'data'." 
 ioctx.write("hw", "Hello World!\n world is not enough"*1000000000) 
 print "Writing XATTR 'lang' with value 'en_US' to object 'hw'" 
 ioctx.set_xattr("hw", "lang", "en_US") 
 ''' 
 print "\nContents of object 'hw'\n------------------------" 
 print ioctx.read("hw") 
 print "\n\nGetting XATTR 'lang' from object 'hw'" 
 print ioctx.get_xattr("hw", "lang") 
 print "\nContents of object 'bm'\n------------------------" 
 print ioctx.read("bm") 
 print "Getting XATTR 'lang' from object 'bm'" 
 print ioctx.get_xattr("bm", "lang") 
 ''' 
 print "Removing object 'bm'" 
 ioctx.remove_object("bm") 
 print "Removing object 'bm1'" 
 ioctx.remove_object("bm1") 
 print "\nRemoving object 'hw'" 
 ioctx.remove_object("hw") 
 </pre> ######################## 
 I got the following output:- 
 <pre> ####################### 
 ems@rack6-ramp-1:~$ python ceph-client.py 
 Created cluster handle. 
 Connected to the cluster. 
 I/O Context and Object Operations 
 ================================= 
 Creating a context for the 'data' pool 
 Writing object 'bm' with contents 'Bonjour tout le monde!' to pool 'data'. 
 Writing XATTR 'lang' with value 'fr_FR' to object 'bm' 
 Writing object 'bm1' with contents 'Bonjour tout le monde!' to pool 'data'. 
 Writing XATTR 'lang' with value 'fr_FR' to object 'bm1' 
 Writing object 'hw' with contents 'Hello World!' to pool 'data'. 
 common/Throttle.cc: In function 'bool Throttle::get_or_fail(int64_t)' thread 7f768a7d0700 time 2014-09-25 18:37:08.003401 
 common/Throttle.cc: 194: FAILED assert(c >= 0) 
 ceph version 0.84 (3215c520e1306f50d0094b5646636c02456c9df4) 
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7f768d73d71b] 
 2: (Throttle::get_or_fail(long)+0x5ae) [0x7f768d733d0e] 
 3: (Objecter::_throttle_op(Objecter::Op*, int)+0x68) [0x7f768d6ae588] 
 4: (Objecter::_op_submit_with_budget(Objecter::Op*, RWLock::Context&)+0x195) [0x7f768d6bf785] 
 5: (Objecter::op_submit(Objecter::Op*)+0x96) [0x7f768d6bf8c6] 
 6: (librados::IoCtxImpl::operate(object_t const&, ObjectOperation*, long*, int)+0x33d) [0x7f768d698a0d] 
 7: (librados::IoCtxImpl::write(object_t const&, ceph::buffer::list&, unsigned long, unsigned long)+0x18d) [0x7f768d69ea2d] 
 8: (rados_write()+0xc1) [0x7f768d66cee1] 
 9: (ffi_call_unix64()+0x4c) [0x7f76903ebadc] 
 10: (ffi_call()+0x1fc) [0x7f76903eb40c] 
 11: (_ctypes_callproc()+0x48e) [0x7f76906025fe] 
 12: (()+0x15f9e) [0x7f7690603f9e] 
 13: (PyEval_EvalFrameEx()+0x1f36) [0x52e1e6] 
 14: (PyEval_EvalFrameEx()+0xc82) [0x52cf32] 
 15: (PyEval_EvalFrameEx()+0xc82) [0x52cf32] 
 16: python() [0x56d0aa] 
 17: python() [0x4d9854] 
 18: (PyEval_CallObjectWithKeywords()+0x6b) [0x4da20b] 
 19: python() [0x5872b2] 
 20: (()+0x8182) [0x7f769168c182] 
 21: (clone()+0x6d) [0x7f76913b930d] 
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. 
 terminate called after throwing an instance of 'ceph::FailedAssertion' 
 Aborted (core dumped) 
 ####################### 
 </pre> 

Back