Actions
Bug #9592
closedlibrados: Not able to create Large Files with Librados
% Done:
100%
Source:
other
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
I find this issue while i was trying to run a 1GB Write Cosbench Workload using librados.(My 1MB write & read run was successful)
The Run hang permanently and nothing written to cluster, however I observed client is sending the data (n/w bandwidth usage was 5-10Gbits).
AFAIK, COSbench uses java binding for Librados. I debugged further with python binding.
After installing ceph-python , I executed following program:-
######################## import rados try: cluster = rados.Rados(conffile='/etc/ceph/ceph.conf') except TypeError as e: print 'Argument validation error: ', e raise e print "Created cluster handle." try: cluster.connect() except Exception as e: print "connection error: ", e raise e finally: print "Connected to the cluster." print "\n\nI/O Context and Object Operations" print "=================================" print "\nCreating a context for the 'data' pool" if not cluster.pool_exists('data'): raise RuntimeError('No data pool exists') ioctx = cluster.open_ioctx('data') print "\nWriting object 'bm' with contents 'Bonjour tout le monde!' to pool 'data'." ioctx.write("bm", "Bonjour tout le monde!") print "Writing XATTR 'lang' with value 'fr_FR' to object 'bm'" ioctx.set_xattr("bm", "lang", "fr_FR") print "\nWriting object 'bm1' with contents 'Bonjour tout le monde!' to pool 'data'." ioctx.write("bm1", "Bonjour tout le monde!"*10000) print "Writing XATTR 'lang' with value 'fr_FR' to object 'bm1'" ioctx.set_xattr("bm1", "lang", "fr_FR") print "\nWriting object 'hw' with contents 'Hello World!' to pool 'data'." ioctx.write("hw", "Hello World!\n world is not enough"*1000000000) print "Writing XATTR 'lang' with value 'en_US' to object 'hw'" ioctx.set_xattr("hw", "lang", "en_US") ''' print "\nContents of object 'hw'\n------------------------" print ioctx.read("hw") print "\n\nGetting XATTR 'lang' from object 'hw'" print ioctx.get_xattr("hw", "lang") print "\nContents of object 'bm'\n------------------------" print ioctx.read("bm") print "Getting XATTR 'lang' from object 'bm'" print ioctx.get_xattr("bm", "lang") ''' print "Removing object 'bm'" ioctx.remove_object("bm") print "Removing object 'bm1'" ioctx.remove_object("bm1") print "\nRemoving object 'hw'" ioctx.remove_object("hw")
I got the following output:-
ems@rack6-ramp-1:~$ python ceph-client.py Created cluster handle. Connected to the cluster. I/O Context and Object Operations ================================= Creating a context for the 'data' pool Writing object 'bm' with contents 'Bonjour tout le monde!' to pool 'data'. Writing XATTR 'lang' with value 'fr_FR' to object 'bm' Writing object 'bm1' with contents 'Bonjour tout le monde!' to pool 'data'. Writing XATTR 'lang' with value 'fr_FR' to object 'bm1' Writing object 'hw' with contents 'Hello World!' to pool 'data'. common/Throttle.cc: In function 'bool Throttle::get_or_fail(int64_t)' thread 7f768a7d0700 time 2014-09-25 18:37:08.003401 common/Throttle.cc: 194: FAILED assert(c >= 0) ceph version 0.84 (3215c520e1306f50d0094b5646636c02456c9df4) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7f768d73d71b] 2: (Throttle::get_or_fail(long)+0x5ae) [0x7f768d733d0e] 3: (Objecter::_throttle_op(Objecter::Op*, int)+0x68) [0x7f768d6ae588] 4: (Objecter::_op_submit_with_budget(Objecter::Op*, RWLock::Context&)+0x195) [0x7f768d6bf785] 5: (Objecter::op_submit(Objecter::Op*)+0x96) [0x7f768d6bf8c6] 6: (librados::IoCtxImpl::operate(object_t const&, ObjectOperation*, long*, int)+0x33d) [0x7f768d698a0d] 7: (librados::IoCtxImpl::write(object_t const&, ceph::buffer::list&, unsigned long, unsigned long)+0x18d) [0x7f768d69ea2d] 8: (rados_write()+0xc1) [0x7f768d66cee1] 9: (ffi_call_unix64()+0x4c) [0x7f76903ebadc] 10: (ffi_call()+0x1fc) [0x7f76903eb40c] 11: (_ctypes_callproc()+0x48e) [0x7f76906025fe] 12: (()+0x15f9e) [0x7f7690603f9e] 13: (PyEval_EvalFrameEx()+0x1f36) [0x52e1e6] 14: (PyEval_EvalFrameEx()+0xc82) [0x52cf32] 15: (PyEval_EvalFrameEx()+0xc82) [0x52cf32] 16: python() [0x56d0aa] 17: python() [0x4d9854] 18: (PyEval_CallObjectWithKeywords()+0x6b) [0x4da20b] 19: python() [0x5872b2] 20: (()+0x8182) [0x7f769168c182] 21: (clone()+0x6d) [0x7f76913b930d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. terminate called after throwing an instance of 'ceph::FailedAssertion' Aborted (core dumped) #######################
Files
Actions