Ceph » bluestore

(gdb) t 16
[Switching to thread 16 (Thread 0x7f6b8e13a240 (LWP 18023))]
#7  0x0000561ba5ead9a5 in StoreTest::doSyntheticTest (this=<optimized out>, num_ops=50000, max_obj=<optimized out>, max_wr=<optimized out>, align=<optimized out>) at /usr/src/debug/ceph-16.0.0-834.g3ef1bee.el8.x86_64/src/test/objectstore/store_test.cc:4664
4664          test_obj.clone_range();
(gdb) f
#7  0x0000561ba5ead9a5 in StoreTest::doSyntheticTest (this=<optimized out>, num_ops=50000, max_obj=<optimized out>, max_wr=<optimized out>, align=<optimized out>) at /usr/src/debug/ceph-16.0.0-834.g3ef1bee.el8.x86_64/src/test/objectstore/store_test.cc:4664
4664          test_obj.clone_range();

I dumped out test_obj and it does show some objects with _len = 4096. Need to verify if this is correct or an error.

    [{
      hobj = {
        static POOL_META = -1,
        static POOL_TEMP_START = -2,
        oid = {
          name = "OBJ_4154" 
        },
        snap = {
          val = 1049056086
        },
        hash = 0,
        max = false,
        nibblewise_key_cache = 0,
        hash_reverse_bits = 0,
        pool = 555,
        nspace = "",
        key = "" 
      },
      generation = 18446744073709551615,
      shard_id = {
        id = -1 '\377',
        static NO_SHARD = {
          id = -1 '\377',
          static NO_SHARD = <same as static member of an already seen type>
        }
      },
      max = false,
      static NO_GEN = 18446744073709551615
    }] = {
      data = {
        _buffers = {
          _root = {
            next = 0x561d1d5bd000
          },
          _tail = 0x561d1d5bd000
        },
        _carriage = 0x561ba67ae540 <ceph::buffer::v15_2_0::list::always_empty_bptr>,
        _len = 4096,
        _num = 1,
        static always_empty_bptr = {
          _raw = 0x0,
          _off = 0,
          _len = 0
        }
      },
      attrs = std::map with 0 elements
    },

The following back trace is likely to be the symptom for the same bug.

-1> 2020-04-24T16:25:20.971+0300 7efe4897e0c0 -1 /home/if/ceph/src/common/Checksummer.h: In function 'static int Checksummer::calculate(typename Alg::init_value_t, size_t, size_t, size_t, const ceph::buffer::v15_2_0::list&, ceph::buffer::v15_2_0::ptr*) [with Alg = Checksummer::crc32c; typename Alg::init_value_t = unsigned int; size_t = long unsigned int]' thread 7efe4897e0c0 time 2020-04-24T16:25:20.968742+0300
/home/if/ceph/src/common/Checksummer.h: 214: FAILED ceph_assert(length % csum_block_size == 0)
 ceph version Development (no_version) pacific (dev)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x151) [0x7efe495d71bb]
 2: (()+0x26e3a5) [0x7efe495d73a5]
 3: (bluestore_blob_t::calc_csum(unsigned long, ceph::buffer::v15_2_0::list const&)+0x546) [0x559f7192b526]
 4: (BlueStore::_do_write_big_apply_deferred(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>, boost::intrusive::tree_iterator<boost::intrusive::bhtraits<BlueStore::Extent, boost::intrusive::rbtree_node_traits<void*, true>, (boost::intrusive::link_mode_type)1, boost::intrusive::dft_tag, 3u>, false>, BlueStore::BigDeferredWriteContext&, ceph::buffer::v15_2_0::list::iterator&, BlueStore::WriteContext*)+0x526) [0x559f718cfdf6]
 5: (BlueStore::_do_write_big(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>, unsigned long, unsigned long, ceph::buffer::v15_2_0::list::iterator&, BlueStore::WriteContext*)+0x18ba) [0x559f718d1ada]
 6: (BlueStore::_do_write_data(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, BlueStore::WriteContext*)+0x176) [0x559f718d6776]

Igor, does this need to be backported? If so could you set the appropriate releases please? I know you set 16 as the affected version but just double checking, thanks.

Hi Brad,
no this is specific to master(Pacific) only

Well, this fix is required if/when 'deferring big writes' feature (https://github.com/ceph/ceph/pull/33434) is backported.