Project

General

Profile

Bug #47211

nautilus: unrecognised rocksdb_option crashes osd process while starting the osd

Added by Pietari Hyvärinen 19 days ago. Updated 1 day ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Target version:
% Done:

0%

Source:
Community (user)
Tags:
Backport:
nautilus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

while upgrading cluster from 14.2.9 to 14.2.11 unrecognised (legacy?) rocksdb_option crashes osd process while starting the osd (nautilus)

Steps to produce the error:

# ceph config  set osd bluestore_rocksdb_options compression=kNoCompression,max_write_buffer_number=32,min_write_buffer_number_to_merge=2,recycle_log_file_num=32,compaction_style=kCompactionStyleLevel,write_buffer_size=67108864,target_file_size_base=67108864,max_background_compactions=31,level0_file_num_compaction_trigger=8,level0_slowdown_writes_trigger=32,level0_stop_writes_trigger=64,max_bytes_for_level_base=536870912,max_bytes_for_level_multiplier=8,flusher_threads=8,compaction_readahead_size=2MB

# systemctl restart ceph-osd@XXX

(osd enters failed state)

dump from cluster:

<code class="text">
    -3> 2020-08-31 13:33:27.933 7efeca353a80 -1 rocksdb: Invalid argument: Can't parse option flusher_threads
    -2> 2020-08-31 13:33:27.933 7efeca353a80  1 bluestore(/var/lib/ceph/osd/ceph-699) _upgrade_super from 0, latest 2
    -1> 2020-08-31 13:33:27.935 7efeca353a80 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/BlueStore.cc: In function 'int BlueStore::_upgrade_super()' thread 7efeca353a80 time 2020-08-31 13:33:27.934832
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/os/bluestore/BlueStore.cc: 10249: FAILED ceph_assert(ondisk_format > 0)

 ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x5651c83840e5]
 2: (()+0x4d72ad) [0x5651c83842ad]
 3: (BlueStore::_upgrade_super()+0x52b) [0x5651c88b0bfb]
 4: (BlueStore::_mount(bool, bool)+0x5d3) [0x5651c88f3453]
 5: (OSD::init()+0x326) [0x5651c848b866]
 6: (main()+0x1bf8) [0x5651c83ebb78]
 7: (__libc_start_main()+0xf5) [0x7efec618f495]
 8: (()+0x573c85) [0x5651c8420c85]

     0> 2020-08-31 13:33:27.938 7efeca353a80 -1 *** Caught signal (Aborted) **
 in thread 7efeca353a80 thread_name:ceph-osd

 ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable)
 1: (()+0xf5d0) [0x7efec73ac5d0]
 2: (gsignal()+0x37) [0x7efec61a32c7]
 3: (abort()+0x148) [0x7efec61a49b8]
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x5651c8384134]
 5: (()+0x4d72ad) [0x5651c83842ad]
 6: (BlueStore::_upgrade_super()+0x52b) [0x5651c88b0bfb]
 7: (BlueStore::_mount(bool, bool)+0x5d3) [0x5651c88f3453]
 8: (OSD::init()+0x326) [0x5651c848b866]
 9: (main()+0x1bf8) [0x5651c83ebb78]
 10: (__libc_start_main()+0xf5) [0x7efec618f495]
 11: (()+0x573c85) [0x5651c8420c85]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
</code>

fix: (remove flusher_threads=8)

<code class="text">
# ceph config  set osd bluestore_rocksdb_options compression=kNoCompression,max_write_buffer_number=32,min_write_buffer_number_to_merge=2,recycle_log_file_num=32,compaction_style=kCompactionStyleLevel,write_buffer_size=67108864,target_file_size_base=67108864,max_background_compactions=31,level0_file_num_compaction_trigger=8,level0_slowdown_writes_trigger=32,level0_stop_writes_trigger=64,max_bytes_for_level_base=536870912,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB
# systemctl restart ceph-osd@XXX
</code>

Same applies to compaction_threads=32 option.


Related issues

Copied to bluestore - Backport #47521: nautilus: unrecognised rocksdb_option crashes osd process while starting the osd Resolved

History

#1 Updated by Igor Fedotov 11 days ago

  • Status changed from New to Fix Under Review
  • Pull request ID set to 37055

Missed backport, specific to Nautilus.

#2 Updated by Neha Ojha 9 days ago

  • Subject changed from unrecognised rocksdb_option crashes osd process while starting the osd to nautilus: unrecognised rocksdb_option crashes osd process while starting the osd

#4 Updated by Igor Fedotov 4 days ago

  • Status changed from Fix Under Review to Resolved

#5 Updated by Nathan Cutler 2 days ago

  • Status changed from Resolved to Pending Backport
  • Backport set to nautilus

#6 Updated by Nathan Cutler 2 days ago

  • Pull request ID changed from 37055 to 32453

#7 Updated by Nathan Cutler 2 days ago

  • Copied to Backport #47521: nautilus: unrecognised rocksdb_option crashes osd process while starting the osd added

#8 Updated by Nathan Cutler 1 day ago

  • Status changed from Pending Backport to Resolved

While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".

Also available in: Atom PDF