Ceph

pulling leveldb 1.12.0 for rhel made it impossible to use/deploy ceph on rhel6/el6 systems.

As already mentioned in #6022.

Same here. Mon doesn't restart after upgrading leveledb.

Have we made sure we are applying the patch mentioned here?

http://tracker.ceph.com/issues/6022#note-9

Did you pull it? I still see 1.12 here: http://ceph.com/rpm-dumpling/el6/x86_64/

I'm concurring. C6 system, Ceph 0.72.2, after reboot oh the host I experienced crashing mon and hanging OSDs. Downgrading leveldb to 1.7 fixed things.

Some info on the crash:
2014-05-02 18:18:02.342119 7ffc738917a0 0 ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60), process ceph-mon, pid 14851
2014-05-02 18:18:02.361653 7ffc738917a0 -1 ** Caught signal (Segmentation fault) *
in thread 7ffc738917a0

ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60)
 1: /usr/bin/ceph-mon() [0x805e69]
 2: /lib64/libpthread.so.0() [0x32c6c0f710]
 3: (leveldb::DBImpl::Get(leveldb::ReadOptions const&, leveldb::Slice const&, leveldb::Value*)+0x83) [0x7ffc738b3153]
 4: (LevelDBStore::_get_iterator()+0x41) [0x7761d1]
 5: (MonitorDBStore::exists(std::string const&, std::string const&)+0x28) [0x5275a8]
 6: (main()+0xac9) [0x521849]
 7: (__libc_start_main()+0xfd) [0x32c641ed1d]
 8: /usr/bin/ceph-mon() [0x51fa19]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
-22> 2014-05-02 18:18:02.339931 7ffc738917a0 5 asok(0x24a8850) register_command perfcounters_dump hook 0x2450010
-21> 2014-05-02 18:18:02.339963 7ffc738917a0 5 asok(0x24a8850) register_command 1 hook 0x2450010
-20> 2014-05-02 18:18:02.339970 7ffc738917a0 5 asok(0x24a8850) register_command perf dump hook 0x2450010
-19> 2014-05-02 18:18:02.339977 7ffc738917a0 5 asok(0x24a8850) register_command perfcounters_schema hook 0x2450010
-18> 2014-05-02 18:18:02.339983 7ffc738917a0 5 asok(0x24a8850) register_command 2 hook 0x2450010
-17> 2014-05-02 18:18:02.339988 7ffc738917a0 5 asok(0x24a8850) register_command perf schema hook 0x2450010
-16> 2014-05-02 18:18:02.339993 7ffc738917a0 5 asok(0x24a8850) register_command config show hook 0x2450010
-15> 2014-05-02 18:18:02.339998 7ffc738917a0 5 asok(0x24a8850) register_command config set hook 0x2450010
-14> 2014-05-02 18:18:02.340004 7ffc738917a0 5 asok(0x24a8850) register_command config get hook 0x2450010
-13> 2014-05-02 18:18:02.340009 7ffc738917a0 5 asok(0x24a8850) register_command log flush hook 0x2450010
-12> 2014-05-02 18:18:02.340014 7ffc738917a0 5 asok(0x24a8850) register_command log dump hook 0x2450010
-11> 2014-05-02 18:18:02.340019 7ffc738917a0 5 asok(0x24a8850) register_command log reopen hook 0x2450010
-10> 2014-05-02 18:18:02.342119 7ffc738917a0 0 ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60), process ceph-mon, pid 14851
-9> 2014-05-02 18:18:02.342735 7ffc738917a0 1 finished global_init_daemonize
-8> 2014-05-02 18:18:02.346330 7ffc738917a0 5 asok(0x24a8850) init /var/run/ceph/ceph-mon.ceph2.asok
-7> 2014-05-02 18:18:02.346353 7ffc738917a0 5 asok(0x24a8850) bind_and_listen /var/run/ceph/ceph-mon.ceph2.asok
-6> 2014-05-02 18:18:02.346443 7ffc738917a0 5 asok(0x24a8850) register_command 0 hook 0x24480b8
-5> 2014-05-02 18:18:02.346456 7ffc738917a0 5 asok(0x24a8850) register_command version hook 0x24480b8
-4> 2014-05-02 18:18:02.346461 7ffc738917a0 5 asok(0x24a8850) register_command git_version hook 0x24480b8
-3> 2014-05-02 18:18:02.346465 7ffc738917a0 5 asok(0x24a8850) register_command help hook 0x24500b0
-2> 2014-05-02 18:18:02.346468 7ffc738917a0 5 asok(0x24a8850) register_command get_command_descriptions hook 0x24500a0
-1> 2014-05-02 18:18:02.346506 7ffc71d3a700 5 asok(0x24a8850) entry start
0> 2014-05-02 18:18:02.361653 7ffc738917a0 -1 ** Caught signal (Segmentation fault) *
in thread 7ffc738917a0

ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60)
 1: /usr/bin/ceph-mon() [0x805e69]
 2: /lib64/libpthread.so.0() [0x32c6c0f710]
 3: (leveldb::DBImpl::Get(leveldb::ReadOptions const&, leveldb::Slice const&, leveldb::Value*)+0x83) [0x7ffc738b3153]
 4: (LevelDBStore::_get_iterator()+0x41) [0x7761d1]
 5: (MonitorDBStore::exists(std::string const&, std::string const&)+0x28) [0x5275a8]
 6: (main()+0xac9) [0x521849]
 7: (__libc_start_main()+0xfd) [0x32c641ed1d]
 8: /usr/bin/ceph-mon() [0x51fa19]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-mon.ceph2.log
-- end dump of recent events ---

1.12 is gone from the dumpling repo now

1.12 is in Fedora, which is what RHEL will be using. Can we close this?

Which RHEL version will be using 1.12? I thought this was about providing something newer than the old/buggy 1.7 in epel for rhel6.

The history of 1.12 on Centos/RHEL has been that it breaks. There were aspersions cast on one of the patches from a Fedora build, but the failure was so drastic that I do not agree we know what was going wrong or why.

A Ceph developer needs to investigate and run a complete regression before we move versions; leveldb is central and the tests need to really stress the system to validate a change. We've been burned in the past.