Ceph

From another osd, two threads actually ran into trouble in that method:

Thread 1209 (Thread 0x7f66d74a4700 (LWP 602689)):
#0 0x00007f66fedc129b in tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) () from /lib64/libtcmalloc.so.4
#1 0x00007f66fedc15b0 in tcmalloc::ThreadCache::Scavenge() () from /lib64/libtcmalloc.so.4
#2 0x00007f66fedd0e67 in tc_delete () from /lib64/libtcmalloc.so.4
#3 0x0000000000b8f0e9 in pretty_version_to_str() ()
#4 0x00000000009f81eb in ceph::BackTrace::print(std::ostream&) ()
#5 0x00000000009f642f in handle_fatal_signal(int) ()
#6 <signal handler called>
#7 0x000000000061d19e in OSD::ms_get_authorizer(int, AuthAuthorizer**, bool) ()
#8 0x0000000000acb5db in SimpleMessenger::get_authorizer(int, bool) ()
#9 0x0000000000bd0cf9 in Pipe::connect() ()
#10 0x0000000000bd4301 in Pipe::writer() ()
#11 0x0000000000bdfbdd in Pipe::Writer::entry() ()
#12 0x00007f66fe622df5 in start_thread () from /lib64/libpthread.so.0
#13 0x00007f66fd1051ad in clone () from /lib64/libc.so.6

in addition to

#6 0x00007f66fd1051ad in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f666c5e1700 (LWP 1007837)):
#0 0x000000000061d19e in OSD::ms_get_authorizer(int, AuthAuthorizer**, bool) ()
#1 0x0000000000acb5db in SimpleMessenger::get_authorizer(int, bool) ()
#2 0x0000000000bd0cf9 in Pipe::connect() ()
#3 0x0000000000bd4301 in Pipe::writer() ()
#4 0x0000000000bdfbdd in Pipe::Writer::entry() ()
#5 0x00007f66fe622df5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f66fd1051ad in clone () from /lib64/libc.so.6

145 (in the summary) also has the same pattern

Thread 1459 (Thread 0x7fa3eb833700 (LWP 604327)):
#0 0x00007fa42acfc819 in _dl_fixup () from /lib64/ld-linux-x86-64.so.2
#1 0x00007fa42ad032e0 in _dl_runtime_resolve () from /lib64/ld-linux-x86-64.so.2
#2 0x00000000009f63f2 in handle_fatal_signal(int) ()
#3 <signal handler called>
#4 0x000000000061d19e in OSD::ms_get_authorizer(int, AuthAuthorizer**, bool) ()
#5 0x0000000000acb5db in SimpleMessenger::get_authorizer(int, bool) ()
#6 0x0000000000bd0cf9 in Pipe::connect() ()
#7 0x0000000000bd4301 in Pipe::writer() ()
#8 0x0000000000bdfbdd in Pipe::Writer::entry() ()
#9 0x00007fa429eb3df5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fa4289961ad in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7fa3f69dd700 (LWP 1002130)):
#0 0x000000000061d19e in OSD::ms_get_authorizer(int, AuthAuthorizer**, bool) ()
#1 0x0000000000acb5db in SimpleMessenger::get_authorizer(int, bool) ()
#2 0x0000000000bd0cf9 in Pipe::connect() ()
#3 0x0000000000bd4301 in Pipe::writer() ()
#4 0x0000000000bdfbdd in Pipe::Writer::entry() ()
#5 0x00007fa429eb3df5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007fa4289961ad in clone () from /lib64/libc.so.6

Is there anything which actually protects from a race between the monc->auth access in OSD::ms_get_authorizor and the bit in MonClient::authenticate where we switch it out?

On one mon node:

[global]
rbd cache writethrough until flush = false
osd crush chooseleaf type = 0
osd crush update on start = false
osd pg bits = 10
osd pgp bits = 10
auth client required = none
auth cluster required = none
auth service required = none
keyring = /tmp/cbt/ceph/keyring
log to syslog = false
log file = /tmp/cbt/ceph/log/$name.log
rbd cache = true
filestore merge threshold = 40
filestore split multiple = 8
osd op threads = 8
mon pg warn max object skew = 100000
mon pg warn min per osd = 0
mon pg warn max per osd = 32768
[mon.a]
mon addr = 192.168.32.1:6789
host = gqac022.sbu.lab.eng.bos.redhat.com
mon data = /tmp/cbt/ceph/mon.$id

[osd.0]
host = gqas012.sbu.lab.eng.bos.redhat.com
osd data = /tmp/cbt/mnt/osd-device-0-data
osd journal = /dev/disk/by-partlabel/osd-device-0-journal
[osd.1]
host = gqas012.sbu.lab.eng.bos.redhat.com
osd data = /tmp/cbt/mnt/osd-device-1-data
osd journal = /dev/disk/by-partlabel/osd-device-1-journal
[osd.2]
host = gqas012.sbu.lab.eng.bos.redhat.com
osd data = /tmp/cbt/mnt/osd-device-2-data
osd journal = /dev/disk/by-partlabel/osd-device-2-journal
[osd.3]
host = gqas012.sbu.lab.eng.bos.redhat.com
osd data = /tmp/cbt/mnt/osd-device-3-data
osd journal = /dev/disk/by-partlabel/osd-device-3-journal
[osd.4]
host = gqas012.sbu.lab.eng.bos.redhat.com
osd data = /tmp/cbt/mnt/osd-device-4-data
osd journal = /dev/disk/by-partlabel/osd-device-4-journal
[osd.5]
host = gqas012.sbu.lab.eng.bos.redhat.com
osd data = /tmp/cbt/mnt/osd-device-5-data
osd journal = /dev/disk/by-partlabel/osd-device-5-journal
[osd.6]
host = gqas012.sbu.lab.eng.bos.redhat.com
osd data = /tmp/cbt/mnt/osd-device-6-data
osd journal = /dev/disk/by-partlabel/osd-device-6-journal
[osd.7]
host = gqas012.sbu.lab.eng.bos.redhat.com
osd data = /tmp/cbt/mnt/osd-device-7-data
osd journal = /dev/disk/by-partlabel/osd-device-7-journal
[osd.8]
host = gqas012.sbu.lab.eng.bos.redhat.com
osd data = /tmp/cbt/mnt/osd-device-8-data
osd journal = /dev/disk/by-partlabel/osd-device-8-journal
[osd.9]
host = gqas012.sbu.lab.eng.bos.redhat.com
osd data = /tmp/cbt/mnt/osd-device-9-data
osd journal = /dev/disk/by-partlabel/osd-device-9-journal
[osd.10]
host = gqas012.sbu.lab.eng.bos.redhat.com
osd data = /tmp/cbt/mnt/osd-device-10-data
osd journal = /dev/disk/by-partlabel/osd-device-10-journal
[osd.11]
host = gqas012.sbu.lab.eng.bos.redhat.com
osd data = /tmp/cbt/mnt/osd-device-11-data
osd journal = /dev/disk/by-partlabel/osd-device-11-journal

[osd.12]
host = gqas003.sbu.lab.eng.bos.redhat.com
osd data = /tmp/cbt/mnt/osd-device-0-data
osd journal = /dev/disk/by-partlabel/osd-device-0-journal
[osd.13]
host = gqas003.sbu.lab.eng.bos.redhat.com
osd data = /tmp/cbt/mnt/osd-device-1-data
osd journal = /dev/disk/by-partlabel/osd-device-1-journal
[osd.14]
host = gqas003.sbu.lab.eng.bos.redhat.com
osd data = /tmp/cbt/mnt/osd-device-2-data
osd journal = /dev/disk/by-partlabel/osd-device-2-journal
[osd.15]
host = gqas003.sbu.lab.eng.bos.redhat.com
osd data = /tmp/cbt/mnt/osd-device-3-data
osd journal = /dev/disk/by-partlabel/osd-device-3-journal
[osd.16]
host = gqas003.sbu.lab.eng.bos.redhat.com
osd data = /tmp/cbt/mnt/osd-device-4-data
osd journal = /dev/disk/by-partlabel/osd-device-4-journal
[osd.17]
host = gqas003.sbu.lab.eng.bos.redhat.com
osd data = /tmp/cbt/mnt/osd-device-5-data
osd journal = /dev/disk/by-partlabel/osd-device-5-journal
[osd.18]
host = gqas003.sbu.lab.eng.bos.redhat.com
osd data = /tmp/cbt/mnt/osd-device-6-data
osd journal = /dev/disk/by-partlabel/osd-device-6-journal
[osd.19]
host = gqas003.sbu.lab.eng.bos.redhat.com
osd data = /tmp/cbt/mnt/osd-device-7-data
osd journal = /dev/disk/by-partlabel/osd-device-7-journal
[osd.20]
host = gqas003.sbu.lab.eng.bos.redhat.com
...

On the other:

[global]
fsid = b65cb329-7e2b-4507-a78a-347095aa9269
mon_initial_members = gqac022-priv, gqac026-priv, gqac014-priv
mon_host = 192.168.32.1,192.168.32.43,192.168.32.44
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true

cephx appears to be enabled on one but not on the other. Can this cause the monc to reset the auth pointer and cause the race?

• harold|brb is now known as harold
  <sjusthm> that triggered a race condition as the monclient switched out the authorizer object
  <sjusthm> and the osd crashed
  <bennyturns> sounds like a good fit!
  <sjusthm> right before it would have rejected the mon (the authorizer would have failed, of course)
• AcroBot___ has quit (Remote host closed the connection)
  <sjusthm> bennyturns: if you can disprove this with clocks, please do so
  <sjusthm> but I think this isn't a big deal
  <sjusthm> (not that we won't fix it...)

The sequence of events as far as I can tell is
1) Cluster happy
2) gqac022 reimaged to serve as mon in smaller cluster with auth disabled
3) Some osds try to connect to that mon and crash due to cephx mismatch, perhaps due a race between OSD::ms_get_authorizer and MonClient::handle_auth (no evidence of MonClient::handle_auth in backtraces).