Ceph » mgr

It seems in each of these nautilus-p2p parallel runs, 'ceph pg dump' was run after the ceph cli was upgraded to nautilus, but ceph-mgr was not restarted, so it was still running 14.2.2.

In each case, the command times out, but the mgr and audit log have no record of this command at the same time.

Yuri is re-running with some client debugging here: http://pulpito.ceph.com/yuriw-2019-08-26_22:39:26-upgrade:nautilus-p2p:nautilus-p2p-parallel-wip_nautilus_14.2.3_RC1-distro-basic-smithi/

Reproduced with a manual run. Automatic respawning failed with no trace, but starting ceph-mgr again manually worked fine.

2019-08-26T23:21:21.949 INFO:teuthology.orchestra.run.smithi071.stdout:  ceph-mgr.x86_64 2:14.2.2-469.ge157074.el7
...
2019-08-26T23:21:21.951 INFO:teuthology.orchestra.run.smithi071.stdout:  librados2.x86_64 2:14.2.2-469.ge157074.el7
...
2019-08-26T23:31:35.431 INFO:tasks.ceph.mgr.x.smithi071.stderr:ceph-mgr: symbol lookup error: ceph-mgr: undefined symbol: _ZTIN4ceph6buffer5errorE

$ c++filt _ZTIN4ceph6buffer5errorE
typeinfo for ceph::buffer::error

master|$ nm -C ./lib/libceph-common.so | grep 'typeinfo for ceph::buffer.*error'
0000000002299df8 V typeinfo for ceph::buffer::v14_2_0::error_code
0000000002299e58 V typeinfo for ceph::buffer::v14_2_0::error

1. in 14.2.2, mgr expects ceph::buffer::error in libceph-common.so
2. in nautilus (14.2.3), we have 4ac8bc79e7fc128f10b38457c93b092b5c8060ad backported in https://github.com/ceph/ceph/pull/29244. so the ceph::buffer::error are actually exported as "ceph::buffer::v14_2_0::error".

and the way how we find the executable prevents ceph-mgr from using the new one just gets installed.

  /* Determine the path to our executable, test if Linux /proc/self/exe exists.
   * This allows us to exec the same executable even if it has since been
   * unlinked.
   */
  char exe_path[PATH_MAX] = "";
  if (readlink(PROCPREFIX "/proc/self/exe", exe_path, PATH_MAX-1) == -1) {
    /* Print CWD for the user's interest */
    char buf[PATH_MAX];
    char *cwd = getcwd(buf, sizeof(buf));
    ceph_assert(cwd);
    dout(1) << " cwd " << cwd << dendl;

    /* Fall back to a best-effort: just running in our CWD */
    strncpy(exe_path, orig_argv[0], PATH_MAX-1);
  } else {
    dout(1) << "respawning with exe " << exe_path << dendl;
    strcpy(exe_path, PROCPREFIX "/proc/self/exe");
  }

that's why the log shows

2019-08-26 23:31:35.310 7f2ba1146700  1 mgr respawn respawning with exe /usr/bin/ceph-mgr;5d646944 (deleted)

we were execv'ing an already deleted executable, like reviving a zombie...

IRC conversation:

2019-08-28 08:08:45 batrick joshd: kefu: it looks like the issue is that the old mgr executable can't use the new ceph-common shared library, right?
2019-08-28 08:09:00 joshd   yes
2019-08-28 08:09:08 batrick mds will probably have similar issues then
2019-08-28 08:09:27 agil    pg 4.1e is stuck inactive for 60643.351410, current state unknown, last acting []
2019-08-28 08:09:32 batrick at the time it seemed like a good use of /proc/self/exe to prevent accidental upgrades of the mds during respawn
2019-08-28 08:09:37 agil    any hint on how to remove them?
2019-08-28 08:09:39 joshd   batrick: can do the same fix there if you like
2019-08-28 08:10:57 batrick it doesn't matter too much
2019-08-28 08:11:10 batrick ceph should really run containerized where this would be a non-issue
2019-08-28 08:16:56 --> vtheile_ (~vtheile@nat.nue.novell.com) has joined #ceph-devel
2019-08-28 08:18:35 --> ivotron (~ivotron@177.228.76.97) has joined #ceph-devel
2019-08-28 08:23:51 batrick joshd: looking at this test more closely
2019-08-28 08:24:04 batrick why are we upgrading the cluster starting with mds, then osd ,then mon?
2019-08-28 08:24:18 batrick  /ceph/teuthology-archive/yuriw-2019-08-26_22:39:26-upgrade:nautilus-p2p:nautilus-p2p-parallel-wip_nautilus_14.2.3_RC1-distro-basic-smithi/4255489
2019-08-28 08:24:28 joshd   batrick: we do different orders in different suites to verify no crashes
2019-08-28 08:24:42 joshd   even if some of them aren't the ideal order
2019-08-28 08:26:09 batrick looks like the MgrMap modules changed which triggered the respawn
2019-08-28 08:26:38 --> jcollin (~jcollin@122.167.109.70) has joined #ceph-devel
2019-08-28 08:26:59 batrick it concerns me that this test doesn't even have an explicit restart the mgr to upgrade
2019-08-28 08:26:59 joshd   that's right
2019-08-28 08:27:10 batrick but the mgr is upgrading as a side-effect of the monitors updating the mgrmap
2019-08-28 08:27:26 joshd   I agree, I think it's an oversight
2019-08-28 08:28:04 batrick since this /proc/self/exe has  not been a problem until now, perhaps the right fix is to update the upgrade sequence to restart the mgr
2019-08-28 08:28:35 batrick separately maybe we can discuss changing the execve executable if it's not desirable
2019-08-28 08:29:31 joshd   that would be even simpler
2019-08-28 08:29:55 joshd   I think that's a good idea, since in practice systemd will end up restarting the failed process as well

https://github.com/ceph/ceph/pull/29977