Ceph » RADOS

root@f23-h17-000-6048r:/var/run/ceph
# ceph daemon osd.98 heap stats
Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 

root@f23-h17-000-6048r:/var/run/ceph
# ceph tell osd.98 heap stats
osd.98 tcmalloc heap stats:------------------------------------------------
MALLOC:     3725949760 ( 3553.3 MiB) Bytes in use by application
MALLOC: +      1056768 (    1.0 MiB) Bytes in page heap freelist
MALLOC: +    370342848 (  353.2 MiB) Bytes in central cache freelist
MALLOC: +     16070656 (   15.3 MiB) Bytes in transfer cache freelist
MALLOC: +     84439296 (   80.5 MiB) Bytes in thread cache freelists
MALLOC: +     33554432 (   32.0 MiB) Bytes in malloc metadata
MALLOC:   ------------
MALLOC: =   4231413760 ( 4035.4 MiB) Actual memory used (physical + swap)
MALLOC: +   4402339840 ( 4198.4 MiB) Bytes released to OS (aka unmapped)
MALLOC:   ------------
MALLOC: =   8633753600 ( 8233.8 MiB) Virtual address space used
MALLOC:
MALLOC:         389490              Spans in use
MALLOC:             39              Thread heaps in use
MALLOC:           8192              Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.

root@f23-h17-000-6048r:/var/run/ceph
# ceph --admin-daemon /var/run/ceph/0b67fc9a-1cc2-11ed-8fb3-000af7995d6c/ceph-osd.98.asok heap stats

root@f23-h17-000-6048r:/var/run/ceph
# ceph --admin-daemon /var/run/ceph/0b67fc9a-1cc2-11ed-8fb3-000af7995d6c/ceph-osd.98.asok heap stats

root@f28-h21-000-r630:~
# ceph tell mon.`hostname -s` heap stats
Error ENOENT: problem getting command descriptions from mon.f28-h21-000-r630

root@f28-h21-000-r630:~
# ceph tell mon.`hostname` heap stats
mon.f28-h21-000-r630.rdu2.scalelab.redhat.com tcmalloc heap stats:------------------------------------------------
MALLOC:      482852640 (  460.5 MiB) Bytes in use by application
MALLOC: +            0 (    0.0 MiB) Bytes in page heap freelist
MALLOC: +     23908208 (   22.8 MiB) Bytes in central cache freelist
MALLOC: +     14829824 (   14.1 MiB) Bytes in transfer cache freelist
MALLOC: +     39135344 (   37.3 MiB) Bytes in thread cache freelists
MALLOC: +      3407872 (    3.2 MiB) Bytes in malloc metadata
MALLOC:   ------------
MALLOC: =    564133888 (  538.0 MiB) Actual memory used (physical + swap)
MALLOC: +     89579520 (   85.4 MiB) Bytes released to OS (aka unmapped)
MALLOC:   ------------
MALLOC: =    653713408 (  623.4 MiB) Virtual address space used
MALLOC:
MALLOC:          15825              Spans in use
MALLOC:             22              Thread heaps in use
MALLOC:           8192              Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.

root@f28-h21-000-r630:~
# ceph --admin-daemon /var/run/ceph/0b67fc9a-1cc2-11ed-8fb3-000af7995d6c/ceph-mon.f28-h21-000-r630.rdu2.scalelab.redhat.com.asok heap stats
mon.f28-h21-000-r630.rdu2.scalelab.redhat.com tcmalloc heap stats:------------------------------------------------
MALLOC:      482828048 (  460.5 MiB) Bytes in use by application
MALLOC: +            0 (    0.0 MiB) Bytes in page heap freelist
MALLOC: +     24090272 (   23.0 MiB) Bytes in central cache freelist
MALLOC: +     14968064 (   14.3 MiB) Bytes in transfer cache freelist
MALLOC: +     38815056 (   37.0 MiB) Bytes in thread cache freelists
MALLOC: +      3407872 (    3.2 MiB) Bytes in malloc metadata
MALLOC:   ------------
MALLOC: =    564109312 (  538.0 MiB) Actual memory used (physical + swap)
MALLOC: +     89604096 (   85.5 MiB) Bytes released to OS (aka unmapped)
MALLOC:   ------------
MALLOC: =    653713408 (  623.4 MiB) Virtual address space used
MALLOC:
MALLOC:          15825              Spans in use
MALLOC:             22              Thread heaps in use
MALLOC:           8192              Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.

root@f28-h21-000-r630:~
# ceph daemon mon.`hostname` heap stats
admin_socket: exception getting command descriptions: [Errno 2] No such file or directory

Appending the stringstream to the "outbl" bufferlist here makes the output print with the "daemon" version, but it is printed twice with the "tell" version. So, not the ideal solution, but clearly the outbl needs to have the stringstream appended to it.
https://github.com/ceph/ceph/blob/042ca579c2e4fece1853503d982b6afc0a0478dc/src/osd/OSD.cc#L3163-L3165

 out:
+  outbl.append(ss.str());
  on_finish(ret, ss.str(), outbl);
}

The same change "works" for mds: https://github.com/ceph/ceph/blob/042ca579c2e4fece1853503d982b6afc0a0478dc/src/mds/MDSDaemon.cc#L209-L211

The interface between `outbl` and the formatter is described at the beginning of the function:

void OSD::asok_command(
  std::string_view prefix, const cmdmap_t& cmdmap,
  Formatter *f,
  const bufferlist& inbl,
  std::function<void(int,const std::string&,bufferlist&)> on_finish)
{
  int ret = 0;
  stringstream ss;   // stderr error message stream
  bufferlist outbl;  // if empty at end, we'll dump formatter as output

The implementation of the heap family of commands doesn't use outbl at all:

  else if (prefix == "heap") {
    ret = ceph::osd_cmds::heap(*cct, cmdmap, *f, ss);
  }

However, it takes the ss (intended to be a counterpart of stderr). Maybe cmds::heap() print there instead of formatted?

Oops, this is the case:

int heap(CephContext& cct, const cmdmap_t& cmdmap, Formatter& f,
         std::ostream& os)
{
  // ...
  ceph_heap_profiler_handle_command(cmd_vec, os);

  return 0;
}

It looks we have a misunderstanding between the caller and callee.

Similar problem may affect e.g. flush_store_cache:

  } else if (prefix == "flush_store_cache") {
    store->flush_cache(&ss);

There is another thing we should unify: daemon and tell differently treat ss (the error stream) when the error code says there was no error.

  hook->call_async(
    prefix, cmdmap, f, inbl,
    [f, on_finish](int r, const std::string& err, bufferlist& out) {
      // handle either existing output in bufferlist *or* via formatter
      if (r >= 0 && out.length() == 0) {
        f->flush(out);
      } 
      delete f;
      on_finish(r, err, out);
    });

For tell we always forward the error stuff to mon:

void AdminSocket::do_tell_queue()
{
  ldout(m_cct,10) << __func__ << dendl;
  std::list<cref_t<MCommand>> q; 
  std::list<cref_t<MMonCommand>> lq;
  { 
    std::lock_guard l(tell_lock);
    q.swap(tell_queue);
    lq.swap(tell_legacy_queue);
  }
  for (auto& m : q) {
    bufferlist outbl;
    execute_command(
      m->cmd,
      m->get_data(),
      [m](int r, const std::string& err, bufferlist& outbl) {
        auto reply = new MCommandReply(r, err);
        reply->set_tid(m->get_tid());
        reply->set_data(outbl);
#ifdef WITH_SEASTAR
        // TODO: crimson: handle asok commmand from alien thread
#else
        m->get_connection()->send_message(reply);
#endif
      });
  }
  for (auto& m : lq) {
    bufferlist outbl;
    execute_command(
      m->cmd,
      m->get_data(),
      [m](int r, const std::string& err, bufferlist& outbl) {
        auto reply = new MMonCommandAck(m->cmd, r, err, 0);
        reply->set_tid(m->get_tid());
        reply->set_data(outbl);
#ifdef WITH_SEASTAR
        // TODO: crimson: handle asok commmand from alien thread
#else
        m->get_connection()->send_message(reply);
#endif
      });
  }
}

For deamon we make use the error stream only if return code is < 0:

void AdminSocket::do_accept()
{
  // ...
  std::vector<std::string> cmdvec = { c };
  bufferlist empty, out;
  ostringstream err;
  int rval = execute_command(cmdvec, empty /* inbl */, err, &out);

  // Unfortunately, the asok wire protocol does not let us pass an error code,
  // and many asok command implementations return helpful error strings.  So,
  // let's prepend an error string to the output if there is an error code.
  if (rval < 0) {
    ostringstream ss;
    ss << "ERROR: " << cpp_strerror(rval) << "\n";
    ss << err.str() << "\n";
    bufferlist o;
    o.append(ss.str());
    o.claim_append(out);
    out.claim_append(o);
  }
  // ...

https://github.com/ceph/ceph/pull/47650 merged