Project

General

Profile

Bug #43582

rebuild-mondb doesn't populate mgr commands -> pg dump EINVAL

Added by Sage Weil about 1 month ago. Updated 6 days ago.

Status:
Pending Backport
Priority:
Urgent
Assignee:
Category:
-
Target version:
% Done:

0%

Source:
Tags:
Backport:
luminous,mimic,nautilus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature:

Description

2020-01-12T22:08:28.772 INFO:teuthology.orchestra.run.smithi199:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json
2020-01-12T22:08:28.951 INFO:teuthology.orchestra.run.smithi199.stderr:Error EINVAL: unrecognized command

it came from the mon:
teuthology:4660678  01:59 PM $ grep command remote/*/log/*mon* | grep -- -22
remote/smithi199/log/ceph-mon.a.log:2020-01-12T22:08:28.947+0000 7fceb6c53700  2 mon.a@0(leader) e1 send_reply 0x5579286a2690 0x5579286a9440 mon_command_ack([{"prefix": "pg dump", "format": "json"}]=-22 unrecognized command v0) v1
remote/smithi199/log/ceph-mon.a.log:2020-01-12T22:08:28.947+0000 7fceb6c53700  1 -- [v2:172.21.15.199:3300/0,v1:172.21.15.199:6789/0] --> 172.21.15.199:0/4164394704 -- mon_command_ack([{"prefix": "pg dump", "format": "json"}]=-22 unrecognized command v0) v1 -- 0x5579286a9440 con 0x557928520d80

presumably because the mgr commands weren't populated as they are on cluster creation, see mon/MgrMonitor.cc
void MgrMonitor::create_initial()
{
  // Take a local copy of initial_modules for tokenizer to iterate over.
  auto initial_modules = g_conf().get_val<std::string>("mgr_initial_modules");
  boost::tokenizer<> tok(initial_modules);
  for (auto& m : tok) {
    pending_map.modules.insert(m);
  }
  pending_map.always_on_modules = always_on_modules;
  pending_command_descs = mgr_commands;
  dout(10) << __func__ << " initial modules " << pending_map.modules
       << ", always on modules " << pending_map.get_always_on_modules()
           << ", " << pending_command_descs.size() << " commands" 
       << dendl;
}

/a/sage-2020-01-12_21:37:03-rados-wip-sage-testing-2020-01-12-0621-distro-basic-smithi/4660678

Related issues

Copied to RADOS - Backport #44084: luminous: rebuild-mondb doesn't populate mgr commands -> pg dump EINVAL New
Copied to RADOS - Backport #44085: nautilus: rebuild-mondb doesn't populate mgr commands -> pg dump EINVAL In Progress
Copied to RADOS - Backport #44086: mimic: rebuild-mondb doesn't populate mgr commands -> pg dump EINVAL New

History

#1 Updated by Kefu Chai about 1 month ago

  • Assignee set to Kefu Chai

#2 Updated by Kefu Chai about 1 month ago

i double checked update_mgrmap() in ceph_monstore_tool.cc. which is called when handling rebuild subcommand. will try to reproduce this issue.

#3 Updated by Kefu Chai about 1 month ago

  • Status changed from New to In Progress

#4 Updated by Sage Weil 24 days ago

/a/sage-2020-01-23_15:27:54-rados-wip-sage2-testing-2020-01-23-0635-distro-basic-smithi/4696610

#5 Updated by Sage Weil 21 days ago

  • Target version set to v15.0.0

#6 Updated by Kefu Chai 7 days ago

  • Pull request ID set to 33172

#7 Updated by Kefu Chai 7 days ago

  • Status changed from In Progress to Fix Under Review

#8 Updated by Kefu Chai 7 days ago

  • Backport set to luminous,mimic,nautilus

#9 Updated by Kefu Chai 6 days ago

  • Status changed from Fix Under Review to Pending Backport

#10 Updated by Nathan Cutler 5 days ago

  • Copied to Backport #44084: luminous: rebuild-mondb doesn't populate mgr commands -> pg dump EINVAL added

#11 Updated by Nathan Cutler 5 days ago

  • Copied to Backport #44085: nautilus: rebuild-mondb doesn't populate mgr commands -> pg dump EINVAL added

#12 Updated by Nathan Cutler 5 days ago

  • Copied to Backport #44086: mimic: rebuild-mondb doesn't populate mgr commands -> pg dump EINVAL added

Also available in: Atom PDF