Project

General

Profile

Bug #37706

list-inconsistent-pg fails with EINVAL

Added by David Zafman over 5 years ago. Updated over 5 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
Scrub/Repair
Target version:
-
% Done:

0%

Source:
Development
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Seen in run-standalone.sh runs of osd-scrub-snaps.sh and osd-scrub-repair.sh:

../qa/standalone/scrub/osd-scrub-repair.sh:3108: corrupt_scrub_erasure:  rados list-inconsistent-pg ecpool
error 22: (22) Invalid argument
../qa/standalone/scrub/osd-scrub-repair.sh:3108: corrupt_scrub_erasure:  return 1
../qa/standalone/scrub/osd-scrub-repair.sh:62: run:  return 1

Teuthology run

/a/sage-2018-12-18_03:59:26-rados-master-distro-basic-smithi/3373050


Related issues

Related to RADOS - Bug #37705: list-inconsistent-pg fails with EINVAL Closed 12/18/2018
Related to RADOS - Bug #37714: test_dump_pgstate_history: Can't find expected values in history object, failing Resolved 12/19/2018

History

#1 Updated by David Zafman over 5 years ago

Thread 1 "rados" hit Breakpoint 2, librados::RadosClient::get_inconsistent_pgs (this=0x5555559f6f20, pool_id=1, pgs=0x7fffffffd6c0)
    at /home/dzafman/ceph/src/librados/RadosClient.cc:1126
1126    {
(gdb) n
1129        "\"pool\": " + std::to_string(pool_id) + "," 
(gdb)
1132      };
(gdb)
1133      bufferlist inbl, outbl;
(gdb)
1134      string outstring;
(gdb)
1135      if (auto ret = mgr_command(cmd, inbl, &outbl, &outstring); ret) {
(gdb)
1138      if (!outbl.length()) {
(gdb)
1142      JSONParser parser;
(gdb)
1143      if (!parser.parse(outbl.c_str(), outbl.length())) {
(gdb)
1146      vector<string> v;
(gdb) print parser
$1 = {<JSONObj> = {_vptr.JSONObj = 0x7fffef1fad28 <vtable for JSONParser+16>, parent = 0x0, name = "", data = {static null = <optimized out>, v_ = {which_ = 0,
        storage_ = {<boost::detail::aligned_storage::aligned_storage_imp<32, 8>> = {data_ = {
              buf = "\340d\251UUU\000\000`\324\377\377\377\177\000\000\000\000\000\000\000\000\000\000\300\344\377\367\377\177\000", align_ = {<No data fields>}}},
          static size = <optimized out>, static alignment = <optimized out>}}}, val = {str = "", quoted = false}, data_quoted = false, children = std::multimap with 2 elements = {
      ["pg_ready"] = 0x555555a998e0, ["pg_stats"] = 0x555555aa1aa0}, attr_map = std::map with 0 elements}, buf_len = 0, json_buffer = "", success = true}
(gdb) n
1147      if (!parser.is_array()) {
(gdb) n
1148        JSONObj *pgstat_obj = parser.find_obj("pg_stats");
(gdb) n
1149        if (!pgstat_obj)
(gdb) print *pgstat_obj
$2 = {_vptr.JSONObj = 0x7fffef1fad48 <vtable for JSONObj+16>, parent = 0x7fffffffd400, name = "pg_stats", data = {static null = <optimized out>, v_ = {which_ = 1,
      storage_ = {<boost::detail::aligned_storage::aligned_storage_imp<32, 8>> = {data_ = {buf = "\320[\252UUU", '\000' <repeats 25 times>, align_ = {<No data fields>}}},
        static size = <optimized out>, static alignment = <optimized out>}}}, val = {str = "", quoted = false}, data_quoted = false, children = std::multimap with 1 element = {
    [""] = 0x555555aabbc0}, attr_map = std::map with 1 element = {["pg_stats"] = {str = "", quoted = false}}}
(gdb) n
1151        auto s = pgstat_obj->get_data();
(gdb) n
1152        JSONParser pg_stats;
(gdb) n
1153        if (!pg_stats.parse(s.c_str(), s.length())) {
(gdb) n
1154          return -EINVAL;
(gdb) print s
$3 = "" 

#2 Updated by David Zafman over 5 years ago

I was suspicious of https://github.com/ceph/ceph/pull/23298, so I reverted all 74 commits and the problem wouldn't reproduce. The JSONParser::parse() function is in libradospp.so which doesn't have debug information so I can't walk through it in gdb easily.

#3 Updated by Yehuda Sadeh over 5 years ago

  • Status changed from In Progress to Fix Under Review

#4 Updated by David Zafman over 5 years ago

  • Assignee changed from David Zafman to Yehuda Sadeh

#5 Updated by Neha Ojha over 5 years ago

  • Related to Bug #37705: list-inconsistent-pg fails with EINVAL added

#6 Updated by Neha Ojha over 5 years ago

  • Related to Bug #37714: test_dump_pgstate_history: Can't find expected values in history object, failing added

#7 Updated by Sage Weil over 5 years ago

  • Status changed from Fix Under Review to Resolved

Also available in: Atom PDF