Bug #37706
list-inconsistent-pg fails with EINVAL
% Done:
0%
Source:
Development
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Seen in run-standalone.sh runs of osd-scrub-snaps.sh and osd-scrub-repair.sh:
../qa/standalone/scrub/osd-scrub-repair.sh:3108: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool error 22: (22) Invalid argument ../qa/standalone/scrub/osd-scrub-repair.sh:3108: corrupt_scrub_erasure: return 1 ../qa/standalone/scrub/osd-scrub-repair.sh:62: run: return 1
Teuthology run
/a/sage-2018-12-18_03:59:26-rados-master-distro-basic-smithi/3373050
Related issues
History
#1 Updated by David Zafman over 5 years ago
Thread 1 "rados" hit Breakpoint 2, librados::RadosClient::get_inconsistent_pgs (this=0x5555559f6f20, pool_id=1, pgs=0x7fffffffd6c0) at /home/dzafman/ceph/src/librados/RadosClient.cc:1126 1126 { (gdb) n 1129 "\"pool\": " + std::to_string(pool_id) + "," (gdb) 1132 }; (gdb) 1133 bufferlist inbl, outbl; (gdb) 1134 string outstring; (gdb) 1135 if (auto ret = mgr_command(cmd, inbl, &outbl, &outstring); ret) { (gdb) 1138 if (!outbl.length()) { (gdb) 1142 JSONParser parser; (gdb) 1143 if (!parser.parse(outbl.c_str(), outbl.length())) { (gdb) 1146 vector<string> v; (gdb) print parser $1 = {<JSONObj> = {_vptr.JSONObj = 0x7fffef1fad28 <vtable for JSONParser+16>, parent = 0x0, name = "", data = {static null = <optimized out>, v_ = {which_ = 0, storage_ = {<boost::detail::aligned_storage::aligned_storage_imp<32, 8>> = {data_ = { buf = "\340d\251UUU\000\000`\324\377\377\377\177\000\000\000\000\000\000\000\000\000\000\300\344\377\367\377\177\000", align_ = {<No data fields>}}}, static size = <optimized out>, static alignment = <optimized out>}}}, val = {str = "", quoted = false}, data_quoted = false, children = std::multimap with 2 elements = { ["pg_ready"] = 0x555555a998e0, ["pg_stats"] = 0x555555aa1aa0}, attr_map = std::map with 0 elements}, buf_len = 0, json_buffer = "", success = true} (gdb) n 1147 if (!parser.is_array()) { (gdb) n 1148 JSONObj *pgstat_obj = parser.find_obj("pg_stats"); (gdb) n 1149 if (!pgstat_obj) (gdb) print *pgstat_obj $2 = {_vptr.JSONObj = 0x7fffef1fad48 <vtable for JSONObj+16>, parent = 0x7fffffffd400, name = "pg_stats", data = {static null = <optimized out>, v_ = {which_ = 1, storage_ = {<boost::detail::aligned_storage::aligned_storage_imp<32, 8>> = {data_ = {buf = "\320[\252UUU", '\000' <repeats 25 times>, align_ = {<No data fields>}}}, static size = <optimized out>, static alignment = <optimized out>}}}, val = {str = "", quoted = false}, data_quoted = false, children = std::multimap with 1 element = { [""] = 0x555555aabbc0}, attr_map = std::map with 1 element = {["pg_stats"] = {str = "", quoted = false}}} (gdb) n 1151 auto s = pgstat_obj->get_data(); (gdb) n 1152 JSONParser pg_stats; (gdb) n 1153 if (!pg_stats.parse(s.c_str(), s.length())) { (gdb) n 1154 return -EINVAL; (gdb) print s $3 = ""
#2 Updated by David Zafman over 5 years ago
I was suspicious of https://github.com/ceph/ceph/pull/23298, so I reverted all 74 commits and the problem wouldn't reproduce. The JSONParser::parse() function is in libradospp.so which doesn't have debug information so I can't walk through it in gdb easily.
#3 Updated by Yehuda Sadeh over 5 years ago
- Status changed from In Progress to Fix Under Review
#4 Updated by David Zafman over 5 years ago
- Assignee changed from David Zafman to Yehuda Sadeh
#5 Updated by Neha Ojha over 5 years ago
- Related to Bug #37705: list-inconsistent-pg fails with EINVAL added
#6 Updated by Neha Ojha over 5 years ago
- Related to Bug #37714: test_dump_pgstate_history: Can't find expected values in history object, failing added
#7 Updated by Sage Weil over 5 years ago
- Status changed from Fix Under Review to Resolved