Project

General

Profile

Bug #24077

test_pool_create_fail (tasks.mgr.dashboard.test_pool.PoolTest) fails

Added by Kefu Chai over 1 year ago. Updated over 1 year ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Tests
Target version:
-
Start date:
05/10/2018
Due date:
% Done:

0%

Source:
Development
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:

Description

2018-05-10T04:37:48.278 INFO:tasks.mgr.dashboard.helper:request POST to https://mira078.front.sepia.ceph.com:7789/api/auth
2018-05-10T04:37:48.585 INFO:tasks.mgr.dashboard.helper:request POST to https://mira078.front.sepia.ceph.com:7789/api/pool/
2018-05-10T04:37:48.586 INFO:tasks.ceph.mgr.x.mira078.stdout:::ffff:172.21.0.51 - - [10/May/2018:04:37:48] "POST /api/auth HTTP/1.1" 201 21 "" "python-requests/2.12.5" 
2018-05-10T04:37:48.597 INFO:tasks.ceph.mgr.x.mira078.stderr:[10/May/2018:04:37:48] HTTP
2018-05-10T04:37:48.597 INFO:tasks.ceph.mgr.x.mira078.stderr:Request Headers:
2018-05-10T04:37:48.597 INFO:tasks.ceph.mgr.x.mira078.stderr:  Content-Length: 79
2018-05-10T04:37:48.597 INFO:tasks.ceph.mgr.x.mira078.stderr:  COOKIE: session_id=dcd6e8ff5833aefb2f7cdf2b75181269f4d2d1e7
2018-05-10T04:37:48.597 INFO:tasks.ceph.mgr.x.mira078.stderr:  HOST: mira078.front.sepia.ceph.com:7789
2018-05-10T04:37:48.598 INFO:tasks.ceph.mgr.x.mira078.stderr:  CONNECTION: keep-alive
2018-05-10T04:37:48.598 INFO:tasks.ceph.mgr.x.mira078.stderr:  Remote-Addr: ::ffff:172.21.0.51
2018-05-10T04:37:48.598 INFO:tasks.ceph.mgr.x.mira078.stderr:  ACCEPT: */*
2018-05-10T04:37:48.598 INFO:tasks.ceph.mgr.x.mira078.stderr:  USER-AGENT: python-requests/2.12.5
2018-05-10T04:37:48.598 INFO:tasks.ceph.mgr.x.mira078.stderr:  Content-Type: application/json
2018-05-10T04:37:48.598 INFO:tasks.ceph.mgr.x.mira078.stderr:  ACCEPT-ENCODING: gzip, deflate
2018-05-10T04:37:48.598 INFO:tasks.ceph.mgr.x.mira078.stderr:[10/May/2018:04:37:48] HTTP Traceback (most recent call last):
2018-05-10T04:37:48.599 INFO:tasks.ceph.mgr.x.mira078.stderr:  File "/usr/lib/python2.7/dist-packages/cherrypy/_cprequest.py", line 670, in respond
2018-05-10T04:37:48.599 INFO:tasks.ceph.mgr.x.mira078.stderr:    response.body = self.handler()
2018-05-10T04:37:48.599 INFO:tasks.ceph.mgr.x.mira078.stderr:  File "/usr/lib/python2.7/dist-packages/cherrypy/lib/encoding.py", line 217, in __call__
2018-05-10T04:37:48.599 INFO:tasks.ceph.mgr.x.mira078.stderr:    self.body = self.oldhandler(*args, **kwargs)
2018-05-10T04:37:48.599 INFO:tasks.ceph.mgr.x.mira078.stderr:  File "/usr/lib/python2.7/dist-packages/cherrypy/_cptools.py", line 231, in wrap
2018-05-10T04:37:48.599 INFO:tasks.ceph.mgr.x.mira078.stderr:    return self.newhandler(innerfunc, *args, **kwargs)
2018-05-10T04:37:48.599 INFO:tasks.ceph.mgr.x.mira078.stderr:  File "/usr/lib/ceph/mgr/dashboard/services/exception.py", line 94, in dashboard_exception_handler
2018-05-10T04:37:48.599 INFO:tasks.ceph.mgr.x.mira078.stderr:    return json.dumps(serialize_dashboard_exception(e)).encode('utf-8')
2018-05-10T04:37:48.599 INFO:tasks.ceph.mgr.x.mira078.stderr:  File "/usr/lib/ceph/mgr/dashboard/services/exception.py", line 73, in serialize_dashboard_exception
2018-05-10T04:37:48.599 INFO:tasks.ceph.mgr.x.mira078.stderr:    out['code'] = e.code
2018-05-10T04:37:48.600 INFO:tasks.ceph.mgr.x.mira078.stderr:  File "/usr/lib/ceph/mgr/dashboard/exceptions.py", line 45, in code
2018-05-10T04:37:48.600 INFO:tasks.ceph.mgr.x.mira078.stderr:    return str(abs(self.errno))
2018-05-10T04:37:48.600 INFO:tasks.ceph.mgr.x.mira078.stderr:TypeError: bad operand type for abs(): 'NoneType'
2018-05-10T04:37:48.600 INFO:tasks.ceph.mgr.x.mira078.stderr:
2018-05-10T04:37:48.606 INFO:tasks.cephfs_test_runner:test_pool_create_fail (tasks.mgr.dashboard.test_pool.PoolTest) ... FAIL

see /a/kchai-2018-05-09_15:14:14-rados-wip-kefu-testing-2018-05-09-2109-distro-basic-mira/2509998

i suspect it's related to https://github.com/ceph/ceph/pull/21066

History

#1 Updated by Josh Durgin over 1 year ago

Looks to have caused a monitor crash:

2018-05-10T04:37:39.127 INFO:tasks.mgr.dashboard.helper:request POST to https://mira078.front.sepia.ceph.com:7789/api/auth
2018-05-10T04:37:39.438 INFO:tasks.mgr.dashboard.helper:request POST to https://mira078.front.sepia.ceph.com:7789/api/pool/
2018-05-10T04:37:39.440 INFO:tasks.ceph.mgr.x.mira078.stdout:::ffff:172.21.0.51 - - [10/May/2018:04:37:39] "POST /api/auth HTTP/1.1" 201 21 "" "python-requests/2.12.5" 
2018-05-10T04:37:43.483 INFO:tasks.ceph.mon.b.mira053.stderr:2018-05-10 04:37:43.477 7f3b27514700 -1 bad boost::get: key val is not type std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >
2018-05-10T04:37:43.487 INFO:tasks.ceph.mon.b.mira053.stderr:2018-05-10 04:37:43.481 7f3b27514700 -1  ceph version 13.0.2-2283-gf16a158 (f16a15847fc04a93cacb395c0e34fadc67c985c8) mimic (dev)
2018-05-10T04:37:43.487 INFO:tasks.ceph.mon.b.mira053.stderr: 1: (bool cmd_getval<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(CephContext*, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, boost::variant<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, double, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::vector<long, std::allocator<long> >, std::vector<double, std::allocator<double> > >, std::less<void>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, boost::variant<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, double, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::vector<long, std::allocator<long> >, std::vector<double, std::allocator<double> > > > > > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)+0xe3) [0x56040b924f13]
2018-05-10T04:37:43.487 INFO:tasks.ceph.mon.b.mira053.stderr: 2: (OSDMonitor::prepare_command_pool_set(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, boost::variant<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, double, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::vector<long, std::allocator<long> >, std::vector<double, std::allocator<double> > >, std::less<void>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, boost::variant<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, double, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::vector<long, std::allocator<long> >, std::vector<double, std::allocator<double> > > > > > const&, std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >&)+0x2e4) [0x56040ba746e4]
2018-05-10T04:37:43.487 INFO:tasks.ceph.mon.b.mira053.stderr: 3: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr<MonOpRequest>, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, boost::variant<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, double, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::vector<long, std::allocator<long> >, std::vector<double, std::allocator<double> > >, std::less<void>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, boost::variant<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, double, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::vector<long, std::allocator<long> >, std::vector<double, std::allocator<double> > > > > > const&)+0x1dc8e) [0x56040ba9488e]
2018-05-10T04:37:43.487 INFO:tasks.ceph.mon.b.mira053.stderr: 4: (OSDMonitor::prepare_command(boost::intrusive_ptr<MonOpRequest>)+0x252) [0x56040ba9a742]
2018-05-10T04:37:43.487 INFO:tasks.ceph.mon.b.mira053.stderr: 5: (OSDMonitor::prepare_update(boost::intrusive_ptr<MonOpRequest>)+0x170) [0x56040ba9aaa0]
2018-05-10T04:37:43.487 INFO:tasks.ceph.mon.b.mira053.stderr: 6: (PaxosService::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x996) [0x56040ba2bb26]
2018-05-10T04:37:43.488 INFO:tasks.ceph.mon.b.mira053.stderr: 7: (Monitor::handle_command(boost::intrusive_ptr<MonOpRequest>)+0x1a12) [0x56040b8f6352]
2018-05-10T04:37:43.488 INFO:tasks.ceph.mon.b.mira053.stderr: 8: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x39d) [0x56040b8fb57d]
2018-05-10T04:37:43.488 INFO:tasks.ceph.mon.b.mira053.stderr: 9: (Monitor::_ms_dispatch(Message*)+0x7a2) [0x56040b8fcc22]
2018-05-10T04:37:43.488 INFO:tasks.ceph.mon.b.mira053.stderr: 10: (Monitor::handle_forward(boost::intrusive_ptr<MonOpRequest>)+0x7a7) [0x56040b8fdef7]
2018-05-10T04:37:43.488 INFO:tasks.ceph.mon.b.mira053.stderr: 11: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0xe71) [0x56040b8fc051]
2018-05-10T04:37:43.488 INFO:tasks.ceph.mon.b.mira053.stderr: 12: (Monitor::_ms_dispatch(Message*)+0x7a2) [0x56040b8fcc22]
2018-05-10T04:37:43.488 INFO:tasks.ceph.mon.b.mira053.stderr: 13: (Monitor::ms_dispatch(Message*)+0x23) [0x56040b9257c3]
2018-05-10T04:37:43.488 INFO:tasks.ceph.mon.b.mira053.stderr: 14: (DispatchQueue::entry()+0xb92) [0x7f3b32aadc72]
2018-05-10T04:37:43.488 INFO:tasks.ceph.mon.b.mira053.stderr: 15: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f3b32b4becd]
2018-05-10T04:37:43.488 INFO:tasks.ceph.mon.b.mira053.stderr: 16: (()+0x76ba) [0x7f3b323336ba]
2018-05-10T04:37:43.489 INFO:tasks.ceph.mon.b.mira053.stderr: 17: (clone()+0x6d) [0x7f3b3169141d]

#2 Updated by Kefu Chai over 1 year ago

  • Assignee changed from Ricardo Dias to Kefu Chai

#3 Updated by Kefu Chai over 1 year ago

  • Category set to Tests
  • Status changed from New to Need Review

Josh, it's not a mon crash. mon was just not happy with this command, please see handle_bad_get() in cmd_getval(CephContext *cct, const cmdmap_t& cmdmap, const std::string& k, T& val)

https://github.com/ceph/ceph/pull/21943

#4 Updated by Kefu Chai over 1 year ago

  • Status changed from Need Review to Resolved

Also available in: Atom PDF