Bug #10372: FAILED assert(check_latest_map_ops.find(op->tid) (firefly,giant) - Ceph - Ceph

Bug #10372

FAILED assert(check_latest_map_ops.find(op->tid) (firefly,giant)

Added by Yuri Weinstein almost 9 years ago. Updated over 8 years ago.

Status:

Resolved

Priority:

Urgent

Assignee:

Sage Weil

Category:

Target version:

% Done:

Source:

Q/A

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

Logs are in http://qa-proxy.ceph.com/teuthology/teuthology-2014-12-17_17:18:01-upgrade:firefly-x-next-distro-basic-vps/666562/

AdminSocket: request '0' '' to 0x23c21d0 returned 1 bytes
2014-12-17T19:16:30.479 INFO:tasks.workunit.client.3.vpm102.stderr:    -1> 2014-12-17 22:16:09.593562 7fd7b6e40700  5 asok(0x23a43b0) AdminSocket: request '0' '' to 0x23c21d0 returned 1 bytes
2014-12-17T19:16:30.479 INFO:tasks.workunit.client.3.vpm102.stderr:     0> 2014-12-17 22:16:30.453129 7fd7acff9700 -1 osdc/Objecter.cc: In function 'void Objecter::finish_op(Objecter::Op*)' thread 7fd7acff9700 time 2014-12-17 22:16:30.452110
2014-12-17T19:16:30.479 INFO:tasks.workunit.client.3.vpm102.stderr:osdc/Objecter.cc: 1618: FAILED assert(check_latest_map_ops.find(op->tid) == check_latest_map_ops.end())
2014-12-17T19:16:30.479 INFO:tasks.workunit.client.3.vpm102.stderr:
2014-12-17T19:16:30.479 INFO:tasks.workunit.client.3.vpm102.stderr: ceph version 0.80.7-157-gb1ab685 (b1ab685e00034751a161a3d5e0325c6581999c75)
2014-12-17T19:16:30.479 INFO:tasks.workunit.client.3.vpm102.stderr: 1: (()+0x6caf1) [0x7fd7bbfe4af1]
2014-12-17T19:16:30.479 INFO:tasks.workunit.client.3.vpm102.stderr: 2: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0x1306) [0x7fd7bbfee966]
2014-12-17T19:16:30.480 INFO:tasks.workunit.client.3.vpm102.stderr: 3: (librados::RadosClient::_dispatch(Message*)+0xcc) [0x7fd7bb1d0b3c]
2014-12-17T19:16:30.480 INFO:tasks.workunit.client.3.vpm102.stderr: 4: (librados::RadosClient::ms_dispatch(Message*)+0xab) [0x7fd7bb1d0d2b]
2014-12-17T19:16:30.480 INFO:tasks.workunit.client.3.vpm102.stderr: 5: (DispatchQueue::entry()+0x57a) [0x7fd7bb386e3a]
2014-12-17T19:16:30.480 INFO:tasks.workunit.client.3.vpm102.stderr: 6: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fd7bb4026bd]
2014-12-17T19:16:30.480 INFO:tasks.workunit.client.3.vpm102.stderr: 7: (()+0x7df3) [0x7fd7bae08df3]
2014-12-17T19:16:30.480 INFO:tasks.workunit.client.3.vpm102.stderr: 8: (clone()+0x6d) [0x7fd7ba6193dd]
2014-12-17T19:16:30.480 INFO:tasks.workunit.client.3.vpm102.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2014-12-17T19:16:30.480 INFO:tasks.workunit.client.3.vpm102.stderr:
2014-12-17T19:16:30.481 INFO:tasks.workunit.client.3.vpm102.stderr:--- logging levels ---
2014-12-17T19:16:30.481 INFO:tasks.workunit.client.3.vpm102.stderr:   0/ 5 none
2014-12-17T19:16:30.481 INFO:tasks.workunit.client.3.vpm102.stderr:   0/ 1 lockdep
2014-12-17T19:16:30.481 INFO:tasks.workunit.client.3.vpm102.stderr:   0/ 1 context
2014-12-17T19:16:30.481 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 1 crush
2014-12-17T19:16:30.481 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 5 mds
2014-12-17T19:16:30.481 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 5 mds_balancer
2014-12-17T19:16:30.482 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 5 mds_locker
2014-12-17T19:16:30.482 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 5 mds_log
2014-12-17T19:16:30.482 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 5 mds_log_expire
2014-12-17T19:16:30.482 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 5 mds_migrator
2014-12-17T19:16:30.482 INFO:tasks.workunit.client.3.vpm102.stderr:   0/ 1 buffer
2014-12-17T19:16:30.482 INFO:tasks.workunit.client.3.vpm102.stderr:   0/ 1 timer
2014-12-17T19:16:30.482 INFO:tasks.workunit.client.3.vpm102.stderr:   0/ 1 filer
2014-12-17T19:16:30.482 INFO:tasks.workunit.client.3.vpm102.stderr:   0/ 1 striper
2014-12-17T19:16:30.483 INFO:tasks.workunit.client.3.vpm102.stderr:   0/ 1 objecter
2014-12-17T19:16:30.483 INFO:tasks.workunit.client.3.vpm102.stderr:   0/ 5 rados
2014-12-17T19:16:30.483 INFO:tasks.workunit.client.3.vpm102.stderr:   0/ 5 rbd
2014-12-17T19:16:30.483 INFO:tasks.workunit.client.3.vpm102.stderr:   0/ 5 journaler
2014-12-17T19:16:30.483 INFO:tasks.workunit.client.3.vpm102.stderr:   0/ 5 objectcacher
2014-12-17T19:16:30.483 INFO:tasks.workunit.client.3.vpm102.stderr:   0/ 5 client
2014-12-17T19:16:30.483 INFO:tasks.workunit.client.3.vpm102.stderr:   0/ 5 osd
2014-12-17T19:16:30.483 INFO:tasks.workunit.client.3.vpm102.stderr:   0/ 5 optracker
2014-12-17T19:16:30.484 INFO:tasks.workunit.client.3.vpm102.stderr:   0/ 5 objclass
2014-12-17T19:16:30.484 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 3 filestore
2014-12-17T19:16:30.484 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 3 keyvaluestore
2014-12-17T19:16:30.484 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 3 journal
2014-12-17T19:16:30.484 INFO:tasks.workunit.client.3.vpm102.stderr:   0/ 5 ms
2014-12-17T19:16:30.484 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 5 mon
2014-12-17T19:16:30.484 INFO:tasks.workunit.client.3.vpm102.stderr:   0/10 monc
2014-12-17T19:16:30.484 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 5 paxos
2014-12-17T19:16:30.485 INFO:tasks.workunit.client.3.vpm102.stderr:   0/ 5 tp
2014-12-17T19:16:30.485 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 5 auth
2014-12-17T19:16:30.485 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 5 crypto
2014-12-17T19:16:30.485 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 1 finisher
2014-12-17T19:16:30.485 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 5 heartbeatmap
2014-12-17T19:16:30.485 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 5 perfcounter
2014-12-17T19:16:30.485 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 5 rgw
2014-12-17T19:16:30.485 INFO:tasks.workunit.client.3.vpm102.stderr:   1/10 civetweb
2014-12-17T19:16:30.486 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 5 javaclient
2014-12-17T19:16:30.486 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 5 asok
2014-12-17T19:16:30.486 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 1 throttle
2014-12-17T19:16:30.486 INFO:tasks.workunit.client.3.vpm102.stderr:  -2/-2 (syslog threshold)
2014-12-17T19:16:30.486 INFO:tasks.workunit.client.3.vpm102.stderr:  99/99 (stderr threshold)
2014-12-17T19:16:30.486 INFO:tasks.workunit.client.3.vpm102.stderr:  max_recent       500
2014-12-17T19:16:30.486 INFO:tasks.workunit.client.3.vpm102.stderr:  max_new         1000
2014-12-17T19:16:30.486 INFO:tasks.workunit.client.3.vpm102.stderr:  log_file /var/log/ceph/ceph-client.admin.4089.log
2014-12-17T19:16:30.487 INFO:tasks.workunit.client.3.vpm102.stderr:--- end dump of recent events ---
2014-12-17T19:16:30.487 INFO:tasks.workunit.client.3.vpm102.stderr:terminate called after throwing an instance of 'ceph::FailedAssertion'
2014-12-17T19:16:30.487 INFO:tasks.workunit.client.3.vpm102.stderr:*** Caught signal (Aborted) **
2014-12-17T19:16:30.487 INFO:tasks.workunit.client.3.vpm102.stderr: in thread 7fd7acff9700
2014-12-17T19:16:30.487 INFO:tasks.workunit.client.3.vpm102.stderr: ceph version 0.80.7-157-gb1ab685 (b1ab685e00034751a161a3d5e0325c6581999c75)
2014-12-17T19:16:30.487 INFO:tasks.workunit.client.3.vpm102.stderr: 1: ceph_test_librbd() [0x474416]
2014-12-17T19:16:30.487 INFO:tasks.workunit.client.3.vpm102.stderr: 2: (()+0xf130) [0x7fd7bae10130]
2014-12-17T19:16:30.487 INFO:tasks.workunit.client.3.vpm102.stderr: 3: (gsignal()+0x39) [0x7fd7ba558989]
2014-12-17T19:16:30.488 INFO:tasks.workunit.client.3.vpm102.stderr: 4: (abort()+0x148) [0x7fd7ba55a098]
2014-12-17T19:16:30.488 INFO:tasks.workunit.client.3.vpm102.stderr: 5: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7fd7bab5a9d5]
2014-12-17T19:16:30.488 INFO:tasks.workunit.client.3.vpm102.stderr: 6: (()+0x5e946) [0x7fd7bab58946]
2014-12-17T19:16:30.488 INFO:tasks.workunit.client.3.vpm102.stderr: 7: (()+0x5e973) [0x7fd7bab58973]
2014-12-17T19:16:30.488 INFO:tasks.workunit.client.3.vpm102.stderr: 8: (()+0x5eb9f) [0x7fd7bab58b9f]
2014-12-17T19:16:30.488 INFO:tasks.workunit.client.3.vpm102.stderr: 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1ef) [0x7fd7bb278b0f]
2014-12-17T19:16:30.488 INFO:tasks.workunit.client.3.vpm102.stderr: 10: (()+0x6caf1) [0x7fd7bbfe4af1]
2014-12-17T19:16:30.489 INFO:tasks.workunit.client.3.vpm102.stderr: 11: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0x1306) [0x7fd7bbfee966]
2014-12-17T19:16:30.489 INFO:tasks.workunit.client.3.vpm102.stderr: 12: (librados::RadosClient::_dispatch(Message*)+0xcc) [0x7fd7bb1d0b3c]
2014-12-17T19:16:30.489 INFO:tasks.workunit.client.3.vpm102.stderr: 13: (librados::RadosClient::ms_dispatch(Message*)+0xab) [0x7fd7bb1d0d2b]
2014-12-17T19:16:30.489 INFO:tasks.workunit.client.3.vpm102.stderr: 14: (DispatchQueue::entry()+0x57a) [0x7fd7bb386e3a]
2014-12-17T19:16:30.489 INFO:tasks.workunit.client.3.vpm102.stderr: 15: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fd7bb4026bd]
2014-12-17T19:16:30.489 INFO:tasks.workunit.client.3.vpm102.stderr: 16: (()+0x7df3) [0x7fd7bae08df3]
2014-12-17T19:16:30.489 INFO:tasks.workunit.client.3.vpm102.stderr: 17: (clone()+0x6d) [0x7fd7ba6193dd]
2014-12-17T19:16:30.489 INFO:tasks.workunit.client.3.vpm102.stderr:2014-12-17 22:16:30.454762 7fd7acff9700 -1 *** Caught signal (Aborted) **
2014-12-17T19:16:30.490 INFO:tasks.workunit.client.3.vpm102.stderr: in thread 7fd7acff9700
2014-12-17T19:16:30.490 INFO:tasks.workunit.client.3.vpm102.stderr:
2014-12-17T19:16:30.490 INFO:tasks.workunit.client.3.vpm102.stderr: ceph version 0.80.7-157-gb1ab685 (b1ab685e00034751a161a3d5e0325c6581999c75)
2014-12-17T19:16:30.490 INFO:tasks.workunit.client.3.vpm102.stderr: 1: ceph_test_librbd() [0x474416]
2014-12-17T19:16:30.490 INFO:tasks.workunit.client.3.vpm102.stderr: 2: (()+0xf130) [0x7fd7bae10130]
2014-12-17T19:16:30.490 INFO:tasks.workunit.client.3.vpm102.stderr: 3: (gsignal()+0x39) [0x7fd7ba558989]
2014-12-17T19:16:30.490 INFO:tasks.workunit.client.3.vpm102.stderr: 4: (abort()+0x148) [0x7fd7ba55a098]
2014-12-17T19:16:30.490 INFO:tasks.workunit.client.3.vpm102.stderr: 5: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7fd7bab5a9d5]
2014-12-17T19:16:30.491 INFO:tasks.workunit.client.3.vpm102.stderr: 6: (()+0x5e946) [0x7fd7bab58946]
2014-12-17T19:16:30.491 INFO:tasks.workunit.client.3.vpm102.stderr: 7: (()+0x5e973) [0x7fd7bab58973]
2014-12-17T19:16:30.491 INFO:tasks.workunit.client.3.vpm102.stderr: 8: (()+0x5eb9f) [0x7fd7bab58b9f]
2014-12-17T19:16:30.491 INFO:tasks.workunit.client.3.vpm102.stderr: 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1ef) [0x7fd7bb278b0f]
2014-12-17T19:16:30.491 INFO:tasks.workunit.client.3.vpm102.stderr: 10: (()+0x6caf1) [0x7fd7bbfe4af1]
2014-12-17T19:16:30.491 INFO:tasks.workunit.client.3.vpm102.stderr: 11: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0x1306) [0x7fd7bbfee966]
2014-12-17T19:16:30.491 INFO:tasks.workunit.client.3.vpm102.stderr: 12: (librados::RadosClient::_dispatch(Message*)+0xcc) [0x7fd7bb1d0b3c]
2014-12-17T19:16:30.491 INFO:tasks.workunit.client.3.vpm102.stderr: 13: (librados::RadosClient::ms_dispatch(Message*)+0xab) [0x7fd7bb1d0d2b]
2014-12-17T19:16:30.492 INFO:tasks.workunit.client.3.vpm102.stderr: 14: (DispatchQueue::entry()+0x57a) [0x7fd7bb386e3a]
2014-12-17T19:16:30.492 INFO:tasks.workunit.client.3.vpm102.stderr: 15: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fd7bb4026bd]
2014-12-17T19:16:30.492 INFO:tasks.workunit.client.3.vpm102.stderr: 16: (()+0x7df3) [0x7fd7bae08df3]
2014-12-17T19:16:30.492 INFO:tasks.workunit.client.3.vpm102.stderr: 17: (clone()+0x6d) [0x7fd7ba6193dd]
2014-12-17T19:16:30.492 INFO:tasks.workunit.client.3.vpm102.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2014-12-17T19:16:30.492 INFO:tasks.workunit.client.3.vpm102.stderr:
2014-12-17T19:16:30.492 INFO:tasks.workunit.client.3.vpm102.stderr:--- begin dump of recent events ---
2014-12-17T19:16:30.492 INFO:tasks.workunit.client.3.vpm102.stderr:     0> 2014-12-17 22:16:30.454762 7fd7acff9700 -1 *** Caught signal (Aborted) **
2014-12-17T19:16:30.493 INFO:tasks.workunit.client.3.vpm102.stderr: in thread 7fd7acff9700
2014-12-17T19:16:30.493 INFO:tasks.workunit.client.3.vpm102.stderr:
2014-12-17T19:16:30.493 INFO:tasks.workunit.client.3.vpm102.stderr: ceph version 0.80.7-157-gb1ab685 (b1ab685e00034751a161a3d5e0325c6581999c75)
2014-12-17T19:16:30.493 INFO:tasks.workunit.client.3.vpm102.stderr: 1: ceph_test_librbd() [0x474416]
2014-12-17T19:16:30.493 INFO:tasks.workunit.client.3.vpm102.stderr: 2: (()+0xf130) [0x7fd7bae10130]
2014-12-17T19:16:30.493 INFO:tasks.workunit.client.3.vpm102.stderr: 3: (gsignal()+0x39) [0x7fd7ba558989]
2014-12-17T19:16:30.493 INFO:tasks.workunit.client.3.vpm102.stderr: 4: (abort()+0x148) [0x7fd7ba55a098]
2014-12-17T19:16:30.494 INFO:tasks.workunit.client.3.vpm102.stderr: 5: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7fd7bab5a9d5]
2014-12-17T19:16:30.494 INFO:tasks.workunit.client.3.vpm102.stderr: 6: (()+0x5e946) [0x7fd7bab58946]
2014-12-17T19:16:30.494 INFO:tasks.workunit.client.3.vpm102.stderr: 7: (()+0x5e973) [0x7fd7bab58973]
2014-12-17T19:16:30.494 INFO:tasks.workunit.client.3.vpm102.stderr: 8: (()+0x5eb9f) [0x7fd7bab58b9f]
2014-12-17T19:16:30.494 INFO:tasks.workunit.client.3.vpm102.stderr: 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1ef) [0x7fd7bb278b0f]
2014-12-17T19:16:30.494 INFO:tasks.workunit.client.3.vpm102.stderr: 10: (()+0x6caf1) [0x7fd7bbfe4af1]
2014-12-17T19:16:30.494 INFO:tasks.workunit.client.3.vpm102.stderr: 11: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0x1306) [0x7fd7bbfee966]
2014-12-17T19:16:30.494 INFO:tasks.workunit.client.3.vpm102.stderr: 12: (librados::RadosClient::_dispatch(Message*)+0xcc) [0x7fd7bb1d0b3c]
2014-12-17T19:16:30.495 INFO:tasks.workunit.client.3.vpm102.stderr: 13: (librados::RadosClient::ms_dispatch(Message*)+0xab) [0x7fd7bb1d0d2b]
2014-12-17T19:16:30.495 INFO:tasks.workunit.client.3.vpm102.stderr: 14: (DispatchQueue::entry()+0x57a) [0x7fd7bb386e3a]
2014-12-17T19:16:30.495 INFO:tasks.workunit.client.3.vpm102.stderr: 15: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fd7bb4026bd]
2014-12-17T19:16:30.495 INFO:tasks.workunit.client.3.vpm102.stderr: 16: (()+0x7df3) [0x7fd7bae08df3]
2014-12-17T19:16:30.495 INFO:tasks.workunit.client.3.vpm102.stderr: 17: (clone()+0x6d) [0x7fd7ba6193dd]
2014-12-17T19:16:30.495 INFO:tasks.workunit.client.3.vpm102.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2014-12-17T19:16:30.495 INFO:tasks.workunit.client.3.vpm102.stderr:
2014-12-17T19:16:30.495 INFO:tasks.workunit.client.3.vpm102.stderr:--- logging levels ---
2014-12-17T19:16:30.496 INFO:tasks.workunit.client.3.vpm102.stderr:   0/ 5 none
2014-12-17T19:16:30.496 INFO:tasks.workunit.client.3.vpm102.stderr:   0/ 1 lockdep
2014-12-17T19:16:30.496 INFO:tasks.workunit.client.3.vpm102.stderr:   0/ 1 context
2014-12-17T19:16:30.496 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 1 crush
2014-12-17T19:16:30.496 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 5 mds
2014-12-17T19:16:30.496 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 5 mds_balancer
2014-12-17T19:16:30.496 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 5 mds_locker
2014-12-17T19:16:30.496 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 5 mds_log
2014-12-17T19:16:30.496 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 5 mds_log_expire
2014-12-17T19:16:30.497 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 5 mds_migrator
2014-12-17T19:16:30.497 INFO:tasks.workunit.client.3.vpm102.stderr:   0/ 1 buffer
2014-12-17T19:16:30.497 INFO:tasks.workunit.client.3.vpm102.stderr:   0/ 1 timer
2014-12-17T19:16:30.497 INFO:tasks.workunit.client.3.vpm102.stderr:   0/ 1 filer
2014-12-17T19:16:30.497 INFO:tasks.workunit.client.3.vpm102.stderr:   0/ 1 striper
2014-12-17T19:16:30.497 INFO:tasks.workunit.client.3.vpm102.stderr:   0/ 1 objecter
2014-12-17T19:16:30.497 INFO:tasks.workunit.client.3.vpm102.stderr:   0/ 5 rados
2014-12-17T19:16:30.497 INFO:tasks.workunit.client.3.vpm102.stderr:   0/ 5 rbd
2014-12-17T19:16:30.498 INFO:tasks.workunit.client.3.vpm102.stderr:   0/ 5 journaler
2014-12-17T19:16:30.498 INFO:tasks.workunit.client.3.vpm102.stderr:   0/ 5 objectcacher
2014-12-17T19:16:30.498 INFO:tasks.workunit.client.3.vpm102.stderr:   0/ 5 client
2014-12-17T19:16:30.498 INFO:tasks.workunit.client.3.vpm102.stderr:   0/ 5 osd
2014-12-17T19:16:30.498 INFO:tasks.workunit.client.3.vpm102.stderr:   0/ 5 optracker
2014-12-17T19:16:30.498 INFO:tasks.workunit.client.3.vpm102.stderr:   0/ 5 objclass
2014-12-17T19:16:30.498 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 3 filestore
2014-12-17T19:16:30.498 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 3 keyvaluestore
2014-12-17T19:16:30.499 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 3 journal
2014-12-17T19:16:30.499 INFO:tasks.workunit.client.3.vpm102.stderr:   0/ 5 ms
2014-12-17T19:16:30.499 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 5 mon
2014-12-17T19:16:30.499 INFO:tasks.workunit.client.3.vpm102.stderr:   0/10 monc
2014-12-17T19:16:30.499 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 5 paxos
2014-12-17T19:16:30.499 INFO:tasks.workunit.client.3.vpm102.stderr:   0/ 5 tp
2014-12-17T19:16:30.499 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 5 auth
2014-12-17T19:16:30.500 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 5 crypto
2014-12-17T19:16:30.500 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 1 finisher
2014-12-17T19:16:30.500 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 5 heartbeatmap
2014-12-17T19:16:30.500 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 5 perfcounter
2014-12-17T19:16:30.500 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 5 rgw
2014-12-17T19:16:30.500 INFO:tasks.workunit.client.3.vpm102.stderr:   1/10 civetweb
2014-12-17T19:16:30.500 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 5 javaclient
2014-12-17T19:16:30.500 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 5 asok
2014-12-17T19:16:30.501 INFO:tasks.workunit.client.3.vpm102.stderr:   1/ 1 throttle
2014-12-17T19:16:30.501 INFO:tasks.workunit.client.3.vpm102.stderr:  -2/-2 (syslog threshold)
2014-12-17T19:16:30.501 INFO:tasks.workunit.client.3.vpm102.stderr:  99/99 (stderr threshold)
2014-12-17T19:16:30.501 INFO:tasks.workunit.client.3.vpm102.stderr:  max_recent       500
2014-12-17T19:16:30.501 INFO:tasks.workunit.client.3.vpm102.stderr:  max_new         1000
2014-12-17T19:16:30.501 INFO:tasks.workunit.client.3.vpm102.stderr:  log_file /var/log/ceph/ceph-client.admin.4089.log
2014-12-17T19:16:30.501 INFO:tasks.workunit.client.3.vpm102.stderr:--- end dump of recent events ---
2014-12-17T19:16:30.501 INFO:tasks.workunit.client.3.vpm102.stderr:/home/ubuntu/cephtest/workunit.client.3/rbd/test_librbd.sh: line 3:  4089 Aborted                 (core dumped) ceph_test_librbd

Related issues

Associated revisions

Revision f20225cb (diff)
Added by Sage Weil almost 9 years ago

osdc/Objecter: handle reply race with pool deletion

We need to handle this scenario:

- send request in epoch X
 - osd replies
 - pool is deleted in epoch X+1
 - client gets map X+1, sends a map check
 - client handles reply
   -> asserts that no map checks are in flight

This isn't the best solution. We could infer that a map check isn't needed
since the pool existed earlier and doesn't now. But this is firefly and
the fix is no more expensive than the old assert.

Fixes: #10372
Signed-off-by: Sage Weil <sage@redhat.com>

Revision d7fd6fcc (diff)
Added by Sage Weil almost 9 years ago

osdc/Objecter: improve pool deletion detection

Currently we can have a race like so:

- send op in epoch X
 - osd replies
 - pool deleted in epoch X+1
 - client gets X+1, sends map epoch check
 - client gets reply
   -> fails assert that there is no map check in flight

Avoid this situation by inferring that the pool is deleted when we see
that we previously sent the request but the pool is no longer present.
Since pool ids are not reused there is no point in doing a synchronous
map check at all.

Backport: giant
Fixes: #10372
Signed-off-by: Sage Weil <sage@redhat.com>

History

#1 Updated by Sage Weil almost 9 years ago

Status changed from New to 12

#2 Updated by Sage Weil almost 9 years ago

Subject changed from "FAILED assert(check_latest_map_ops.find(op->tid)" in upgrade:firefly-x-next-distro-basic-vps run to FAILED assert(check_latest_map_ops.find(op->tid)
Priority changed from Normal to Urgent

2014-12-18T22:35:04.078 INFO:tasks.workunit.client.0.plana51.stderr:osdc/Objecter.cc: In function 'void Objecter::_finish_op(Objecter::Op*)' thread 7fc3e2efc700 time 2014-12-18 22:35:04.076166
2014-12-18T22:35:04.078 INFO:tasks.workunit.client.0.plana51.stderr:osdc/Objecter.cc: 2689: FAILED assert(check_latest_map_ops.find(op->tid) == check_latest_map_ops.end())
2014-12-18T22:35:04.078 INFO:tasks.workunit.client.0.plana51.stderr: ceph version 0.89-813-gaf18cf8 (af18cf8d61ce5edb5fb421d136cd354bdb4eedd3)
2014-12-18T22:35:04.079 INFO:tasks.workunit.client.0.plana51.stderr: 1: (()+0x12199f) [0x7fc3ffe1999f]
2014-12-18T22:35:04.079 INFO:tasks.workunit.client.0.plana51.stderr: 2: (()+0x94657) [0x7fc3ffd8c657]
2014-12-18T22:35:04.079 INFO:tasks.workunit.client.0.plana51.stderr: 3: (()+0xa159d) [0x7fc3ffd9959d]
2014-12-18T22:35:04.079 INFO:tasks.workunit.client.0.plana51.stderr: 4: (()+0xad44b) [0x7fc3ffda544b]
2014-12-18T22:35:04.080 INFO:tasks.workunit.client.0.plana51.stderr: 5: (()+0x282629) [0x7fc3fff7a629]
2014-12-18T22:35:04.080 INFO:tasks.workunit.client.0.plana51.stderr: 6: (()+0x2a267f) [0x7fc3fff9a67f]
2014-12-18T22:35:04.080 INFO:tasks.workunit.client.0.plana51.stderr: 7: (()+0x2a540d) [0x7fc3fff9d40d]
2014-12-18T22:35:04.080 INFO:tasks.workunit.client.0.plana51.stderr: 8: (()+0x7e9a) [0x7fc3ffae2e9a]
2014-12-18T22:35:04.081 INFO:tasks.workunit.client.0.plana51.stderr: 9: (clone()+0x6d) [0x7fc3ff2f93fd]
2014-12-18T22:35:04.081 INFO:tasks.workunit.client.0.plana51.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

ubuntu@teuthology:/a/sage-2014-12-18_20:33:36-rados-master-distro-basic-multi/667748

#3 Updated by Sage Weil almost 9 years ago

Subject changed from FAILED assert(check_latest_map_ops.find(op->tid) to FAILED assert(check_latest_map_ops.find(op->tid) (firefly)
Status changed from 12 to In Progress
Assignee set to Sage Weil
Backport set to giant,firefly

https://github.com/ceph/ceph/pull/3249 is a minimal firefly fix

#4 Updated by Sage Weil almost 9 years ago

Status changed from In Progress to Fix Under Review

#5 Updated by Loïc Dachary over 8 years ago

Status changed from Fix Under Review to Resolved

#6 Updated by Josh Durgin over 8 years ago

Status changed from Resolved to Pending Backport

#7 Updated by Yuri Weinstein over 8 years ago

Run: http://pulpito.ceph.com/teuthology-2015-03-14_17:05:02-upgrade:giant-x-hammer-distro-basic-vps/
Job: ['804012']
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2015-03-14_17:05:02-upgrade:giant-x-hammer-distro-basic-vps/804012/

#8 Updated by Yuri Weinstein over 8 years ago

Subject changed from FAILED assert(check_latest_map_ops.find(op->tid) (firefly) to FAILED assert(check_latest_map_ops.find(op->tid) (firefly,giant)

#9 Updated by Loïc Dachary over 8 years ago

Backport changed from giant,firefly to firefly

#10 Updated by Loïc Dachary over 8 years ago

No need to backport : https://github.com/ceph/ceph/pull/4032#issuecomment-82436252

#11 Updated by Loïc Dachary over 8 years ago

Status changed from Pending Backport to Resolved
Backport deleted (~~firefly~~)

Also available in: Atom PDF

	Related to Ceph - Bug #11427: "FAILED assert(s != __null)" in rados-giant-distro-basic-multi run	Duplicate	04/19/2015
	Duplicated by Ceph - Bug #12517: assert failure in Objecter::_finish_op(Objecter::Op*)	Duplicate	07/29/2015

Project

General

Profile

Ceph

Issues

Tags

Custom queries