Bug #2834
closed
osd/ReplicatedPG.cc: 3577: FAILED assert(waiting_for_ack.begin()->first == repop->v)
Added by Sage Weil almost 12 years ago.
Updated over 11 years ago.
Description
-5> 2012-07-24 14:06:49.664524 7f1bb757b700 -1 osd/ReplicatedPG.cc: In function 'void ReplicatedPG::eval_repop(ReplicatedPG::RepGather*)' thread 7f1bb757b700 time 2012-07-24 14:06:49.549566
osd/ReplicatedPG.cc: 3577: FAILED assert(waiting_for_ack.begin()->first == repop->v)
ceph version 0.49-299-g9ecc5c2 (commit:9ecc5c2c9c31f9ab8a01cba47690c32b6792b9c5)
1: (ReplicatedPG::eval_repop(ReplicatedPG::RepGather*)+0x75e) [0x55122e]
2: (ReplicatedPG::op_applied(ReplicatedPG::RepGather*)+0x57b) [0x554f6b]
3: (C_OSD_OpApplied::finish(int)+0x11) [0x5a62d1]
4: (Finisher::finisher_thread_entry()+0x218) [0x7315b8]
5: (()+0x7e9a) [0x7f1bc32e6e9a]
6: (clone()+0x6d) [0x7f1bc189b4bd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
ubuntu@teuthology:/a/sage-2012-07-24_13:13:37-regression-wip-msgr-testing-basic/17035$ cat config.yaml
kernel: &id001
kdb: true
sha1: 77dca1ac33894de22b1740bb9cf6b8ef6429c700
nuke-on-error: true
overrides:
ceph:
conf:
global:
ms inject socket failures: 200
fs: btrfs
log-whitelist:
- slow request
sha1: 9ecc5c2c9c31f9ab8a01cba47690c32b6792b9c5
workunit:
sha1: 9ecc5c2c9c31f9ab8a01cba47690c32b6792b9c5
roles:
- - mon.a
- mon.c
- osd.0
- osd.1
- osd.2
- - mon.b
- mds.a
- osd.3
- osd.4
- osd.5
- - client.0
targets:
ubuntu@plana25.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDuXajaQgHe9XnbLOzI8WWFYVz6+TnOiTzbkIJPGOZpzQEjnUtJraQIEt5ABSeovMjiEj+V4XvunfyuSmEd0H9giRSyjmCHTPGlpndfTeCdVtCBpNqf5GkUqHaEY1Hp57XPbya2rGlwtFm0NeIDYx6pfkejKnsTOUqwhgUb6950TRhjHQhMjFgyALSyfAm/4y6vGZfjm57+yyih6XgDkqWiiQ6Y/aJVR2n+iCzvqEzV7JSCU+Brn+k8IQLHho1fadYqc5PjYct5BaVlHcP6c+T8nJE/DvqGwZ4gQaVJcuWJiDfLOPPYo1g/0AFicxauLwVNJ6HFR9FjLLGtGU+2DcVN
ubuntu@plana61.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOTCMIScDTmD9NkfsWU7xeyZ+WOXai5izYeliiXDSjJC3bT6r8Fp+rhPfcHCVHiw++VsbvKZtkhjCSnJTVPWCdpRDghzJ3nZUBImWRo3PmHo1etQpCeimaOrIJ2q0ChN5jmSOqy5B+Z4om2vXBtBY6nkdTxDOr2+MH3NrSPkQSFB0zO+VPuwKXsemeUC6urb2IZZpxY3cxNq4fafTF9PROpgOnIA+o3igyU4duKEjnCzTHZjw/PL7Eph/7p6+UQgrUwe7pgVzT+2MM0zcBtBSXNqs3dCGmpvUapOkBlDoIX02EkWRNpkM3vfeFt1EFC17B5vd61Kg40bYUG8qWGR0T
ubuntu@plana64.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDk4GmsUmC8svnRI6Xd+mRX2MwKb4RHECAeLfqTm2COfqfolS2wKGw3U92eJcyvpZ+2p82X7uBrimjZh5JgRtxJ1aGUG4Pi60+JBYF0WpohM/3aYISFegVNET9rcapdDaAi6fFB5vhT06Q/cYEO0tPrdqGb/O3oiDSurtqtfOzkdwSPWSTY/hSegXgOeG6EjuEfvnU4BbgXWkLlDQRXCdgQd35F0SlKJVgMo+J1MgMCEK4qnBMFN614P1gBSzZCBsSUGQdjYBOzZfCRlI2bUdPDtB0kyjp7o5Ns9gLd07TLw8h9oxvI7wxG16XnLOAIzPBNOaH4OztTMGg3wJ/1e26t
task:
- ceph:
conf:
client:
rbd cache: true
tasks:
- internal.lock_machines: 3
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.check_conflict: null
- kernel: *id001
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock: null
- ceph: null
- qemu:
all:
test: https://raw.github.com/ceph/ceph/master/qa/workunits/suites/tiobench.sh
- Assignee deleted (
Sage Weil)
- Status changed from New to Resolved
Recent log: ubuntu@teuthology:/a/teuthology-2012-09-07_00:00:07-regression-next-testing-basic/17906
Recent log: ubuntu@teuthology:/a/teuthology-2012-09-07_00:00:07-regression-next-testing-basic/17906
2012-09-07 03:43:22.940159 7f748e13d700 -1 osd/ReplicatedPG.cc: In function 'void ReplicatedPG::eval_repo
p(ReplicatedPG::RepGather*)' thread 7f748e13d700 time 2012-09-07 03:43:22.938766
osd/ReplicatedPG.cc: 3582: FAILED assert(waiting_for_ack.begin()->first == repop->v)
ceph version 0.51-274-g5f36b8d (commit:5f36b8d78416b7a1d1bbefecddfcee00b7bfcfa3)
1: (ReplicatedPG::eval_repop(ReplicatedPG::RepGather*)+0x790) [0x551560]
2: (ReplicatedPG::repop_ack(ReplicatedPG::RepGather*, int, int, int, eversion_t)+0x1d4) [0x5527b4]
3: (ReplicatedPG::sub_op_modify_reply(std::tr1::shared_ptr<OpRequest>)+0x172) [0x554f12]
4: (ReplicatedPG::do_sub_op_reply(std::tr1::shared_ptr<OpRequest>)+0x82) [0x585ab2]
5: (PG::do_request(std::tr1::shared_ptr<OpRequest>)+0x325) [0x6556c5]
6: (OSD::dequeue_op(PG*)+0x27c) [0x5ba1ec]
7: (ThreadPool::worker()+0x523) [0x7cc343]
8: (ThreadPool::WorkThread::entry()+0xd) [0x5fa52d]
9: (()+0x7e9a) [0x7f749e9c3e9a]
10: (clone()+0x6d) [0x7f749cd674bd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
ubuntu@teuthology:/a/teuthology-2012-09-07_00:00:07-regression-next-testing-basic/17906$ cat config.yaml
kernel: &id001
kdb: true
sha1: e81d5d695a03a141b9a4a4e75b8e009ecba43c64
nuke-on-error: true
overrides:
ceph:
conf:
global:
ms inject socket failures: 5000
fs: btrfs
log-whitelist:
- slow request
sha1: 5f36b8d78416b7a1d1bbefecddfcee00b7bfcfa3
workunit:
sha1: 5f36b8d78416b7a1d1bbefecddfcee00b7bfcfa3
roles:
- - mon.a
- osd.0
- osd.1
- osd.2
- - mds.a
- osd.3
- osd.4
- osd.5
- - client.0
targets:
ubuntu@plana23.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDiwH5Qz5dXnbtYRiTk0QVNNyZQWYcardED+AqVLxoz5h/z/tPUyt6VTcrNvyFiKZcrz70vKy/1S1JNmt74gSc0KE8YhLjvuCaTwJDw1LOTNzc5b074zfnjeNGKqb0L3BefbFFOMh/ZuxGbTJWZXdD1DwP2VWxGdhtHAxglgLjt5541nxw41vT+dVMgQMt7Lv5P3MXl+IY58LUzYC9EkOvgZTPTfRx7IptkDSEmbYGL7dQE6H9VyoukOejj4jgg8ZWhPR9e39OhB/Vh7qtiCTRfapovh1zrfCM/b7O1/nMisUmfOK+nF2ruiTefEA14u59uxlfpaRLQDtyv6b5aPour
ubuntu@plana42.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzQfmtpfECJ+NZaaiSH/R8X+dGXHH+aDTCKGLLiHhW9fttxzfzcJJaBx1b664D3ynZAC7NiaegfLDTCMW7FFVDUltMQcWjsM4BqfFipIquDP4KOclCc6EwG5aYG/MLCJwL6sovt1uKg00bSkVQsUSHBgZbMJKCjCbBb0XPxfuS4dppA3diEZBOMt1YHr+NdV7sace/Gc7YBlGsNOinnqkKfVWIpfYCiTQ18cvaisSEHsQR6zhKqrX4afQk13cTjdvZeQp9AXxRIf1g9fq2zHVWMdJdVNR8D0BSBtfAzMqIqZ8qcJqmzQN0Zq9Wk9Y021vMFORZy2SFI6c7yBWDJLdT
ubuntu@plana52.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9kswBp2g5ZV1Qrvlee8MvUOCNdubQFqUBr5WSsmFBODqEuiitWbhuBu2Ucz0lBMf41DpMKLeYDN0lIC94GZmGaiCN+Ak9Ia05d/uRvesT2nDgHB3Z9J/zEFlY8RVxL3xhD+hq4u8dbASlqqoMDiBP+7efZMxt4Ndnzr/yOxge3KenxyQImBUS+OV+BqnfCOHf6BqM33U1leXz2kng7ocxoE91DAMslKD/2DPRSYEhfucUJZk6IYevr/g0JVhbfvjSlZzwUEfTyVmPeqNyls/U+azhKlvQbqpb+ttc02RNydQ1YgOgHFCaqd9Vm8XjUU6vYGlkFHZ+BMJuEwA9AH/D
tasks:
- internal.lock_machines: 3
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.check_conflict: null
- kernel: *id001
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock: null
- ceph:
log-whitelist:
- wrongly marked me down
- objects unfound and apparently lost
- thrashosds:
timeout: 1200
- rados:
clients:
- client.0
objects: 50
op_weights:
delete: 50
read: 100
snap_create: 50
snap_remove: 50
snap_rollback: 50
write: 100
ops: 4000
ubuntu@teuthology:/a/teuthology-2012-09-07_00:00:07-regression-next-testing-basic/17906$ cat summary.yaml
ceph-sha1: 5f36b8d78416b7a1d1bbefecddfcee00b7bfcfa3
client.0-kernel-sha1: e81d5d695a03a141b9a4a4e75b8e009ecba43c64
description: collection:rados-thrash clusters:6-osd-3-machine.yaml fs:btrfs.yaml msgr-failures:few.yaml
thrashers:default.yaml workloads:snaps-few-objects.yaml
duration: 1845.6092298030853
failure_reason: 'Command failed with status 1: ''/tmp/cephtest/enable-coredump /tmp/cephtest/binary/usr/local/bin/ceph-coverage
/tmp/cephtest/archive/coverage /tmp/cephtest/daemon-helper kill /tmp/cephtest/binary/usr/local/bin/ceph-osd
-f -i 3 -c /tmp/cephtest/ceph.conf'''
flavor: basic
mds.a-kernel-sha1: e81d5d695a03a141b9a4a4e75b8e009ecba43c64
mon.a-kernel-sha1: e81d5d695a03a141b9a4a4e75b8e009ecba43c64
owner: scheduled_teuthology@teuthology
success: false
Also available in: Atom
PDF