Project

General

Profile

Actions
Copy link

Bug #2834

closed

osd/ReplicatedPG.cc: 3577: FAILED assert(waiting_for_ack.begin()->first == repop->v)

Added by Sage Weil over 11 years ago. Updated over 11 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
-
Category:
OSD
Target version:
-
% Done:

0%

Source:
Development
Tags:
Backport:
Regression:
Severity:
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description


    -5> 2012-07-24 14:06:49.664524 7f1bb757b700 -1 osd/ReplicatedPG.cc: In function 'void ReplicatedPG::eval_repop(ReplicatedPG::RepGather*)' thread 7f1bb757b700 time 2012-07-24 14:06:49.549566
osd/ReplicatedPG.cc: 3577: FAILED assert(waiting_for_ack.begin()->first == repop->v)

 ceph version 0.49-299-g9ecc5c2 (commit:9ecc5c2c9c31f9ab8a01cba47690c32b6792b9c5)
 1: (ReplicatedPG::eval_repop(ReplicatedPG::RepGather*)+0x75e) [0x55122e]
 2: (ReplicatedPG::op_applied(ReplicatedPG::RepGather*)+0x57b) [0x554f6b]
 3: (C_OSD_OpApplied::finish(int)+0x11) [0x5a62d1]
 4: (Finisher::finisher_thread_entry()+0x218) [0x7315b8]
 5: (()+0x7e9a) [0x7f1bc32e6e9a]
 6: (clone()+0x6d) [0x7f1bc189b4bd]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

ubuntu@teuthology:/a/sage-2012-07-24_13:13:37-regression-wip-msgr-testing-basic/17035$ cat config.yaml 
kernel: &id001
  kdb: true
  sha1: 77dca1ac33894de22b1740bb9cf6b8ef6429c700
nuke-on-error: true
overrides:
  ceph:
    conf:
      global:
        ms inject socket failures: 200
    fs: btrfs
    log-whitelist:
    - slow request
    sha1: 9ecc5c2c9c31f9ab8a01cba47690c32b6792b9c5
  workunit:
    sha1: 9ecc5c2c9c31f9ab8a01cba47690c32b6792b9c5
roles:
- - mon.a
  - mon.c
  - osd.0
  - osd.1
  - osd.2
- - mon.b
  - mds.a
  - osd.3
  - osd.4
  - osd.5
- - client.0
targets:
  ubuntu@plana25.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDuXajaQgHe9XnbLOzI8WWFYVz6+TnOiTzbkIJPGOZpzQEjnUtJraQIEt5ABSeovMjiEj+V4XvunfyuSmEd0H9giRSyjmCHTPGlpndfTeCdVtCBpNqf5GkUqHaEY1Hp57XPbya2rGlwtFm0NeIDYx6pfkejKnsTOUqwhgUb6950TRhjHQhMjFgyALSyfAm/4y6vGZfjm57+yyih6XgDkqWiiQ6Y/aJVR2n+iCzvqEzV7JSCU+Brn+k8IQLHho1fadYqc5PjYct5BaVlHcP6c+T8nJE/DvqGwZ4gQaVJcuWJiDfLOPPYo1g/0AFicxauLwVNJ6HFR9FjLLGtGU+2DcVN
  ubuntu@plana61.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOTCMIScDTmD9NkfsWU7xeyZ+WOXai5izYeliiXDSjJC3bT6r8Fp+rhPfcHCVHiw++VsbvKZtkhjCSnJTVPWCdpRDghzJ3nZUBImWRo3PmHo1etQpCeimaOrIJ2q0ChN5jmSOqy5B+Z4om2vXBtBY6nkdTxDOr2+MH3NrSPkQSFB0zO+VPuwKXsemeUC6urb2IZZpxY3cxNq4fafTF9PROpgOnIA+o3igyU4duKEjnCzTHZjw/PL7Eph/7p6+UQgrUwe7pgVzT+2MM0zcBtBSXNqs3dCGmpvUapOkBlDoIX02EkWRNpkM3vfeFt1EFC17B5vd61Kg40bYUG8qWGR0T
  ubuntu@plana64.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDk4GmsUmC8svnRI6Xd+mRX2MwKb4RHECAeLfqTm2COfqfolS2wKGw3U92eJcyvpZ+2p82X7uBrimjZh5JgRtxJ1aGUG4Pi60+JBYF0WpohM/3aYISFegVNET9rcapdDaAi6fFB5vhT06Q/cYEO0tPrdqGb/O3oiDSurtqtfOzkdwSPWSTY/hSegXgOeG6EjuEfvnU4BbgXWkLlDQRXCdgQd35F0SlKJVgMo+J1MgMCEK4qnBMFN614P1gBSzZCBsSUGQdjYBOzZfCRlI2bUdPDtB0kyjp7o5Ns9gLd07TLw8h9oxvI7wxG16XnLOAIzPBNOaH4OztTMGg3wJ/1e26t
task:
- ceph:
    conf:
      client:
        rbd cache: true
tasks:
- internal.lock_machines: 3
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.check_conflict: null
- kernel: *id001
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock: null
- ceph: null
- qemu:
    all:
      test: https://raw.github.com/ceph/ceph/master/qa/workunits/suites/tiobench.sh
Actions
Copy link
 #1

Updated by Sage Weil over 11 years ago

  • Assignee deleted (Sage Weil)
Actions
Copy link
 #2

Updated by Sage Weil over 11 years ago

  • Status changed from New to Resolved

hasn't come up recently

Actions
Copy link
 #3

Updated by Tamilarasi muthamizhan over 11 years ago

Recent log: ubuntu@teuthology:/a/teuthology-2012-09-07_00:00:07-regression-next-testing-basic/17906

Actions
Copy link
 #4

Updated by Tamilarasi muthamizhan over 11 years ago

Recent log: ubuntu@teuthology:/a/teuthology-2012-09-07_00:00:07-regression-next-testing-basic/17906

2012-09-07 03:43:22.940159 7f748e13d700 -1 osd/ReplicatedPG.cc: In function 'void ReplicatedPG::eval_repo
p(ReplicatedPG::RepGather*)' thread 7f748e13d700 time 2012-09-07 03:43:22.938766
osd/ReplicatedPG.cc: 3582: FAILED assert(waiting_for_ack.begin()->first == repop->v)

 ceph version 0.51-274-g5f36b8d (commit:5f36b8d78416b7a1d1bbefecddfcee00b7bfcfa3)
 1: (ReplicatedPG::eval_repop(ReplicatedPG::RepGather*)+0x790) [0x551560]
 2: (ReplicatedPG::repop_ack(ReplicatedPG::RepGather*, int, int, int, eversion_t)+0x1d4) [0x5527b4]
 3: (ReplicatedPG::sub_op_modify_reply(std::tr1::shared_ptr<OpRequest>)+0x172) [0x554f12]
 4: (ReplicatedPG::do_sub_op_reply(std::tr1::shared_ptr<OpRequest>)+0x82) [0x585ab2]
 5: (PG::do_request(std::tr1::shared_ptr<OpRequest>)+0x325) [0x6556c5]
 6: (OSD::dequeue_op(PG*)+0x27c) [0x5ba1ec]
 7: (ThreadPool::worker()+0x523) [0x7cc343]
 8: (ThreadPool::WorkThread::entry()+0xd) [0x5fa52d]
 9: (()+0x7e9a) [0x7f749e9c3e9a]
 10: (clone()+0x6d) [0x7f749cd674bd]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

ubuntu@teuthology:/a/teuthology-2012-09-07_00:00:07-regression-next-testing-basic/17906$ cat config.yaml 
kernel: &id001
  kdb: true
  sha1: e81d5d695a03a141b9a4a4e75b8e009ecba43c64
nuke-on-error: true
overrides:
  ceph:
    conf:
      global:
        ms inject socket failures: 5000
    fs: btrfs
    log-whitelist:
    - slow request
    sha1: 5f36b8d78416b7a1d1bbefecddfcee00b7bfcfa3
  workunit:
    sha1: 5f36b8d78416b7a1d1bbefecddfcee00b7bfcfa3
roles:
- - mon.a
  - osd.0
  - osd.1
  - osd.2
- - mds.a
  - osd.3
  - osd.4
  - osd.5
- - client.0
targets:
  ubuntu@plana23.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDiwH5Qz5dXnbtYRiTk0QVNNyZQWYcardED+AqVLxoz5h/z/tPUyt6VTcrNvyFiKZcrz70vKy/1S1JNmt74gSc0KE8YhLjvuCaTwJDw1LOTNzc5b074zfnjeNGKqb0L3BefbFFOMh/ZuxGbTJWZXdD1DwP2VWxGdhtHAxglgLjt5541nxw41vT+dVMgQMt7Lv5P3MXl+IY58LUzYC9EkOvgZTPTfRx7IptkDSEmbYGL7dQE6H9VyoukOejj4jgg8ZWhPR9e39OhB/Vh7qtiCTRfapovh1zrfCM/b7O1/nMisUmfOK+nF2ruiTefEA14u59uxlfpaRLQDtyv6b5aPour
  ubuntu@plana42.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzQfmtpfECJ+NZaaiSH/R8X+dGXHH+aDTCKGLLiHhW9fttxzfzcJJaBx1b664D3ynZAC7NiaegfLDTCMW7FFVDUltMQcWjsM4BqfFipIquDP4KOclCc6EwG5aYG/MLCJwL6sovt1uKg00bSkVQsUSHBgZbMJKCjCbBb0XPxfuS4dppA3diEZBOMt1YHr+NdV7sace/Gc7YBlGsNOinnqkKfVWIpfYCiTQ18cvaisSEHsQR6zhKqrX4afQk13cTjdvZeQp9AXxRIf1g9fq2zHVWMdJdVNR8D0BSBtfAzMqIqZ8qcJqmzQN0Zq9Wk9Y021vMFORZy2SFI6c7yBWDJLdT
  ubuntu@plana52.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9kswBp2g5ZV1Qrvlee8MvUOCNdubQFqUBr5WSsmFBODqEuiitWbhuBu2Ucz0lBMf41DpMKLeYDN0lIC94GZmGaiCN+Ak9Ia05d/uRvesT2nDgHB3Z9J/zEFlY8RVxL3xhD+hq4u8dbASlqqoMDiBP+7efZMxt4Ndnzr/yOxge3KenxyQImBUS+OV+BqnfCOHf6BqM33U1leXz2kng7ocxoE91DAMslKD/2DPRSYEhfucUJZk6IYevr/g0JVhbfvjSlZzwUEfTyVmPeqNyls/U+azhKlvQbqpb+ttc02RNydQ1YgOgHFCaqd9Vm8XjUU6vYGlkFHZ+BMJuEwA9AH/D
tasks:
- internal.lock_machines: 3
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.check_conflict: null
- kernel: *id001
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock: null
- ceph:
    log-whitelist:
    - wrongly marked me down
    - objects unfound and apparently lost
- thrashosds:
    timeout: 1200
- rados:
    clients:
    - client.0
    objects: 50
    op_weights:
      delete: 50
      read: 100
      snap_create: 50
      snap_remove: 50
      snap_rollback: 50
      write: 100
    ops: 4000

ubuntu@teuthology:/a/teuthology-2012-09-07_00:00:07-regression-next-testing-basic/17906$ cat summary.yaml 
ceph-sha1: 5f36b8d78416b7a1d1bbefecddfcee00b7bfcfa3
client.0-kernel-sha1: e81d5d695a03a141b9a4a4e75b8e009ecba43c64
description: collection:rados-thrash clusters:6-osd-3-machine.yaml fs:btrfs.yaml msgr-failures:few.yaml
  thrashers:default.yaml workloads:snaps-few-objects.yaml
duration: 1845.6092298030853
failure_reason: 'Command failed with status 1: ''/tmp/cephtest/enable-coredump /tmp/cephtest/binary/usr/local/bin/ceph-coverage
  /tmp/cephtest/archive/coverage /tmp/cephtest/daemon-helper kill /tmp/cephtest/binary/usr/local/bin/ceph-osd
  -f -i 3 -c /tmp/cephtest/ceph.conf'''
flavor: basic
mds.a-kernel-sha1: e81d5d695a03a141b9a4a4e75b8e009ecba43c64
mon.a-kernel-sha1: e81d5d695a03a141b9a4a4e75b8e009ecba43c64
owner: scheduled_teuthology@teuthology
success: false
Actions
Copy link

Also available in: Atom PDF