Bug #4110: assertion in DeleteOp::_begin - Ceph - Ceph

Actions

Copy link

Bug #4110

closed

assertion in DeleteOp::_begin

Added by Tamilarasi muthamizhan about 11 years ago. Updated about 11 years ago.

Status:

Resolved

Priority:

Urgent

Assignee:

Sage Weil

Category:

Target version:

% Done:

Source:

Q/A

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

log: ubuntu@teuthology:/a/teuthology-2013-02-12_01:00:03-regression-next-testing-basic/5133

2013-02-12T03:51:22.629 INFO:teuthology.task.rados.rados.0.err: ceph version 0.56-651-g31e911b (31e911b63d326bdd06981ec4029ad71b7479ed70)
2013-02-12T03:51:22.629 INFO:teuthology.task.rados.rados.0.err: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x95) [0x7fc0d59f673d]
2013-02-12T03:51:22.629 INFO:teuthology.task.rados.rados.0.err: 2: (DeleteOp::_begin()+0x492) [0x44727a]
2013-02-12T03:51:22.629 INFO:teuthology.task.rados.rados.0.err: 3: (TestOp::begin()+0x23) [0x466587]
2013-02-12T03:51:22.630 INFO:teuthology.task.rados.rados.0.err: 4: (RadosTestContext::loop(TestOpGenerator*)+0xd8) [0x4420aa]
2013-02-12T03:51:22.630 INFO:teuthology.task.rados.rados.0.err: 5: (main()+0x9dd) [0x440251]
2013-02-12T03:51:22.630 INFO:teuthology.task.rados.rados.0.err: 6: (__libc_start_main()+0xed) [0x7fc0d476576d]
2013-02-12T03:51:22.630 INFO:teuthology.task.rados.rados.0.err: 7: /tmp/cephtest/binary/usr/local/bin/testrados() [0x43f7b9]
2013-02-12T03:51:22.631 INFO:teuthology.task.rados.rados.0.err: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

ubuntu@teuthology:/a/teuthology-2013-02-12_01:00:03-regression-next-testing-basic/5133$ cat config.yaml 
kernel: &id001
  kdb: true
  sha1: 4ad2b189c1a52ce8ae1d6d2528c512021a2f1654
nuke-on-error: true
overrides:
  ceph:
    conf:
      global:
        ms inject socket failures: 5000
    fs: ext4
    log-whitelist:
    - slow request
    sha1: 31e911b63d326bdd06981ec4029ad71b7479ed70
  s3tests:
    branch: next
  workunit:
    sha1: 31e911b63d326bdd06981ec4029ad71b7479ed70
roles:
- - mon.a
  - mon.b
  - osd.0
  - osd.1
  - osd.2
- - mon.c
  - mds.a
  - osd.3
  - osd.4
  - osd.5
- - client.0
targets:
  ubuntu@plana19.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0IExVUuQEnwBzUbN6+jgEb/qLLsSDFLAe0OPv+R3Q3uUZn+4QjB+FQ3sMMEwaGpEhUfbpWn3xZbCmi4NngTsgPDjImYJPMeMecaxvVXqAJt4IM6gdBN0415lrKXtbXaGBJCmeFFDB+xN+JN6Qhk8DWN62DnJB8MS5vKn9u0S3HvtGzY/QrnutT3AX9I4097isbTepLFRC4n2CoAC9srXaxAprFgLgOIYHm386B3W7yK3yfIImWofvZxYpPCsr/7ws8DSgdRX9eDucGQfY2WcYKCEIgZclGPBhExm7q/ahHUqYKrzWY3RD93AuXJVgOJk5Yp8C1Ryx8vyJkAxBW4nb
  ubuntu@plana85.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCQQz4HH+uZV5duWKnH+ZVOyBzgUIw3K4G31TlhhiDrxLQJIBFeYHacS9+TzSdTSsgDvikBmX43J/VUdEE+W6jL8bZ5cwXcULXdiaO6eFbnZOld76WdI3Rb6+F2cAQUBAFEmy4FvdGVteMYlLZVo+3VvMckM1C09RDlhA8AHyNf6AisZEkV0gu6ba7BKmfj8CxGI+Hs/zFRa0wyLmJgJIgPqvo5yikBAAZmVPyfOt8jUe3sfhf5R8o2hlr8ucBc9jHlpbspc1fJJ0HMOE2cIjCN5vup2CtjTmnxhghAeYujIJL8mkT81RiCPrXxngg8X8CkaH2KevviMElXjK/dtLL
  ubuntu@plana91.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCwq9PdHwBKUjNa86vjFTDwYcnXBXF81QqgOmXBN8NNbePIGhjY9Ji6IvvGwrywnCi9a46tY9hFPCvq6y+6tC7Pfjf248C7KYWAxqA4UPCkQQMvedJCk2RAH7oljHqXGuyfFgsveq11bWjTWRYZPpENwfjdB0okvr8TZrZePhJNYgMY2smiSDBQ96Cr211KnQk/cvGFC0u8x/R/MJaZBVD0GANH8xLpe8PpND3uuNIEox/CmXpC+XdXVGnm1AXpF8Bb4gDSgwF8NUyedj8c8GfC0iI1ISg96WYFfNgS4m9ulOlXl0SSdYQ3+DCr6aOz7Nk+2r84Vq8E5wpiKD9z+raJ
tasks:
- internal.lock_machines:
  - 3
  - plana
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.check_conflict: null
- kernel: *id001
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock: null
- ceph:
    log-whitelist:
    - wrongly marked me down
    - objects unfound and apparently lost
- thrashosds:
    timeout: 1200
- rados:
    clients:
    - client.0
    objects: 500
    op_weights:
      delete: 10
      read: 45
      write: 45
    ops: 4000
ubuntu@teuthology:/a/teuthology-2013-02-12_01:00:03-regression-next-testing-basic/5133$ cat summary.yaml 
ceph-sha1: 31e911b63d326bdd06981ec4029ad71b7479ed70
client.0-kernel-sha1: 4ad2b189c1a52ce8ae1d6d2528c512021a2f1654
description: collection:rados-thrash clusters:6-osd-3-machine.yaml fs:ext4.yaml msgr-failures:few.yaml
  thrashers:default.yaml workloads:readwrite.yaml
duration: 2230.463079929352
failure_reason: 'Command crashed: ''CEPH_CLIENT_ID=0 CEPH_CONF=/tmp/cephtest/ceph.conf
  LD_LIBRARY_PATH=/tmp/cephtest/binary/usr/local/lib /tmp/cephtest/enable-coredump
  /tmp/cephtest/binary/usr/local/bin/ceph-coverage /tmp/cephtest/archive/coverage
  /tmp/cephtest/binary/usr/local/bin/testrados --op read 45 --op write 45 --op delete
  10 --op snap_create 0 --op snap_remove 0 --op rollback 0 --op setattr 0 --op rmattr
  0 --op watch 0 --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size
  400000 --max-stride-size 800000 --max-seconds 0'''
flavor: basic
mon.a-kernel-sha1: 4ad2b189c1a52ce8ae1d6d2528c512021a2f1654
mon.c-kernel-sha1: 4ad2b189c1a52ce8ae1d6d2528c512021a2f1654
owner: scheduled_teuthology@teuthology
success: false

Actions

Copy link

Updated by Ian Colle about 11 years ago

Assignee set to Samuel Just
Priority changed from Normal to Urgent

Actions

Copy link

Updated by Sage Weil about 11 years ago

Similar:

2013-02-18T06:37:34.711 INFO:teuthology.task.rados.rados.0.out:Writing 5 current snap is 0
2013-02-18T06:37:34.712 INFO:teuthology.task.rados.rados.0.out:waiting_on = 2
2013-02-18T06:37:34.712 INFO:teuthology.task.rados.rados.0.out:Writing plana3412525-5 from 0 to 55 tid 1 ranges are [0~55,566564~400000]
2013-02-18T06:37:34.720 INFO:teuthology.task.rados.rados.0.err:Found incorrect object contents ObjNum: 1559 snap: 0 seqnum: 1559 prefix: plana3412525-OID: 387 snap 0
2013-02-18T06:37:34.720 INFO:teuthology.task.rados.rados.0.err:, expected ObjNum: 2168 snap: 0 seqnum: 2168 prefix: plana3412525-OID: 387 snap 0
2013-02-18T06:37:34.720 INFO:teuthology.task.rados.rados.0.err: oid 387
2013-02-18T06:37:34.720 INFO:teuthology.task.rados.rados.0.err:Object 387 contents ObjNum: 1559 snap: 0 seqnum: 1559 prefix: plana3412525-OID: 387 snap 0
2013-02-18T06:37:34.721 INFO:teuthology.task.rados.rados.0.err: corrupt
2013-02-18T06:37:34.721 INFO:teuthology.task.rados.rados.0.out:incorrect buffer at pos 4
2013-02-18T06:37:34.726 INFO:teuthology.task.rados.rados.0.err:./test/osd/RadosModel.h: In function 'virtual void ReadOp::_finish(TestOp::CallbackInfo*)' thread 7f7f1effd700 time 2013-02-18 06:37:29.549218
2013-02-18T06:37:34.727 INFO:teuthology.task.rados.rados.0.err:./test/osd/RadosModel.h: 1057: FAILED assert(0)
2013-02-18T06:37:34.727 INFO:teuthology.task.rados.rados.0.err: ceph version 0.56-664-g3f5c252 (3f5c25279622ea59624ae689cd21e9d69c852c0c)
2013-02-18T06:37:34.728 INFO:teuthology.task.rados.rados.0.err: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x95) [0x7f7f29d9573d]
2013-02-18T06:37:34.728 INFO:teuthology.task.rados.rados.0.err: 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x46a) [0x448390]
2013-02-18T06:37:34.728 INFO:teuthology.task.rados.rados.0.err: 3: (TestOp::finish(TestOp::CallbackInfo*)+0x2e) [0x4665b8]
2013-02-18T06:37:34.728 INFO:teuthology.task.rados.rados.0.err: 4: (read_callback(void*, void*)+0x29) [0x4665e3]
2013-02-18T06:37:34.729 INFO:teuthology.task.rados.rados.0.err: 5: (librados::C_AioComplete::finish(int)+0x4d) [0x7f7f29d1fced]
2013-02-18T06:37:34.729 INFO:teuthology.task.rados.rados.0.err: 6: (Finisher::finisher_thread_entry()+0x342) [0x7f7f29d948ae]
2013-02-18T06:37:34.729 INFO:teuthology.task.rados.rados.0.err: 7: (Finisher::FinisherThread::entry()+0x1c) [0x7f7f29d08d88]
2013-02-18T06:37:34.729 INFO:teuthology.task.rados.rados.0.err: 8: (Thread::_entry_func(void*)+0x23) [0x7f7f29f38fc1]
2013-02-18T06:37:34.730 INFO:teuthology.task.rados.rados.0.err: 9: (()+0x7e9a) [0x7f7f293bde9a]
2013-02-18T06:37:34.730 INFO:teuthology.task.rados.rados.0.err: 10: (clone()+0x6d) [0x7f7f28bd54bd]
2013-02-18T06:37:34.730 INFO:teuthology.task.rados.rados.0.err: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

ubuntu@teuthology:/a/sage-2013-02-17_22:40:36-regression-next-testing-basic/4220
job was

kernel:
  kdb: true
  sha1: 077413082f9ade9ca4d9774dbdc81ee7256d8089
nuke-on-error: true
overrides:
  ceph:
    conf:
      global:
        ms inject delay max: 1
        ms inject delay probability: 0.005
        ms inject delay type: osd
        ms inject socket failures: 2500
    fs: xfs
    log-whitelist:
    - slow request
    sha1: 3f5c25279622ea59624ae689cd21e9d69c852c0c
  s3tests:
    branch: next
  workunit:
    sha1: 3f5c25279622ea59624ae689cd21e9d69c852c0c
roles:
- - mon.a
  - mon.b
  - osd.0
  - osd.1
  - osd.2
- - mon.c
  - mds.a
  - osd.3
  - osd.4
  - osd.5
- - client.0
tasks:
- chef: null
- clock: null
- ceph:
    log-whitelist:
    - wrongly marked me down
    - objects unfound and apparently lost
- thrashosds:
    timeout: 1200
- rados:
    clients:
    - client.0
    objects: 500
    op_weights:
      delete: 10
      read: 45
      write: 45
    ops: 4000

Actions

Copy link