Project

General

Profile

Bug #46323

thrash_cache_writeback_proxy_none: FAILED ceph_assert(version == old_value.version) in src/test/osd/RadosModel.h

Added by Neha Ojha 3 months ago. Updated 8 days ago.

Status:
New
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
octopus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature:

Description

2020-06-08T11:51:16.668 INFO:tasks.rados.rados.0.smithi104.stderr:10488: oid 9152 version is 8009 and expected 8808
2020-06-08T11:51:16.668 INFO:tasks.rados.rados.0.smithi104.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/
gigantic/release/16.0.0-2335-g2cb9872a203/rpm/el8/BUILD/ceph-16.0.0-2335-g2cb9872a203/src/test/osd/RadosModel.h: In function 'virtual void ReadOp::_finish(TestOp::CallbackInfo*)' thread 7fcb167fc700 time
2020-06-08T11:51:16.662931+0000
2020-06-08T11:51:16.669 INFO:tasks.rados.rados.0.smithi104.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/
gigantic/release/16.0.0-2335-g2cb9872a203/rpm/el8/BUILD/ceph-16.0.0-2335-g2cb9872a203/src/test/osd/RadosModel.h: 1397: FAILED ceph_assert(version == old_value.version)
2020-06-08T11:51:16.669 INFO:tasks.rados.rados.0.smithi104.stderr: ceph version 16.0.0-2335-g2cb9872a203 (2cb9872a203194b8c6ee2bf947577ee78c7dd961) pacific (dev)
2020-06-08T11:51:16.669 INFO:tasks.rados.rados.0.smithi104.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7fcb29d563a8]
2020-06-08T11:51:16.669 INFO:tasks.rados.rados.0.smithi104.stderr: 2: (()+0x2885c2) [0x7fcb29d565c2]
2020-06-08T11:51:16.669 INFO:tasks.rados.rados.0.smithi104.stderr: 3: (ReadOp::_finish(TestOp::CallbackInfo*)+0x215) [0x55590cdccd35]
2020-06-08T11:51:16.670 INFO:tasks.rados.rados.0.smithi104.stderr: 4: (()+0xa48bb) [0x7fcb332a68bb]
2020-06-08T11:51:16.670 INFO:tasks.rados.rados.0.smithi104.stderr: 5: (()+0xbf325) [0x7fcb332c1325]
2020-06-08T11:51:16.670 INFO:tasks.rados.rados.0.smithi104.stderr: 6: (()+0xc157a) [0x7fcb332c357a]
2020-06-08T11:51:16.670 INFO:tasks.rados.rados.0.smithi104.stderr: 7: (()+0xc5e5a) [0x7fcb332c7e5a]
2020-06-08T11:51:16.670 INFO:tasks.rados.rados.0.smithi104.stderr: 8: (()+0xc2b23) [0x7fcb2863fb23]
2020-06-08T11:51:16.671 INFO:tasks.rados.rados.0.smithi104.stderr: 9: (()+0x82de) [0x7fcb2918e2de]
2020-06-08T11:51:16.671 INFO:tasks.rados.rados.0.smithi104.stderr: 10: (clone()+0x43) [0x7fcb27d1c133]

/a/kchai-2020-06-08_10:56:36-rados-wip-kefu-testing-2020-06-08-1713-distro-basic-smithi/5128820
/a/dis-2020-06-28_18:43:20-rados-wip-msgr21-fix-reuse-rebuildci-distro-basic-smithi/5186890
/a/yuriw-2020-06-04_18:03:48-rados-wip-yuri2-testing-2020-06-03-2341-MASTER-distro-basic-smithi/5118028
/a/nojha-2020-05-21_19:33:40-rados-wip-32601-distro-basic-smithi/5077147

May or may not be related to https://github.com/ceph/ceph/pull/35015.

History

#1 Updated by Neha Ojha 3 months ago

  • Backport set to octopus

rados/singleton/{all/thrash_cache_writeback_proxy_none msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest}

2020-07-06T21:22:25.610 INFO:tasks.rados.rados.0.smithi089.stderr:Error: racing read on 3505 returned version 8643 rather than version 8668
2020-07-06T21:22:25.610 INFO:tasks.rados.rados.0.smithi089.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.4-31-g54563c40723/rpm/el8/BUILD/ceph-15.2.4-31-g54563c40723/src/test/osd/RadosModel.h: In function 'virtual void WriteOp::_finish(TestOp::CallbackInfo*)' thread 7fba6a7fc700 time 2020-07-06T21:22:25.607847+0000
2020-07-06T21:22:25.611 INFO:tasks.rados.rados.0.smithi089.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.4-31-g54563c40723/rpm/el8/BUILD/ceph-15.2.4-31-g54563c40723/src/test/osd/RadosModel.h: 934: ceph_abort_msg("racing read got wrong version")
2020-07-06T21:22:25.611 INFO:tasks.rados.rados.0.smithi089.stderr: ceph version 15.2.4-31-g54563c40723 (54563c407238a10af9366d1a19805d53f9f8bfb7) octopus (stable)
2020-07-06T21:22:25.611 INFO:tasks.rados.rados.0.smithi089.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe5) [0x7fba8917423c]
2020-07-06T21:22:25.611 INFO:tasks.rados.rados.0.smithi089.stderr: 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5b6) [0x55db15f39d16]
2020-07-06T21:22:25.611 INFO:tasks.rados.rados.0.smithi089.stderr: 3: (write_callback(void*, void*)+0x1d) [0x55db15f5a33d]
2020-07-06T21:22:25.612 INFO:tasks.rados.rados.0.smithi089.stderr: 4: (()+0x9e45e) [0x7fba9268045e]
2020-07-06T21:22:25.612 INFO:tasks.rados.rados.0.smithi089.stderr: 5: (()+0x619cd) [0x7fba926439cd]
2020-07-06T21:22:25.612 INFO:tasks.rados.rados.0.smithi089.stderr: 6: (Finisher::finisher_thread_entry()+0x1a5) [0x7fba89203955]
2020-07-06T21:22:25.612 INFO:tasks.rados.rados.0.smithi089.stderr: 7: (()+0x82de) [0x7fba885bd2de]
2020-07-06T21:22:25.612 INFO:tasks.rados.rados.0.smithi089.stderr: 8: (clone()+0x43) [0x7fba87150133]

/a/yuriw-2020-07-06_17:23:10-rados-wip-yuri8-testing-2020-07-01-2358-octopus-distro-basic-smithi/5203870

Pretty sure these are related

#2 Updated by Kefu Chai 2 months ago

/a//kchai-2020-07-15_09:19:03-rados-wip-kefu-testing-2020-07-13-2108-distro-basic-smithi/5228761

#3 Updated by Brad Hubbard 2 months ago

/a/yuriw-2020-07-13_23:00:15-rados-wip-yuri8-testing-2020-07-13-1946-octopus-distro-basic-smithi/5224163

#4 Updated by Kefu Chai about 2 months ago

/a/kchai-2020-07-31_01:42:48-rados-wip-kefu-testing-2020-07-30-2107-distro-basic-smithi/5271969

#5 Updated by Deepika Upadhyay 30 days ago

/a/yuriw-2020-08-26_18:16:40-rados-wip-yuri-testing-2020-08-26-1631-octopus-distro-basic-smithi/5378493

2020-08-27T01:45:11.785 INFO:tasks.rados.rados.0.smithi083.stdout:update_object_version oid 7722 v 8834 (ObjNum 10441 snap 0 seq_num 10441) dirty exists
2020-08-27T01:45:11.785 INFO:tasks.rados.rados.0.smithi083.stderr:Error: racing read on 7722 returned version 8775 rather than version 8834
2020-08-27T01:45:11.786 INFO:tasks.rados.rados.0.smithi083.stderr:/build/ceph-15.2.4-804-g1ed4f8a5e55/src/test/osd/RadosModel.h: In function 'virtual void WriteOp::_finish(TestOp::CallbackInfo*)' thread 7f988d7fa700 time 2020-08-27T01:45:11.784846+0000
2020-08-27T01:45:11.786 INFO:tasks.rados.rados.0.smithi083.stderr:/build/ceph-15.2.4-804-g1ed4f8a5e55/src/test/osd/RadosModel.h: 934: ceph_abort_msg("racing read got wrong version")
2020-08-27T01:45:11.786 INFO:tasks.rados.rados.0.smithi083.stderr: ceph version 15.2.4-804-g1ed4f8a5e55 (1ed4f8a5e559f4f1d47ae41b42e5fd1371af2e2a) octopus (stable)
2020-08-27T01:45:11.787 INFO:tasks.rados.rados.0.smithi083.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe1) [0x7f989c0d0c8e]
2020-08-27T01:45:11.787 INFO:tasks.rados.rados.0.smithi083.stderr: 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6b0) [0x5636491647e0]
2020-08-27T01:45:11.787 INFO:tasks.rados.rados.0.smithi083.stderr: 3: (write_callback(void*, void*)+0x19) [0x563649180f39]
2020-08-27T01:45:11.787 INFO:tasks.rados.rados.0.smithi083.stderr: 4: (()+0x9181a) [0x7f98a4d6981a]
2020-08-27T01:45:11.787 INFO:tasks.rados.rados.0.smithi083.stderr: 5: (()+0x54769) [0x7f98a4d2c769]
2020-08-27T01:45:11.788 INFO:tasks.rados.rados.0.smithi083.stderr: 6: (Finisher::finisher_thread_entry()+0x195) [0x7f989c123885]
2020-08-27T01:45:11.788 INFO:tasks.rados.rados.0.smithi083.stderr: 7: (()+0x76db) [0x7f989aeae6db]
2020-08-27T01:45:11.788 INFO:tasks.rados.rados.0.smithi083.stderr: 8: (clone()+0x3f) [0x7f989b5f3a3f]
2020-08-27T01:45:12.295 DEBUG:teuthology.orchestra.run:got remote process result: None
2020-08-27T01:45:12.299 ERROR:teuthology:Uncaught exception (Hub)
Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 766, in gevent._greenlet.Greenlet.run
  File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-yuri-testing-2020-08-26-1631-octopus/qa/tasks/rados.py", line 266, in thread
    run.wait(tests.values())
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 470, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 178, in _raise_for_status
    raise CommandCrashedError(command=self.command)
teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base'

/a/yuriw-2020-08-27_00:49:53-rados-wip-yuri8-testing-2020-08-26-2329-octopus-distro-basic-smithi/5379221

#6 Updated by Neha Ojha 24 days ago

/a/yuriw-2020-08-31_19:07:15-rados-octopus-distro-basic-smithi/5396037

#7 Updated by Neha Ojha 8 days ago

this still fails consistently

/a/teuthology-2020-09-17_07:01:02-rados-master-distro-basic-smithi/5443303

Also available in: Atom PDF