Project

General

Profile

Actions
Copy link

Bug #13971

closed

"test/multi_stress_watch.cc: 61: FAILED assert(!ret)" in rados-hammer-distro-basic-openstack

Added by Yuri Weinstein over 8 years ago. Updated over 7 years ago.

Status:
Can't reproduce
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
rados
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Run: http://pulpito.ovh.sepia.ceph.com:8081/teuthology-2015-12-02_20:55:02-rados-hammer-distro-basic-openstack/
Job: 25961
Logs: http://teuthology.ovh.sepia.ceph.com/teuthology/teuthology-2015-12-02_20:55:02-rados-hammer-distro-basic-openstack/25961/teuthology.log

2015-12-02T21:49:47.004 INFO:tasks.workunit.client.0.target071208.stderr:Iteration 930
2015-12-02T21:49:47.037 INFO:tasks.workunit.client.0.target071208.stderr:Iteration 931
2015-12-02T21:49:57.067 INFO:tasks.workunit.client.0.target071208.stderr:test/multi_stress_watch.cc: In function 'void test_loop(librados::Rados&, std::string, std::string)' thread 7fa41dd00840 time 2015-12-02 21:49:56.964549
2015-12-02T21:49:57.067 INFO:tasks.workunit.client.0.target071208.stderr:test/multi_stress_watch.cc: 61: FAILED assert(!ret)
2015-12-02T21:49:57.068 INFO:tasks.workunit.client.0.target071208.stderr: ceph version 0.94.5-163-g8c4145e (8c4145ecc4a68accdb2120889fd933e8f6630dba)
2015-12-02T21:49:57.069 INFO:tasks.workunit.client.0.target071208.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x40746b]
2015-12-02T21:49:57.069 INFO:tasks.workunit.client.0.target071208.stderr: 2: (test_loop(librados::Rados&, std::string, std::string)+0x229) [0x406b09]
2015-12-02T21:49:57.069 INFO:tasks.workunit.client.0.target071208.stderr: 3: (test_replicated(librados::Rados&, std::string, std::string)+0x45) [0x406c25]
2015-12-02T21:49:57.070 INFO:tasks.workunit.client.0.target071208.stderr: 4: (main()+0x34e) [0x4063ee]
2015-12-02T21:49:57.070 INFO:tasks.workunit.client.0.target071208.stderr: 5: (__libc_start_main()+0xf5) [0x7fa41a4e6ec5]
2015-12-02T21:49:57.070 INFO:tasks.workunit.client.0.target071208.stderr: 6: ceph_multi_stress_watch() [0x406817]
2015-12-02T21:49:57.070 INFO:tasks.workunit.client.0.target071208.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2015-12-02T21:49:57.071 INFO:tasks.workunit.client.0.target071208.stderr:terminate called after throwing an instance of 'ceph::FailedAssertion'
2015-12-02T21:49:57.107 INFO:tasks.workunit.client.0.target071208.stderr:Aborted (core dumped)
2015-12-02T21:49:57.108 INFO:tasks.workunit:Stopping ['rados/stress_watch.sh'] on client.0...

Related issues 2 (0 open — 2 closed)

Related to Ceph - Bug #10441: osd: dup watch can reply before watch is persistedResolvedSamuel Just12/30/2014

Actions
Related to Ceph - Bug #10564: test/multi_stress_watch.cc: 60: FAILED assert(!ret)DuplicateSage Weil01/18/2015

Actions
Actions
Copy link
 #1

Updated by Yuri Weinstein over 8 years ago

  • Related to Bug #10441: osd: dup watch can reply before watch is persisted added
Actions
Copy link
 #2

Updated by Yuri Weinstein over 8 years ago

  • Related to Bug #10564: test/multi_stress_watch.cc: 60: FAILED assert(!ret) added
Actions
Copy link
 #3

Updated by Yuri Weinstein over 8 years ago

Run: http://pulpito.ovh.sepia.ceph.com:8081/teuthology-2016-01-20_02:00:02-rados-infernalis-distro-basic-openstack/
Job: 7169
Logs: http://teuthology.ovh.sepia.ceph.com/teuthology/teuthology-2016-01-20_02:00:02-rados-infernalis-distro-basic-openstack/7169/teuthology.log

2016-01-20T04:09:56.319 INFO:tasks.workunit.client.0.target085247.stderr:test/multi_stress_watch.cc: In function 'void test_loop(librados::Rados&, std::string, std::string)' thread 7f60d2f1a7c0 time 2016-01-20 04:09:56.276685
2016-01-20T04:09:56.319 INFO:tasks.workunit.client.0.target085247.stderr:test/multi_stress_watch.cc: 61: FAILED assert(!ret)
2016-01-20T04:09:56.323 INFO:tasks.workunit.client.0.target085247.stderr: ceph version 9.2.0-39-g1296c2b (1296c2baef3412f462ee2124af747a892ea8b7a9)
2016-01-20T04:09:56.324 INFO:tasks.workunit.client.0.target085247.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7f60d2d2ca5b]
2016-01-20T04:09:56.324 INFO:tasks.workunit.client.0.target085247.stderr: 2: (test_loop(librados::Rados&, std::string, std::string)+0x26f) [0x7f60d2d2c12f]
2016-01-20T04:09:56.324 INFO:tasks.workunit.client.0.target085247.stderr: 3: (test_replicated(librados::Rados&, std::string, std::string)+0x49) [0x7f60d2d2c259]
2016-01-20T04:09:56.325 INFO:tasks.workunit.client.0.target085247.stderr: 4: (main()+0x36d) [0x7f60d2d2b97d]
2016-01-20T04:09:56.325 INFO:tasks.workunit.client.0.target085247.stderr: 5: (__libc_start_main()+0xf5) [0x7f60cf5a2ec5]
2016-01-20T04:09:56.325 INFO:tasks.workunit.client.0.target085247.stderr: 6: (()+0x6dc7) [0x7f60d2d2bdc7]
2016-01-20T04:09:56.326 INFO:tasks.workunit.client.0.target085247.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Actions
Copy link
 #4

Updated by Sage Weil about 8 years ago

  • Priority changed from Normal to High
Actions
Copy link
 #5

Updated by Samuel Just about 8 years ago

  • Priority changed from High to Urgent
Actions
Copy link
 #6

Updated by Sage Weil about 8 years ago

  • Status changed from New to Need More Info

Odd we haven't seen this on master. I don't see anything that may have inadvertantly fixed it except haomai's recent rewrite of the watch/notify to allow async ops.

Actions
Copy link
 #7

Updated by Samuel Just about 8 years ago

  • Status changed from Need More Info to Can't reproduce
Actions
Copy link
 #8

Updated by Yuri Weinstein about 8 years ago

  • Status changed from Can't reproduce to New

still see it
Run: http://pulpito.ceph.com/teuthology-2016-04-06_09:00:02-rados-hammer-distro-basic-vps/
Job: 111404
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2016-04-06_09:00:02-rados-hammer-distro-basic-vps/111404/teuthology.log

2016-04-06T09:49:10.035 INFO:tasks.workunit.client.0.vpm054.stderr:Iteration 5460
2016-04-06T09:49:10.091 INFO:tasks.workunit.client.0.vpm054.stderr:Iteration 5461
2016-04-06T09:49:10.210 INFO:tasks.workunit.client.0.vpm054.stderr:Iteration 5462
2016-04-06T09:49:20.337 INFO:tasks.workunit.client.0.vpm054.stderr:test/multi_stress_watch.cc: In function 'void test_loop(librados::Rados&, std::string, std::string)' thread 7fc98711d7c0 time 2016-04-06 16:49:20.262411
2016-04-06T09:49:20.337 INFO:tasks.workunit.client.0.vpm054.stderr:test/multi_stress_watch.cc: 61: FAILED assert(!ret)
2016-04-06T09:49:20.348 INFO:tasks.workunit.client.0.vpm054.stderr: ceph version 0.94.6-206-g77fbf58 (77fbf581cb2259146938a737c299d6cf762303d1)
2016-04-06T09:49:20.348 INFO:tasks.workunit.client.0.vpm054.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x40746b]
2016-04-06T09:49:20.349 INFO:tasks.workunit.client.0.vpm054.stderr: 2: (test_loop(librados::Rados&, std::string, std::string)+0x229) [0x406b09]
2016-04-06T09:49:20.349 INFO:tasks.workunit.client.0.vpm054.stderr: 3: (test_replicated(librados::Rados&, std::string, std::string)+0x45) [0x406c25]
2016-04-06T09:49:20.349 INFO:tasks.workunit.client.0.vpm054.stderr: 4: (main()+0x34e) [0x4063ee]
2016-04-06T09:49:20.349 INFO:tasks.workunit.client.0.vpm054.stderr: 5: (__libc_start_main()+0xf5) [0x7fc9839a1ec5]
2016-04-06T09:49:20.350 INFO:tasks.workunit.client.0.vpm054.stderr: 6: ceph_multi_stress_watch() [0x406817]
2016-04-06T09:49:20.350 INFO:tasks.workunit.client.0.vpm054.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2016-04-06T09:49:20.350 INFO:tasks.workunit.client.0.vpm054.stderr:terminate called after throwing an instance of 'ceph::FailedAssertion'
Actions
Copy link
 #9

Updated by Samuel Just over 7 years ago

  • Status changed from New to Can't reproduce
Actions
Copy link

Also available in: Atom PDF