Bug #10165
closedceph_test_rados got short read
0%
Description
Runs:
http://pulpito.front.sepia.ceph.com/teuthology-2014-11-20_17:05:01-upgrade:giant-x-next-distro-basic-vps/
Jobs: ['611379', '611383', '611386', '611388', '611390']
http://pulpito.front.sepia.ceph.com/teuthology-2014-11-20_17:05:01-upgrade:giant-x-next-distro-basic-multi/
Jobs: ['611356', '611358']
Logs for one are in http://qa-proxy.ceph.com/teuthology/teuthology-2014-11-20_17:05:01-upgrade:giant-x-next-distro-basic-vps/611379/
2014-11-20T21:38:22.130 INFO:tasks.rados.rados.0.vpm200.stderr:./test/osd/RadosModel.h: 1077: FAILED assert(0) 2014-11-20T21:38:22.182 INFO:tasks.rados.rados.0.vpm200.stderr: ceph version 0.87-34-g37ffccb (37ffccbd57e7e441e0eb1499e5c173aa9c375d35) 2014-11-20T21:38:22.182 INFO:tasks.rados.rados.0.vpm200.stderr: 1: (ReadOp::_finish(TestOp::CallbackInfo*)+0x17d7) [0x422ca7] 2014-11-20T21:38:22.182 INFO:tasks.rados.rados.0.vpm200.stderr: 2: (librados::C_AioComplete::finish(int)+0x18) [0x7f34f28d5898] 2014-11-20T21:38:22.182 INFO:tasks.rados.rados.0.vpm200.stderr: 3: (Context::complete(int)+0x9) [0x7f34f28af899] 2014-11-20T21:38:22.183 INFO:tasks.rados.rados.0.vpm200.stderr: 4: (Finisher::finisher_thread_entry()+0x188) [0x7f34f296b7b8] 2014-11-20T21:38:22.183 INFO:tasks.rados.rados.0.vpm200.stderr: 5: (()+0x79d1) [0x7f34f24b09d1] 2014-11-20T21:38:22.183 INFO:tasks.rados.rados.0.vpm200.stderr: 6: (clone()+0x6d) [0x7f34f1ce186d]
Updated by Sage Weil over 9 years ago
- Subject changed from "Assertion: ./test/osd/RadosModel.h" in upgrade:giant-x-next-distro-basic-vps run to ceph_test_rados got short read
2014-11-20T21:38:21.790 INFO:tasks.rados.rados.0.vpm200.stdout:only read 3829760 out of size 3832037
Updated by Yuri Weinstein over 9 years ago
Same problem in run http://pulpito.front.sepia.ceph.com/teuthology-2014-11-23_09:54:59-powercycle-giant-distro-basic-multi/
Jobs: ['615448', '615476', '615477']
Assertion: ./test/osd/RadosModel.h: 1077: FAILED assert(0) ceph version 0.87-34-g37ffccb (37ffccbd57e7e441e0eb1499e5c173aa9c375d35) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7f340e1c32fb] 2: ceph_test_rados() [0x421111] 3: (librados::C_AioComplete::finish(int)+0x1d) [0x7f340e1254fd] 4: (Context::complete(int)+0x9) [0x7f340e100fd9] 5: (Finisher::finisher_thread_entry()+0x158) [0x7f340e1c23e8] 6: (()+0x8182) [0x7f340dcff182] 7: (clone()+0x6d) [0x7f340d51238d]
Updated by Yuri Weinstein over 9 years ago
Repeated issue in run http://pulpito.ceph.com/teuthology-2014-11-22_17:05:01-upgrade:giant-x-next-distro-basic-multi/
Jobs ['615149', '615151']
2014-11-22T18:32:06.190 INFO:tasks.rados.rados.0.plana09.stdout:3744: rollback oid 3 current snap is 485 2014-11-22T18:32:06.190 INFO:tasks.rados.rados.0.plana09.stdout:rollback oid 3 to 380 2014-11-22T18:32:06.191 INFO:tasks.rados.rados.0.plana09.stdout:3742: expect (ObjNum 1191 snap 425 seq_num 1191) 2014-11-22T18:32:06.266 INFO:teuthology.task.swift:Configuring testswift... 2014-11-22T18:32:06.266 INFO:teuthology.task.swift:client=client.0 2014-11-22T18:32:06.267 INFO:teuthology.task.swift:config={'clients': {'client.0': {'rgw_server': 'client.0'}}, 'testswift_conf': {'client.0': ConfigObj({'func_test': {'auth_port': 7280, 'auth_prefix': '/auth/', 'auth_ssl': 'no', 'account': 'foo.client.0', 'username': 'foo', 'email': 'foo.client.0+test@test.test', 'display_name': 'Mr. foo.client.0 foo', 'password': 'OgozC0daOewhOcfNjei75dka7swfcwBHt5C8UlxFPwV9EVAmwDtF2Q==', 'account2': 'bar.client.0', 'username2': 'bar', 'email2': 'bar.client.0+test@test.test', 'display_name2': 'Mr. bar.client.0 bar', 'password2': 'RJpa1NGEuXcnThh00Jg2tFhO+4ASv2J8VAPOgmMdIUl9kBNtosxw3w=='}})}} 2014-11-22T18:32:06.267 INFO:teuthology.task.swift:roles: ['mon.a', 'mon.b', 'mon.c', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'osd.4', 'osd.5', 'osd.6'] 2014-11-22T18:32:06.267 INFO:teuthology.task.swift:target: ubuntu@plana13.front.sepia.ceph.com 2014-11-22T18:32:06.267 INFO:teuthology.task.swift:roles: ['osd.7', 'osd.8', 'osd.9', 'osd.10', 'osd.11', 'osd.12', 'osd.13'] 2014-11-22T18:32:06.267 INFO:teuthology.task.swift:target: ubuntu@plana42.front.sepia.ceph.com 2014-11-22T18:32:06.267 INFO:teuthology.task.swift:roles: ['client.0'] 2014-11-22T18:32:06.268 INFO:teuthology.task.swift:target: ubuntu@plana09.front.sepia.ceph.com 2014-11-22T18:32:06.268 INFO:teuthology.task.swift:client.0 2014-11-22T18:32:06.268 INFO:teuthology.orchestra.run.plana09:Running: 'cd /home/ubuntu/cephtest/swift && ./bootstrap' 2014-11-22T18:32:06.714 INFO:teuthology.orchestra.run.plana09.stdout:The --no-site-packages flag is deprecated; it is now the default behavior. 2014-11-22T18:32:06.714 INFO:teuthology.orchestra.run.plana09.stdout:New python executable in virtualenv/bin/python 2014-11-22T18:32:06.941 INFO:teuthology.orchestra.run.plana13:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json' 2014-11-22T18:32:07.256 INFO:teuthology.orchestra.run.plana13.stderr:dumped all in format json 2014-11-22T18:32:07.441 INFO:teuthology.orchestra.run.plana13:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph status --format=json-pretty' 2014-11-22T18:32:07.444 INFO:tasks.rados.rados.0.plana09.stdout:incorrect buffer at pos 1322908 2014-11-22T18:32:07.445 INFO:tasks.rados.rados.0.plana09.stderr:3742: oid 7 contents (ObjNum 1191 snap 425 seq_num 1191) corrupt 2014-11-22T18:32:07.445 INFO:tasks.rados.rados.0.plana09.stderr:./test/osd/RadosModel.h: In function 'virtual void ReadOp::_finish(TestOp::CallbackInfo*)' thread 7f54e57fa700 time 2014-11-22 18:32:07.383754 2014-11-22T18:32:07.445 INFO:tasks.rados.rados.0.plana09.stderr:./test/osd/RadosModel.h: 1077: FAILED assert(0) 2014-11-22T18:32:07.445 INFO:tasks.rados.rados.0.plana09.stderr: ceph version 0.87-34-g37ffccb (37ffccbd57e7e441e0eb1499e5c173aa9c375d35) 2014-11-22T18:32:07.445 INFO:tasks.rados.rados.0.plana09.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x7f) [0x7f54f19f8a0f] 2014-11-22T18:32:07.445 INFO:tasks.rados.rados.0.plana09.stderr: 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x14de) [0x41c27e] 2014-11-22T18:32:07.446 INFO:tasks.rados.rados.0.plana09.stderr: 3: (librados::C_AioComplete::finish(int)+0x1d) [0x7f54f19593cd] 2014-11-22T18:32:07.446 INFO:tasks.rados.rados.0.plana09.stderr: 4: (Context::complete(int)+0x9) [0x7f54f19358e9] 2014-11-22T18:32:07.446 INFO:tasks.rados.rados.0.plana09.stderr: 5: (Finisher::finisher_thread_entry()+0x160) [0x7f54f19f7c00] 2014-11-22T18:32:07.446 INFO:tasks.rados.rados.0.plana09.stderr: 6: (()+0x7e9a) [0x7f54f1597e9a] 2014-11-22T18:32:07.447 INFO:tasks.rados.rados.0.plana09.stderr: 7: (clone()+0x6d) [0x7f54f0dae3fd]
Updated by Samuel Just over 9 years ago
I think this also is due to enabling fiemap in the nightlies, teuthology commit:
0f97481ce44e0487ac6cffa051a05590f907a081