https://tracker.ceph.com/
https://tracker.ceph.com/favicon.ico
2015-07-26T10:15:02Z
Ceph
Ceph - Bug #12469: RadosModel.h: read returned error code -2 (hammer)
https://tracker.ceph.com/issues/12469?journal_id=55556
2015-07-26T10:15:02Z
Abhishek Lekshmanan
abhishek.lekshmanan@gmail.com
<ul></ul><p>Rescheduled a run against hammer branch (as per Loic's comments) in order to ensure whether the bug is reproducible against hammer branch as well</p>
<p><a class="external" href="http://pulpito.ceph.com/abhi-2015-07-26_15:39:50-rados-hammer---basic-multi/">http://pulpito.ceph.com/abhi-2015-07-26_15:39:50-rados-hammer---basic-multi/</a></p>
Ceph - Bug #12469: RadosModel.h: read returned error code -2 (hammer)
https://tracker.ceph.com/issues/12469?journal_id=55558
2015-07-26T10:38:02Z
Loïc Dachary
loic@dachary.org
<ul></ul><pre>
run=loic-2015-07-20_16:39:35-rados-hammer-backports---basic-multi
eval filter=$(curl --silent http://paddles.front.sepia.ceph.com/runs/$run/ | jq '.jobs[] | select(.status == "fail") | .description' | while read description ; do echo -n $description, ; done | sed -e 's/,$//')
echo $filter
rados/thrash/{0-size-min-size-overrides/2-size-1-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml clusters/fixed-2.yaml fs/btrfs.yaml msgr-failures/fastclose.yaml thrashers/pggrow.yaml workloads/snaps-few-objects.yaml}
teuthology-openstack --verbose --key-name loic --priority 50 --suite rados --filter="$filter" --suite-branch hammer --distro ubuntu --ceph hammer
teuthology-openstack --verbose --key-name loic --priority 50 --suite rados --filter="$filter" --suite-branch hammer --distro ubuntu --ceph hammer-backports
</pre>
<ul>
<li><strong>pass</strong> <a href="http://149.202.174.228:8081/ubuntu-2015-07-26_11:35:42-rados-hammer---basic-openstack/" class="external">hammer run</a></li>
<li><strong>pass</strong> <a href="http://149.202.174.228:8081/ubuntu-2015-07-26_11:35:14-rados-hammer-backports---basic-openstack/" class="external">hammer-backports run</a></li>
</ul>
Ceph - Bug #12469: RadosModel.h: read returned error code -2 (hammer)
https://tracker.ceph.com/issues/12469?journal_id=55559
2015-07-26T10:56:32Z
Loïc Dachary
loic@dachary.org
<ul><li><strong>Subject</strong> changed from <i>Failed assert(0) in rados-hammer-multi run</i> to <i>RadosModel.h: read returned error code -2</i></li><li><strong>Status</strong> changed from <i>New</i> to <i>12</i></li></ul><p><a class="external" href="https://github.com/ceph/ceph/blob/hammer/src/test/osd/RadosModel.h#L1091">https://github.com/ceph/ceph/blob/hammer/src/test/osd/RadosModel.h#L1091</a><br /><pre>
if (!(err == -ENOENT && old_value.deleted())) {
cerr << num << ": Error: oid " << oid << " read returned error code "
<< err << std::endl;
assert(0);
}
</pre><br />The err -2 means err == -ENOENT, therefore old_value.deleted() is false and triggers the assert.</p>
Ceph - Bug #12469: RadosModel.h: read returned error code -2 (hammer)
https://tracker.ceph.com/issues/12469?journal_id=55560
2015-07-26T10:58:10Z
Loïc Dachary
loic@dachary.org
<ul></ul><p>I don't see anything in the commits from the integration branch of hammer that could cause that kind of problem (it's almost entirely rgw / librbd / build). Since a similar error (<a class="issue tracker-1 status-6 priority-6 priority-high2 closed" title="Bug: 2014-04-02T20:36:41.677 INFO:teuthology.task.rados.rados.0.err:[10.214.131.18]: 2479: Error: oid ... (Rejected)" href="https://tracker.ceph.com/issues/7985">#7985</a>) was rejected a while ago by Sam, it's worth asking him why it was rejected before digging further.</p>
Ceph - Bug #12469: RadosModel.h: read returned error code -2 (hammer)
https://tracker.ceph.com/issues/12469?journal_id=56152
2015-08-04T15:54:23Z
Loïc Dachary
loic@dachary.org
<ul><li><strong>Subject</strong> changed from <i>RadosModel.h: read returned error code -2</i> to <i>RadosModel.h: read returned error code -2 (hammer)</i></li><li><strong>Target version</strong> deleted (<del><i>v0.94.3</i></del>)</li><li><strong>Release</strong> set to <i>hammer</i></li></ul>
Ceph - Bug #12469: RadosModel.h: read returned error code -2 (hammer)
https://tracker.ceph.com/issues/12469?journal_id=56773
2015-08-18T20:51:18Z
Sage Weil
sage@newdream.net
<ul></ul><p>This appears to be caused by bad past_intervals on PG 1.16, resulting from the ceph-objectstore-tool import/export:</p>
<pre>
remote/plana11/log/ceph-osd.5.log.gz:2015-07-23 05:10:25.871127 7fb97a47d700 10 osd.5 pg_epoch: 165 pg[1.16( empty local-les=0 n=0 ec=5 les/c 104/104 165/165/165) [5] r=0 lpr=165 pi=104-164/25 crt=0'0 mlcod 0'0 peering] PriorSet: build_prior interval(156-164 up [](-1) acting [](-1))
remote/plana11/log/ceph-osd.5.log.gz:2015-07-23 05:10:25.871157 7fb97a47d700 10 osd.5 pg_epoch: 165 pg[1.16( empty local-les=0 n=0 ec=5 les/c 104/104 165/165/165) [5] r=0 lpr=165 pi=104-164/25 crt=0'0 mlcod 0'0 peering] PriorSet: build_prior interval(155-155 up [](5) acting [](5))
remote/plana11/log/ceph-osd.5.log.gz:2015-07-23 05:10:25.871177 7fb97a47d700 10 osd.5 pg_epoch: 165 pg[1.16( empty local-les=0 n=0 ec=5 les/c 104/104 165/165/165) [5] r=0 lpr=165 pi=104-164/25 crt=0'0 mlcod 0'0 peering] PriorSet: build_prior interval(154-154 up [](5) acting [](5))
remote/plana11/log/ceph-osd.5.log.gz:2015-07-23 05:10:25.871197 7fb97a47d700 10 osd.5 pg_epoch: 165 pg[1.16( empty local-les=0 n=0 ec=5 les/c 104/104 165/165/165) [5] r=0 lpr=165 pi=104-164/25 crt=0'0 mlcod 0'0 peering] PriorSet: build_prior interval(153-153 up [](5) acting [](5))
remote/plana11/log/ceph-osd.5.log.gz:2015-07-23 05:10:25.871217 7fb97a47d700 10 osd.5 pg_epoch: 165 pg[1.16( empty local-les=0 n=0 ec=5 les/c 104/104 165/165/165) [5] r=0 lpr=165 pi=104-164/25 crt=0'0 mlcod 0'0 peering] PriorSet: build_prior interval(152-152 up [](5) acting [](5))
remote/plana11/log/ceph-osd.5.log.gz:2015-07-23 05:10:25.871236 7fb97a47d700 10 osd.5 pg_epoch: 165 pg[1.16( empty local-les=0 n=0 ec=5 les/c 104/104 165/165/165) [5] r=0 lpr=165 pi=104-164/25 crt=0'0 mlcod 0'0 peering] PriorSet: build_prior interval(151-151 up [](5) acting [](5))
remote/plana11/log/ceph-osd.5.log.gz:2015-07-23 05:10:25.871256 7fb97a47d700 10 osd.5 pg_epoch: 165 pg[1.16( empty local-les=0 n=0 ec=5 les/c 104/104 165/165/165) [5] r=0 lpr=165 pi=104-164/25 crt=0'0 mlcod 0'0 peering] PriorSet: build_prior interval(150-150 up [](5) acting [](5))
remote/plana11/log/ceph-osd.5.log.gz:2015-07-23 05:10:25.871275 7fb97a47d700 10 osd.5 pg_epoch: 165 pg[1.16( empty local-les=0 n=0 ec=5 les/c 104/104 165/165/165) [5] r=0 lpr=165 pi=104-164/25 crt=0'0 mlcod 0'0 peering] PriorSet: build_prior interval(149-149 up [](5) acting [](5))
remote/plana11/log/ceph-osd.5.log.gz:2015-07-23 05:10:25.871299 7fb97a47d700 10 osd.5 pg_epoch: 165 pg[1.16( empty local-les=0 n=0 ec=5 les/c 104/104 165/165/165) [5] r=0 lpr=165 pi=104-164/25 crt=0'0 mlcod 0'0 peering] PriorSet: build_prior interval(148-148 up [](5) acting [](5))
remote/plana11/log/ceph-osd.5.log.gz:2015-07-23 05:10:25.871319 7fb97a47d700 10 osd.5 pg_epoch: 165 pg[1.16( empty local-les=0 n=0 ec=5 les/c 104/104 165/165/165) [5] r=0 lpr=165 pi=104-164/25 crt=0'0 mlcod 0'0 peering] PriorSet: build_prior interval(147-147 up [](5) acting [](5))
remote/plana11/log/ceph-osd.5.log.gz:2015-07-23 05:10:25.871339 7fb97a47d700 10 osd.5 pg_epoch: 165 pg[1.16( empty local-les=0 n=0 ec=5 les/c 104/104 165/165/165) [5] r=0 lpr=165 pi=104-164/25 crt=0'0 mlcod 0'0 peering] PriorSet: build_prior interval(146-146 up [0](5) acting [0](5))
remote/plana11/log/ceph-osd.5.log.gz:2015-07-23 05:10:25.871360 7fb97a47d700 10 osd.5 pg_epoch: 165 pg[1.16( empty local-les=0 n=0 ec=5 les/c 104/104 165/165/165) [5] r=0 lpr=165 pi=104-164/25 crt=0'0 mlcod 0'0 peering] PriorSet: build_prior interval(145-145 up [0](5) acting [0](5))
remote/plana11/log/ceph-osd.5.log.gz:2015-07-23 05:10:25.871390 7fb97a47d700 10 osd.5 pg_epoch: 165 pg[1.16( empty local-les=0 n=0 ec=5 les/c 104/104 165/165/165) [5] r=0 lpr=165 pi=104-164/25 crt=0'0 mlcod 0'0 peering] PriorSet: build_prior interval(144-144 up [0](5) acting [0](5))
remote/plana11/log/ceph-osd.5.log.gz:2015-07-23 05:10:25.871410 7fb97a47d700 10 osd.5 pg_epoch: 165 pg[1.16( empty local-les=0 n=0 ec=5 les/c 104/104 165/165/165) [5] r=0 lpr=165 pi=104-164/25 crt=0'0 mlcod 0'0 peering] PriorSet: build_prior interval(143-143 up [0](5) acting [0](5))
remote/plana11/log/ceph-osd.5.log.gz:2015-07-23 05:10:25.871430 7fb97a47d700 10 osd.5 pg_epoch: 165 pg[1.16( empty local-les=0 n=0 ec=5 les/c 104/104 165/165/165) [5] r=0 lpr=165 pi=104-164/25 crt=0'0 mlcod 0'0 peering] PriorSet: build_prior interval(142-142 up [0](5) acting [0](5))
remote/plana11/log/ceph-osd.5.log.gz:2015-07-23 05:10:25.871451 7fb97a47d700 10 osd.5 pg_epoch: 165 pg[1.16( empty local-les=0 n=0 ec=5 les/c 104/104 165/165/165) [5] r=0 lpr=165 pi=104-164/25 crt=0'0 mlcod 0'0 peering] PriorSet: build_prior interval(141-141 up [0](5) acting [0](5))
remote/plana11/log/ceph-osd.5.log.gz:2015-07-23 05:10:25.871471 7fb97a47d700 10 osd.5 pg_epoch: 165 pg[1.16( empty local-les=0 n=0 ec=5 les/c 104/104 165/165/165) [5] r=0 lpr=165 pi=104-164/25 crt=0'0 mlcod 0'0 peering] PriorSet: build_prior interval(140-140 up [0](5) acting [0](5))
remote/plana11/log/ceph-osd.5.log.gz:2015-07-23 05:10:25.871491 7fb97a47d700 10 osd.5 pg_epoch: 165 pg[1.16( empty local-les=0 n=0 ec=5 les/c 104/104 165/165/165) [5] r=0 lpr=165 pi=104-164/25 crt=0'0 mlcod 0'0 peering] PriorSet: build_prior interval(139-139 up [0](5) acting [0](5))
remote/plana11/log/ceph-osd.5.log.gz:2015-07-23 05:10:25.871518 7fb97a47d700 10 osd.5 pg_epoch: 165 pg[1.16( empty local-les=0 n=0 ec=5 les/c 104/104 165/165/165) [5] r=0 lpr=165 pi=104-164/25 crt=0'0 mlcod 0'0 peering] PriorSet: build_prior interval(138-138 up [0](5) acting [0](5))
remote/plana11/log/ceph-osd.5.log.gz:2015-07-23 05:10:25.871539 7fb97a47d700 10 osd.5 pg_epoch: 165 pg[1.16( empty local-les=0 n=0 ec=5 les/c 104/104 165/165/165) [5] r=0 lpr=165 pi=104-164/25 crt=0'0 mlcod 0'0 peering] PriorSet: build_prior interval(137-137 up [0](5) acting [0](5))
remote/plana11/log/ceph-osd.5.log.gz:2015-07-23 05:10:25.871560 7fb97a47d700 10 osd.5 pg_epoch: 165 pg[1.16( empty local-les=0 n=0 ec=5 les/c 104/104 165/165/165) [5] r=0 lpr=165 pi=104-164/25 crt=0'0 mlcod 0'0 peering] PriorSet: build_prior interval(136-136 up [0](5) acting [0](5))
remote/plana11/log/ceph-osd.5.log.gz:2015-07-23 05:10:25.871580 7fb97a47d700 10 osd.5 pg_epoch: 165 pg[1.16( empty local-les=0 n=0 ec=5 les/c 104/104 165/165/165) [5] r=0 lpr=165 pi=104-164/25 crt=0'0 mlcod 0'0 peering] PriorSet: build_prior interval(135-135 up [0](5) acting [0](5))
remote/plana11/log/ceph-osd.5.log.gz:2015-07-23 05:10:25.871600 7fb97a47d700 10 osd.5 pg_epoch: 165 pg[1.16( empty local-les=0 n=0 ec=5 les/c 104/104 165/165/165) [5] r=0 lpr=165 pi=104-164/25 crt=0'0 mlcod 0'0 peering] PriorSet: build_prior interval(134-134 up [0](5) acting [0](5))
remote/plana11/log/ceph-osd.5.log.gz:2015-07-23 05:10:25.871620 7fb97a47d700 10 osd.5 pg_epoch: 165 pg[1.16( empty local-les=0 n=0 ec=5 les/c 104/104 165/165/165) [5] r=0 lpr=165 pi=104-164/25 crt=0'0 mlcod 0'0 peering] PriorSet: build_prior interval(133-133 up [0](5) acting [0](5))
remote/plana11/log/ceph-osd.5.log.gz:2015-07-23 05:10:25.871640 7fb97a47d700 10 osd.5 pg_epoch: 165 pg[1.16( empty local-les=0 n=0 ec=5 les/c 104/104 165/165/165) [5] r=0 lpr=165 pi=104-164/25 crt=0'0 mlcod 0'0 peering] PriorSet: build_prior interval(104-132 up [5,0](5) acting [5,0](5) maybe_went_rw)
remote/plana11/log/ceph-osd.5.log.gz:2015-07-23 05:10:25.871661 7fb97a47d700 10 osd.5 pg_epoch: 165 pg[1.16( empty local-les=0 n=0 ec=5 les/c 104/104 165/165/165) [5] r=0 lpr=165 pi=104-164/25 crt=0'0 mlcod 0'0 peering] PriorSet: build_prior prior osd.0 is down
remote/plana11/log/ceph-osd.5.log.gz:2015-07-23 05:10:25.871679 7fb97a47d700 10 osd.5 pg_epoch: 165 pg[1.16( empty local-les=0 n=0 ec=5 les/c 104/104 165/165/165) [5] r=0 lpr=165 pi=104-164/25 crt=0'0 mlcod 0'0 peering] PriorSet: build_prior final: probe 5 down 0 blocked_by {}
</pre>
<p>is totally wrong.</p>
Ceph - Bug #12469: RadosModel.h: read returned error code -2 (hammer)
https://tracker.ceph.com/issues/12469?journal_id=56776
2015-08-18T21:01:08Z
Loïc Dachary
loic@dachary.org
<ul></ul><p>It's a bug with ceph-objectstore-tool that is not a blocker for a hammer release.</p>
Ceph - Bug #12469: RadosModel.h: read returned error code -2 (hammer)
https://tracker.ceph.com/issues/12469?journal_id=58070
2015-09-08T14:49:17Z
Loïc Dachary
loic@dachary.org
<ul><li><strong>Assignee</strong> set to <i>David Zafman</i></li></ul><p>Feel free to unassign yourself if that's not for you :-)</p>
Ceph - Bug #12469: RadosModel.h: read returned error code -2 (hammer)
https://tracker.ceph.com/issues/12469?journal_id=67766
2016-03-18T01:45:13Z
David Zafman
dzafman@redhat.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-9 status-3 priority-6 priority-high2 closed" href="/issues/15171">Backport #15171</a>: hammer: osd: corruption when min_read_recency_for_promote > 1</i> added</li></ul>
Ceph - Bug #12469: RadosModel.h: read returned error code -2 (hammer)
https://tracker.ceph.com/issues/12469?journal_id=67773
2016-03-18T01:55:22Z
David Zafman
dzafman@redhat.com
<ul><li><strong>Assignee</strong> deleted (<del><i>David Zafman</i></del>)</li></ul><p>Added related to 15171 at this point only because it exhibited the same symptom.</p>
Ceph - Bug #12469: RadosModel.h: read returned error code -2 (hammer)
https://tracker.ceph.com/issues/12469?journal_id=89090
2017-04-12T17:27:06Z
Sage Weil
sage@newdream.net
<ul><li><strong>Status</strong> changed from <i>12</i> to <i>Can't reproduce</i></li></ul>