Bug #8522
closedENOENT on read from ceph_test_rados + osd thrashing
0%
Description
2014-06-02T17:24:13.234 INFO:teuthology.task.rados.rados.0.plana50.stderr:2494: Error: oid 42 read returned error code -2
ubuntu@teuthology:/a/teuthology-2014-06-02_02:30:05-rados-master-testing-basic-plana/285679
Updated by Samuel Just almost 10 years ago
2014-06-02 17:24:13.162057 7fd39a414700 10 filestore(/var/lib/ceph/osd/ceph-2) remove 3.4e_head/96d896ce/plana505832-42/head//3 = -2
2014-06-02 17:24:13.162063 7fd39a414700 15 filestore(/var/lib/ceph/osd/ceph-2) remove 3.4e_head/96d896ce/plana505832-42/head//3
2014-06-02 17:24:13.162100 7fd39a414700 10 filestore(/var/lib/ceph/osd/ceph-2) remove 3.4e_head/96d896ce/plana505832-42/head//3 = -2
2014-06-02 17:24:13.162106 7fd39a414700 15 filestore(/var/lib/ceph/osd/ceph-2) remove 3.4e_head/96d896ce/plana505832-42/snapdir//3
2014-06-02 17:24:13.162151 7fd39a414700 10 filestore(/var/lib/ceph/osd/ceph-2) remove 3.4e_head/96d896ce/plana505832-42/snapdir//3 = -2
2014-06-02 17:24:13.162156 7fd39a414700 15 filestore(/var/lib/ceph/osd/ceph-2) remove 3.4e_head/96d896ce/plana505832-42/snapdir//3
2014-06-02 17:24:13.162194 7fd39a414700 10 filestore(/var/lib/ceph/osd/ceph-2) remove 3.4e_head/96d896ce/plana505832-42/snapdir//3 = -2
2014-06-02 17:24:13.210491 7fd38cacb700 10 osd.2 709 dequeue_op 0x3cc1f00 prio 63 cost 32 latency 22.072409 osd_op(client.4115.0:4712 plana505832-42 [read 0~3141145,omap-get-vals-by-keys 0~4,omap-get-keys 0~12,omap-get-vals 0~16,omap-get-header 0~0,getxattrs] 3.96d896ce ack+read e696) v4 pg pg[3.e( v 657'537 lc 607'508 (0'0,657'537] local-les=709 n=2 ec=6 les/c 709/666 695/699/695) [2,4] r=0 lpr=699 pi=665-698/2 crt=657'5
37 lcod 0'0 mlcod 0'0 inactive m=3 snaptrimq=[1b9~1,1bc~1,1c2~1,1e1~1,1e6~1,1e9~4,1ee~1,1f0~1,1f2~2,1f8~2,1fb~2,1fe~1,200~1,205~3]]
2014-06-02 17:24:13.233532 7fd38cacb700 10 osd.2 709 dequeue_op 0x3cc1f00 prio 63 cost 32 latency 22.095451 osd_op(client.4115.0:4712 plana505832-42 [read 0~3141145,omap-get-vals-by-keys 0~4,omap-get-keys 0~12,omap-get-vals 0~16,omap-get-header 0~0,getxattrs] 3.96d896ce ack+read e696) v4 pg pg[3.e( v 657'537 lc 607'508 (0'0,657'537] local-les=709 n=2 ec=6 les/c 709/666 695/699/695) [2,4] r=0 lpr=699 pi=665-698/2 crt=657'5
37 lcod 0'0 mlcod 0'0 active m=3 snaptrimq=[1b9~1,1bc~1,1c2~1,1e1~1,1e6~1,1e9~4,1ee~1,1f0~1,1f2~2,1f8~2,1fb~2,1fe~1,200~1,205~3]]
2014-06-02 17:24:13.233656 7fd38cacb700 10 osd.2 pg_epoch: 709 pg[3.e( v 657'537 lc 607'508 (0'0,657'537] local-les=709 n=2 ec=6 les/c 709/666 695/699/695) [2,4] r=0 lpr=699 pi=665-698/2 crt=657'537 lcod 0'0 mlcod 0'0 active m=3 snaptrimq=[1b9~1,1bc~1,1c2~1,1e1~1,1e6~1,1e9~4,1ee~1,1f0~1,1f2~2,1f8~2,1fb~2,1fe~1,200~1,205~3]] do_op osd_op(client.4115.0:4712 plana505832-42 [read 0~3141145,omap-get-vals-by-keys 0~4,omap-get-k
eys 0~12,omap-get-vals 0~16,omap-get-header 0~0,getxattrs] 3.96d896ce ack+read e696) v4 may_read -> read-ordered flags ack+read
2014-06-02 17:24:13.233708 7fd38cacb700 15 filestore(/var/lib/ceph/osd/ceph-2) getattr 3.e_head/96d896ce/plana505832-42/head//3 '_'
2014-06-02 17:24:13.233779 7fd38cacb700 10 filestore(/var/lib/ceph/osd/ceph-2) error opening file /var/lib/ceph/osd/ceph-2/current/3.e_head/plana505832-42__head_96D896CE__3 with flags=2: (2) No such file or directory
2014-06-02 17:24:13.233793 7fd38cacb700 10 filestore(/var/lib/ceph/osd/ceph-2) getattr 3.e_head/96d896ce/plana505832-42/head//3 '_' = -
Op processed on wrong pg, dup of 8396