Bug #7367
Updated by Loïc Dachary about 10 years ago
<pre>
ceph-mon-lmb-B-1:~# ceph -s
cluster 0b68be85-f5a1-4565-9ab1-6625b8a13597
health HEALTH_WARN mds chab1 is laggy
monmap e5: 3 mons at {chab1=172.20.106.84:6789/0,lmbb1=172.20.107.84:6789/0,loib1=172.20.108.84:6789/0}, election epoch 576, quorum 0,1,2 chab1,lmbb1,loib1
mdsmap e24563: 1/1/1 up {0=chab1=up:active(laggy or crashed)}
osdmap e54512: 21 osds: 21 up, 21 in
pgmap v8039014: 6432 pgs, 6 pools, 3271 GB data, 1232 kobjects
6470 GB used, 4239 GB / 10710 GB avail
6432 active+clean
</pre>
rbd pools are available:
<pre>
root@machriemoor:~# rbd ls
bench
bench2
</pre>
but mounting the images they contain fail
<pre>
root@machriemoor:~# mount /dev/rbd/rbd/bench /mnt/tempo
mount: wrong fs type, bad option, bad superblock on /dev/rbd1,
dmesg : 1747.349670] rbd: rbd1: write 1000 at 4100000000 (0)
[ 1747.349670]
[ 1747.349819] rbd: rbd1: result -6 xferred 1000
[ 1747.349819]
[ 1747.349963] blk_update_request: 127 callbacks suppressed
[ 1747.350082] end_request: I/O error, dev rbd1, sector 545259520
[ 1747.350203] quiet_error: 127 callbacks suppressed
[ 1747.350321] Buffer I/O error on device rbd1, logical block 68157440
[ 1747.350442] lost page write due to I/O error on rbd1
[ 1747.350744] rbd: rbd1: write 1000 at 4100020000 (20000)
[ 1747.350744]
[ 1747.350889] rbd: rbd1: result -6 xferred 1000
[ 1747.350889]
[ 1747.351031] end_request: I/O error, dev rbd1, sector 545259776
[ 1747.351153] Buffer I/O error on device rbd1, logical block 68157472
[ 1747.351273] lost page write due to I/O error on rbd1
[ 1747.351669] rbd: rbd1: write 3000 at 4100040000 (40000)
[ 1747.351669]
[ 1747.351870] rbd: rbd1: result -6 xferred 3000
[ 1747.351870]
[ 1747.352013] end_request: I/O error, dev rbd1, sector 545260032
[ 1747.352134] Buffer I/O error on device rbd1, logical block 68157504
[ 1747.352255] lost page write due to I/O error on rbd1
[ 1747.352374] Buffer I/O error on device rbd1, logical block 68157505
</pre>
<pre>
root@machriemoor:~# uname -a
Linux machriemoor 3.13.1-dsiun-130719 #12 SMP Fri Jan 31 12:08:15 CET 2014 x86_64 GNU/Linux
</pre>
on the machine trying to mount
</pre>
root@machriemoor:~# ceph --version
ceph version 0.70 (e3bb0656d92e74ead0342ae696039a51170fe941)
</pre>
and it was rebooted recently
<pre>
root@machriemoor:~# uptime
23:09:09 up 34 min, 1 user, load average: 0,21, 0,08, 0,08
</pre>
on the machines running the osds and the mds
<pre>
ceph-mds-loi-B-1:~# ceph --version
ceph version 0.76 (3b990136bfab74249f166dd742fd8e61637e63d9)
</pre>
the mds refuses to start with the error
<pre>
2014-02-07 23:12:37.038781 7f7538327780 -1 mds.-1.-1 *** one or more OSDs do not support TMAP2OMAP; upgrade OSDs before starting MDS (or downgrade MDS) ***
</pre>
the cluster was upgraded to 0.76 and downgraded to 0.72 after seeing some problems and upgraded again to 0.76 in an attempt to fix them