Bug #7367
Updated by Loïc Dachary about 10 years ago
<pre> ceph-mon-lmb-B-1:~# ceph -s cluster 0b68be85-f5a1-4565-9ab1-6625b8a13597 health HEALTH_WARN mds chab1 is laggy monmap e5: 3 mons at {chab1=172.20.106.84:6789/0,lmbb1=172.20.107.84:6789/0,loib1=172.20.108.84:6789/0}, election epoch 576, quorum 0,1,2 chab1,lmbb1,loib1 mdsmap e24563: 1/1/1 up {0=chab1=up:active(laggy or crashed)} osdmap e54512: 21 osds: 21 up, 21 in pgmap v8039014: 6432 pgs, 6 pools, 3271 GB data, 1232 kobjects 6470 GB used, 4239 GB / 10710 GB avail 6432 active+clean </pre> rbd pools are available: <pre> root@machriemoor:~# rbd ls bench bench2 </pre> but mounting the images they contain fail <pre> root@machriemoor:~# mount /dev/rbd/rbd/bench /mnt/tempo mount: wrong fs type, bad option, bad superblock on /dev/rbd1, dmesg : 1747.349670] rbd: rbd1: write 1000 at 4100000000 (0) [ 1747.349670] [ 1747.349819] rbd: rbd1: result -6 xferred 1000 [ 1747.349819] [ 1747.349963] blk_update_request: 127 callbacks suppressed [ 1747.350082] end_request: I/O error, dev rbd1, sector 545259520 [ 1747.350203] quiet_error: 127 callbacks suppressed [ 1747.350321] Buffer I/O error on device rbd1, logical block 68157440 [ 1747.350442] lost page write due to I/O error on rbd1 [ 1747.350744] rbd: rbd1: write 1000 at 4100020000 (20000) [ 1747.350744] [ 1747.350889] rbd: rbd1: result -6 xferred 1000 [ 1747.350889] [ 1747.351031] end_request: I/O error, dev rbd1, sector 545259776 [ 1747.351153] Buffer I/O error on device rbd1, logical block 68157472 [ 1747.351273] lost page write due to I/O error on rbd1 [ 1747.351669] rbd: rbd1: write 3000 at 4100040000 (40000) [ 1747.351669] [ 1747.351870] rbd: rbd1: result -6 xferred 3000 [ 1747.351870] [ 1747.352013] end_request: I/O error, dev rbd1, sector 545260032 [ 1747.352134] Buffer I/O error on device rbd1, logical block 68157504 [ 1747.352255] lost page write due to I/O error on rbd1 [ 1747.352374] Buffer I/O error on device rbd1, logical block 68157505 </pre> <pre> root@machriemoor:~# uname -a Linux machriemoor 3.13.1-dsiun-130719 #12 SMP Fri Jan 31 12:08:15 CET 2014 x86_64 GNU/Linux </pre> on the machine trying to mount </pre> root@machriemoor:~# ceph --version ceph version 0.70 (e3bb0656d92e74ead0342ae696039a51170fe941) </pre> and it was rebooted recently <pre> root@machriemoor:~# uptime 23:09:09 up 34 min, 1 user, load average: 0,21, 0,08, 0,08 </pre> on the machines running the osds and the mds <pre> ceph-mds-loi-B-1:~# ceph --version ceph version 0.76 (3b990136bfab74249f166dd742fd8e61637e63d9) </pre> the mds refuses to start with the error <pre> 2014-02-07 23:12:37.038781 7f7538327780 -1 mds.-1.-1 *** one or more OSDs do not support TMAP2OMAP; upgrade OSDs before starting MDS (or downgrade MDS) *** </pre> the cluster was upgraded to 0.75 0.76 and downgraded to 0.72 after seeing some problems and upgraded again to 0.76 in an attempt to fix them