Ceph

RBD got silent after about 1 month running.
Although I restarted the daemons, the symptom doesn't go away.
Attached a log from /var/log/ceph/osd.0.log. It seems that mon got
wrong at 2011-05-30 09:08:19.
Before then, it had no problem.

Using the Debian package version, 0.26-1~bpo60+1.

2011-05-30 09:05:51.258591 7f0213209700 osd0 2 OSD::ms_handle_reset()
2011-05-30 09:05:51.258620 7f0213209700 osd0 2 OSD::ms_handle_reset()
s=0xafeac60
2011-05-30 09:08:19.143404 7f0213209700 monclient: hunting for new mon
2011-05-30 09:08:19.153778 7f0213209700 osd0 2 OSD::ms_handle_reset()
2011-05-30 09:08:19.157018 7f0211205700 -- 192.168.100.7:6801/27764 >>
192.168.100.7:6789/0 pipe(0xac4ea00 sd=12 pgs=0 cs=0 l=0).fault first
fault
2011-05-30 09:08:20.220057 7f0213209700 osd0 2 OSD::ms_handle_reset()
2011-05-30 09:08:20.220086 7f0213209700 osd0 2 OSD::ms_handle_reset()
s=0xafead80
ceph version 0.26.commit: 9981ff90968398da43c63106694d661f5e3d07d5.
process: cosd. pid: 1120
2011-05-30 09:12:12.346156 7fdf06fa8760 filestore(/data/osd0) mount
FIEMAP ioctl is NOT supported
2011-05-30 09:12:12.346260 7fdf06fa8760 filestore(/data/osd0) mount
did NOT detect btrfs
2011-05-30 09:12:12.353443 7fdf06fa8760 filestore(/data/osd0) mount
found snaps <>
2011-05-30 09:12:12.384327 7fdf06fa8760 filestore(/data/osd0) mount:
WRITEAHEAD journal mode explicitly enabled in conf
2011-05-30 09:12:12.384359 7fdf06fa8760 filestore(/data/osd0) mount
WARNING: not btrfs or ext3; data may be lost
2011-05-30 09:12:12.384438 7fdf06fa8760 journal _open
/data/osd0/journal fd 11: 1048576000 bytes, block size 4096 bytes,
directio = 1
2011-05-30 09:12:12.423439 7fdf06fa8760 journal read_entry 727867392 :
seq 555687 1045 bytes
2011-05-30 09:12:12.423522 7fdf06fa8760 journal _open
/data/osd0/journal fd 11: 1048576000 bytes, block size 4096 bytes,
directio = 1
2011-05-30 16:07:09.388801 7fdefbab9700 osd0 6 OSD::ms_handle_reset()
2011-05-30 16:07:09.388843 7fdefbab9700 osd0 6 OSD::ms_handle_reset()
s=0x2b247e0
2011-05-30 16:09:18.108682 7fdefbab9700 osd0 6 OSD::ms_handle_reset()
2011-05-30 16:09:18.108706 7fdefbab9700 osd0 6 OSD::ms_handle_reset()
s=0x2b24a20
2011-05-30 16:13:08.438687 7fdefbab9700 osd0 6 OSD::ms_handle_reset()
2011-05-30 16:13:08.438713 7fdefbab9700 osd0 6 OSD::ms_handle_reset()
s=0x2b24b40
2011-05-30 16:15:01.244245 7fdefbab9700 monclient: hunting for new mon
2011-05-30 16:15:01.244379 7fdefbab9700 osd0 6 OSD::ms_handle_reset()
2011-05-30 16:15:01.244449 7fdef9ab5700 -- 192.168.100.7:6801/1120 >>
192.168.100.7:6789/0 pipe(0x2b28780 sd=12 pgs=0 cs=0 l=0).fault first
fault
2011-05-30 16:15:02.593141 7fdefbab9700 osd0 6 OSD::ms_handle_reset()
2011-05-30 16:15:02.593170 7fdefbab9700 osd0 6 OSD::ms_handle_reset()
s=0x2b24240
ceph version 0.26.commit: 9981ff90968398da43c63106694d661f5e3d07d5.
process: cosd. pid: 2665
2011-05-30 16:15:05.177239 7f6375dab760 filestore(/data/osd0) mount
FIEMAP ioctl is NOT supported
2011-05-30 16:15:05.177340 7f6375dab760 filestore(/data/osd0) mount
did NOT detect btrfs
2011-05-30 16:15:05.177377 7f6375dab760 filestore(/data/osd0) mount
found snaps <>
2011-05-30 16:15:05.177411 7f6375dab760 filestore(/data/osd0) mount:
WRITEAHEAD journal mode explicitly enabled in conf
2011-05-30 16:15:05.177423 7f6375dab760 filestore(/data/osd0) mount
WARNING: not btrfs or ext3; data may be lost
2011-05-30 16:15:05.177468 7f6375dab760 journal _open
/data/osd0/journal fd 11: 1048576000 bytes, block size 4096 bytes,
directio = 1
2011-05-30 16:15:05.203112 7f6375dab760 journal read_entry 727900160 :
seq 555691 1757 bytes
2011-05-30 16:15:05.203206 7f6375dab760 journal _open
/data/osd0/journal fd 11: 1048576000 bytes, block size 4096 bytes,
directio = 1
2011-05-30 16:15:29.368764 7f636a8bc700 osd0 9 OSD::ms_handle_reset()
2011-05-30 16:15:29.368793 7f636a8bc700 osd0 9 OSD::ms_handle_reset()
s=0x1c9e5a0
2011-05-30 16:16:58.408591 7f636a8bc700 osd0 9 OSD::ms_handle_reset()
2011-05-30 16:16:58.408627 7f636a8bc700 osd0 9 OSD::ms_handle_reset()
s=0x1c9e7e0
2011-05-30 16:17:27.008678 7f636a8bc700 osd0 9 OSD::ms_handle_reset()
2011-05-30 16:17:27.008697 7f636a8bc700 osd0 9 OSD::ms_handle_reset()
s=0x1c9e900
2011-05-30 16:19:42.595055 7f636a8bc700 monclient: hunting for new mon
2011-05-30 16:19:42.595112 7f636a8bc700 osd0 9 OSD::ms_handle_reset()
2011-05-30 16:19:42.595246 7f63688b8700 -- 192.168.100.7:6801/2665 >>
192.168.100.7:6789/0 pipe(0x1ca4780 sd=12 pgs=0 cs=0 l=0).fault first
fault
2011-05-30 16:19:43.691896 7f636a8bc700 osd0 9 OSD::ms_handle_reset()
2011-05-30 16:19:43.691920 7f636a8bc700 osd0 9 OSD::ms_handle_reset()
s=0x1c9e6c0
ceph version 0.26.commit: 9981ff90968398da43c63106694d661f5e3d07d5.
process: cosd. pid: 3470
2011-05-30 16:22:38.347927 7f7658be4760 filestore(/data/osd0) mount
FIEMAP ioctl is NOT supported
2011-05-30 16:22:38.348029 7f7658be4760 filestore(/data/osd0) mount
did NOT detect btrfs
2011-05-30 16:22:38.348067 7f7658be4760 filestore(/data/osd0) mount
found snaps <>
2011-05-30 16:22:38.348099 7f7658be4760 filestore(/data/osd0) mount:
WRITEAHEAD journal mode explicitly enabled in conf
2011-05-30 16:22:38.348119 7f7658be4760 filestore(/data/osd0) mount
WARNING: not btrfs or ext3; data may be lost
2011-05-30 16:22:38.348169 7f7658be4760 journal _open
/data/osd0/journal fd 11: 1048576000 bytes, block size 4096 bytes,
directio = 1
2011-05-30 16:22:38.395563 7f7658be4760 journal read_entry 727924736 :
seq 555694 1909 bytes
2011-05-30 16:22:38.395646 7f7658be4760 journal _open
/data/osd0/journal fd 11: 1048576000 bytes, block size 4096 bytes,
directio = 1
2011-05-30 16:22:52.468693 7f764d6f5700 osd0 11 OSD::ms_handle_reset()
2011-05-30 16:22:52.468728 7f764d6f5700 osd0 11 OSD::ms_handle_reset()
s=0xdf35a0
2011-05-30 16:31:46.888685 7f764d6f5700 osd0 12 OSD::ms_handle_reset()
2011-05-30 16:31:46.888710 7f764d6f5700 osd0 12 OSD::ms_handle_reset()
s=0xdf37e0