2012-10-11 13:43:44.105828 7f8acff2f780 0 ceph version 0.52 (commit:e48859474c4944d4ff201ddc9f5fd400e8898173), process ceph-osd, pid 1657 2012-10-11 13:43:44.148518 7f8acff2f780 0 filestore(/data/osd.2) mount FIEMAP ioctl is NOT supported 2012-10-11 13:43:44.148531 7f8acff2f780 0 filestore(/data/osd.2) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option 2012-10-11 13:43:44.148631 7f8acff2f780 0 filestore(/data/osd.2) mount did NOT detect btrfs 2012-10-11 13:43:44.164914 7f8acff2f780 0 filestore(/data/osd.2) mount syncfs(2) syscall fully supported (by glibc and kernel) 2012-10-11 13:43:44.165042 7f8acff2f780 0 filestore(/data/osd.2) mount found snaps <> 2012-10-11 13:43:44.184623 7f8acff2f780 0 filestore(/data/osd.2) mount: enabling WRITEAHEAD journal mode: btrfs not detected 2012-10-11 13:43:44.184924 7f8acff2f780 1 journal _open /data/osd.2/journal fd 20: 1048576000 bytes, block size 131072 bytes, directio = 0, aio = 0 2012-10-11 13:43:44.185176 7f8acff2f780 1 journal _open /data/osd.2/journal fd 20: 1048576000 bytes, block size 131072 bytes, directio = 0, aio = 0 2012-10-11 13:43:44.185604 7f8acff2f780 1 journal close /data/osd.2/journal 2012-10-11 13:43:44.226632 7f8acff2f780 0 filestore(/data/osd.2) mount FIEMAP ioctl is NOT supported 2012-10-11 13:43:44.226646 7f8acff2f780 0 filestore(/data/osd.2) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option 2012-10-11 13:43:44.226821 7f8acff2f780 0 filestore(/data/osd.2) mount did NOT detect btrfs 2012-10-11 13:43:44.234856 7f8acff2f780 0 filestore(/data/osd.2) mount syncfs(2) syscall fully supported (by glibc and kernel) 2012-10-11 13:43:44.234945 7f8acff2f780 0 filestore(/data/osd.2) mount found snaps <> 2012-10-11 13:43:44.254590 7f8acff2f780 0 filestore(/data/osd.2) mount: enabling WRITEAHEAD journal mode: btrfs not detected 2012-10-11 13:43:44.254777 7f8acff2f780 1 journal _open /data/osd.2/journal fd 29: 1048576000 bytes, block size 131072 bytes, directio = 0, aio = 0 2012-10-11 13:43:44.254924 7f8acff2f780 1 journal _open /data/osd.2/journal fd 29: 1048576000 bytes, block size 131072 bytes, directio = 0, aio = 0 2012-10-11 13:43:54.233546 7f8abb3ca700 0 -- 149.165.228.10:6808/1657 >> 149.165.228.10:6802/338 pipe(0x22ebd80 sd=32 :58518 pgs=2 cs=1 l=0).fault with nothing to send, going to standby 2012-10-11 13:43:59.822919 7f8abbece700 0 -- 149.165.228.10:0/1657 submit_message osd_ping(ping e8 stamp 2012-10-11 13:43:59.822865) v2 remote, 149.165.228.10:6803/338, failed lossy con, dropping message 0x267ec40 2012-10-11 13:44:02.123163 7f8abbece700 0 -- 149.165.228.10:0/1657 submit_message osd_ping(ping e8 stamp 2012-10-11 13:44:02.123149) v2 remote, 149.165.228.10:6803/338, failed lossy con, dropping message 0x29af700 2012-10-11 13:44:04.423313 7f8abbece700 0 -- 149.165.228.10:0/1657 submit_message osd_ping(ping e8 stamp 2012-10-11 13:44:04.423297) v2 remote, 149.165.228.10:6803/338, failed lossy con, dropping message 0x2692e00 2012-10-11 13:44:04.923484 7f8abbece700 0 -- 149.165.228.10:0/1657 submit_message osd_ping(ping e8 stamp 2012-10-11 13:44:04.923468) v2 remote, 149.165.228.10:6803/338, failed lossy con, dropping message 0x310ec40 2012-10-11 13:44:06.623631 7f8abbece700 0 -- 149.165.228.10:0/1657 submit_message osd_ping(ping e8 stamp 2012-10-11 13:44:06.623613) v2 remote, 149.165.228.10:6803/338, failed lossy con, dropping message 0x30668c0 2012-10-11 13:44:09.523764 7f8abbece700 0 -- 149.165.228.10:0/1657 submit_message osd_ping(ping e8 stamp 2012-10-11 13:44:09.523749) v2 remote, 149.165.228.10:6803/338, failed lossy con, dropping message 0x3066a80 2012-10-11 13:44:13.023988 7f8abbece700 -1 osd.2 8 heartbeat_check: no reply from osd.0 since 2012-10-11 13:43:52.822562 (cutoff 2012-10-11 13:43:53.023987) 2012-10-11 13:44:13.425446 7f8ac8fe9700 -1 osd.2 8 heartbeat_check: no reply from osd.0 since 2012-10-11 13:43:52.822562 (cutoff 2012-10-11 13:43:53.425445) 2012-10-11 13:44:14.425651 7f8ac8fe9700 -1 osd.2 8 heartbeat_check: no reply from osd.0 since 2012-10-11 13:43:52.822562 (cutoff 2012-10-11 13:43:54.425650) 2012-10-11 13:44:15.425772 7f8ac8fe9700 -1 osd.2 8 heartbeat_check: no reply from osd.0 since 2012-10-11 13:43:52.822562 (cutoff 2012-10-11 13:43:55.425771) 2012-10-11 13:44:16.425907 7f8ac8fe9700 -1 osd.2 8 heartbeat_check: no reply from osd.0 since 2012-10-11 13:43:52.822562 (cutoff 2012-10-11 13:43:56.425906) 2012-10-11 13:44:17.426027 7f8ac8fe9700 -1 osd.2 8 heartbeat_check: no reply from osd.0 since 2012-10-11 13:43:52.822562 (cutoff 2012-10-11 13:43:57.426026) 2012-10-11 13:44:18.324259 7f8abbece700 0 -- 149.165.228.10:0/1657 submit_message osd_ping(ping e8 stamp 2012-10-11 13:44:18.324206) v2 remote, 149.165.228.10:6803/338, failed lossy con, dropping message 0x29af700 2012-10-11 13:44:18.324289 7f8abbece700 -1 osd.2 8 heartbeat_check: no reply from osd.0 since 2012-10-11 13:43:52.822562 (cutoff 2012-10-11 13:43:58.324288) 2012-10-11 13:44:18.426144 7f8ac8fe9700 -1 osd.2 8 heartbeat_check: no reply from osd.0 since 2012-10-11 13:43:52.822562 (cutoff 2012-10-11 13:43:58.426143) 2012-10-11 13:44:18.824423 7f8abbece700 0 -- 149.165.228.10:0/1657 submit_message osd_ping(ping e8 stamp 2012-10-11 13:44:18.824407) v2 remote, 149.165.228.10:6803/338, failed lossy con, dropping message 0x267ec40 2012-10-11 13:44:18.824441 7f8abbece700 -1 osd.2 8 heartbeat_check: no reply from osd.0 since 2012-10-11 13:43:52.822562 (cutoff 2012-10-11 13:43:58.824441) 2012-10-11 13:44:19.426277 7f8ac8fe9700 -1 osd.2 8 heartbeat_check: no reply from osd.0 since 2012-10-11 13:43:52.822562 (cutoff 2012-10-11 13:43:59.426276) 2012-10-11 13:44:24.427934 7f8ac8fe9700 0 log [WRN] : 2 slow requests, 2 included below; oldest blocked for > 30.192631 secs 2012-10-11 13:44:24.427966 7f8ac8fe9700 0 log [WRN] : slow request 30.192631 seconds old, received at 2012-10-11 13:43:54.235189: osd_op(mds.0.1:1 200.00000000 [writefull 0~84] 1.844f3494 RETRY) v4 currently reached pg 2012-10-11 13:44:24.427976 7f8ac8fe9700 0 log [WRN] : slow request 30.192580 seconds old, received at 2012-10-11 13:43:54.235240: osd_op(mds.0.1:21 100.00000000 [setxattr path (5),setxattr parent (13),tmapput 0~4089] 1.c5265ab3 RETRY) v4 currently delayed 2012-10-11 13:44:54.436613 7f8ac8fe9700 0 log [WRN] : 2 slow requests, 2 included below; oldest blocked for > 60.201373 secs 2012-10-11 13:44:54.436624 7f8ac8fe9700 0 log [WRN] : slow request 60.201373 seconds old, received at 2012-10-11 13:43:54.235189: osd_op(mds.0.1:1 200.00000000 [writefull 0~84] 1.844f3494 RETRY) v4 currently reached pg 2012-10-11 13:44:54.436631 7f8ac8fe9700 0 log [WRN] : slow request 60.201322 seconds old, received at 2012-10-11 13:43:54.235240: osd_op(mds.0.1:21 100.00000000 [setxattr path (5),setxattr parent (13),tmapput 0~4089] 1.c5265ab3 RETRY) v4 currently delayed 2012-10-11 13:45:54.452099 7f8ac8fe9700 0 log [WRN] : 2 slow requests, 2 included below; oldest blocked for > 120.216859 secs 2012-10-11 13:45:54.452110 7f8ac8fe9700 0 log [WRN] : slow request 120.216859 seconds old, received at 2012-10-11 13:43:54.235189: osd_op(mds.0.1:1 200.00000000 [writefull 0~84] 1.844f3494 RETRY) v4 currently reached pg 2012-10-11 13:45:54.452116 7f8ac8fe9700 0 log [WRN] : slow request 120.216808 seconds old, received at 2012-10-11 13:43:54.235240: osd_op(mds.0.1:21 100.00000000 [setxattr path (5),setxattr parent (13),tmapput 0~4089] 1.c5265ab3 RETRY) v4 currently delayed