Bug #1098
closedmds never coming "up:active" awaits in "up:creating"
Added by shyamali mukherjee almost 13 years ago. Updated almost 13 years ago.
0%
Description
After upgrading to ceph 0.27 and latest ceph-client-standalone tree I am unable to mount FS. Intial debugging in kernel module reveals that
======
libceph: client4158 fsid 6ff03cdd-3381-fdba-0673-f85542c086ba
ceph: handle_map epoch 3 len 499
ceph: check_new_map new 3 old 0
ceph: do_request on ffff8800528a4800
ceph: __register_request ffff8800528a4800 tid 1
libceph: mon0 192.168.2.101:6789 session established
ceph: __choose_mds (null) is_hash=0 (0) mode 0
ceph: choose_mds chose random mds-1
ceph: do_request no mds or not active, waiting for map
ceph: do_request waiting
ceph: mdsc delayed_work
ceph: mdsc delayed_work
ceph: mdsc delayed_work
ceph: mdsc delayed_work
ceph: mdsc delayed_work
ceph: mdsc delayed_work
ceph: mdsc delayed_work
ceph: mdsc delayed_work
ceph: mdsc delayed_work
ceph: mdsc delayed_work
ceph: mdsc delayed_work
ceph: do_request waited, got -5
ceph: aborted request 1 with -5
ceph: do_request ffff8800528a4800 done, result -5
ceph: open_root_dentry after ceph_mdsc_do_request
ceph: open_root_dentry after ceph_mdsc_put_request
ceph: mount error opening root -5
ceph: close_sessions
ceph: waiting for sessions to close
watching "ceph -w" in real time shows:
[root@bz1 ceph_bup]# ceph -w
2011-05-19 11:18:10.026537 pg v1959: 15840 pgs: 8379 active+clean, 849 peering, 6612 active+clean+degraded; 17 KB data, 22118 MB used, 16845 GB / 16866 GB avail
2011-05-19 11:18:10.061804 mds e3: 1/1/1 up {0=up:creating}
2011-05-19 11:18:10.062099 osd e117: 60 osds: 60 up, 60 in
2011-05-19 11:18:10.062319 log 2011-05-19 10:46:38.975844 mon0 192.168.2.101:6789/0 100 : [INF] osd5 192.168.2.102:6815/4881 boot
2011-05-19 11:18:10.062391 mon e1: 1 mons at {0=192.168.2.101:6789/0}
Files
| ceph.conf (6 KB) ceph.conf | shyamali mukherjee, 05/19/2011 11:30 AM | ||
| ceph_msd_mon_osd.log (25.4 KB) ceph_msd_mon_osd.log | shyamali mukherjee, 05/19/2011 11:30 AM |
Updated by Sage Weil almost 13 years ago
- Category set to OSD
- Target version set to v0.29
2011-05-18 16:06:18.643599 41ece940 -- 192.168.2.101:6800/8459 --> 192.168.2.107:6812/18794 -- osd_op(mds0.1:10 604.00000000 [setxattr path (12),setxattr parent (39),tmapup 0~0] 1.5c95 RETRY) v1 -- ?+0 0x7feba000b2e0 con 0x7feba000aef0
This one isn't getting a reply. Can you attach the complete log for that osd? (whoever 192.168.2.107:6812/18794 is... ceph osd dump -o - | grep 192.168.2.107:6812/18794 to see)
Are there any message in kern.log / dmesg on that node?
Updated by shyamali mukherjee almost 13 years ago
Log file is too big.. It can not be attached ( > 5 MB). 192.168.2.101:6800/8459 --> 192.168.2.107 means bz1 ( MDS node to bz7).
I have looked at dmesg on node7:
INFO: task cosd:17629 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
cosd D ffff8801fc64f310 0 17629 1 0x00000000
ffff880137b12830 0000000000000086 ffff8801fc64efa0 000000031ee16000
ffff8801e8481bb8 ffff8801f79ee250 0000000000000002 0000000181051412
0000000000000000 ffff8801f79ee200 ffff8801f79ee250 000000001ee16000
Call Trace:
[<ffffffffa02aebce>] ? btrfs_start_ordered_extent+0x8d/0xa3 [btrfs]
[<ffffffff81051243>] ? autoremove_wake_function+0x0/0x2e
[<ffffffffa02aecc7>] ? btrfs_wait_ordered_range+0x9c/0x107 [btrfs]
[<ffffffffa02a3a8e>] ? btrfs_file_aio_write+0x6cf/0x800 [btrfs]
[<ffffffffa028917a>] ? btrfs_block_rsv_check+0x42/0x128 [btrfs]
[<ffffffffa02a3b11>] ? btrfs_file_aio_write+0x752/0x800 [btrfs]
[<ffffffffa02165e9>] ? xfs_iunlock+0x21/0xb9 [xfs]
[<ffffffffa02a33bf>] ? btrfs_file_aio_write+0x0/0x800 [btrfs]
[<ffffffff810c6eca>] ? do_sync_readv_writev+0x9a/0xde
[<ffffffff810c6fb9>] ? do_sync_write+0xab/0xeb
[<ffffffff811686b0>] ? security_file_permission+0x18/0x6a
[<ffffffff810c7436>] ? do_readv_writev+0xb4/0x188
[<ffffffff810c7d38>] ? sys_writev+0x45/0x6e
[<ffffffff8100292b>] ? system_call_fastpath+0x16/0x1b
INFO: task cosd:17629 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
cosd D ffff8801fc64f310 0 17629 1 0x00000000
ffff880137b12830 0000000000000086 ffff8801fc64efa0 000000031ee16000
ffff8801e8481bb8 ffff8801f79ee250 0000000000000002 0000000181051412
0000000000000000 ffff8801f79ee200 ffff8801f79ee250 000000001ee16000
Call Trace:
[<ffffffffa02aebce>] ? btrfs_start_ordered_extent+0x8d/0xa3 [btrfs]
[<ffffffff81051243>] ? autoremove_wake_function+0x0/0x2e
[<ffffffffa02aecc7>] ? btrfs_wait_ordered_range+0x9c/0x107 [btrfs]
[<ffffffffa02a3a8e>] ? btrfs_file_aio_write+0x6cf/0x800 [btrfs]
[<ffffffffa028917a>] ? btrfs_block_rsv_check+0x42/0x128 [btrfs]
[<ffffffffa02a3b11>] ? btrfs_file_aio_write+0x752/0x800 [btrfs]
[<ffffffffa02165e9>] ? xfs_iunlock+0x21/0xb9 [xfs]
[<ffffffffa02a33bf>] ? btrfs_file_aio_write+0x0/0x800 [btrfs]
[<ffffffff810c6eca>] ? do_sync_readv_writev+0x9a/0xde
[<ffffffff810c6fb9>] ? do_sync_write+0xab/0xeb
[<ffffffff811686b0>] ? security_file_permission+0x18/0x6a
[<ffffffff810c7436>] ? do_readv_writev+0xb4/0x188
[<ffffffff810c7d38>] ? sys_writev+0x45/0x6e
[<ffffffff8100292b>] ? system_call_fastpath+0x16/0x1b
INFO: task cosd:17629 blocked for more than 120 seconds.
It shows FS sync failure or hung.. Is it due to xfs or btrfs. I need a some ideas to debug why mds is not coming up. Also too much of ping ping messages in OSD log so very hard to track. I just did "grep osd_op ceph.log" and sent you that log.
Updated by Sage Weil almost 13 years ago
The MDS isn't coming up because teh OSD requests aren't completing because btrfs is wedged. Which kernel are you using on the storage nodes? There have been a lot of btrfs fixes during the last two cycles.
Updated by shyamali mukherjee almost 13 years ago
Hi Sage,
I know you have closed the issue. But I could not attach the logfile as it is too huge. I have got few lines which may be useful using "awk" and "sed". Please let me know if it would work.
2011-05-26 09:31:04.861808 498ea940 -- 192.168.2.101:6807/7118 <== mds0 192.168.2.101:6800/6486 2 ==== osd_op(mds0.1:17 100.00000000.inode [writeful
l 0~396] 1.e07f) v2 ==== 133+0+396 (4022725998 0 1112785557) 0x9a9230 con 0x9a81e0
2011-05-26 09:31:04.861849 498ea940 -- 192.168.2.101:6807/7118 <== mds0 192.168.2.101:6800/6486 1 ==== osd_op(mds0.1:13 607.00000000 [setxattr path
(12),setxattr parent (39),tmapup 0~0] 1.b7c) v2 ==== 203+0+263 (2428217695 0 2655079671) 0x9a8a80 con 0x9a81e0
2011-05-26 09:31:04.861875 498ea940 -- 192.168.2.101:6807/7118 <== mds0 192.168.2.101:6800/6486 3 ==== osd_op(mds0.1:15 609.00000000 [setxattr path
(12),setxattr parent (39),tmapup 0~0] 1.2d07) v2 ==== 203+0+263 (1960066852 0 3508436675) 0x9a9920 con 0x9a81e0
2011-05-26 09:31:04.861898 498ea940 -- 192.168.2.101:6807/7118 <== mds0 192.168.2.101:6800/6486 4 ==== osd_op(mds0.1:16 100.00000000 [setxattr path
(5),setxattr parent (13),tmapup 0~0] 1.5ab3) v2 ==== 203+0+230 (2326146290 0 3970804453) 0x9aa030 con 0x9a81e0
2011-05-26 09:31:04.963211 49d69940 -- 192.168.2.101:6804/6911 <== mds0 192.168.2.101:6800/6486 1 ==== osd_op(mds0.1:5 1.00000000.inode [writefull 0
~425] 1.e14) v2 ==== 131+0+425 (2167959030 0 1507599605) 0x9e3c80 con 0x9e3260
2011-05-26 09:31:04.963246 49d69940 -- 192.168.2.101:6804/6911 <== mds0 192.168.2.101:6800/6486 2 ==== osd_op(mds0.1:19 mds0_sessionmap [writefull 0
~17] 1.c60b) v2 ==== 130+0+17 (2269158516 0 2334645356) 0x9e4100 con 0x9e3260
2011-05-26 09:31:04.963283 49d69940 -- 192.168.2.101:6804/6911 <== mds0 192.168.2.101:6800/6486 3 ==== osd_op(mds0.1:9 603.00000000 [setxattr path (
12),setxattr parent (39),tmapup 0~0] 1.91b0) v2 ==== 203+0+263 (1245453232 0 414148690) 0x9e46e0 con 0x9e3260
2011-05-26 09:31:04.963310 49d69940 -- 192.168.2.101:6804/6911 <== mds0 192.168.2.101:6800/6486 4 ==== osd_op(mds0.1:3 2.00000000 [setxattr path (6)
,setxattr parent (38),tmapup 0~0] 1.3707) v2 ==== 201+0+256 (2608959529 0 3602622908) 0x9e4f10 con 0x9e3260
2011-05-26 09:31:04.963333 49d69940 -- 192.168.2.101:6804/6911 <== mds0 192.168.2.101:6800/6486 5 ==== osd_op(mds0.1:12 606.00000000 [setxattr path
(12),setxattr parent (39),tmapup 0~0] 1.aaf4) v2 ==== 203+0+263 (3989332855 0 4269922402) 0x9e5640 con 0x9e3260
2011-05-26 09:31:04.963366 49d69940 -- 192.168.2.101:6804/6911 <== mds0 192.168.2.101:6800/6486 6 ==== osd_op(mds0.1:6 600.00000000 [setxattr path (
12),setxattr parent (39),tmapup 0~0] 1.41b0) v2 ==== 203+0+263 (2625221295 0 3119437293) 0x9e5d70 con 0x9e3260
2011-05-26 09:31:04.963392 49d69940 -- 192.168.2.101:6804/6911 <== mds0 192.168.2.101:6800/6486 7 ==== osd_op(mds0.1:7 601.00000000 [setxattr path
2011-05-26 09:31:17.281367 42c93940 -- 192.168.2.101:6800/6486 --> 192.168.2.102:6806/5395 -- osd_op(mds0.1:1 200.00000000 [writefull 0~84] 1.3494 R
ETRY) v1 -- ?+0 0x7fe5a80033d0 con 0xab6ce0
2011-05-26 09:31:17.281504 42c93940 -- 192.168.2.101:6800/6486 --> 192.168.2.103:6800/5190 -- osd_op(mds0.1:2 200.00000001 [write 0~115] 1.f474 RETR
Y) v1 -- ?+0 0x7fe5a8003890 con 0xb42da0
2011-05-26 09:31:17.281529 42c93940 -- 192.168.2.101:6800/6486 --> 192.168.2.103:6806/5511 -- osd_op(mds0.1:9 603.00000000 [setxattr path (12),setxa
ttr parent (39),tmapup 0~0] 1.91b0 RETRY) v1 -- ?+0 0x7fe5a8003c00 con 0x7fe5a8003090
2011-05-26 09:31:17.281550 42c93940 -- 192.168.2.101:6800/6486 --> 192.168.2.103:6803/5360 -- osd_op(mds0.1:3 2.00000000 [setxattr path (6),setxattr
parent (38),tmapup 0~0] 1.3707 RETRY) v1 -- ?+0 0x7fe5a80050d0 con 0xab6650
2011-05-26 09:31:17.281581 42c93940 -- 192.168.2.101:6800/6486 --> 192.168.2.102:6806/5395 -- osd_op(mds0.1:4 1.00000000 [setxattr path,setxattr par
ent (13),tmapput 0~553] 1.daff RETRY) v1 -- ?+0 0x7fe5a8003430 con 0xab6ce0
2011-05-26 09:31:17.281632 42c93940 -- 192.168.2.101:6800/6486 --> 192.168.2.103:6800/5190 -- osd_op(mds0.1:11 605.00000000 [setxattr path (12),setx
attr parent (39),tmapup 0~0] 1.8e1d RETRY) v1 -- ?+0 0x7fe5a8005890 con 0xb42da0
2011-05-26 09:31:17.281725 42c93940 -- 192.168.2.101:6800/6486 --> 192.168.2.103:6806/5511 -- osd_op(mds0.1:12 606.00000000 [setxattr path (12),setx
attr parent (39),tmapup 0~0] 1.aaf4 RETRY) v1 -- ?+0 0xac0a10 con 0x7fe5a8003090
2011-05-26 09:31:17.281783 42c93940 -- 192.168.2.101:6800/6486 --> 192.168.2.103:6800/5190 -- osd_op(mds0.1:13 607.00000000 [setxattr path (12),setx
attr parent (39),tmapup 0~0] 1.b7c RETRY) v1 -- ?+0 0xac2010 con 0xb42da0
2011-05-26 09:31:17.281823 42c93940 -- 192.168.2.101:6800/6486 --> 192.168.2.102:6800/5088 -- osd_op(mds0.1:14 608.00000000 [setxattr path (12),setx
attr parent (39),tmapup 0~0] 1.5e8e RETRY) v1 -- ?+0 0xac2240 con 0xb524e0
2011-05-26 09:31:17.281920 42c93940 -- 192.168.2.101:6800/6486 --> 192.168.2.103:6806/5511 -- osd_op(mds0.1:15 609.00000000 [setxattr path (12),setx
attr parent (39),tmapup 0~0] 1.2d07 RETRY) v1 -- ?+0 0xac28c0 con 0x7fe5a8003090
2011-05-26 09:31:17.281945 42c93940 -- 192.168.2.101:6800/6486 --> 192.168.2.103:6800/5190 -- osd_op(mds0.1:16 100.00000000 [setxattr path (5),setxa
ttr parent (13),tmapup 0~0] 1.5ab3 RETRY) v1 -- ?+0 0x7fe5a0000a00 con 0xb42da0
2011-05-26 09:31:17.281987 42c93940 -- 192.168.2.101:6800/6486 --> 192.168.2.102:6800/5088 -- osd_op(mds0.1:17 100.00000000.inode [writefull 0~396]
1.e07f RETRY) v1 -- ?+0 0x7fe5a0000ee0 con 0xb524e0
2011-05-26 09:31:17.282021 42c93940 -- 192.168.2.101:6800/6486 --> 192.168.2.103:6806/5511 -- osd_op(mds0.1:6 600.00000000 [setxattr path (12),setxa
ttr parent (39),tmapup 0~0] 1.41b0 RETRY) v1 -- ?+0 0x7fe5a0001250 con 0x7fe5a8003090
2011-05-26 09:31:17.282041 42c93940 -- 192.168.2.101:6800/6486 --> 192.168.2.102:6803/5251 -- osd_op(mds0.1:7 601.00000000 [setxattr path (12),setxa
ttr parent (39),tmapup 0~0] 1.290 RETRY) v1 -- ?+0 0x7fe5a0001710 con 0xb58ec0
2011-05-26 09:31:17.282071 42c93940 -- 192.168.2.101:6800/6486 --> 192.168.2.102:6803/5251 -- osd_op(mds0.1:8 602.00000000 [setxattr path (12),setxa
ttr parent (39),tmapup 0~0] 1.6bd0 RETRY) v1 -- ?+0 0x7fe5a0000a60 con 0xb58ec0
2011-05-26 09:31:17.282088 42c93940 -- 192.168.2.101:6800/6486 --> 192.168.2.102:6800/5088 -- osd_op(mds0.1:18 mds0_inotable [writefull 0~29] 1.b893
RETRY) v1 -- ?+0 0x7fe5a0001c30 con 0xb524e0
2011-05-26 09:31:17.282105 42c93940 -- 192.168.2.101:6800/6486 --> 192.168.2.103:6800/5190 -- osd_op(mds0.1:20 mds_anchortable [writefull 0~29] 1.f6
a7 RETRY) v1 -- ?+0 0x7fe5a0001710 con 0xb42da0
2011-05-26 09:31:17.282125 42c93940 -- 192.168.2.101:6800/6486 --> 192.168.2.103:6800/5190 -- osd_op(mds0.1:21 mds_snaptable [writefull 0~41] 1.70ad
RETRY) v1 -- ?+0 0x7fe5a0000a60 con 0xb42da0
2011-05-26 09:31:47.272118 42c93940 -- 192.168.2.101:6800/6486 --> 192.168.2.104:6800/4871 -- osd_op(mds0.1:2 200.00000001 [write 0~115] 1.f474 RETR
Y) v1 -- ?+0 0x7fe5a0001360 con 0x7fe5a0000a60
2011-05-26 09:31:47.272138 42c93940 -- 192.168.2.101:6800/6486 --> 192.168.2.104:6800/4871 -- osd_op(mds0.1:5 1.00000000.inode [writefull 0~425] 1.e
14 RETRY) v1 -- ?+0 0x7fe5a0001600 con 0x7fe5a0000a60
2011-05-26 09:31:47.272155 42c93940 -- 192.168.2.101:6800/6486 --> 192.168.2.104:6800/4871 -- osd_op(mds0.1:21 mds_snaptable [writefull 0~41] 1.70ad
RETRY) v1 -- ?+0 0x7fe5a0001940 con 0x7fe5a0000a60
2011-05-26 09:32:32.327469 42c93940 -- 192.168.2.101:6800/6486 --> 192.168.2.104:6803/5093 -- osd_op(mds0.1:5 1.00000000.inode [writefull 0~425] 1.e
14 RETRY) v1 -- ?+0 0x7fe5a0001ae0 con 0x7fe5a0000c50
2011-05-26 09:32:34.190780 42c93940 -- 192.168.2.101:6800/6486 <== osd8 192.168.2.102:6806/5395 1 ==== osd_op_reply(4 1.00000000 [setxattr,setxattr
(13),tmapput 0~553] ondisk = 0) v1 ==== 172+0+0 (1644226004 0 0) 0xa7bc90 con 0xab6ce0
2011-05-26 09:32:34.415615 42c93940 -- 192.168.2.101:6800/6486 <== osd6 192.168.2.102:6800/5088 1 ==== osd_op_reply(14 608.00000000 [setxattr (12),s
etxattr (39),tmapup 0~0] ondisk = 0) v1 ==== 174+0+0 (571501435 0 0) 0xaac9b0 con 0xb524e0
2011-05-26 09:32:34.807704 42c93940 -- 192.168.2.101:6800/6486 <== osd12 192.168.2.103:6800/5190 1 ==== osd_op_reply(13 607.00000000 [setxattr (12),
setxattr (39),tmapup 0~0] ondisk = 0) v1 ==== 174+0+0 (1348644492 0 0) 0xa7bc90 con 0xb42da0
2011-05-26 09:32:34.815586 42c93940 -- 192.168.2.101:6800/6486 <== osd12 192.168.2.103:6800/5190 2 ==== osd_op_reply(16 100.00000000 [setxattr (5),s
etxattr (13),tmapup 0~0] ondisk = 0) v1 ==== 174+0+0 (1682881669 0 0) 0xac0480 con 0xb42da0
2011-05-26 09:32:34.914791 42c93940 -- 192.168.2.101:6800/6486 <== osd14 192.168.2.103:6806/5511 1 ==== osd_op_reply(9 603.00000000 [setxattr (12),s
etxattr (39),tmapup 0~0] ondisk = 0) v1 ==== 174+0+0 (91175798 0 0) 0xa7bc90 con 0x7fe5a8003090
2011-05-26 09:32:34.921529 42c93940 -- 192.168.2.101:6800/6486 <== osd14 192.168.2.103:6806/5511 2 ==== osd_op_reply(6 600.00000000 [setxattr (12),s
etxattr (39),tmapup 0~0] ondisk = 0) v1 ==== 174+0+0 (2772476054 0 0) 0xab2a90 con 0x7fe5a8003090
2011-05-26 09:32:35.261580 42c93940 -- 192.168.2.101:6800/6486 <== osd7 192.168.2.102:6803/5251 1 ==== osd_op_reply(7 601.00000000 [setxattr (12),se
txattr (39),tmapup 0~0] ondisk = 0) v1 ==== 174+0+0 (2688518047 0 0) 0xa7bc90 con 0xb58ec0
2011-05-26 09:32:38.335505 42c93940 -- 192.168.2.101:6800/6486 <== osd14 192.168.2.103:6806/5511 3 ==== osd_op_reply(15 609.00000000 [setxattr (12),
setxattr (39),tmapup 0~0] ondisk = 0) v1 ==== 174+0+0 (3378144364 0 0) 0xa7bc90 con 0x7fe5a8003090
2011-05-26 09:32:38.343165 42c93940 -- 192.168.2.101:6800/6486 <== osd14 192.168.2.103:6806/5511 4 ==== osd_op_reply(12 606.00000000 [setxattr (12),
setxattr (39),tmapup 0~0] ondisk = 0) v1 ==== 174+0+0 (2386751939 0 0) 0xac0480 con 0x7fe5a8003090
Please let me know if this is user_xattr realted. I have mounted ext3 ( for journal and log) it with -o user_xattr option though.
Updated by Greg Farnum almost 13 years ago
The cosd has blocked on a btrfs bug; it doesn't have much to do with Ceph.
Eventually your cluster should declare that OSD down, reassign the data storage elsewhere, and allow your MDS to come up. Let us know if it doesn't do that!
Updated by Sage Weil almost 13 years ago
You switched everything over to ext3?
It doesn't look like a user_xattr issue; the cosd daemon will error out and exit on startup if they aren't working.
From that log segment it looks like some osd requests still aren't getting responses. You can look in the logs OSDs they are being sent to to see why. You can also run something like 'rados -p data bench 60' to verify that the OSD cluster is working in general (I suspect that will fail, given the above). Did you switch all of the OSDs over to ext3? Are they all up? Does 'ceph -s' show them all in the active+clean state?
Updated by shyamali mukherjee almost 13 years ago
I have put OSd logfile and journal to ext3. osd data still comes from "btrfs".
I have tried atleast about 50 times creatibng newfs and restarting. OSd logs shows just scrubbing and osd_ping.
"ceph osd stat" shows all up and clean. mon0 log also shows it is getting ping from all OSDs. all pgs also seems to be clean+active. Only
msd will stay in up:creating state.
I have scaled fown from 60 OSD (which I think caused a suicide because of heartbeat miss and eventually marking all down during heavy write).
to ONLY 10 OSD ( 1 osd / node). But I still see MDS never coming up.
"rados -p data bench 60" were failing too.. it was just going on forever with 0 in every column. So I suspect no write was being done.
Updated by Sage Weil almost 13 years ago
- Status changed from Closed to In Progress
shyamali mukherjee wrote:
I have put OSd logfile and journal to ext3. osd data still comes from "btrfs".
I have tried atleast about 50 times creatibng newfs and restarting. OSd logs shows just scrubbing and osd_ping.
"ceph osd stat" shows all up and clean. mon0 log also shows it is getting ping from all OSDs. all pgs also seems to be clean+active. Only
msd will stay in up:creating state.
Ok, you need to use ext3 for osd data or switch to a newer kernel for a newer btrfs. Whicheve ris easier for you!
I have scaled fown from 60 OSD (which I think caused a suicide because of heartbeat miss and eventually marking all down during heavy write).
to ONLY 10 OSD ( 1 osd / node). But I still see MDS never coming up.
I'm extremely interested in hearing about this particular problem (the heartbeat failures). Are you running the stable branch? Master branch? Please try the latest of either as there were a bunch of recent fixes in that code. If you're still seeing problems logs would be most appreciated.
"rados -p data bench 60" were failing too.. it was just going on forever with 0 in every column. So I suspect no write was being done.
Updated by Sage Weil almost 13 years ago
- Target version changed from v0.29 to v0.30
Updated by Sage Weil almost 13 years ago
- Status changed from In Progress to Closed