Ceph

It looks like the OSD crashed but there is no more information than the following log at the moment. It was created with ../qa/workunits/cephtool/test.sh --test-osd --asok-does-not-need-root on a vstart cluster run with rm -fr dev out ; mkdir -p dev ; MON=1 OSD=3 ./vstart.sh -d -n -l mon osd

./init-ceph: ceph conf ./ceph.conf not found; system is not configured.
ip 127.0.0.1

NOTE: hostname resolves to loopback; remote hosts will not be able to
  connect.  either adjust /etc/hosts, or edit this script to use your
  machine's real IP.

creating /home/loic/software/ceph/ceph/src/test-7201/keyring
./monmaptool --create --clobber --add a 127.0.0.1:7201 --print /tmp/ceph_monmap.24231
./monmaptool: monmap file /tmp/ceph_monmap.24231
./monmaptool: generated fsid ab98fe36-73be-4f91-aeb0-76fc694cce32
epoch 0
fsid ab98fe36-73be-4f91-aeb0-76fc694cce32
last_changed 2014-10-18 22:25:23.807384
created 2014-10-18 22:25:23.807384
0: 127.0.0.1:7201/0 mon.a
./monmaptool: writing epoch 0 to /tmp/ceph_monmap.24231 (1 monitors)
rm -rf /home/loic/software/ceph/ceph/src/test-7201/dev/mon.a
mkdir -p /home/loic/software/ceph/ceph/src/test-7201/dev/mon.a
./ceph-mon --mkfs -c /home/loic/software/ceph/ceph/src/test-7201/ceph.conf -i a --monmap=/tmp/ceph_monmap.24231 --keyring=/home/loic/software/ceph/ceph/src/test-7201/keyring
./ceph-mon: set fsid to c4a0e8e4-f301-41f3-9ca6-80053b2f8c8b
./ceph-mon: created monfs at /home/loic/software/ceph/ceph/src/test-7201/dev/mon.a for mon.a
./ceph-mon -i a -c /home/loic/software/ceph/ceph/src/test-7201/ceph.conf
ERROR: error accessing '/home/loic/software/ceph/ceph/src/test-7201/dev/osd0/*'
add osd0 06dd10a7-1224-4e58-aa8a-0e4c7e972135
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
0
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
add item id 0 name 'osd.0' weight 1 at location {host=fold,root=default} to crush map
2014-10-18 22:25:44.821984 2ba97c7d24c0 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
2014-10-18 22:25:50.255423 2ba97c7d24c0 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
2014-10-18 22:25:50.339583 2ba97c7d24c0 -1 filestore(/home/loic/software/ceph/ceph/src/test-7201/dev/osd0) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory
2014-10-18 22:25:55.208941 2ba97c7d24c0 -1 created object store /home/loic/software/ceph/ceph/src/test-7201/dev/osd0 journal /home/loic/software/ceph/ceph/src/test-7201/dev/osd0.journal for osd.0 fsid c4a0e8e4-f301-41f3-9ca6-80053b2f8c8b
2014-10-18 22:25:55.208994 2ba97c7d24c0 -1 auth: error reading file: /home/loic/software/ceph/ceph/src/test-7201/dev/osd0/keyring: can't open /home/loic/software/ceph/ceph/src/test-7201/dev/osd0/keyring: (2) No such file or directory
2014-10-18 22:25:55.209056 2ba97c7d24c0 -1 created new key in keyring /home/loic/software/ceph/ceph/src/test-7201/dev/osd0/keyring
adding osd0 key to auth repository
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
added key for osd.0
start osd0
./ceph-osd -i 0 -c /home/loic/software/ceph/ceph/src/test-7201/ceph.conf
starting osd.0 at :/0 osd_data /home/loic/software/ceph/ceph/src/test-7201/dev/osd0 /home/loic/software/ceph/ceph/src/test-7201/dev/osd0.journal
ERROR: error accessing '/home/loic/software/ceph/ceph/src/test-7201/dev/osd1/*'
add osd1 b35f3e09-7b74-4ac3-a94c-5515de511fc8
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
1
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
add item id 1 name 'osd.1' weight 1 at location {host=fold,root=default} to crush map
2014-10-18 22:26:03.131454 2b88ca2e04c0 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
2014-10-18 22:26:07.377648 2b88ca2e04c0 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
2014-10-18 22:26:07.453137 2b88ca2e04c0 -1 filestore(/home/loic/software/ceph/ceph/src/test-7201/dev/osd1) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory
2014-10-18 22:26:12.724027 2b88ca2e04c0 -1 created object store /home/loic/software/ceph/ceph/src/test-7201/dev/osd1 journal /home/loic/software/ceph/ceph/src/test-7201/dev/osd1.journal for osd.1 fsid c4a0e8e4-f301-41f3-9ca6-80053b2f8c8b
2014-10-18 22:26:12.724080 2b88ca2e04c0 -1 auth: error reading file: /home/loic/software/ceph/ceph/src/test-7201/dev/osd1/keyring: can't open /home/loic/software/ceph/ceph/src/test-7201/dev/osd1/keyring: (2) No such file or directory
2014-10-18 22:26:12.758995 2b88ca2e04c0 -1 created new key in keyring /home/loic/software/ceph/ceph/src/test-7201/dev/osd1/keyring
adding osd1 key to auth repository
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
added key for osd.1
start osd1
./ceph-osd -i 1 -c /home/loic/software/ceph/ceph/src/test-7201/ceph.conf
starting osd.1 at :/0 osd_data /home/loic/software/ceph/ceph/src/test-7201/dev/osd1 /home/loic/software/ceph/ceph/src/test-7201/dev/osd1.journal
ERROR: error accessing '/home/loic/software/ceph/ceph/src/test-7201/dev/osd2/*'
add osd2 c250ca3a-7aee-4808-818c-b1df4591c413
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
2
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
add item id 2 name 'osd.2' weight 1 at location {host=fold,root=default} to crush map
2014-10-18 22:26:20.290486 2af785e684c0 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
2014-10-18 22:27:21.402106 2af785e684c0 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
2014-10-18 22:27:21.449712 2af785e684c0 -1 filestore(/home/loic/software/ceph/ceph/src/test-7201/dev/osd2) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory
2014-10-18 22:27:24.929625 2af785e684c0 -1 created object store /home/loic/software/ceph/ceph/src/test-7201/dev/osd2 journal /home/loic/software/ceph/ceph/src/test-7201/dev/osd2.journal for osd.2 fsid c4a0e8e4-f301-41f3-9ca6-80053b2f8c8b
2014-10-18 22:27:24.929670 2af785e684c0 -1 auth: error reading file: /home/loic/software/ceph/ceph/src/test-7201/dev/osd2/keyring: can't open /home/loic/software/ceph/ceph/src/test-7201/dev/osd2/keyring: (2) No such file or directory
2014-10-18 22:27:24.929732 2af785e684c0 -1 created new key in keyring /home/loic/software/ceph/ceph/src/test-7201/dev/osd2/keyring
adding osd2 key to auth repository
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
added key for osd.2
start osd2
./ceph-osd -i 2 -c /home/loic/software/ceph/ceph/src/test-7201/ceph.conf
starting osd.2 at :/0 osd_data /home/loic/software/ceph/ceph/src/test-7201/dev/osd2 /home/loic/software/ceph/ceph/src/test-7201/dev/osd2.journal
creating /home/loic/software/ceph/ceph/src/test-7201/dev/mds.a/keyring
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
added key for mds.a
./ceph -c /home/loic/software/ceph/ceph/src/test-7201/ceph.conf -k /home/loic/software/ceph/ceph/src/test-7201/keyring osd pool create cephfs_data 8
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
pool 'cephfs_data' created
./ceph -c /home/loic/software/ceph/ceph/src/test-7201/ceph.conf -k /home/loic/software/ceph/ceph/src/test-7201/keyring osd pool create cephfs_metadata 8
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
pool 'cephfs_metadata' created
./ceph -c /home/loic/software/ceph/ceph/src/test-7201/ceph.conf -k /home/loic/software/ceph/ceph/src/test-7201/keyring fs new cephfs cephfs_metadata cephfs_data
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
new fs with metadata pool 2 and data pool 1
./ceph-mds -i a -c /home/loic/software/ceph/ceph/src/test-7201/ceph.conf
starting mds.a at :/0
creating /home/loic/software/ceph/ceph/src/test-7201/dev/mds.b/keyring
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
added key for mds.b
./ceph -c /home/loic/software/ceph/ceph/src/test-7201/ceph.conf -k /home/loic/software/ceph/ceph/src/test-7201/keyring osd pool create cephfs_data 8
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
pool 'cephfs_data' already exists
./ceph -c /home/loic/software/ceph/ceph/src/test-7201/ceph.conf -k /home/loic/software/ceph/ceph/src/test-7201/keyring osd pool create cephfs_metadata 8
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
pool 'cephfs_metadata' already exists
./ceph -c /home/loic/software/ceph/ceph/src/test-7201/ceph.conf -k /home/loic/software/ceph/ceph/src/test-7201/keyring fs new cephfs cephfs_metadata cephfs_data
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
filesystem 'cephfs' already exists
./ceph-mds -i b -c /home/loic/software/ceph/ceph/src/test-7201/ceph.conf
starting mds.b at :/0
creating /home/loic/software/ceph/ceph/src/test-7201/dev/mds.c/keyring
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
added key for mds.c
./ceph -c /home/loic/software/ceph/ceph/src/test-7201/ceph.conf -k /home/loic/software/ceph/ceph/src/test-7201/keyring osd pool create cephfs_data 8
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
pool 'cephfs_data' already exists
./ceph -c /home/loic/software/ceph/ceph/src/test-7201/ceph.conf -k /home/loic/software/ceph/ceph/src/test-7201/keyring osd pool create cephfs_metadata 8
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
pool 'cephfs_metadata' already exists
./ceph -c /home/loic/software/ceph/ceph/src/test-7201/ceph.conf -k /home/loic/software/ceph/ceph/src/test-7201/keyring fs new cephfs cephfs_metadata cephfs_data
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
filesystem 'cephfs' already exists
./ceph-mds -i c -c /home/loic/software/ceph/ceph/src/test-7201/ceph.conf
starting mds.c at :/0
./ceph -c /home/loic/software/ceph/ceph/src/test-7201/ceph.conf -k /home/loic/software/ceph/ceph/src/test-7201/keyring mds set max_mds 3
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
started.  stop.sh to stop.  see out/* (e.g. 'tail -f out/????') for debug output.

export PYTHONPATH=./pybind
export LD_LIBRARY_PATH=.libs
export CEPH_CONF=/home/loic/software/ceph/ceph/src/test-7201/ceph.conf
export CEPH_KEYRING=/home/loic/software/ceph/ceph/src/test-7201/keyring
+ set -e
+ set -o functrace
+ PS4=' ${FUNCNAME[0]}: $LINENO: '
 main: 6: SUDO=sudo
 main: 49: TMPDIR=/tmp/cephtool20859
 main: 50: mkdir /tmp/cephtool20859
 main: 51: trap 'rm -fr /tmp/cephtool20859' 0
 main: 53: TMPFILE=/tmp/cephtool20859/test_invalid.20859
 main: 1265: set +x
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
 main: 1376: test_osd_bench
 test_osd_bench: 1217: ceph tell osd.0 injectargs '    --osd-bench-duration 10     --osd-bench-max-block-size 2097152     --osd-bench-large-size-max-throughput 10485760     --osd-bench-small-size-max-iops 10'
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
 failed to parse arguments: 
osd_bench_duration = '10' osd_bench_large_size_max_throughput = '10485760' osd_bench_max_block_size = '2097152' osd_bench_small_size_max_iops = '10' 
 test_osd_bench: 1224: expect_false ceph tell osd.0 bench 1 2097153
 expect_false: 45: set -x
 expect_false: 46: ceph tell osd.0 bench 1 2097153
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
Error EINVAL: block 'size' values are capped at 2048 kB. If you wish to use a higher value, please adjust 'osd_bench_max_block_size'
 expect_false: 46: return 0
 test_osd_bench: 1226: ceph tell osd.0 bench 1 2097152
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
{ "bytes_written": 1,
  "blocksize": 2097152,
  "bytes_per_sec": "4.000000"}
 test_osd_bench: 1233: expect_false ceph tell osd.0 bench 409601 4096
 expect_false: 45: set -x
 expect_false: 46: ceph tell osd.0 bench 409601 4096
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
Error EINVAL: 'count' values greater than 409600 for a block size of 4096 bytes, assuming 10 IOPS, for 10 seconds, can cause ill effects on osd.  Please adjust 'osd_bench_small_size_max_iops' with a higher value if you wish to use a higher 'count'.
 expect_false: 46: return 0
 test_osd_bench: 1235: ceph tell osd.0 bench 409600 4096
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
{ "bytes_written": 409600,
  "blocksize": 4096,
  "bytes_per_sec": "26440.000000"}
 test_osd_bench: 1243: expect_false ceph tell osd.0 bench 104857601 2097152
 expect_false: 45: set -x
 expect_false: 46: ceph tell osd.0 bench 104857601 2097152
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
Error EINVAL: 'count' values greater than 104857600 for a block size of 2048 kB, assuming 10240 kB/s, for 10 seconds, can cause ill effects on osd.  Please adjust 'osd_bench_large_size_max_throughput' with a higher value if you wish to use a higher 'count'.
 expect_false: 46: return 0
 test_osd_bench: 1245: ceph tell osd.0 bench 104857600 2097152
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
Error ENXIO: osd down
 test_osd_bench: 1: rm -fr /tmp/cephtool20859