Actions
Bug #9818
closedENXIO qa/workunits/cephtool/test.sh:test_osd_bench
% Done:
0%
Source:
Development
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
It looks like the OSD crashed but there is no more information than the following log at the moment. It was created with ../qa/workunits/cephtool/test.sh --test-osd --asok-does-not-need-root on a vstart cluster run with rm -fr dev out ; mkdir -p dev ; MON=1 OSD=3 ./vstart.sh -d -n -l mon osd
./init-ceph: ceph conf ./ceph.conf not found; system is not configured. ip 127.0.0.1 NOTE: hostname resolves to loopback; remote hosts will not be able to connect. either adjust /etc/hosts, or edit this script to use your machine's real IP. creating /home/loic/software/ceph/ceph/src/test-7201/keyring ./monmaptool --create --clobber --add a 127.0.0.1:7201 --print /tmp/ceph_monmap.24231 ./monmaptool: monmap file /tmp/ceph_monmap.24231 ./monmaptool: generated fsid ab98fe36-73be-4f91-aeb0-76fc694cce32 epoch 0 fsid ab98fe36-73be-4f91-aeb0-76fc694cce32 last_changed 2014-10-18 22:25:23.807384 created 2014-10-18 22:25:23.807384 0: 127.0.0.1:7201/0 mon.a ./monmaptool: writing epoch 0 to /tmp/ceph_monmap.24231 (1 monitors) rm -rf /home/loic/software/ceph/ceph/src/test-7201/dev/mon.a mkdir -p /home/loic/software/ceph/ceph/src/test-7201/dev/mon.a ./ceph-mon --mkfs -c /home/loic/software/ceph/ceph/src/test-7201/ceph.conf -i a --monmap=/tmp/ceph_monmap.24231 --keyring=/home/loic/software/ceph/ceph/src/test-7201/keyring ./ceph-mon: set fsid to c4a0e8e4-f301-41f3-9ca6-80053b2f8c8b ./ceph-mon: created monfs at /home/loic/software/ceph/ceph/src/test-7201/dev/mon.a for mon.a ./ceph-mon -i a -c /home/loic/software/ceph/ceph/src/test-7201/ceph.conf ERROR: error accessing '/home/loic/software/ceph/ceph/src/test-7201/dev/osd0/*' add osd0 06dd10a7-1224-4e58-aa8a-0e4c7e972135 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** 0 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** add item id 0 name 'osd.0' weight 1 at location {host=fold,root=default} to crush map 2014-10-18 22:25:44.821984 2ba97c7d24c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-10-18 22:25:50.255423 2ba97c7d24c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-10-18 22:25:50.339583 2ba97c7d24c0 -1 filestore(/home/loic/software/ceph/ceph/src/test-7201/dev/osd0) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory 2014-10-18 22:25:55.208941 2ba97c7d24c0 -1 created object store /home/loic/software/ceph/ceph/src/test-7201/dev/osd0 journal /home/loic/software/ceph/ceph/src/test-7201/dev/osd0.journal for osd.0 fsid c4a0e8e4-f301-41f3-9ca6-80053b2f8c8b 2014-10-18 22:25:55.208994 2ba97c7d24c0 -1 auth: error reading file: /home/loic/software/ceph/ceph/src/test-7201/dev/osd0/keyring: can't open /home/loic/software/ceph/ceph/src/test-7201/dev/osd0/keyring: (2) No such file or directory 2014-10-18 22:25:55.209056 2ba97c7d24c0 -1 created new key in keyring /home/loic/software/ceph/ceph/src/test-7201/dev/osd0/keyring adding osd0 key to auth repository *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** added key for osd.0 start osd0 ./ceph-osd -i 0 -c /home/loic/software/ceph/ceph/src/test-7201/ceph.conf starting osd.0 at :/0 osd_data /home/loic/software/ceph/ceph/src/test-7201/dev/osd0 /home/loic/software/ceph/ceph/src/test-7201/dev/osd0.journal ERROR: error accessing '/home/loic/software/ceph/ceph/src/test-7201/dev/osd1/*' add osd1 b35f3e09-7b74-4ac3-a94c-5515de511fc8 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** 1 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** add item id 1 name 'osd.1' weight 1 at location {host=fold,root=default} to crush map 2014-10-18 22:26:03.131454 2b88ca2e04c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-10-18 22:26:07.377648 2b88ca2e04c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-10-18 22:26:07.453137 2b88ca2e04c0 -1 filestore(/home/loic/software/ceph/ceph/src/test-7201/dev/osd1) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory 2014-10-18 22:26:12.724027 2b88ca2e04c0 -1 created object store /home/loic/software/ceph/ceph/src/test-7201/dev/osd1 journal /home/loic/software/ceph/ceph/src/test-7201/dev/osd1.journal for osd.1 fsid c4a0e8e4-f301-41f3-9ca6-80053b2f8c8b 2014-10-18 22:26:12.724080 2b88ca2e04c0 -1 auth: error reading file: /home/loic/software/ceph/ceph/src/test-7201/dev/osd1/keyring: can't open /home/loic/software/ceph/ceph/src/test-7201/dev/osd1/keyring: (2) No such file or directory 2014-10-18 22:26:12.758995 2b88ca2e04c0 -1 created new key in keyring /home/loic/software/ceph/ceph/src/test-7201/dev/osd1/keyring adding osd1 key to auth repository *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** added key for osd.1 start osd1 ./ceph-osd -i 1 -c /home/loic/software/ceph/ceph/src/test-7201/ceph.conf starting osd.1 at :/0 osd_data /home/loic/software/ceph/ceph/src/test-7201/dev/osd1 /home/loic/software/ceph/ceph/src/test-7201/dev/osd1.journal ERROR: error accessing '/home/loic/software/ceph/ceph/src/test-7201/dev/osd2/*' add osd2 c250ca3a-7aee-4808-818c-b1df4591c413 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** 2 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** add item id 2 name 'osd.2' weight 1 at location {host=fold,root=default} to crush map 2014-10-18 22:26:20.290486 2af785e684c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-10-18 22:27:21.402106 2af785e684c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2014-10-18 22:27:21.449712 2af785e684c0 -1 filestore(/home/loic/software/ceph/ceph/src/test-7201/dev/osd2) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory 2014-10-18 22:27:24.929625 2af785e684c0 -1 created object store /home/loic/software/ceph/ceph/src/test-7201/dev/osd2 journal /home/loic/software/ceph/ceph/src/test-7201/dev/osd2.journal for osd.2 fsid c4a0e8e4-f301-41f3-9ca6-80053b2f8c8b 2014-10-18 22:27:24.929670 2af785e684c0 -1 auth: error reading file: /home/loic/software/ceph/ceph/src/test-7201/dev/osd2/keyring: can't open /home/loic/software/ceph/ceph/src/test-7201/dev/osd2/keyring: (2) No such file or directory 2014-10-18 22:27:24.929732 2af785e684c0 -1 created new key in keyring /home/loic/software/ceph/ceph/src/test-7201/dev/osd2/keyring adding osd2 key to auth repository *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** added key for osd.2 start osd2 ./ceph-osd -i 2 -c /home/loic/software/ceph/ceph/src/test-7201/ceph.conf starting osd.2 at :/0 osd_data /home/loic/software/ceph/ceph/src/test-7201/dev/osd2 /home/loic/software/ceph/ceph/src/test-7201/dev/osd2.journal creating /home/loic/software/ceph/ceph/src/test-7201/dev/mds.a/keyring *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** added key for mds.a ./ceph -c /home/loic/software/ceph/ceph/src/test-7201/ceph.conf -k /home/loic/software/ceph/ceph/src/test-7201/keyring osd pool create cephfs_data 8 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** pool 'cephfs_data' created ./ceph -c /home/loic/software/ceph/ceph/src/test-7201/ceph.conf -k /home/loic/software/ceph/ceph/src/test-7201/keyring osd pool create cephfs_metadata 8 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** pool 'cephfs_metadata' created ./ceph -c /home/loic/software/ceph/ceph/src/test-7201/ceph.conf -k /home/loic/software/ceph/ceph/src/test-7201/keyring fs new cephfs cephfs_metadata cephfs_data *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** new fs with metadata pool 2 and data pool 1 ./ceph-mds -i a -c /home/loic/software/ceph/ceph/src/test-7201/ceph.conf starting mds.a at :/0 creating /home/loic/software/ceph/ceph/src/test-7201/dev/mds.b/keyring *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** added key for mds.b ./ceph -c /home/loic/software/ceph/ceph/src/test-7201/ceph.conf -k /home/loic/software/ceph/ceph/src/test-7201/keyring osd pool create cephfs_data 8 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** pool 'cephfs_data' already exists ./ceph -c /home/loic/software/ceph/ceph/src/test-7201/ceph.conf -k /home/loic/software/ceph/ceph/src/test-7201/keyring osd pool create cephfs_metadata 8 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** pool 'cephfs_metadata' already exists ./ceph -c /home/loic/software/ceph/ceph/src/test-7201/ceph.conf -k /home/loic/software/ceph/ceph/src/test-7201/keyring fs new cephfs cephfs_metadata cephfs_data *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** filesystem 'cephfs' already exists ./ceph-mds -i b -c /home/loic/software/ceph/ceph/src/test-7201/ceph.conf starting mds.b at :/0 creating /home/loic/software/ceph/ceph/src/test-7201/dev/mds.c/keyring *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** added key for mds.c ./ceph -c /home/loic/software/ceph/ceph/src/test-7201/ceph.conf -k /home/loic/software/ceph/ceph/src/test-7201/keyring osd pool create cephfs_data 8 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** pool 'cephfs_data' already exists ./ceph -c /home/loic/software/ceph/ceph/src/test-7201/ceph.conf -k /home/loic/software/ceph/ceph/src/test-7201/keyring osd pool create cephfs_metadata 8 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** pool 'cephfs_metadata' already exists ./ceph -c /home/loic/software/ceph/ceph/src/test-7201/ceph.conf -k /home/loic/software/ceph/ceph/src/test-7201/keyring fs new cephfs cephfs_metadata cephfs_data *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** filesystem 'cephfs' already exists ./ceph-mds -i c -c /home/loic/software/ceph/ceph/src/test-7201/ceph.conf starting mds.c at :/0 ./ceph -c /home/loic/software/ceph/ceph/src/test-7201/ceph.conf -k /home/loic/software/ceph/ceph/src/test-7201/keyring mds set max_mds 3 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** started. stop.sh to stop. see out/* (e.g. 'tail -f out/????') for debug output. export PYTHONPATH=./pybind export LD_LIBRARY_PATH=.libs export CEPH_CONF=/home/loic/software/ceph/ceph/src/test-7201/ceph.conf export CEPH_KEYRING=/home/loic/software/ceph/ceph/src/test-7201/keyring + set -e + set -o functrace + PS4=' ${FUNCNAME[0]}: $LINENO: ' main: 6: SUDO=sudo main: 49: TMPDIR=/tmp/cephtool20859 main: 50: mkdir /tmp/cephtool20859 main: 51: trap 'rm -fr /tmp/cephtool20859' 0 main: 53: TMPFILE=/tmp/cephtool20859/test_invalid.20859 main: 1265: set +x *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** main: 1376: test_osd_bench test_osd_bench: 1217: ceph tell osd.0 injectargs ' --osd-bench-duration 10 --osd-bench-max-block-size 2097152 --osd-bench-large-size-max-throughput 10485760 --osd-bench-small-size-max-iops 10' *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** failed to parse arguments: osd_bench_duration = '10' osd_bench_large_size_max_throughput = '10485760' osd_bench_max_block_size = '2097152' osd_bench_small_size_max_iops = '10' test_osd_bench: 1224: expect_false ceph tell osd.0 bench 1 2097153 expect_false: 45: set -x expect_false: 46: ceph tell osd.0 bench 1 2097153 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** Error EINVAL: block 'size' values are capped at 2048 kB. If you wish to use a higher value, please adjust 'osd_bench_max_block_size' expect_false: 46: return 0 test_osd_bench: 1226: ceph tell osd.0 bench 1 2097152 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** { "bytes_written": 1, "blocksize": 2097152, "bytes_per_sec": "4.000000"} test_osd_bench: 1233: expect_false ceph tell osd.0 bench 409601 4096 expect_false: 45: set -x expect_false: 46: ceph tell osd.0 bench 409601 4096 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** Error EINVAL: 'count' values greater than 409600 for a block size of 4096 bytes, assuming 10 IOPS, for 10 seconds, can cause ill effects on osd. Please adjust 'osd_bench_small_size_max_iops' with a higher value if you wish to use a higher 'count'. expect_false: 46: return 0 test_osd_bench: 1235: ceph tell osd.0 bench 409600 4096 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** { "bytes_written": 409600, "blocksize": 4096, "bytes_per_sec": "26440.000000"} test_osd_bench: 1243: expect_false ceph tell osd.0 bench 104857601 2097152 expect_false: 45: set -x expect_false: 46: ceph tell osd.0 bench 104857601 2097152 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** Error EINVAL: 'count' values greater than 104857600 for a block size of 2048 kB, assuming 10240 kB/s, for 10 seconds, can cause ill effects on osd. Please adjust 'osd_bench_large_size_max_throughput' with a higher value if you wish to use a higher 'count'. expect_false: 46: return 0 test_osd_bench: 1245: ceph tell osd.0 bench 104857600 2097152 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** Error ENXIO: osd down test_osd_bench: 1: rm -fr /tmp/cephtool20859
Updated by Loïc Dachary over 9 years ago
- Status changed from Need More Info to Resolved
did not happen for a long time, looks like it's stable at last
Actions