Actions
Bug #9700
closedcephtool mon_osd intermittent failure
% Done:
80%
Source:
other
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Hit this one time on a gitbuilder: it's not clear to me why we have a 5-time retry here: some timeout raciness in the test? Log attached, grep for "echo retried with non zero exit status, 5 times".
Test code was added in:
commit f1becf9ad7237f36cf65e2b8dc95ee43946fe1fd
Author: Loic Dachary <loic-201408@dachary.org>
Date: Sat Oct 4 11:34:27 2014 +0200
qa: ceph tell must retry on ENXIO
It is expected for ceph tell to fail with ENXIO if the daemon it is
trying to join is not ready for some reason. This should be handled as a
transient error instead of a fatal error.
Add two shell functions to help with retry. They may prove useful if
other cases requiring a few retries show up.
http://tracker.ceph.com/issues/9655 Fixes: #9655
Signed-off-by: Loic Dachary <loic-201408@dachary.org>
Or perhaps this is just the same thing as #9655 and the change to the test wasn't a sufficient wait?
Files
Updated by Loïc Dachary over 9 years ago
ENXIO is expected when ceph tell tries to join an osd that is not ready and it should be treated as EAGAIN. If it happens multiple times in a row, it no longer is a matter of waiting for the OSD to come up, it means it is down and the script should fail. I'll something must have caused the OSD to fail. And there hopefully is a clue earlier in the log. Note that at the beginning of each function a check is done to assert that no osd is down . That was introduced to limit the scope of investigation to the run of a single function when trying to find how an OSD could have died.
Updated by Loïc Dachary over 9 years ago
The osd 1 goes down during the following. Reading the script and what it does I can imagine why. Unless osd.1 dies because there is a late side effect of the previous test.
: 1340: test_mon_osd
test_mon_osd: 751: bl=192.168.0.1:0/1000
test_mon_osd: 752: ceph osd blacklist add 192.168.0.1:0/1000
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
blacklisting 192.168.0.1:0/1000 until 2014-10-08 12:21:01.429977 (3600 sec)
test_mon_osd: 753: ceph osd blacklist ls
test_mon_osd: 753: grep 192.168.0.1:0/1000
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
listed 2 entries
192.168.0.1:0/1000 2014-10-08 12:21:01.429977
test_mon_osd: 754: ceph osd blacklist rm 192.168.0.1:0/1000
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
un-blacklisting 192.168.0.1:0/1000
test_mon_osd: 755: expect_false 'ceph osd blacklist ls | grep 192.168.0.1:0/1000'
expect_false: 45: set -x
expect_false: 46: 'ceph osd blacklist ls | grep 192.168.0.1:0/1000'
../qa/workunits/cephtool/test.sh: line 46: ceph osd blacklist ls | grep 192.168.0.1:0/1000: No such file or directory
expect_false: 46: return 0
test_mon_osd: 757: bl=192.168.0.1
test_mon_osd: 759: ceph osd blacklist add 192.168.0.1
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
blacklisting 192.168.0.1:0/0 until 2014-10-08 12:21:02.539453 (3600 sec)
test_mon_osd: 760: ceph osd blacklist ls
test_mon_osd: 760: grep 192.168.0.1
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
listed 2 entries
192.168.0.1:0/0 2014-10-08 12:21:02.539453
test_mon_osd: 761: ceph osd blacklist rm 192.168.0.1
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
un-blacklisting 192.168.0.1:0/0
test_mon_osd: 762: expect_false 'ceph osd blacklist ls | grep 192.168.0.1'
expect_false: 45: set -x
expect_false: 46: 'ceph osd blacklist ls | grep 192.168.0.1'
../qa/workunits/cephtool/test.sh: line 46: ceph osd blacklist ls | grep 192.168.0.1: command not found
expect_false: 46: return 0
test_mon_osd: 763: expect_false 'ceph osd blacklist 192.168.0.1/-1'
expect_false: 45: set -x
expect_false: 46: 'ceph osd blacklist 192.168.0.1/-1'
../qa/workunits/cephtool/test.sh: line 46: ceph osd blacklist 192.168.0.1/-1: No such file or directory
expect_false: 46: return 0
test_mon_osd: 764: expect_false 'ceph osd blacklist 192.168.0.1/foo'
expect_false: 45: set -x
expect_false: 46: 'ceph osd blacklist 192.168.0.1/foo'
../qa/workunits/cephtool/test.sh: line 46: ceph osd blacklist 192.168.0.1/foo: No such file or directory
expect_false: 46: return 0
test_mon_osd: 769: ceph osd crush tunables legacy
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
adjusted tunables profile to legacy
test_mon_osd: 770: ceph osd crush show-tunables
test_mon_osd: 770: grep argonaut
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
"profile": "argonaut",
test_mon_osd: 771: ceph osd crush tunables bobtail
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
adjusted tunables profile to bobtail
test_mon_osd: 772: ceph osd crush show-tunables
test_mon_osd: 772: grep bobtail
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
"profile": "bobtail",
test_mon_osd: 773: ceph osd crush tunables firefly
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
adjusted tunables profile to firefly
test_mon_osd: 774: ceph osd crush show-tunables
test_mon_osd: 774: grep firefly
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
"profile": "firefly",
test_mon_osd: 780: ceph osd scrub 0
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
osd.0 instructed to scrub
test_mon_osd: 781: ceph osd deep-scrub 0
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
osd.0 instructed to deep-scrub
test_mon_osd: 782: ceph osd repair 0
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
osd.0 instructed to repair
test_mon_osd: 784: for f in noup nodown noin noout noscrub nodeep-scrub nobackfill norecover notieragent
test_mon_osd: 786: ceph osd set noup
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
set noup
test_mon_osd: 787: ceph osd unset noup
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
unset noup
test_mon_osd: 784: for f in noup nodown noin noout noscrub nodeep-scrub nobackfill norecover notieragent
test_mon_osd: 786: ceph osd set nodown
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
set nodown
test_mon_osd: 787: ceph osd unset nodown
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
unset nodown
test_mon_osd: 784: for f in noup nodown noin noout noscrub nodeep-scrub nobackfill norecover notieragent
test_mon_osd: 786: ceph osd set noin
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
set noin
test_mon_osd: 787: ceph osd unset noin
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
unset noin
test_mon_osd: 784: for f in noup nodown noin noout noscrub nodeep-scrub nobackfill norecover notieragent
test_mon_osd: 786: ceph osd set noout
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
set noout
test_mon_osd: 787: ceph osd unset noout
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
unset noout
test_mon_osd: 784: for f in noup nodown noin noout noscrub nodeep-scrub nobackfill norecover notieragent
test_mon_osd: 786: ceph osd set noscrub
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
set noscrub
test_mon_osd: 787: ceph osd unset noscrub
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
unset noscrub
test_mon_osd: 784: for f in noup nodown noin noout noscrub nodeep-scrub nobackfill norecover notieragent
test_mon_osd: 786: ceph osd set nodeep-scrub
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
set nodeep-scrub
test_mon_osd: 787: ceph osd unset nodeep-scrub
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
unset nodeep-scrub
test_mon_osd: 784: for f in noup nodown noin noout noscrub nodeep-scrub nobackfill norecover notieragent
test_mon_osd: 786: ceph osd set nobackfill
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
set nobackfill
test_mon_osd: 787: ceph osd unset nobackfill
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
unset nobackfill
test_mon_osd: 784: for f in noup nodown noin noout noscrub nodeep-scrub nobackfill norecover notieragent
test_mon_osd: 786: ceph osd set norecover
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
set norecover
test_mon_osd: 787: ceph osd unset norecover
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
unset norecover
test_mon_osd: 784: for f in noup nodown noin noout noscrub nodeep-scrub nobackfill norecover notieragent
test_mon_osd: 786: ceph osd set notieragent
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
set notieragent
test_mon_osd: 787: ceph osd unset notieragent
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
unset notieragent
test_mon_osd: 789: expect_false ceph osd set bogus
expect_false: 45: set -x
expect_false: 46: ceph osd set bogus
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
Invalid command: bogus not in pause|noup|nodown|noout|noin|nobackfill|norecover|noscrub|nodeep-scrub|notieragent
osd set pause|noup|nodown|noout|noin|nobackfill|norecover|noscrub|nodeep-scrub|notieragent : set
Error EINVAL: invalid command
expect_false: 46: return 0
test_mon_osd: 790: expect_false ceph osd unset bogus
expect_false: 45: set -x
expect_false: 46: ceph osd unset bogus
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
Invalid command: bogus not in pause|noup|nodown|noout|noin|nobackfill|norecover|noscrub|nodeep-scrub|notieragent
osd unset pause|noup|nodown|noout|noin|nobackfill|norecover|noscrub|nodeep-scrub|notieragent : unset
Error EINVAL: invalid command
expect_false: 46: return 0
test_mon_osd: 792: ceph osd set noup
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
set noup
test_mon_osd: 793: ceph osd down 0
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
marked down osd.0.
test_mon_osd: 794: ceph osd dump
test_mon_osd: 794: grep 'osd.0 down'
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
osd.0 down in weight 1 up_from 4 up_thru 120 down_at 158 last_clean_interval [0,0) 127.0.0.1:6800/30001 127.0.0.1:6801/30001 127.0.0.1:6802/30001 127.0.0.1:6803/30001 exists 6528f0a4-dabf-46fa-ab4c-36a7e41f0742
test_mon_osd: 795: ceph osd unset noup
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
unset noup
test_mon_osd: 796: (( i=0 ))
test_mon_osd: 796: (( i < 100 ))
test_mon_osd: 797: ceph osd dump
test_mon_osd: 797: grep 'osd.0 up'
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
test_mon_osd: 798: echo 'waiting for osd.0 to come back up'
waiting for osd.0 to come back up
test_mon_osd: 799: sleep 10
test_mon_osd: 796: (( i++ ))
test_mon_osd: 796: (( i < 100 ))
test_mon_osd: 797: ceph osd dump
test_mon_osd: 797: grep 'osd.0 up'
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
osd.0 up in weight 1 up_from 161 up_thru 161 down_at 158 last_clean_interval [4,160) 127.0.0.1:6800/30001 127.0.0.1:6814/1030001 127.0.0.1:6815/1030001 127.0.0.1:6816/1030001 exists,up 6528f0a4-dabf-46fa-ab4c-36a7e41f0742
test_mon_osd: 801: break
test_mon_osd: 804: ceph osd dump
test_mon_osd: 804: grep 'osd.0 up'
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
osd.0 up in weight 1 up_from 161 up_thru 161 down_at 158 last_clean_interval [4,160) 127.0.0.1:6800/30001 127.0.0.1:6814/1030001 127.0.0.1:6815/1030001 127.0.0.1:6816/1030001 exists,up 6528f0a4-dabf-46fa-ab4c-36a7e41f0742
test_mon_osd: 806: ceph osd thrash 0
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
will thrash map for 0 epochs
test_mon_osd: 808: ceph osd dump
test_mon_osd: 808: grep 'osd.0 up'
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
osd.0 up in weight 1 up_from 161 up_thru 161 down_at 158 last_clean_interval [4,160) 127.0.0.1:6800/30001 127.0.0.1:6814/1030001 127.0.0.1:6815/1030001 127.0.0.1:6816/1030001 exists,up 6528f0a4-dabf-46fa-ab4c-36a7e41f0742
test_mon_osd: 809: ceph osd find 1
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
{ "osd": 1,
"ip": "127.0.0.1:6804\/30226",
"crush_location": { "host": "gitbuilder-ceph-tarball-precise-amd64-basic",
"root": "default"}} test_mon_osd: 810: ceph --format plain osd find 1
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
{ "osd": 1,
"ip": "127.0.0.1:6804\/30226",
"crush_location": { "host": "gitbuilder-ceph-tarball-precise-amd64-basic",
"root": "default"}} test_mon_osd: 811: ceph osd metadata 1
test_mon_osd: 811: grep distro
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
"distro": "Ubuntu",
"distro_codename": "precise",
"distro_description": "Ubuntu 12.04.2 LTS",
"distro_version": "12.04",
test_mon_osd: 812: ceph --format plain osd metadata 1
test_mon_osd: 812: grep distro
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
"distro": "Ubuntu",
"distro_codename": "precise",
"distro_description": "Ubuntu 12.04.2 LTS",
"distro_version": "12.04",
test_mon_osd: 813: ceph osd out 0
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
marked out osd.0.
test_mon_osd: 814: ceph osd dump
test_mon_osd: 814: grep 'osd.0.*out'
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
osd.0 up out weight 0 up_from 161 up_thru 161 down_at 158 last_clean_interval [4,160) 127.0.0.1:6800/30001 127.0.0.1:6814/1030001 127.0.0.1:6815/1030001 127.0.0.1:6816/1030001 exists,up 6528f0a4-dabf-46fa-ab4c-36a7e41f0742
test_mon_osd: 815: ceph osd in 0
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
marked in osd.0.
test_mon_osd: 816: ceph osd dump
test_mon_osd: 816: grep 'osd.0.*in'
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
osd.0 up in weight 1 up_from 161 up_thru 161 down_at 158 last_clean_interval [4,160) 127.0.0.1:6800/30001 127.0.0.1:6814/1030001 127.0.0.1:6815/1030001 127.0.0.1:6816/1030001 exists,up 6528f0a4-dabf-46fa-ab4c-36a7e41f0742
test_mon_osd: 817: ceph osd find 0
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
{ "osd": 0,
"ip": "127.0.0.1:6800\/30001",
"crush_location": { "host": "gitbuilder-ceph-tarball-precise-amd64-basic",
"root": "default"}} test_mon_osd: 819: f=/tmp/cephtool30851/map.30851
test_mon_osd: 820: ceph osd getcrushmap -o /tmp/cephtool30851/map.30851
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
got crush map from osdmap epoch 165
test_mon_osd: 821: '[' -s /tmp/cephtool30851/map.30851 ']'
test_mon_osd: 822: rm /tmp/cephtool30851/map.30851
test_mon_osd: 823: ceph osd getmap -o /tmp/cephtool30851/map.30851
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
got osdmap epoch 165
test_mon_osd: 824: '[' -s /tmp/cephtool30851/map.30851 ']'
test_mon_osd: 825: rm /tmp/cephtool30851/map.30851
test_mon_osd: 826: ceph osd getmaxosd
test_mon_osd: 826: sed -e 's/max_osd = //' -e 's/ in epoch.*//'
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
test_mon_osd: 826: save=3
test_mon_osd: 827: ceph osd setmaxosd 10
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
set new max_osd = 10
test_mon_osd: 828: ceph osd getmaxosd
test_mon_osd: 828: grep 'max_osd = 10'
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
max_osd = 10 in epoch 167
test_mon_osd: 829: ceph osd setmaxosd 3
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
set new max_osd = 3
test_mon_osd: 830: ceph osd getmaxosd
test_mon_osd: 830: grep 'max_osd = 3'
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
max_osd = 3 in epoch 168
test_mon_osd: 747: ceph osd ls
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
test_mon_osd: 832: for id in '`ceph osd ls`'
test_mon_osd: 833: retry_eagain 5 map_enxio_to_eagain ceph tell osd.0 version
retry_eagain: 63: local max=5
retry_eagain: 64: shift
retry_eagain: 65: local status
retry_eagain: 66: local tmpfile=/tmp/cephtool30851/retry_eagain.30851
retry_eagain: 67: local count
retry_eagain: 62: seq 1 5
retry_eagain: 68: for count in '$(seq 1 $max)'
retry_eagain: 69: status=0
retry_eagain: 70: map_enxio_to_eagain ceph tell osd.0 version
retry_eagain: 71: test 0 = 0
retry_eagain: 73: break
retry_eagain: 77: test 1 = 5
retry_eagain: 80: cat /tmp/cephtool30851/retry_eagain.30851
map_enxio_to_eagain: 93: local status=0
map_enxio_to_eagain: 94: local tmpfile=/tmp/cephtool30851/map_enxio_to_eagain.30851
map_enxio_to_eagain: 96: ceph tell osd.0 version
map_enxio_to_eagain: 97: test 0 '!=' 0
map_enxio_to_eagain: 101: cat /tmp/cephtool30851/map_enxio_to_eagain.30851
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
{ "version": "ceph version 0.86-267-ge27cf41 (e27cf4139fbe895ef4d1817365275e6a50d603d8)"}
map_enxio_to_eagain: 102: rm /tmp/cephtool30851/map_enxio_to_eagain.30851
map_enxio_to_eagain: 103: return 0
retry_eagain: 81: rm /tmp/cephtool30851/retry_eagain.30851
retry_eagain: 82: return 0
test_mon_osd: 832: for id in '`ceph osd ls`'
test_mon_osd: 833: retry_eagain 5 map_enxio_to_eagain ceph tell osd.1 version
retry_eagain: 63: local max=5
retry_eagain: 64: shift
retry_eagain: 65: local status
retry_eagain: 66: local tmpfile=/tmp/cephtool30851/retry_eagain.30851
retry_eagain: 67: local count
retry_eagain: 62: seq 1 5
retry_eagain: 68: for count in '$(seq 1 $max)'
retry_eagain: 69: status=0
retry_eagain: 70: map_enxio_to_eagain ceph tell osd.1 version
retry_eagain: 70: status=6
retry_eagain: 71: test 6 = 0
retry_eagain: 72: grep --quiet EAGAIN /tmp/cephtool30851/retry_eagain.30851
retry_eagain: 75: sleep 1
retry_eagain: 68: for count in '$(seq 1 $max)'
retry_eagain: 69: status=0
retry_eagain: 70: map_enxio_to_eagain ceph tell osd.1 version
retry_eagain: 70: status=6
retry_eagain: 71: test 6 = 0
retry_eagain: 72: grep --quiet EAGAIN /tmp/cephtool30851/retry_eagain.30851
retry_eagain: 75: sleep 1
retry_eagain: 68: for count in '$(seq 1 $max)'
retry_eagain: 69: status=0
retry_eagain: 70: map_enxio_to_eagain ceph tell osd.1 version
retry_eagain: 70: status=6
retry_eagain: 71: test 6 = 0
retry_eagain: 72: grep --quiet EAGAIN /tmp/cephtool30851/retry_eagain.30851
retry_eagain: 75: sleep 1
retry_eagain: 68: for count in '$(seq 1 $max)'
retry_eagain: 69: status=0
retry_eagain: 70: map_enxio_to_eagain ceph tell osd.1 version
retry_eagain: 70: status=6
retry_eagain: 71: test 6 = 0
retry_eagain: 72: grep --quiet EAGAIN /tmp/cephtool30851/retry_eagain.30851
retry_eagain: 75: sleep 1
retry_eagain: 68: for count in '$(seq 1 $max)'
retry_eagain: 69: status=0
retry_eagain: 70: map_enxio_to_eagain ceph tell osd.1 version
retry_eagain: 70: status=6
retry_eagain: 71: test 6 = 0
retry_eagain: 72: grep --quiet EAGAIN /tmp/cephtool30851/retry_eagain.30851
retry_eagain: 75: sleep 1
retry_eagain: 77: test 5 = 5
retry_eagain: 78: echo retried with non zero exit status, 5 times: map_enxio_to_eagain ceph tell osd.1 version
retried with non zero exit status, 5 times: map_enxio_to_eagain ceph tell osd.1 version
retry_eagain: 80: cat /tmp/cephtool30851/retry_eagain.30851
map_enxio_to_eagain: 93: local status=0
map_enxio_to_eagain: 94: local tmpfile=/tmp/cephtool30851/map_enxio_to_eagain.30851
map_enxio_to_eagain: 96: ceph tell osd.1 version
map_enxio_to_eagain: 96: status=6
map_enxio_to_eagain: 97: test 6 '!=' 0
map_enxio_to_eagain: 98: grep --quiet ENXIO /tmp/cephtool30851/map_enxio_to_eagain.30851
map_enxio_to_eagain: 99: echo 'EAGAIN added by ../qa/workunits/cephtool/test.sh::map_enxio_to_eagain'
map_enxio_to_eagain: 101: cat /tmp/cephtool30851/map_enxio_to_eagain.30851
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
Error ENXIO: problem getting command descriptions from osd.1
EAGAIN added by ../qa/workunits/cephtool/test.sh::map_enxio_to_eagain
map_enxio_to_eagain: 102: rm /tmp/cephtool30851/map_enxio_to_eagain.30851
Updated by Loïc Dachary over 9 years ago
- Category set to qa
- Status changed from New to Fix Under Review
- % Done changed from 0 to 80
https://github.com/ceph/ceph/pull/2670
<jcsp1> loicd: hmm, what in mon_mds is leaving something in a bad state? <jcsp1> I don't mind the MDS tests moving to last, just curious <loicd> I have absolutely no clue. <loicd> only that mon_osd contains nothing that could shut osd.1 down <loicd> the test that precedes it is mon_mon which is ... trivial and harmless <loicd> and mon_mds is before it and does a number of things I do not understand, hence my semi irrational proposal ;-)
Updated by Loïc Dachary over 9 years ago
- Status changed from Fix Under Review to 7
Waiting about a week to see if it shows up again.
Updated by Loïc Dachary over 9 years ago
- Status changed from 7 to Resolved
I've not seen errors since this patch, except for firefly builds because this was not backported. Feel free to re-open with a link to a failure log if you see one !
Actions