Bug #9700
closedcephtool mon_osd intermittent failure
Added by John Spray over 9 years ago. Updated over 9 years ago.
80%
Description
Hit this one time on a gitbuilder: it's not clear to me why we have a 5-time retry here: some timeout raciness in the test? Log attached, grep for "echo retried with non zero exit status, 5 times".
Test code was added in:
commit f1becf9ad7237f36cf65e2b8dc95ee43946fe1fd Author: Loic Dachary <loic-201408@dachary.org> Date: Sat Oct 4 11:34:27 2014 +0200 qa: ceph tell must retry on ENXIO It is expected for ceph tell to fail with ENXIO if the daemon it is trying to join is not ready for some reason. This should be handled as a transient error instead of a fatal error. Add two shell functions to help with retry. They may prove useful if other cases requiring a few retries show up. http://tracker.ceph.com/issues/9655 Fixes: #9655 Signed-off-by: Loic Dachary <loic-201408@dachary.org>
Or perhaps this is just the same thing as #9655 and the change to the test wasn't a sufficient wait?
Files
Updated by Loïc Dachary over 9 years ago
ENXIO is expected when ceph tell tries to join an osd that is not ready and it should be treated as EAGAIN. If it happens multiple times in a row, it no longer is a matter of waiting for the OSD to come up, it means it is down and the script should fail. I'll something must have caused the OSD to fail. And there hopefully is a clue earlier in the log. Note that at the beginning of each function a check is done to assert that no osd is down . That was introduced to limit the scope of investigation to the run of a single function when trying to find how an OSD could have died.
Updated by Loïc Dachary over 9 years ago
The osd 1 goes down during the following. Reading the script and what it does I can imagine why. Unless osd.1 dies because there is a late side effect of the previous test.
: 1340: test_mon_osd test_mon_osd: 751: bl=192.168.0.1:0/1000 test_mon_osd: 752: ceph osd blacklist add 192.168.0.1:0/1000 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** blacklisting 192.168.0.1:0/1000 until 2014-10-08 12:21:01.429977 (3600 sec) test_mon_osd: 753: ceph osd blacklist ls test_mon_osd: 753: grep 192.168.0.1:0/1000 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** listed 2 entries 192.168.0.1:0/1000 2014-10-08 12:21:01.429977 test_mon_osd: 754: ceph osd blacklist rm 192.168.0.1:0/1000 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** un-blacklisting 192.168.0.1:0/1000 test_mon_osd: 755: expect_false 'ceph osd blacklist ls | grep 192.168.0.1:0/1000' expect_false: 45: set -x expect_false: 46: 'ceph osd blacklist ls | grep 192.168.0.1:0/1000' ../qa/workunits/cephtool/test.sh: line 46: ceph osd blacklist ls | grep 192.168.0.1:0/1000: No such file or directory expect_false: 46: return 0 test_mon_osd: 757: bl=192.168.0.1 test_mon_osd: 759: ceph osd blacklist add 192.168.0.1 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** blacklisting 192.168.0.1:0/0 until 2014-10-08 12:21:02.539453 (3600 sec) test_mon_osd: 760: ceph osd blacklist ls test_mon_osd: 760: grep 192.168.0.1 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** listed 2 entries 192.168.0.1:0/0 2014-10-08 12:21:02.539453 test_mon_osd: 761: ceph osd blacklist rm 192.168.0.1 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** un-blacklisting 192.168.0.1:0/0 test_mon_osd: 762: expect_false 'ceph osd blacklist ls | grep 192.168.0.1' expect_false: 45: set -x expect_false: 46: 'ceph osd blacklist ls | grep 192.168.0.1' ../qa/workunits/cephtool/test.sh: line 46: ceph osd blacklist ls | grep 192.168.0.1: command not found expect_false: 46: return 0 test_mon_osd: 763: expect_false 'ceph osd blacklist 192.168.0.1/-1' expect_false: 45: set -x expect_false: 46: 'ceph osd blacklist 192.168.0.1/-1' ../qa/workunits/cephtool/test.sh: line 46: ceph osd blacklist 192.168.0.1/-1: No such file or directory expect_false: 46: return 0 test_mon_osd: 764: expect_false 'ceph osd blacklist 192.168.0.1/foo' expect_false: 45: set -x expect_false: 46: 'ceph osd blacklist 192.168.0.1/foo' ../qa/workunits/cephtool/test.sh: line 46: ceph osd blacklist 192.168.0.1/foo: No such file or directory expect_false: 46: return 0 test_mon_osd: 769: ceph osd crush tunables legacy *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** adjusted tunables profile to legacy test_mon_osd: 770: ceph osd crush show-tunables test_mon_osd: 770: grep argonaut *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** "profile": "argonaut", test_mon_osd: 771: ceph osd crush tunables bobtail *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** adjusted tunables profile to bobtail test_mon_osd: 772: ceph osd crush show-tunables test_mon_osd: 772: grep bobtail *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** "profile": "bobtail", test_mon_osd: 773: ceph osd crush tunables firefly *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** adjusted tunables profile to firefly test_mon_osd: 774: ceph osd crush show-tunables test_mon_osd: 774: grep firefly *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** "profile": "firefly", test_mon_osd: 780: ceph osd scrub 0 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** osd.0 instructed to scrub test_mon_osd: 781: ceph osd deep-scrub 0 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** osd.0 instructed to deep-scrub test_mon_osd: 782: ceph osd repair 0 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** osd.0 instructed to repair test_mon_osd: 784: for f in noup nodown noin noout noscrub nodeep-scrub nobackfill norecover notieragent test_mon_osd: 786: ceph osd set noup *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** set noup test_mon_osd: 787: ceph osd unset noup *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** unset noup test_mon_osd: 784: for f in noup nodown noin noout noscrub nodeep-scrub nobackfill norecover notieragent test_mon_osd: 786: ceph osd set nodown *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** set nodown test_mon_osd: 787: ceph osd unset nodown *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** unset nodown test_mon_osd: 784: for f in noup nodown noin noout noscrub nodeep-scrub nobackfill norecover notieragent test_mon_osd: 786: ceph osd set noin *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** set noin test_mon_osd: 787: ceph osd unset noin *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** unset noin test_mon_osd: 784: for f in noup nodown noin noout noscrub nodeep-scrub nobackfill norecover notieragent test_mon_osd: 786: ceph osd set noout *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** set noout test_mon_osd: 787: ceph osd unset noout *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** unset noout test_mon_osd: 784: for f in noup nodown noin noout noscrub nodeep-scrub nobackfill norecover notieragent test_mon_osd: 786: ceph osd set noscrub *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** set noscrub test_mon_osd: 787: ceph osd unset noscrub *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** unset noscrub test_mon_osd: 784: for f in noup nodown noin noout noscrub nodeep-scrub nobackfill norecover notieragent test_mon_osd: 786: ceph osd set nodeep-scrub *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** set nodeep-scrub test_mon_osd: 787: ceph osd unset nodeep-scrub *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** unset nodeep-scrub test_mon_osd: 784: for f in noup nodown noin noout noscrub nodeep-scrub nobackfill norecover notieragent test_mon_osd: 786: ceph osd set nobackfill *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** set nobackfill test_mon_osd: 787: ceph osd unset nobackfill *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** unset nobackfill test_mon_osd: 784: for f in noup nodown noin noout noscrub nodeep-scrub nobackfill norecover notieragent test_mon_osd: 786: ceph osd set norecover *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** set norecover test_mon_osd: 787: ceph osd unset norecover *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** unset norecover test_mon_osd: 784: for f in noup nodown noin noout noscrub nodeep-scrub nobackfill norecover notieragent test_mon_osd: 786: ceph osd set notieragent *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** set notieragent test_mon_osd: 787: ceph osd unset notieragent *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** unset notieragent test_mon_osd: 789: expect_false ceph osd set bogus expect_false: 45: set -x expect_false: 46: ceph osd set bogus *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** Invalid command: bogus not in pause|noup|nodown|noout|noin|nobackfill|norecover|noscrub|nodeep-scrub|notieragent osd set pause|noup|nodown|noout|noin|nobackfill|norecover|noscrub|nodeep-scrub|notieragent : set Error EINVAL: invalid command expect_false: 46: return 0 test_mon_osd: 790: expect_false ceph osd unset bogus expect_false: 45: set -x expect_false: 46: ceph osd unset bogus *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** Invalid command: bogus not in pause|noup|nodown|noout|noin|nobackfill|norecover|noscrub|nodeep-scrub|notieragent osd unset pause|noup|nodown|noout|noin|nobackfill|norecover|noscrub|nodeep-scrub|notieragent : unset Error EINVAL: invalid command expect_false: 46: return 0 test_mon_osd: 792: ceph osd set noup *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** set noup test_mon_osd: 793: ceph osd down 0 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** marked down osd.0. test_mon_osd: 794: ceph osd dump test_mon_osd: 794: grep 'osd.0 down' *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** osd.0 down in weight 1 up_from 4 up_thru 120 down_at 158 last_clean_interval [0,0) 127.0.0.1:6800/30001 127.0.0.1:6801/30001 127.0.0.1:6802/30001 127.0.0.1:6803/30001 exists 6528f0a4-dabf-46fa-ab4c-36a7e41f0742 test_mon_osd: 795: ceph osd unset noup *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** unset noup test_mon_osd: 796: (( i=0 )) test_mon_osd: 796: (( i < 100 )) test_mon_osd: 797: ceph osd dump test_mon_osd: 797: grep 'osd.0 up' *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** test_mon_osd: 798: echo 'waiting for osd.0 to come back up' waiting for osd.0 to come back up test_mon_osd: 799: sleep 10 test_mon_osd: 796: (( i++ )) test_mon_osd: 796: (( i < 100 )) test_mon_osd: 797: ceph osd dump test_mon_osd: 797: grep 'osd.0 up' *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** osd.0 up in weight 1 up_from 161 up_thru 161 down_at 158 last_clean_interval [4,160) 127.0.0.1:6800/30001 127.0.0.1:6814/1030001 127.0.0.1:6815/1030001 127.0.0.1:6816/1030001 exists,up 6528f0a4-dabf-46fa-ab4c-36a7e41f0742 test_mon_osd: 801: break test_mon_osd: 804: ceph osd dump test_mon_osd: 804: grep 'osd.0 up' *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** osd.0 up in weight 1 up_from 161 up_thru 161 down_at 158 last_clean_interval [4,160) 127.0.0.1:6800/30001 127.0.0.1:6814/1030001 127.0.0.1:6815/1030001 127.0.0.1:6816/1030001 exists,up 6528f0a4-dabf-46fa-ab4c-36a7e41f0742 test_mon_osd: 806: ceph osd thrash 0 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** will thrash map for 0 epochs test_mon_osd: 808: ceph osd dump test_mon_osd: 808: grep 'osd.0 up' *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** osd.0 up in weight 1 up_from 161 up_thru 161 down_at 158 last_clean_interval [4,160) 127.0.0.1:6800/30001 127.0.0.1:6814/1030001 127.0.0.1:6815/1030001 127.0.0.1:6816/1030001 exists,up 6528f0a4-dabf-46fa-ab4c-36a7e41f0742 test_mon_osd: 809: ceph osd find 1 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** { "osd": 1, "ip": "127.0.0.1:6804\/30226", "crush_location": { "host": "gitbuilder-ceph-tarball-precise-amd64-basic", "root": "default"}} test_mon_osd: 810: ceph --format plain osd find 1 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** { "osd": 1, "ip": "127.0.0.1:6804\/30226", "crush_location": { "host": "gitbuilder-ceph-tarball-precise-amd64-basic", "root": "default"}} test_mon_osd: 811: ceph osd metadata 1 test_mon_osd: 811: grep distro *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** "distro": "Ubuntu", "distro_codename": "precise", "distro_description": "Ubuntu 12.04.2 LTS", "distro_version": "12.04", test_mon_osd: 812: ceph --format plain osd metadata 1 test_mon_osd: 812: grep distro *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** "distro": "Ubuntu", "distro_codename": "precise", "distro_description": "Ubuntu 12.04.2 LTS", "distro_version": "12.04", test_mon_osd: 813: ceph osd out 0 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** marked out osd.0. test_mon_osd: 814: ceph osd dump test_mon_osd: 814: grep 'osd.0.*out' *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** osd.0 up out weight 0 up_from 161 up_thru 161 down_at 158 last_clean_interval [4,160) 127.0.0.1:6800/30001 127.0.0.1:6814/1030001 127.0.0.1:6815/1030001 127.0.0.1:6816/1030001 exists,up 6528f0a4-dabf-46fa-ab4c-36a7e41f0742 test_mon_osd: 815: ceph osd in 0 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** marked in osd.0. test_mon_osd: 816: ceph osd dump test_mon_osd: 816: grep 'osd.0.*in' *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** osd.0 up in weight 1 up_from 161 up_thru 161 down_at 158 last_clean_interval [4,160) 127.0.0.1:6800/30001 127.0.0.1:6814/1030001 127.0.0.1:6815/1030001 127.0.0.1:6816/1030001 exists,up 6528f0a4-dabf-46fa-ab4c-36a7e41f0742 test_mon_osd: 817: ceph osd find 0 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** { "osd": 0, "ip": "127.0.0.1:6800\/30001", "crush_location": { "host": "gitbuilder-ceph-tarball-precise-amd64-basic", "root": "default"}} test_mon_osd: 819: f=/tmp/cephtool30851/map.30851 test_mon_osd: 820: ceph osd getcrushmap -o /tmp/cephtool30851/map.30851 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** got crush map from osdmap epoch 165 test_mon_osd: 821: '[' -s /tmp/cephtool30851/map.30851 ']' test_mon_osd: 822: rm /tmp/cephtool30851/map.30851 test_mon_osd: 823: ceph osd getmap -o /tmp/cephtool30851/map.30851 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** got osdmap epoch 165 test_mon_osd: 824: '[' -s /tmp/cephtool30851/map.30851 ']' test_mon_osd: 825: rm /tmp/cephtool30851/map.30851 test_mon_osd: 826: ceph osd getmaxosd test_mon_osd: 826: sed -e 's/max_osd = //' -e 's/ in epoch.*//' *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** test_mon_osd: 826: save=3 test_mon_osd: 827: ceph osd setmaxosd 10 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** set new max_osd = 10 test_mon_osd: 828: ceph osd getmaxosd test_mon_osd: 828: grep 'max_osd = 10' *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** max_osd = 10 in epoch 167 test_mon_osd: 829: ceph osd setmaxosd 3 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** set new max_osd = 3 test_mon_osd: 830: ceph osd getmaxosd test_mon_osd: 830: grep 'max_osd = 3' *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** max_osd = 3 in epoch 168 test_mon_osd: 747: ceph osd ls *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** test_mon_osd: 832: for id in '`ceph osd ls`' test_mon_osd: 833: retry_eagain 5 map_enxio_to_eagain ceph tell osd.0 version retry_eagain: 63: local max=5 retry_eagain: 64: shift retry_eagain: 65: local status retry_eagain: 66: local tmpfile=/tmp/cephtool30851/retry_eagain.30851 retry_eagain: 67: local count retry_eagain: 62: seq 1 5 retry_eagain: 68: for count in '$(seq 1 $max)' retry_eagain: 69: status=0 retry_eagain: 70: map_enxio_to_eagain ceph tell osd.0 version retry_eagain: 71: test 0 = 0 retry_eagain: 73: break retry_eagain: 77: test 1 = 5 retry_eagain: 80: cat /tmp/cephtool30851/retry_eagain.30851 map_enxio_to_eagain: 93: local status=0 map_enxio_to_eagain: 94: local tmpfile=/tmp/cephtool30851/map_enxio_to_eagain.30851 map_enxio_to_eagain: 96: ceph tell osd.0 version map_enxio_to_eagain: 97: test 0 '!=' 0 map_enxio_to_eagain: 101: cat /tmp/cephtool30851/map_enxio_to_eagain.30851 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** { "version": "ceph version 0.86-267-ge27cf41 (e27cf4139fbe895ef4d1817365275e6a50d603d8)"} map_enxio_to_eagain: 102: rm /tmp/cephtool30851/map_enxio_to_eagain.30851 map_enxio_to_eagain: 103: return 0 retry_eagain: 81: rm /tmp/cephtool30851/retry_eagain.30851 retry_eagain: 82: return 0 test_mon_osd: 832: for id in '`ceph osd ls`' test_mon_osd: 833: retry_eagain 5 map_enxio_to_eagain ceph tell osd.1 version retry_eagain: 63: local max=5 retry_eagain: 64: shift retry_eagain: 65: local status retry_eagain: 66: local tmpfile=/tmp/cephtool30851/retry_eagain.30851 retry_eagain: 67: local count retry_eagain: 62: seq 1 5 retry_eagain: 68: for count in '$(seq 1 $max)' retry_eagain: 69: status=0 retry_eagain: 70: map_enxio_to_eagain ceph tell osd.1 version retry_eagain: 70: status=6 retry_eagain: 71: test 6 = 0 retry_eagain: 72: grep --quiet EAGAIN /tmp/cephtool30851/retry_eagain.30851 retry_eagain: 75: sleep 1 retry_eagain: 68: for count in '$(seq 1 $max)' retry_eagain: 69: status=0 retry_eagain: 70: map_enxio_to_eagain ceph tell osd.1 version retry_eagain: 70: status=6 retry_eagain: 71: test 6 = 0 retry_eagain: 72: grep --quiet EAGAIN /tmp/cephtool30851/retry_eagain.30851 retry_eagain: 75: sleep 1 retry_eagain: 68: for count in '$(seq 1 $max)' retry_eagain: 69: status=0 retry_eagain: 70: map_enxio_to_eagain ceph tell osd.1 version retry_eagain: 70: status=6 retry_eagain: 71: test 6 = 0 retry_eagain: 72: grep --quiet EAGAIN /tmp/cephtool30851/retry_eagain.30851 retry_eagain: 75: sleep 1 retry_eagain: 68: for count in '$(seq 1 $max)' retry_eagain: 69: status=0 retry_eagain: 70: map_enxio_to_eagain ceph tell osd.1 version retry_eagain: 70: status=6 retry_eagain: 71: test 6 = 0 retry_eagain: 72: grep --quiet EAGAIN /tmp/cephtool30851/retry_eagain.30851 retry_eagain: 75: sleep 1 retry_eagain: 68: for count in '$(seq 1 $max)' retry_eagain: 69: status=0 retry_eagain: 70: map_enxio_to_eagain ceph tell osd.1 version retry_eagain: 70: status=6 retry_eagain: 71: test 6 = 0 retry_eagain: 72: grep --quiet EAGAIN /tmp/cephtool30851/retry_eagain.30851 retry_eagain: 75: sleep 1 retry_eagain: 77: test 5 = 5 retry_eagain: 78: echo retried with non zero exit status, 5 times: map_enxio_to_eagain ceph tell osd.1 version retried with non zero exit status, 5 times: map_enxio_to_eagain ceph tell osd.1 version retry_eagain: 80: cat /tmp/cephtool30851/retry_eagain.30851 map_enxio_to_eagain: 93: local status=0 map_enxio_to_eagain: 94: local tmpfile=/tmp/cephtool30851/map_enxio_to_eagain.30851 map_enxio_to_eagain: 96: ceph tell osd.1 version map_enxio_to_eagain: 96: status=6 map_enxio_to_eagain: 97: test 6 '!=' 0 map_enxio_to_eagain: 98: grep --quiet ENXIO /tmp/cephtool30851/map_enxio_to_eagain.30851 map_enxio_to_eagain: 99: echo 'EAGAIN added by ../qa/workunits/cephtool/test.sh::map_enxio_to_eagain' map_enxio_to_eagain: 101: cat /tmp/cephtool30851/map_enxio_to_eagain.30851 *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** Error ENXIO: problem getting command descriptions from osd.1 EAGAIN added by ../qa/workunits/cephtool/test.sh::map_enxio_to_eagain map_enxio_to_eagain: 102: rm /tmp/cephtool30851/map_enxio_to_eagain.30851
Updated by Loïc Dachary over 9 years ago
- Category set to qa
- Status changed from New to Fix Under Review
- % Done changed from 0 to 80
https://github.com/ceph/ceph/pull/2670
<jcsp1> loicd: hmm, what in mon_mds is leaving something in a bad state? <jcsp1> I don't mind the MDS tests moving to last, just curious <loicd> I have absolutely no clue. <loicd> only that mon_osd contains nothing that could shut osd.1 down <loicd> the test that precedes it is mon_mon which is ... trivial and harmless <loicd> and mon_mds is before it and does a number of things I do not understand, hence my semi irrational proposal ;-)
Updated by Loïc Dachary over 9 years ago
- Status changed from Fix Under Review to 7
Waiting about a week to see if it shows up again.
Updated by Loïc Dachary over 9 years ago
- Status changed from 7 to Resolved
I've not seen errors since this patch, except for firefly builds because this was not backported. Feel free to re-open with a link to a failure log if you see one !