Ceph » RADOS

/a/sseshasa-2021-04-23_18:11:53-rados-wip-sseshasa-testing-2021-04-23-2212-distro-basic-smithi/6068991

Noticed a test failure shown below:

2021-04-24T02:57:01.237 INFO:tasks.workunit.client.0.smithi003.stdout:                      api_service_pp: [==========] Running 4 tests from 1 test suite.
2021-04-24T02:57:01.237 INFO:tasks.workunit.client.0.smithi003.stdout:           api_service_pp: [----------] Global test environment set-up.
2021-04-24T02:57:01.237 INFO:tasks.workunit.client.0.smithi003.stdout:           api_service_pp: [----------] 4 tests from LibRadosServicePP
2021-04-24T02:57:01.238 INFO:tasks.workunit.client.0.smithi003.stdout:           api_service_pp: [ RUN      ] LibRadosServicePP.RegisterEarly
2021-04-24T02:57:01.238 INFO:tasks.workunit.client.0.smithi003.stdout:           api_service_pp: [       OK ] LibRadosServicePP.RegisterEarly (5076 ms)
2021-04-24T02:57:01.238 INFO:tasks.workunit.client.0.smithi003.stdout:           api_service_pp: [ RUN      ] LibRadosServicePP.RegisterLate
2021-04-24T02:57:01.238 INFO:tasks.workunit.client.0.smithi003.stdout:           api_service_pp: [       OK ] LibRadosServicePP.RegisterLate (50 ms)
2021-04-24T02:57:01.238 INFO:tasks.workunit.client.0.smithi003.stdout:           api_service_pp: [ RUN      ] LibRadosServicePP.Status
2021-04-24T02:57:01.238 INFO:tasks.workunit.client.0.smithi003.stdout:           api_service_pp: [       OK ] LibRadosServicePP.Status (20021 ms)
2021-04-24T02:57:01.238 INFO:tasks.workunit.client.0.smithi003.stdout:           api_service_pp: [ RUN      ] LibRadosServicePP.Close
2021-04-24T02:57:01.238 INFO:tasks.workunit.client.0.smithi003.stdout:           api_service_pp: attempt 0 of 20
2021-04-24T02:57:01.238 INFO:tasks.workunit.client.0.smithi003.stdout:           api_service_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-3552-g2a676895/rpm/el8/BUILD/ceph-17.0.0-3552-g2a676895/src/test/librados/service_cxx.cc:73: Failure
2021-04-24T02:57:01.239 INFO:tasks.workunit.client.0.smithi003.stdout:           api_service_pp: Expected equality of these values:
2021-04-24T02:57:01.239 INFO:tasks.workunit.client.0.smithi003.stdout:           api_service_pp:   0
2021-04-24T02:57:01.239 INFO:tasks.workunit.client.0.smithi003.stdout:           api_service_pp:   cluster.connect()
2021-04-24T02:57:01.239 INFO:tasks.workunit.client.0.smithi003.stdout:           api_service_pp:     Which is: -110
2021-04-24T02:57:01.239 INFO:tasks.workunit.client.0.smithi003.stdout:           api_service_pp: [  FAILED  ] LibRadosServicePP.Close (3000021 ms)
2021-04-24T02:57:01.239 INFO:tasks.workunit.client.0.smithi003.stdout:           api_service_pp: [----------] 4 tests from LibRadosServicePP (3025168 ms total)
2021-04-24T02:57:01.239 INFO:tasks.workunit.client.0.smithi003.stdout:           api_service_pp:
2021-04-24T02:57:01.240 INFO:tasks.workunit.client.0.smithi003.stdout:           api_service_pp: [----------] Global test environment tear-down
2021-04-24T02:57:01.240 INFO:tasks.workunit.client.0.smithi003.stdout:           api_service_pp: [==========] 4 tests from 1 test suite ran. (3025168 ms total)
2021-04-24T02:57:01.240 INFO:tasks.workunit.client.0.smithi003.stdout:           api_service_pp: [  PASSED  ] 3 tests.
2021-04-24T02:57:01.240 INFO:tasks.workunit.client.0.smithi003.stdout:           api_service_pp: [  FAILED  ] 1 test, listed below:
2021-04-24T02:57:01.240 INFO:tasks.workunit.client.0.smithi003.stdout:           api_service_pp: [  FAILED  ] LibRadosServicePP.Close
2021-04-24T02:57:01.240 INFO:tasks.workunit.client.0.smithi003.stdout:           api_service_pp:
2021-04-24T02:57:01.240 INFO:tasks.workunit.client.0.smithi003.stdout:           api_service_pp:  1 FAILED TEST

Much later the test times out:

2021-04-24T05:06:35.852 INFO:tasks.workunit.client.0.smithi003.stderr:++ cleanup
2021-04-24T05:06:35.852 INFO:tasks.workunit.client.0.smithi003.stderr:++ pkill -P 40307
2021-04-24T05:06:35.863 INFO:teuthology.orchestra.run.smithi037.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused
2021-04-24T05:06:35.864 DEBUG:teuthology.orchestra.run:got remote process result: 22
2021-04-24T05:06:35.865 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops
2021-04-24T05:06:35.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh: line 10: 40312 Terminated              bash -o pipefail -exc "ceph_test_rados_$f $color 2>&1 | tee ceph_test_rados_$ff.log | sed \"s/^/$r: /\"" 
2021-04-24T05:06:35.870 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh: line 10: 40320 Terminated              bash -o pipefail -exc "ceph_test_rados_$f $color 2>&1 | tee ceph_test_rados_$ff.log | sed \"s/^/$r: /\"" 
2021-04-24T05:06:35.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh: line 10: 40352 Terminated              bash -o pipefail -exc "ceph_test_rados_$f $color 2>&1 | tee ceph_test_rados_$ff.log | sed \"s/^/$r: /\"" 
2021-04-24T05:06:35.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh: line 10: 40364 Terminated              bash -o pipefail -exc "ceph_test_rados_$f $color 2>&1 | tee ceph_test_rados_$ff.log | sed \"s/^/$r: /\"" 
2021-04-24T05:06:35.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh: line 10: 40436 Terminated              bash -o pipefail -exc "ceph_test_rados_$f $color 2>&1 | tee ceph_test_rados_$ff.log | sed \"s/^/$r: /\"" 
2021-04-24T05:06:35.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh: line 10: 40465 Terminated              bash -o pipefail -exc "ceph_test_rados_$f $color 2>&1 | tee ceph_test_rados_$ff.log | sed \"s/^/$r: /\"" 
2021-04-24T05:06:35.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh: line 10: 40480 Terminated              bash -o pipefail -exc "ceph_test_rados_$f $color 2>&1 | tee ceph_test_rados_$ff.log | sed \"s/^/$r: /\"" 
2021-04-24T05:06:35.871 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh: line 10: 40501 Terminated              bash -o pipefail -exc "ceph_test_rados_$f $color 2>&1 | tee ceph_test_rados_$ff.log | sed \"s/^/$r: /\"" 
2021-04-24T05:06:35.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh: line 10: 40524 Terminated              bash -o pipefail -exc "ceph_test_rados_$f $color 2>&1 | tee ceph_test_rados_$ff.log | sed \"s/^/$r: /\"" 
2021-04-24T05:06:35.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh: line 10: 40551 Terminated              bash -o pipefail -exc "ceph_test_rados_$f $color 2>&1 | tee ceph_test_rados_$ff.log | sed \"s/^/$r: /\"" 
2021-04-24T05:06:35.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh: line 10: 40599 Terminated              bash -o pipefail -exc "ceph_test_rados_$f $color 2>&1 | tee ceph_test_rados_$ff.log | sed \"s/^/$r: /\"" 
2021-04-24T05:06:35.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh: line 10: 40629 Terminated              bash -o pipefail -exc "ceph_test_rados_$f $color 2>&1 | tee ceph_test_rados_$ff.log | sed \"s/^/$r: /\"" 
2021-04-24T05:06:35.872 INFO:tasks.workunit.client.0.smithi003.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh: line 10: 40758 Terminated              bash -o pipefail -exc "ceph_test_rados_$f $color 2>&1 | tee ceph_test_rados_$ff.log | sed \"s/^/$r: /\"" 
2021-04-24T05:06:35.873 DEBUG:teuthology.orchestra.run:got remote process result: 124
2021-04-24T05:06:35.874 INFO:tasks.workunit.client.0.smithi003.stderr:++ true
2021-04-24T05:06:35.874 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0...
2021-04-24T05:06:35.874 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0
2021-04-24T05:06:35.890 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/teuthology/orchestra/daemon/state.py", line 108, in signal
    self.proc.stdin.write(struct.pack('!b', sig))
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 405, in write
    self._write_all(data)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 522, in _write_all
    count = self._write(data)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1364, in _write
    self.channel.sendall(data)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 846, in sendall
    sent = self.send(s)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 801, in send
    return self._send(s, m)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1198, in _send
    raise socket.error("Socket is closed")
OSError: Socket is closed