Ceph » RADOS

This failure was previously tracked in issue #50042, but it has come up enough that it warrants its own Tracker. See issue #50042 for earlier instances.

/a/yuriw-2022-03-19_14:37:23-rados-wip-yuri6-testing-2022-03-18-1104-distro-default-smithi/6747001

2022-03-19T19:56:53.724 INFO:tasks.workunit.client.0.smithi053.stdout:              api_tier_pp: [----------] Global test environment tear-down
2022-03-19T19:56:53.725 INFO:tasks.workunit.client.0.smithi053.stdout:              api_tier_pp: [==========] 77 tests from 4 test suites ran. (3861916 ms total)
2022-03-19T19:56:53.725 INFO:tasks.workunit.client.0.smithi053.stdout:              api_tier_pp: [  PASSED  ] 77 tests.
2022-03-19T19:56:53.726 INFO:tasks.workunit.client.0.smithi053.stderr:+ exit 1
2022-03-19T19:56:53.726 INFO:tasks.workunit.client.0.smithi053.stderr:+ cleanup
2022-03-19T19:56:53.726 INFO:tasks.workunit.client.0.smithi053.stderr:+ pkill -P 48582
2022-03-19T19:56:53.732 INFO:tasks.workunit.client.0.smithi053.stderr:+ true
2022-03-19T19:56:53.733 DEBUG:teuthology.orchestra.run:got remote process result: 1
2022-03-19T19:56:53.734 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0...
2022-03-19T19:56:53.734 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0
2022-03-19T19:56:53.776 INFO:tasks.ceph.osd.1.smithi053.stderr:2022-03-19T19:56:53.773+0000 ee74700 -1 received  signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1  (PID: 268905) UID: 0
2022-03-19T19:56:53.778 INFO:teuthology.orchestra.run.smithi053.stdout:{
2022-03-19T19:56:53.778 INFO:teuthology.orchestra.run.smithi053.stdout:    "success": "" 
2022-03-19T19:56:53.779 INFO:teuthology.orchestra.run.smithi053.stdout:}
2022-03-19T19:56:53.843 INFO:tasks.ceph.osd.5.smithi125.stderr:2022-03-19T19:56:53.840+0000 ee74700 -1 received  signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5  (PID: 107842) UID: 0
2022-03-19T19:56:53.861 INFO:teuthology.orchestra.run.smithi053.stderr:2022-03-19T19:56:53.858+0000 7f4aeaffd700  1 -- 172.21.15.53:0/1210551706 <== mon.1 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0  v0) v1 ==== 72+0+162503 (secure 0 0 0) 0x7f4aec04fc60 con 0x7f4b040ae240
2022-03-19T19:56:53.862 INFO:teuthology.orchestra.run.smithi053.stderr:2022-03-19T19:56:53.858+0000 7f4aeaffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}]
2022-03-19T19:56:53.862 INFO:teuthology.orchestra.run.smithi053.stderr:2022-03-19T19:56:53.858+0000 7f4aeaffd700 10 monclient: _finish_command 1 = system:0
2022-03-19T19:56:53.912 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument
2022-03-19T19:56:53.912 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards.
2022-03-19T19:56:53.925 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight
2022-03-19T19:56:53.944 INFO:tasks.ceph.osd.0.smithi053.stderr:2022-03-19T19:56:53.940+0000 ee74700 -1 received  signal: Hangup from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0  (PID: 373503) UID: 0
2022-03-19T19:56:54.039 ERROR:teuthology.run_tasks:Saw exception from tasks.