Ceph : Issues
https://tracker.ceph.com/
https://tracker.ceph.com/favicon.ico
2023-11-20T04:09:58Z
Ceph
Redmine
Dashboard - Bug #63591 (Fix Under Review): mgr/dashboard: pyyaml==6.0 installation fails with "At...
https://tracker.ceph.com/issues/63591
2023-11-20T04:09:58Z
Brad Hubbard
bhubbard@redhat.com
<a name="Description-of-problem"></a>
<h3 >Description of problem<a href="#Description-of-problem" class="wiki-anchor">¶</a></h3>
<p>pyyaml==6.0 installation fails with "AttributeError: cython_sources" and is an instance of <a class="external" href="https://github.com/yaml/pyyaml/issues/601">https://github.com/yaml/pyyaml/issues/601</a></p>
<a name="Environment"></a>
<h3 >Environment<a href="#Environment" class="wiki-anchor">¶</a></h3>
<ul>
<li><code>ceph version</code> string: v18.0.0-7432-g329e2a1e046</li>
<li>Platform (OS/distro/release): Fedora 39</li>
<li>Cluster details (nodes, monitors, OSDs): N/A</li>
<li>Did it happen on a stable environment or after a migration/upgrade?: N/A</li>
<li>Browser used (e.g.: <code>Version 86.0.4240.198 (Official Build) (64-bit)</code>): N/A</li>
</ul>
<a name="How-reproducible"></a>
<h3 >How reproducible<a href="#How-reproducible" class="wiki-anchor">¶</a></h3>
<p>100%</p>
<p>Steps:</p>
<p>Create a new fedora 39 environment and run install-deps.sh</p>
<a name="Actual-results"></a>
<h3 >Actual results<a href="#Actual-results" class="wiki-anchor">¶</a></h3>
<pre>
+ pip --timeout 300 --exists-action i wheel -w wheelhouse-wip -r requirements-alerts.txt -r requirements-grafonnet.txt -r requirements-lint.txt
Collecting pyyaml==6.0 (from -r requirements-alerts.txt (line 1))
Downloading PyYAML-6.0.tar.gz (124 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 125.0/125.0 kB 252.3 kB/s eta 0:00:00
Installing build dependencies ... done
Getting requirements to build wheel ... error
...
File "/tmp/pip-build-env-t3m5whfk/overlay/lib/python3.12/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
raise AttributeError(attr)
AttributeError: cython_sources
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
</pre>
<a name="Expected-results"></a>
<h3 >Expected results<a href="#Expected-results" class="wiki-anchor">¶</a></h3>
<p>Successful completion of install-deps.sh</p>
<a name="Additional-info"></a>
<h3 >Additional info<a href="#Additional-info" class="wiki-anchor">¶</a></h3>
<p>The following patch resolves the failure in install-deps.sh and allows it to complete but has not been validated in any other way.</p>
<pre><code class="diff syntaxhl"><span class="CodeRay"><span class="line comment">diff --git a/install-deps.sh b/install-deps.sh</span>
<span class="line comment">index d01bc6464e6..1d6ba5b8cbf 100755</span>
<span class="line head"><span class="head">--- </span><span class="filename">a/install-deps.sh</span></span>
<span class="line head"><span class="head">+++ </span><span class="filename">b/install-deps.sh</span></span>
<span class="change"><span class="change">@@</span> -304,7 +304,8 <span class="change">@@</span></span> function populate_wheelhouse() {
pip $PIP_OPTS $install \
'setuptools >= 0.8' 'pip >= 21.0' 'wheel >= 0.24' 'tox >= 2.9.1' || return 1
if test $# != 0 ; then
<span class="line delete"><span class="delete">-</span> pip $PIP_OPTS $install $@ || return 1</span>
<span class="line insert"><span class="insert">+</span> echo "cython<3" > /tmp/constraint.txt</span>
<span class="line insert"><span class="insert">+</span> PIP_CONSTRAINT=/tmp/constraint.txt pip $PIP_OPTS $install $@ || return 1</span>
fi
}
</span></code></pre>
Infrastructure - Bug #55443 (New): "SELinux denials found.." in rados run
https://tracker.ceph.com/issues/55443
2022-04-25T23:39:38Z
Yuri Weinstein
yweinste@redhat.com
<p>Run: <a class="external" href="http://pulpito.front.sepia.ceph.com/yuriw-2022-04-23_16:12:08-rados-wip-55324-pacific-backport-distro-default-smithi/">http://pulpito.front.sepia.ceph.com/yuriw-2022-04-23_16:12:08-rados-wip-55324-pacific-backport-distro-default-smithi/</a><br />Jobs: ['6803080', '6803087', '6803132', '6803104', '6803113', '6803128', '6803112', '6803134', '6803107', '6803103']<br />Logs: <a class="external" href="http://qa-proxy.ceph.com/teuthology/yuriw-2022-04-23_16:12:08-rados-wip-55324-pacific-backport-distro-default-smithi/6803080/teuthology.log">http://qa-proxy.ceph.com/teuthology/yuriw-2022-04-23_16:12:08-rados-wip-55324-pacific-backport-distro-default-smithi/6803080/teuthology.log</a></p>
<pre>
SELinux denials found on ubuntu@smithi120.front.sepia.ceph.com: ['type=AVC msg=audit(1650731309.430:200): avc: denied { node_bind } for pid=1997 comm="ping" saddr=172.21.15.120 scontext=system_u:system_r:ping_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1']
</pre>
Infrastructure - Bug #55347 (New): SELinux Denials during cephadm/workunits/test_cephadm
https://tracker.ceph.com/issues/55347
2022-04-17T16:12:28Z
Adam King
<p>Of the form</p>
<pre>
SELinux denials found on ubuntu@smithi125.front.sepia.ceph.com: ['type=AVC msg=audit(1649370825.813:7601): avc: denied { ioctl } for pid=50190 comm="iptables" path="/var/lib/containers/storage/overlay/37ab0e80f2df160e4640eb9ac25f85ce4b420ac86d66b8cf07a0bbd403b7b840/merged" dev="overlay" ino=3279249 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:container_file_t:s0:c1022,c1023 tclass=dir permissive=1']
</pre>
<p><a class="external" href="http://pulpito.ceph.com/yuriw-2022-04-07_20:11:42-orch-wip-yuri2-testing-2022-04-05-1453-pacific-distro-default-smithi/6781558">http://pulpito.ceph.com/yuriw-2022-04-07_20:11:42-orch-wip-yuri2-testing-2022-04-05-1453-pacific-distro-default-smithi/6781558</a><br /><a class="external" href="http://pulpito.ceph.com/adking-2022-04-17_04:09:51-orch:cephadm-wip-adk2-testing-2022-04-16-2212-pacific-distro-basic-smithi/6793836">http://pulpito.ceph.com/adking-2022-04-17_04:09:51-orch:cephadm-wip-adk2-testing-2022-04-16-2212-pacific-distro-basic-smithi/6793836</a><br /><a class="external" href="http://pulpito.ceph.com/adking-2022-04-17_04:09:51-orch:cephadm-wip-adk2-testing-2022-04-16-2212-pacific-distro-basic-smithi/6793880">http://pulpito.ceph.com/adking-2022-04-17_04:09:51-orch:cephadm-wip-adk2-testing-2022-04-16-2212-pacific-distro-basic-smithi/6793880</a></p>
RADOS - Bug #52319 (New): LibRadosWatchNotify.WatchNotify2 fails
https://tracker.ceph.com/issues/52319
2021-08-19T03:54:32Z
Aishwarya Mathuria
<p>2021-08-17T01:34:43.023 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2<br />2021-08-17T01:34:43.023 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: watch_notify2_test_errcb cookie 93835321637840 err -107<br />2021-08-17T01:34:43.023 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-7008-ga00b5732/rpm/el8/BUILD/ceph-17.0.0-7008-ga00b5732/src/test/librados/watch_notify.cc:218: Failure<br />2021-08-17T01:34:43.024 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: Expected: (rados_watch_check(ioctx, handle)) > (0), actual: -107 vs 0<br />2021-08-17T01:34:43.024 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: [ FAILED ] LibRadosWatchNotify.WatchNotify2 (86339 ms)<br />2021-08-17T01:34:43.024 INFO:tasks.workunit.client.0.smithi111.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2</p>
<p><a class="external" href="https://pulpito.ceph.com/yuriw-2021-08-16_21:15:00-rados-wip-yuri-testing-master-8.16.21-distro-basic-smithi/6342072">https://pulpito.ceph.com/yuriw-2021-08-16_21:15:00-rados-wip-yuri-testing-master-8.16.21-distro-basic-smithi/6342072</a></p>
teuthology - Bug #50922 (Fix Under Review): Can not schedule build on seemly successful build
https://tracker.ceph.com/issues/50922
2021-05-21T00:26:17Z
Yuri Weinstein
yweinste@redhat.com
<p>Build <a class="external" href="https://shaman.ceph.com/builds/ceph/wip-yuri7-testing-2021-05-20-0755-octopus/0259fef7bc243fd9f41c3336fe745d42b3b95352/">https://shaman.ceph.com/builds/ceph/wip-yuri7-testing-2021-05-20-0755-octopus/0259fef7bc243fd9f41c3336fe745d42b3b95352/</a></p>
<p>Only centos 8 arm64 failed</p>
<p>Command lines:</p>
<pre><code>yuriw@teuthology ~ [00:07:40]> SHA1=0259fef7bc243fd9f41c3336fe745d42b3b95352<br /> yuriw@teuthology ~ [00:07:40]> CEPH_BRANCH=wip-yuri7-testing-2021-05-20-0755-octopus<br /> yuriw@teuthology ~ [00:07:40]> CEPH_QA_MAIL="<a class="email" href="mailto:ceph-qa@ceph.io">ceph-qa@ceph.io</a>" <br /> yuriw@teuthology ~ [00:07:40]> CEPH_REPO=https://github.com/ceph/ceph-ci.git<br /> yuriw@teuthology ~ [00:07:40]> SUITE_REPO=https://github.com/ceph/ceph-ci.git<br /> yuriw@teuthology ~ [00:07:40]> LIMIT=10000<br /> yuriw@teuthology ~ [00:07:40]> DISTRO=distro<br /> yuriw@teuthology ~ [00:07:40]> TEUTH=master<br /> yuriw@teuthology ~ [00:07:40]> MACHINE_NAME=smithi<br /> yuriw@teuthology ~ [00:07:40]> PRIO=100<br /> yuriw@teuthology ~ [00:07:40]> echo $SHA1<br />0259fef7bc243fd9f41c3336fe745d42b3b95352<br /> yuriw@teuthology ~ [00:07:40]> echo $CEPH_BRANCH<br />wip-yuri7-testing-2021-05-20-0755-octopus<br /> yuriw@teuthology ~ [00:07:40]> echo $MACHINE_NAME<br />smithi</code></pre>
<pre><code>yuriw@teuthology ~ [00:25:11]> teuthology-suite -v --ceph-repo $CEPH_REPO --suite-repo $CEPH_REPO -c $CEPH_BRANCH -m $MACHINE_NAME -s rgw -k $DISTRO -p $PRIO -e $CEPH_QA_MAIL --suite-branch $CEPH_BRANCH -l $LIMIT -S $SHA1 --force-priority --subset 111/120000</code></pre>
<pre>
2021-05-21 00:25:40,759.759 INFO:teuthology.suite:Using random seed=674
2021-05-21 00:25:40,761.761 INFO:teuthology.suite.run:kernel sha1: distro
2021-05-21 00:25:41,113.113 INFO:teuthology.suite.run:ceph sha1 explicitly supplied
2021-05-21 00:25:41,113.113 INFO:teuthology.suite.run:ceph sha1: 0259fef7bc243fd9f41c3336fe745d42b3b95352
2021-05-21 00:25:41,113.113 DEBUG:teuthology.suite.util:Defaults for machine_type smithi distro centos: arch=x86_64, release=centos/7, pkg_type=rpm
2021-05-21 00:25:41,113.113 INFO:teuthology.suite.util:container build centos/8, checking for build_complete
2021-05-21 00:25:41,114.114 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=0259fef7bc243fd9f41c3336fe745d42b3b95352
2021-05-21 00:25:41,427.427 INFO:teuthology.suite.util:build not complete
2021-05-21 00:25:41,427.427 INFO:teuthology.suite.run:ceph version: None
2021-05-21 00:25:41,616.616 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci.git wip-yuri7-testing-2021-05-20-0755-octopus -> 0259fef7bc243fd9f41c3336fe745d42b3b95352
2021-05-21 00:25:41,811.811 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci.git wip-yuri7-testing-2021-05-20-0755-octopus -> 0259fef7bc243fd9f41c3336fe745d42b3b95352
2021-05-21 00:25:41,812.812 INFO:teuthology.suite.run:ceph-ci branch: wip-yuri7-testing-2021-05-20-0755-octopus 0259fef7bc243fd9f41c3336fe745d42b3b95352
2021-05-21 00:25:41,819.819 DEBUG:teuthology.repo_utils:Setting repo remote to https://github.com/ceph/ceph-ci.git
2021-05-21 00:25:41,830.830 INFO:teuthology.repo_utils:Fetching wip-yuri7-testing-2021-05-20-0755-octopus from origin
2021-05-21 00:25:42,284.284 INFO:teuthology.repo_utils:Resetting repo at /home/yuriw/src/github.com_ceph_ceph-c_wip-yuri7-testing-2021-05-20-0755-octopus to origin/wip-yuri7-testing-2021-05-20-0755-octopus
2021-05-21 00:25:43,191.191 DEBUG:teuthology.suite.run:Check file /home/yuriw/src/github.com_ceph_ceph-c_wip-yuri7-testing-2021-05-20-0755-octopus/qa/.teuthology_branch exists
2021-05-21 00:25:43,192.192 DEBUG:teuthology.suite.run:Found teuthology branch config file /home/yuriw/src/github.com_ceph_ceph-c_wip-yuri7-testing-2021-05-20-0755-octopus/qa/.teuthology_branch
2021-05-21 00:25:43,193.193 DEBUG:teuthology.suite.run:The teuthology branch is overridden with master
2021-05-21 00:25:43,373.373 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/teuthology master -> 471a2f1efab907807e62c17be86bf6fb1a2f4522
2021-05-21 00:25:43,373.373 INFO:teuthology.suite.run:teuthology branch: master 471a2f1efab907807e62c17be86bf6fb1a2f4522
2021-05-21 00:25:43,386.386 DEBUG:teuthology.suite.run:Suite rgw in /home/yuriw/src/github.com_ceph_ceph-c_wip-yuri7-testing-2021-05-20-0755-octopus/qa/suites/rgw
2021-05-21 00:25:43,387.387 INFO:teuthology.suite.build_matrix:Subset=111/120000
2021-05-21 00:25:50,188.188 INFO:teuthology.suite.run:Suite rgw in /home/yuriw/src/github.com_ceph_ceph-c_wip-yuri7-testing-2021-05-20-0755-octopus/qa/suites/rgw generated 39 jobs (not yet filtered)
2021-05-21 00:25:50,213.213 DEBUG:teuthology.suite.util:Defaults for machine_type smithi distro centos: arch=x86_64, release=centos/7, pkg_type=rpm
2021-05-21 00:25:50,213.213 INFO:teuthology.suite.util:container build centos/8, checking for build_complete
2021-05-21 00:25:50,214.214 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=0259fef7bc243fd9f41c3336fe745d42b3b95352
2021-05-21 00:25:50,513.513 INFO:teuthology.suite.util:build not complete
2021-05-21 00:25:50,513.513 ERROR:teuthology.suite.run:Packages for os_type 'None', flavor basic and ceph hash '0259fef7bc243fd9f41c3336fe745d42b3b95352' not found
2021-05-21 00:25:50,521.521 ERROR:teuthology.suite.run:Packages for os_type 'None', flavor basic and ceph hash '0259fef7bc243fd9f41c3336fe745d42b3b95352' not found
2021-05-21 00:25:50,538.538 DEBUG:teuthology.suite.util:Defaults for machine_type smithi distro ubuntu: arch=x86_64, release=ubuntu/16.04, pkg_type=deb
2021-05-21 00:25:50,538.538 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=0259fef7bc243fd9f41c3336fe745d42b3b95352
2021-05-21 00:25:50,725.725 DEBUG:teuthology.suite.util:Defaults for machine_type smithi distro centos: arch=x86_64, release=centos/7, pkg_type=rpm
2021-05-21 00:25:50,725.725 INFO:teuthology.suite.util:container build centos/8, checking for build_complete
2021-05-21 00:25:50,726.726 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=0259fef7bc243fd9f41c3336fe745d42b3b95352
2021-05-21 00:25:51,024.024 INFO:teuthology.suite.util:build not complete
2021-05-21 00:25:51,024.024 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor basic and ceph hash '0259fef7bc243fd9f41c3336fe745d42b3b95352' not found
2021-05-21 00:25:51,039.039 DEBUG:teuthology.suite.util:Defaults for machine_type smithi distro ubuntu: arch=x86_64, release=ubuntu/16.04, pkg_type=deb
2021-05-21 00:25:51,039.039 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=0259fef7bc243fd9f41c3336fe745d42b3b95352
2021-05-21 00:25:51,207.207 DEBUG:teuthology.suite.util:Defaults for machine_type smithi distro centos: arch=x86_64, release=centos/7, pkg_type=rpm
2021-05-21 00:25:51,207.207 INFO:teuthology.suite.util:container build centos/8, checking for build_complete
2021-05-21 00:25:51,207.207 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=0259fef7bc243fd9f41c3336fe745d42b3b95352
2021-05-21 00:25:51,507.507 INFO:teuthology.suite.util:build not complete
2021-05-21 00:25:51,507.507 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor basic and ceph hash '0259fef7bc243fd9f41c3336fe745d42b3b95352' not found
2021-05-21 00:25:51,530.530 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor basic and ceph hash '0259fef7bc243fd9f41c3336fe745d42b3b95352' not found
2021-05-21 00:25:51,566.566 ERROR:teuthology.suite.run:Packages for os_type 'None', flavor basic and ceph hash '0259fef7bc243fd9f41c3336fe745d42b3b95352' not found
2021-05-21 00:25:51,573.573 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor basic and ceph hash '0259fef7bc243fd9f41c3336fe745d42b3b95352' not found
2021-05-21 00:25:51,581.581 ERROR:teuthology.suite.run:Packages for os_type 'None', flavor basic and ceph hash '0259fef7bc243fd9f41c3336fe745d42b3b95352' not found
2021-05-21 00:25:51,611.611 ERROR:teuthology.suite.run:Packages for os_type 'None', flavor basic and ceph hash '0259fef7bc243fd9f41c3336fe745d42b3b95352' not found
2021-05-21 00:25:51,637.637 DEBUG:teuthology.suite.util:Defaults for machine_type smithi distro centos: arch=x86_64, release=centos/7, pkg_type=rpm
2021-05-21 00:25:51,637.637 INFO:teuthology.suite.util:container build centos/8, checking for build_complete
2021-05-21 00:25:51,637.637 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=0259fef7bc243fd9f41c3336fe745d42b3b95352
2021-05-21 00:25:51,941.941 INFO:teuthology.suite.util:build not complete
2021-05-21 00:25:51,941.941 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor basic and ceph hash '0259fef7bc243fd9f41c3336fe745d42b3b95352' not found
2021-05-21 00:25:51,955.955 ERROR:teuthology.suite.run:Packages for os_type 'None', flavor basic and ceph hash '0259fef7bc243fd9f41c3336fe745d42b3b95352' not found
2021-05-21 00:25:51,966.966 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor basic and ceph hash '0259fef7bc243fd9f41c3336fe745d42b3b95352' not found
2021-05-21 00:25:51,986.986 ERROR:teuthology.suite.run:Packages for os_type 'None', flavor basic and ceph hash '0259fef7bc243fd9f41c3336fe745d42b3b95352' not found
2021-05-21 00:25:51,992.992 ERROR:teuthology.suite.run:Packages for os_type 'None', flavor basic and ceph hash '0259fef7bc243fd9f41c3336fe745d42b3b95352' not found
2021-05-21 00:25:52,023.023 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor basic and ceph hash '0259fef7bc243fd9f41c3336fe745d42b3b95352' not found
2021-05-21 00:25:52,060.060 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor basic and ceph hash '0259fef7bc243fd9f41c3336fe745d42b3b95352' not found
2021-05-21 00:25:52,122.122 ERROR:teuthology.suite.run:Packages for os_type 'None', flavor basic and ceph hash '0259fef7bc243fd9f41c3336fe745d42b3b95352' not found
2021-05-21 00:25:52,146.146 ERROR:teuthology.suite.run:Packages for os_type 'None', flavor basic and ceph hash '0259fef7bc243fd9f41c3336fe745d42b3b95352' not found
2021-05-21 00:25:52,170.170 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor basic and ceph hash '0259fef7bc243fd9f41c3336fe745d42b3b95352' not found
2021-05-21 00:25:52,186.186 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor basic and ceph hash '0259fef7bc243fd9f41c3336fe745d42b3b95352' not found
Job scheduled with name yuriw-2021-05-21_00:25:40-rgw-wip-yuri7-testing-2021-05-20-0755-octopus-distro-basic-smithi and ID 6125641
2021-05-21 00:25:53,095.095 INFO:teuthology.suite.run:Scheduling rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests}}
Traceback (most recent call last):
File "/home/yuriw/teuthology/virtualenv/bin/teuthology-suite", line 33, in <module>
sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology-suite')())
File "/home/yuriw/teuthology/scripts/suite.py", line 189, in main
return teuthology.suite.main(args)
File "/home/yuriw/teuthology/teuthology/suite/__init__.py", line 143, in main
run.prepare_and_schedule()
File "/home/yuriw/teuthology/teuthology/suite/run.py", line 397, in prepare_and_schedule
num_jobs = self.schedule_suite()
File "/home/yuriw/teuthology/teuthology/suite/run.py", line 644, in schedule_suite
self.schedule_jobs(jobs_missing_packages, jobs_to_schedule, name)
File "/home/yuriw/teuthology/teuthology/suite/run.py", line 496, in schedule_jobs
name,
File "/home/yuriw/teuthology/teuthology/suite/util.py", line 76, in schedule_fail
raise ScheduleFailError(message, name)
teuthology.exceptions.ScheduleFailError: Scheduling yuriw-2021-05-21_00:25:40-rgw-wip-yuri7-testing-2021-05-20-0755-octopus-distro-basic-smithi failed: At least one job needs packages that don't exist for hash 0259fef7bc243fd9f41c3336fe745d42b3b95352.
</pre>
RADOS - Bug #50371 (New): Segmentation fault (core dumped) ceph_test_rados_api_watch_notify_pp
https://tracker.ceph.com/issues/50371
2021-04-15T03:43:50Z
Brad Hubbard
bhubbard@redhat.com
<p>/a/nojha-2021-04-14_00:54:53-rados-master-distro-basic-smithi/6044164</p>
<pre>
2021-04-14T01:40:16.439 INFO:tasks.workunit.client.0.smithi079.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1
2021-04-14T01:40:16.477 INFO:tasks.workunit.client.0.smithi079.stderr:bash: line 1: 36189 Segmentation fault (core dumped) ceph_test_rados_api_watch_notify_pp 2>&1
2021-04-14T01:40:16.477 INFO:tasks.workunit.client.0.smithi079.stderr: 36190 Done | tee ceph_test_rados_api_watch_notify_pp.log
2021-04-14T01:40:16.477 INFO:tasks.workunit.client.0.smithi079.stderr: 36192 Done | sed "s/^/ api_watch_notify_pp: /"
</pre>
RADOS - Bug #49809 (Need More Info): 1 out of 3 mon crashed in MonitorDBStore::get_synchronizer
https://tracker.ceph.com/issues/49809
2021-03-15T17:16:31Z
Christian Rohmann
<p>We experienced a single mon crash (out of 3 mons) - We observed no other issues on the machine or the cluster.</p>
<p>I attached the ceph-mon.log to this ticket.</p>
<p>ceph crash info:</p>
<pre>
{
"os_version_id": "18.04",
"utsname_release": "4.15.0-136-generic",
"os_name": "Ubuntu",
"entity_name": "mon.ctrl-01",
"timestamp": "2021-03-14 18:00:28.686018Z",
"process_name": "ceph-mon",
"utsname_machine": "x86_64",
"utsname_sysname": "Linux",
"os_version": "18.04.5 LTS (Bionic Beaver)",
"os_id": "ubuntu",
"utsname_version": "#140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021",
"backtrace": [
"(()+0x12980) [0x7f1c0cfbe980]",
"(MonitorDBStore::get_synchronizer(std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >&, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&)+0x3f) [0x555e110c756f]",
"(Monitor::_scrub(ScrubResult*, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >*, int*)+0xe9) [0x555e11089379]",
"(Monitor::scrub()+0x298) [0x555e11092658]",
"(Monitor::scrub_start()+0x137) [0x555e11092847]",
"(C_MonContext::finish(int)+0x39) [0x555e1106a899]",
"(Context::complete(int)+0x9) [0x555e110adc69]",
"(SafeTimer::timer_thread()+0x190) [0x7f1c0e1dbd20]",
"(SafeTimerThread::entry()+0xd) [0x7f1c0e1dd5ed]",
"(()+0x76db) [0x7f1c0cfb36db]",
"(clone()+0x3f) [0x7f1c0c19971f]"
],
"utsname_hostname": "ctrl-01",
"crash_id": "2021-03-14_18:00:28.686018Z_274d6160-d49e-4395-9d8c-64d4dcad53f5",
"ceph_version": "14.2.16"
}
</pre>
<p>ceph.log reported:</p>
<pre>
2021-03-14 18:00:00.000161 mon.ctrl-01 (mon.0) 224915 : cluster [INF] overall HEALTH_OK
2021-03-14 18:00:27.513970 mon.ctrl-03 (mon.2) 146401 : cluster [INF] mon.ctrl-03 calling monitor election
2021-03-14 18:00:27.516169 mon.ctrl-02 (mon.1) 122690 : cluster [INF] mon.ctrl-02 calling monitor election
2021-03-14 18:00:39.412761 mon.ctrl-02 (mon.1) 122691 : cluster [INF] mon.ctrl-02 is new leader, mons ctrl-02,ctrl-03 in quorum (ranks 1,2)
2021-03-14 18:00:39.423504 mon.ctrl-02 (mon.1) 122696 : cluster [WRN] Health check failed: 1/3 mons down, quorum ctrl-02,ctrl-03 (MON_DOWN)
2021-03-14 18:00:39.431667 mon.ctrl-02 (mon.1) 122697 : cluster [WRN] overall HEALTH_WARN 1/3 mons down, quorum ctrl-02,ctrl-03
2021-03-14 18:00:40.733430 mon.ctrl-01 (mon.0) 1 : cluster [INF] mon.ctrl-01 calling monitor election
2021-03-14 18:00:40.742270 mon.ctrl-01 (mon.0) 2 : cluster [INF] mon.ctrl-01 calling monitor election
2021-03-14 18:00:45.745709 mon.ctrl-01 (mon.0) 3 : cluster [INF] mon.ctrl-01 is new leader, mons ctrl-01,ctrl-03 in quorum (ranks 0,2)
2021-03-14 18:00:51.930256 mon.ctrl-02 (mon.1) 122710 : cluster [INF] mon.ctrl-02 calling monitor election
2021-03-14 18:00:51.934337 mon.ctrl-01 (mon.0) 4 : cluster [WRN] overall HEALTH_WARN 1/3 mons down, quorum ctrl-02,ctrl-03
2021-03-14 18:00:51.934422 mon.ctrl-01 (mon.0) 5 : cluster [INF] mon.ctrl-01 calling monitor election
2021-03-14 18:00:51.960175 mon.ctrl-01 (mon.0) 6 : cluster [INF] mon.ctrl-01 is new leader, mons ctrl-01,ctrl-02,ctrl-03 in quorum (ranks 0,1,2)
2021-03-14 18:00:51.970642 mon.ctrl-01 (mon.0) 11 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum ctrl-02,ctrl-03)
2021-03-14 18:00:51.970670 mon.ctrl-01 (mon.0) 12 : cluster [INF] Cluster is now healthy
2021-03-14 18:00:51.979150 mon.ctrl-01 (mon.0) 13 : cluster [INF] overall HEALTH_OK
2021-03-14 18:03:47.551678 mon.ctrl-01 (mon.0) 49 : cluster [WRN] Health check failed: 1 daemons have recently crashed (RECENT_CRASH)
</pre>
Ceph - Bug #47932 (In Progress): perf_counter_data_any_d.u64 can wrap
https://tracker.ceph.com/issues/47932
2020-10-22T03:48:16Z
Brad Hubbard
bhubbard@redhat.com
<pre>
$ cat ceph_mgr.perf.dump.2020-10-21_20-48-16
{
"AsyncMessenger::Worker-0": {
"msgr_recv_messages": 40543424,
"msgr_send_messages": 247898,
"msgr_recv_bytes": 808394289834,
"msgr_send_bytes": 53012008,
"msgr_created_connections": 643631,
"msgr_active_connections": 18446744073709547145, <--- ***
"msgr_running_total_time": 57373.378688153,
"msgr_running_send_time": 15.301807783,
"msgr_running_recv_time": 31548.281129471,
"msgr_running_fast_dispatch_time": 0.000063072
},
</pre>
RADOS - Bug #47025 (Fix Under Review): rados/test.sh: api_watch_notify_pp LibRadosWatchNotifyECPP...
https://tracker.ceph.com/issues/47025
2020-08-19T00:36:57Z
Neha Ojha
nojha@redhat.com
<pre>
2020-08-14T23:23:58.787 INFO:tasks.workunit.client.0.smithi112.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify
2020-08-14T23:23:58.788 INFO:tasks.workunit.client.0.smithi112.stdout: api_watch_notify_pp: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.0.0-4375-g4fea9a8be3/rpm/el8/BUILD/ceph-16.0.0-4375-g4fea9a8be3/src/test/librados/watch_notify_cxx.cc:124: Failure
2020-08-14T23:23:58.788 INFO:tasks.workunit.client.0.smithi112.stdout: api_watch_notify_pp: Expected equality of these values:
2020-08-14T23:23:58.788 INFO:tasks.workunit.client.0.smithi112.stdout: api_watch_notify_pp: 1u
2020-08-14T23:23:58.788 INFO:tasks.workunit.client.0.smithi112.stdout: api_watch_notify_pp: Which is: 1
2020-08-14T23:23:58.789 INFO:tasks.workunit.client.0.smithi112.stdout: api_watch_notify_pp: watches.size()
2020-08-14T23:23:58.789 INFO:tasks.workunit.client.0.smithi112.stdout: api_watch_notify_pp: Which is: 0
2020-08-14T23:23:58.789 INFO:tasks.workunit.client.0.smithi112.stdout: api_watch_notify_pp: [ FAILED ] LibRadosWatchNotifyECPP.WatchNotify (159389 ms)
</pre>
<p>/a/teuthology-2020-08-14_07:01:01-rados-master-distro-basic-smithi/5346903</p>
Ceph - Bug #45902 (New): thrashosds hits watchdog_daemon_timeout during powercycle
https://tracker.ceph.com/issues/45902
2020-06-05T01:22:13Z
Brad Hubbard
bhubbard@redhat.com
<p>/a/teuthology-2020-06-02_11:15:03-powercycle-master-testing-basic-smithi/5111856</p>
<p>The run fails with "teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/ffsb.sh) on smithi116 with status 1" but earlier we see the following.</p>
<pre>
2020-06-04T02:30:30.708 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~303s
2020-06-04T02:30:30.709 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons
</pre>
<p>The last entry in osd.1s log is it being told to shut down at 2020-06-04T02:25:24</p>
<pre>
2020-06-04T02:25:24.909+0000 7f5f38726700 -1 osd.1 31 *** Got signal Terminated ***
2020-06-04T02:25:24.909+0000 7f5f38726700 -1 osd.1 31 *** Immediate shutdown (osd_fast_shutdown=true) ***
</pre>
<p>It looks like it was shutdown due to a power cycle on smithi036.</p>
<pre>
2020-06-04T02:25:20.803 INFO:tasks.ceph.ceph_manager.ceph:kill_osd on osd.1 doing powercycle of ubuntu@smithi036.front.sepia.ceph.com
</pre>
<p>Power on completed at 02:29:02.</p>
<pre>
2020-06-04T02:29:02.878 INFO:teuthology.orchestra.console:Power on for smithi036 completed
</pre>
<p>But it's not until 02:30:43 that we try to start osd.1 and by then we've already timed out.</p>
<pre>
2020-06-04T02:30:43.072 DEBUG:teuthology.orchestra.console:expect after: b'smithi036 login: '
2020-06-04T02:30:43.226 INFO:teuthology.misc:Re-opening connections...
2020-06-04T02:30:43.226 INFO:teuthology.misc:trying to connect to ubuntu@smithi036.front.sepia.ceph.com
2020-06-04T02:30:43.227 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2020-06-04T02:30:43.228 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi036.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2020-06-04T02:30:43.676 INFO:teuthology.orchestra.run.smithi036:> true
2020-06-04T02:30:44.245 DEBUG:teuthology.misc:waited 1.0186738967895508
2020-06-04T02:30:45.247 DEBUG:tasks.ceph_manager:Mounting data for osd.1 on ubuntu@smithi036.front.sepia.ceph.com
</pre>
<p>I think we should adjust the watchdog_daemon_timeout for the powercycle tests.</p>
Ceph - Bug #45900 (New): "ERROR: (22) Invalid argument" in powercycle
https://tracker.ceph.com/issues/45900
2020-06-04T18:50:44Z
Yuri Weinstein
yweinste@redhat.com
<p>Run: <a class="external" href="http://pulpito.ceph.com/teuthology-2020-06-02_11:15:03-powercycle-master-testing-basic-smithi/">http://pulpito.ceph.com/teuthology-2020-06-02_11:15:03-powercycle-master-testing-basic-smithi/</a><br />Job: 511856<br />Logs: /a/teuthology-2020-06-02_11:15:03-powercycle-master-testing-basic-smithi/5111856/teuthology.log</p>
<pre>
2020-06-04T02:30:30.240 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2: {}
2020-06-04T02:30:30.240 INFO:teuthology.orchestra.run.smithi116.stderr:osd.0: osd_enable_op_tracker = 'false'
2020-06-04T02:30:30.241 INFO:teuthology.orchestra.run.smithi116.stderr:Error ENXIO: problem getting command descriptions from osd.1
2020-06-04T02:30:30.241 INFO:teuthology.orchestra.run.smithi116.stderr:osd.1: problem getting command descriptions from osd.1
2020-06-04T02:30:30.241 INFO:teuthology.orchestra.run.smithi116.stderr:osd.2: osd_enable_op_tracker = 'false'
2020-06-04T02:30:30.243 INFO:tasks.ceph.osd.2.smithi003.stderr:2020-06-04T02:30:30.241+0000 7fa068b4a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22070) UID: 0
2020-06-04T02:30:30.251 DEBUG:teuthology.orchestra.run:got remote process result: 6
2020-06-04T02:30:30.251 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring
2020-06-04T02:30:30.344 INFO:tasks.ceph.osd.2.smithi003.stderr:2020-06-04T02:30:30.341+0000 7fa068b4a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22070) UID: 0
2020-06-04T02:30:30.445 INFO:tasks.ceph.osd.0.smithi005.stderr:2020-06-04T02:30:30.443+0000 7f5832534700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21975) UID: 0
2020-06-04T02:30:30.471 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument
2020-06-04T02:30:30.471 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards.
2020-06-04T02:30:30.485 INFO:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight
2020-06-04T02:30:30.547 INFO:tasks.ceph.osd.2.smithi003.stderr:2020-06-04T02:30:30.544+0000 7fa068b4a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22070) UID: 0
2020-06-04T02:30:30.648 INFO:tasks.ceph.osd.2.smithi003.stderr:2020-06-04T02:30:30.645+0000 7fa068b4a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22070) UID: 0
2020-06-04T02:30:30.708 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~303s
2020-06-04T02:30:30.709 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons
2020-06-04T02:30:30.709 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi116.front.sepia.ceph.com...
2020-06-04T02:30:30.710 INFO:teuthology.orchestra.run:Running command with timeout 1800
2020-06-04T02:30:30.710 INFO:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0)
2020-06-04T02:30:30.735 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument
2020-06-04T02:30:30.736 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards.
2020-06-04T02:30:30.746 INFO:tasks.workunit.client.0.smithi116.stdout:Wrote -1 instead of 4096 bytes.
2020-06-04T02:30:30.747 INFO:tasks.workunit.client.0.smithi116.stdout:Probably out of disk space
2020-06-04T02:30:30.747 INFO:tasks.workunit.client.0.smithi116.stderr:write: Transport endpoint is not connected
2020-06-04T02:30:30.748 DEBUG:teuthology.orchestra.run:got remote process result: 1
2020-06-04T02:30:30.749 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.0...
2020-06-04T02:30:30.749 INFO:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0
2020-06-04T02:30:30.750 INFO:tasks.ceph.osd.0.smithi005.stderr:2020-06-04T02:30:30.746+0000 7f5832534700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 21975) UID: 0
2020-06-04T02:30:30.751 INFO:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops
2020-06-04T02:30:30.850 INFO:tasks.ceph.osd.2.smithi003.stderr:2020-06-04T02:30:30.847+0000 7fa068b4a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22070) UID: 0
2020-06-04T02:30:30.950 INFO:tasks.ceph.osd.2.smithi003.stderr:2020-06-04T02:30:30.948+0000 7fa068b4a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22070) UID: 0
2020-06-04T02:30:30.988 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument
2020-06-04T02:30:30.989 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards.
2020-06-04T02:30:31.003 INFO:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops
2020-06-04T02:30:31.045 INFO:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd unset nodeep-scrub
2020-06-04T02:30:31.051 INFO:tasks.ceph.osd.2.smithi003.stderr:2020-06-04T02:30:31.048+0000 7fa068b4a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22070) UID: 0
2020-06-04T02:30:31.086 INFO:teuthology.orchestra.run:waiting for 900
2020-06-04T02:30:31.087 INFO:teuthology.orchestra.run:Running command with timeout 300
2020-06-04T02:30:31.087 INFO:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0)
2020-06-04T02:30:31.156 INFO:tasks.ceph.osd.2.smithi003.stderr:2020-06-04T02:30:31.151+0000 7fa068b4a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22070) UID: 0
2020-06-04T02:30:31.227 INFO:tasks.cephfs.mount:Cleaning the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0'
2020-06-04T02:30:31.228 INFO:tasks.cephfs.mount:Removing the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0'
2020-06-04T02:30:31.228 INFO:teuthology.orchestra.run:Running command with timeout 300
2020-06-04T02:30:31.228 INFO:teuthology.orchestra.run.smithi116:> sudo bash -c 'ip link set brx.0 down'
2020-06-04T02:30:31.237 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument
2020-06-04T02:30:31.238 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards.
2020-06-04T02:30:31.253 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight
2020-06-04T02:30:31.256 INFO:tasks.ceph.osd.2.smithi003.stderr:2020-06-04T02:30:31.253+0000 7fa068b4a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22070) UID: 0
2020-06-04T02:30:31.339 INFO:teuthology.orchestra.run:Running command with timeout 300
2020-06-04T02:30:31.340 INFO:teuthology.orchestra.run.smithi116:> sudo bash -c 'ip link delete brx.0'
2020-06-04T02:30:31.355 INFO:tasks.ceph.osd.2.smithi003.stderr:2020-06-04T02:30:31.353+0000 7fa068b4a700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 22070) UID: 0
2020-06-04T02:30:31.428 ERROR:teuthology.run_tasks:Saw exception from tasks.
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 91, in run_tasks
manager = run_one_task(taskname, ctx=ctx, config=config)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 70, in run_one_task
return task(**kwargs)
File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/workunit.py", line 140, in task
cleanup=cleanup)
File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/workunit.py", line 290, in _spawn_on_all_clients
timeout=timeout)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 87, in __exit__
for result in self:
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 101, in __next__
resurrect_traceback(result)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 37, in resurrect_traceback
reraise(*exc_info)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/six.py", line 703, in reraise
raise value
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 24, in capture_traceback
return func(*args, **kwargs)
File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/workunit.py", line 415, in _run_tests
label="workunit test {workunit}".format(workunit=workunit)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 206, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 475, in run
r.wait()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 162, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 184, in _raise_for_status
node=self.hostname, label=self.label
teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/ffsb.sh) on smithi116 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=cca6533da2dbb756769bf3640b19705a1d0ea1fa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh'
2020-06-04T02:30:31.490 ERROR:teuthology.run_tasks: Sentry event: http://sentry.ceph.com/sepia/teuthology/?q=4fdfbaaaade6448d8cf112d6741b265f
</pre>
Ceph - Bug #44399 (Triaged): qa: /usr/libexec/platform-python fails / core dumps
https://tracker.ceph.com/issues/44399
2020-03-03T18:50:59Z
Patrick Donnelly
pdonnell@redhat.com
<pre>
2020-03-03T16:28:22.300 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive
2020-03-03T16:28:22.320 INFO:teuthology.task.internal:Transferring archived files...
2020-03-03T16:28:22.320 DEBUG:teuthology.misc:Transferring archived files from smithi063:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2020-03-03_14:22:02-kcephfs-master-distro-basic-smithi/4820744/remote/smithi063
2020-03-03T16:28:22.320 INFO:teuthology.orchestra.run.smithi063:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- .
2020-03-03T16:28:24.154 INFO:teuthology.task.internal:Transferring binaries for coredumps...
2020-03-03T16:28:24.193 INFO:teuthology.orchestra.run.smithi063:> which /usr/libexec/platform-python
2020-03-03T16:28:24.232 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/libexec/platform-python
2020-03-03T16:28:24.289 DEBUG:teuthology.orchestra.remote:smithi063:/usr/libexec/platform-python is 12KB
2020-03-03T16:28:24.335 ERROR:teuthology.run_tasks:Manager failed: internal.archive
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 159, in run_tasks
suppress = manager.__exit__(*exc_info)
File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
self.gen.next()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 363, in archive
fetch_binaries_for_coredumps(path, rem)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 318, in fetch_binaries_for_coredumps
dump_program))
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 320, in _sftp_get_file
sftp.get(remote_path, local_path)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 801, in get
with open(localpath, "wb") as fl:
IOError: [Errno 13] Permission denied: '/usr/libexec/platform-python'
2020-03-03T16:28:24.346 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload
2020-03-03T16:28:24.360 INFO:teuthology.task.internal:Not uploading archives.
2020-03-03T16:28:24.360 DEBUG:teuthology.run_tasks:Unwinding manager internal.base
2020-03-03T16:28:24.374 INFO:teuthology.task.internal:Tidying up after the test...
2020-03-03T16:28:24.374 INFO:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest
2020-03-03T16:28:24.381 INFO:teuthology.orchestra.run.smithi063:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest
2020-03-03T16:28:24.384 INFO:teuthology.orchestra.run.smithi099:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest
2020-03-03T16:28:24.396 INFO:teuthology.orchestra.run.smithi063.stdout: 262149 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 3 16:26 /home/ubuntu/cephtest
2020-03-03T16:28:24.396 INFO:teuthology.orchestra.run.smithi063.stdout: 262150 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 3 16:28 /home/ubuntu/cephtest/archive
2020-03-03T16:28:24.396 INFO:teuthology.orchestra.run.smithi063.stdout: 262170 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Mar 3 16:28 /home/ubuntu/cephtest/archive/audit
2020-03-03T16:28:24.396 INFO:teuthology.orchestra.run.smithi063.stdout: 262174 92 -rw------- 1 ubuntu ubuntu 91917 Mar 3 16:28 /home/ubuntu/cephtest/archive/audit/audit.log.gz
2020-03-03T16:28:24.396 INFO:teuthology.orchestra.run.smithi063.stdout: 262152 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 3 16:28 /home/ubuntu/cephtest/archive/syslog
2020-03-03T16:28:24.397 INFO:teuthology.orchestra.run.smithi063.stdout: 262154 4 -rw-rw-rw- 1 ubuntu ubuntu 678 Mar 3 16:19 /home/ubuntu/cephtest/archive/syslog/kern.log.gz
2020-03-03T16:28:24.397 INFO:teuthology.orchestra.run.smithi063.stdout: 262172 68 -rw-rw-rw- 1 ubuntu ubuntu 65894 Mar 3 16:28 /home/ubuntu/cephtest/archive/syslog/misc.log.gz
2020-03-03T16:28:24.397 INFO:teuthology.orchestra.run.smithi063.stdout: 262151 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 3 15:54 /home/ubuntu/cephtest/archive/coredump
2020-03-03T16:28:24.397 INFO:teuthology.orchestra.run.smithi063.stdout: 262159 47596 -rw------- 1 root root 57581568 Mar 3 15:54 /home/ubuntu/cephtest/archive/coredump/1583250867.14575.core
2020-03-03T16:28:24.398 INFO:teuthology.orchestra.run.smithi063.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty
2020-03-03T16:28:24.401 INFO:teuthology.orchestra.run.smithi099.stdout: 262153 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 3 16:26 /home/ubuntu/cephtest
2020-03-03T16:28:24.401 INFO:teuthology.orchestra.run.smithi099.stdout: 262154 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 3 16:28 /home/ubuntu/cephtest/archive
2020-03-03T16:28:24.401 INFO:teuthology.orchestra.run.smithi099.stdout: 262170 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Mar 3 16:28 /home/ubuntu/cephtest/archive/audit
2020-03-03T16:28:24.402 INFO:teuthology.orchestra.run.smithi099.stdout: 262174 100 -rw------- 1 ubuntu ubuntu 100166 Mar 3 16:28 /home/ubuntu/cephtest/archive/audit/audit.log.gz
2020-03-03T16:28:24.402 INFO:teuthology.orchestra.run.smithi099.stdout: 262156 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 3 16:28 /home/ubuntu/cephtest/archive/syslog
2020-03-03T16:28:24.402 INFO:teuthology.orchestra.run.smithi099.stdout: 262158 4 -rw-rw-rw- 1 ubuntu ubuntu 686 Mar 3 16:19 /home/ubuntu/cephtest/archive/syslog/kern.log.gz
2020-03-03T16:28:24.402 INFO:teuthology.orchestra.run.smithi099.stdout: 262172 72 -rw-rw-rw- 1 ubuntu ubuntu 69870 Mar 3 16:28 /home/ubuntu/cephtest/archive/syslog/misc.log.gz
2020-03-03T16:28:24.404 INFO:teuthology.orchestra.run.smithi099.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty
2020-03-03T16:28:24.407 INFO:teuthology.orchestra.run.smithi040.stdout: 393248 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 3 16:26 /home/ubuntu/cephtest
2020-03-03T16:28:24.407 INFO:teuthology.orchestra.run.smithi040.stdout: 393249 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 3 16:28 /home/ubuntu/cephtest/archive
2020-03-03T16:28:24.407 INFO:teuthology.orchestra.run.smithi040.stdout: 524332 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Mar 3 16:28 /home/ubuntu/cephtest/archive/audit
2020-03-03T16:28:24.407 INFO:teuthology.orchestra.run.smithi040.stdout: 524609 92 -rw------- 1 ubuntu ubuntu 90692 Mar 3 16:28 /home/ubuntu/cephtest/archive/audit/audit.log.gz
2020-03-03T16:28:24.408 INFO:teuthology.orchestra.run.smithi040.stdout: 393251 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 3 16:28 /home/ubuntu/cephtest/archive/syslog
2020-03-03T16:28:24.408 INFO:teuthology.orchestra.run.smithi040.stdout: 393237 4 -rw-rw-rw- 1 ubuntu ubuntu 2921 Mar 3 16:02 /home/ubuntu/cephtest/archive/syslog/kern.log.gz
2020-03-03T16:28:24.408 INFO:teuthology.orchestra.run.smithi040.stdout: 393236 68 -rw-rw-rw- 1 ubuntu ubuntu 66499 Mar 3 16:28 /home/ubuntu/cephtest/archive/syslog/misc.log.gz
2020-03-03T16:28:24.409 INFO:teuthology.orchestra.run.smithi040.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty
2020-03-03T16:28:24.409 DEBUG:teuthology.orchestra.run:got remote process result: 1
2020-03-03T16:28:24.409 ERROR:teuthology.run_tasks:Manager failed: internal.base
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 159, in run_tasks
suppress = manager.__exit__(*exc_info)
File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
self.gen.throw(type, value, traceback)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 49, in base
wait=False,
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 457, in wait
proc.wait()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 158, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 180, in _raise_for_status
node=self.hostname, label=self.label
CommandFailedError: Command failed on smithi040 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest'
2020-03-03T16:28:24.411 DEBUG:teuthology.run_tasks:Unwinding manager console_log
2020-03-03T16:28:24.427 DEBUG:teuthology.run_tasks:Unwinding manager internal.lock_machines
2020-03-03T16:28:24.443 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: IOError: [Errno 13] Permission denied: '/usr/libexec/platform-python'
</pre>
<p>From: /ceph/teuthology-archive/pdonnell-2020-03-03_14:22:02-kcephfs-master-distro-basic-smithi/4820744/teuthology.log</p>
<p>Core: /ceph/teuthology-archive/pdonnell-2020-03-03_14:22:02-kcephfs-master-distro-basic-smithi/4820744/remote/smithi063/coredump/1583250867.14575.core</p>
<pre>
./remote/smithi063/coredump/1583250867.14575.core: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from '/usr/libexec/platform-python /usr/bin/dnf makecache --timer'
</pre>
Infrastructure - Bug #41663 (New): ceph-post-file creates files that are inaccessible to non-priv...
https://tracker.ceph.com/issues/41663
2019-09-05T05:45:44Z
Brad Hubbard
bhubbard@redhat.com
<p>Several times now I've been asked to look at data uploaded with ceph-post-file only to find it is not readable by my login. At other times it is readable and everything is fine. Whether this bug belongs in the sepia project depends where we fix it but I thought here was a good place to start anyway.</p>
RADOS - Bug #38135 (New): Ceph is in HEALTH_ERR status with inconsistent PG after some rbd snapsh...
https://tracker.ceph.com/issues/38135
2019-01-31T23:53:46Z
Bengen Tan
<p>We observe Ceph is in HEALTH_ERR status with inconsistent PG after some rbd snapshot creating/removing task. Here are the environments and steps:<br />1, The ceph cluster has 108 OSDs.<br />2, Create a pool with 2048 PGs.<br />3, Generate 500K RBDs in the pool, each RBD is 20G<br />4, After the ceph performs some deep-scrub, the cluster is in HEALTH_OK status<br />5, Create snapshots for those RBDs, total snapshots are around 1.2M.<br />6, Make sure the Ceph cluster is in HEALTH_OK<br />7, Randomly creating and removing snapshot in parallel. We have about 6 clients do the creating/removing<br />8, We observe some snaptrim_wait, after about 12 hrs, we got about 3 inconsistent PGs.<br />Comparing to Ceph 12, we have 100K RBDs, with about 2M snapshots, we only get 1 inconsistent PG, with some crashed OSDs.<br />If need more details, please kindly let me know and I am happy to provide the test script and the detail.</p>
RADOS - Bug #6297 (In Progress): ceph osd tell * will break when FD limit reached, messenger shou...
https://tracker.ceph.com/issues/6297
2013-09-12T14:34:22Z
Brian Andrus
brian.andrus@inktank.com
<p>In environments with a large number of OSD's (approaching or exceeding the file descriptor limit set), ceph osd tell * will start throwing errors once the fd limit is reached.</p>
<p><code>osd.1018: filestore_wbthrottle_enable = 'false' <br />osd.1019: filestore_wbthrottle_enable = 'false' <br />2013-09-11 11:24:30.007712 7f61b4453700 -1 -- x.x.23.100:0/1030951 >> x.x.23.102:6899/92790 pipe(0x7f6618002800 sd=-1 :0 s=1 pgs=0 cs=0 l=1 c=0x7f6618002a60).connect couldn't created socket Too many open files </code></p>