Ceph : Issues
https://tracker.ceph.com/
https://tracker.ceph.com/favicon.ico
2024-01-31T22:53:19Z
Ceph
Redmine
Ceph - Bug #64279 (New): "Error ENOTSUP: Warning: due to ceph-mgr restart" in octopus-x/pacific s...
https://tracker.ceph.com/issues/64279
2024-01-31T22:53:19Z
Yuri Weinstein
yweinste@redhat.com
<p>This is for 16.2.15</p>
<p>Run: <a class="external" href="https://pulpito.ceph.com/yuriw-2024-01-31_16:13:03-upgrade:octopus-x-pacific-release-distro-default-smithi/">https://pulpito.ceph.com/yuriw-2024-01-31_16:13:03-upgrade:octopus-x-pacific-release-distro-default-smithi/</a><br />Jobs: 7540432 7540434 7540435<br />Logs: <a class="external" href="https://pulpito.ceph.com/yuriw-2024-01-31_16:13:03-upgrade:octopus-x-pacific-release-distro-default-smithi/7540432/">https://pulpito.ceph.com/yuriw-2024-01-31_16:13:03-upgrade:octopus-x-pacific-release-distro-default-smithi/7540432/</a></p>
<pre>
2024-01-31T17:36:17.722 INFO:teuthology.orchestra.run.smithi159.stderr:Error ENOTSUP: Warning: due to ceph-mgr restart, some PG states may not be up to date
2024-01-31T17:36:17.722 INFO:teuthology.orchestra.run.smithi159.stderr:Module 'orchestrator' is not enabled/loaded (required by command 'orch ps'): use `ceph mgr module enable orchestrator` to enable it
</pre>
<pre>
024-01-31T17:36:17.933 INFO:journalctl@ceph.mon.a.smithi159.stdout:Jan 31 17:36:17 smithi159 ceph-157b0ed4-c05e-11ee-95b3-87774f69a715-mon-a[70364]: cluster 2024-01-31T17:36:17.656902+0000 mon.a (mon.0) 62 : cluster [INF] Manager daemon y is now available
2024-01-31T17:36:18.034 DEBUG:teuthology.orchestra.run:got remote process result: 95
2024-01-31T17:36:18.035 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 105, in run_tasks
manager = run_one_task(taskname, ctx=ctx, config=config)
File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 83, in run_one_task
return task(**kwargs)
File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/parallel.py", line 56, in task
p.spawn(_run_spawned, ctx, confg, taskname)
File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 84, in __exit__
for result in self:
File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 98, in __next__
resurrect_traceback(result)
File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 30, in resurrect_traceback
raise exc.exc_info[1]
File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 23, in capture_traceback
return func(*args, **kwargs)
File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/parallel.py", line 64, in _run_spawned
mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config)
File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 83, in run_one_task
return task(**kwargs)
File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/sequential.py", line 47, in task
mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg)
File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 83, in run_one_task
return task(**kwargs)
File "/home/teuthworker/src/github.com_ceph_ceph-c_88fb4c6adb4bae6b1cdf34fca7eae2dacb06cc7b/qa/tasks/cephadm.py", line 1058, in shell
_shell(ctx, cluster_name, remote,
File "/home/teuthworker/src/github.com_ceph_ceph-c_88fb4c6adb4bae6b1cdf34fca7eae2dacb06cc7b/qa/tasks/cephadm.py", line 34, in _shell
return remote.run(
File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run
r.wait()
File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status
raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi159 with status 95: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:octopus shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 157b0ed4-c05e-11ee-95b3-87774f69a715 -e sha1=88fb4c6adb4bae6b1cdf34fca7eae2dacb06cc7b -- bash -c 'ceph orch ps'"
2024-01-31T17:36:18.263 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=05855bfb45164e1fa69621bf999495c7
</pre>
Ceph - Bug #64256 (New): "Cannot download repodata/repomd.xml: All mirrors were tried' rc: 1 resu...
https://tracker.ceph.com/issues/64256
2024-01-30T20:17:42Z
Yuri Weinstein
yweinste@redhat.com
<p>This is for 16.2.15 release</p>
<p>Run: <a class="external" href="https://pulpito.ceph.com/yuriw-2024-01-30_15:54:26-upgrade:pacific-x-quincy-distro-default-smithi/">https://pulpito.ceph.com/yuriw-2024-01-30_15:54:26-upgrade:pacific-x-quincy-distro-default-smithi/</a><br />Jobs: '7538534', '7538533', '7538535', '7538528'</p>
<p>Logs: <a class="external" href="http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-30_15:54:26-upgrade:pacific-x-quincy-distro-default-smithi/7538528/teuthology.log">http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-30_15:54:26-upgrade:pacific-x-quincy-distro-default-smithi/7538528/teuthology.log</a></p>
<pre>
2024-01-30T16:09:48.249 INFO:teuthology.task.ansible:Archiving ansible failure log at: /home/teuthworker/archive/yuriw-2024-01-30_15:54:26-upgrade:pacific-x-quincy-distro-default-smithi/7538528/ansible_failures.yaml
2024-01-30T16:09:48.251 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/run_tasks.py", line 109, in run_tasks
manager.__enter__()
File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/task/__init__.py", line 123, in __enter__
self.begin()
File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/task/ansible.py", line 508, in begin
super(CephLab, self).begin()
File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/task/ansible.py", line 342, in begin
self.execute_playbook()
File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/task/ansible.py", line 374, in execute_playbook
self._handle_failure(command, status)
File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/task/ansible.py", line 402, in _handle_failure
raise AnsibleFailedError(failures)
teuthology.exceptions.AnsibleFailedError: smithi174.front.sepia.ceph.com: _ansible_no_log: null changed: false invocation: module_args: allow_downgrade: false allowerasing: false autoremove: false bugfix: false cacheonly: false conf_file: null disable_excludes: null disable_gpg_check: false disable_plugin: [] disablerepo: [] download_dir: null download_only: false enable_plugin: [] enablerepo: [] exclude: [] install_repoquery: true install_weak_deps: true installroot: / list: null lock_timeout: 30 name: - krb5-workstation nobest: false releasever: null security: false skip_broken: false sslverify: true state: present update_cache: false update_only: false validate_certs: true msg: 'Failed to download metadata for repo ''codeready-builder-for-rhel-8-x86_64-rpms'': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried' rc: 1 results: []smithi006.front.sepia.ceph.com: _ansible_no_log: null changed: false invocation: module_args: allow_downgrade: false allowerasing: false autoremove: false bugfix: false cacheonly: false conf_file: null disable_excludes: null disable_gpg_check: false disable_plugin: [] disablerepo: [] download_dir: null download_only: false enable_plugin: [] enablerepo: [] exclude: [] install_repoquery: true install_weak_deps: true installroot: / list: null lock_timeout: 30 name: - krb5-workstation nobest: false releasever: null security: false skip_broken: false sslverify: true state: present update_cache: false update_only: false validate_certs: true msg: 'Failed to download metadata for repo ''codeready-builder-for-rhel-8-x86_64-rpms'': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried' rc: 1 results: []
2024-01-30T16:09:48.630 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=3cf9a763f1224702b05b0ad425b4fe45
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/run_tasks.py", line 109, in run_tasks
manager.__enter__()
File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/task/__init__.py", line 123, in __enter__
self.begin()
File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/task/ansible.py", line 508, in begin
super(CephLab, self).begin()
File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/task/ansible.py", line 342, in begin
self.execute_playbook()
File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/task/ansible.py", line 374, in execute_playbook
self._handle_failure(command, status)
File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/task/ansible.py", line 402, in _handle_failure
raise AnsibleFailedError(failures)
teuthology.exceptions.AnsibleFailedError: smithi174.front.sepia.ceph.com: _ansible_no_log: null changed: false invocation: module_args: allow_downgrade: false allowerasing: false autoremove: false bugfix: false cacheonly: false conf_file: null disable_excludes: null disable_gpg_check: false disable_plugin: [] disablerepo: [] download_dir: null download_only: false enable_plugin: [] enablerepo: [] exclude: [] install_repoquery: true install_weak_deps: true installroot: / list: null lock_timeout: 30 name: - krb5-workstation nobest: false releasever: null security: false skip_broken: false sslverify: true state: present update_cache: false update_only: false validate_certs: true msg: 'Failed to download metadata for repo ''codeready-builder-for-rhel-8-x86_64-rpms'': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried' rc: 1 results: []smithi006.front.sepia.ceph.com: _ansible_no_log: null changed: false invocation: module_args: allow_downgrade: false allowerasing: false autoremove: false bugfix: false cacheonly: false conf_file: null disable_excludes: null disable_gpg_check: false disable_plugin: [] disablerepo: [] download_dir: null download_only: false enable_plugin: [] enablerepo: [] exclude: [] install_repoquery: true install_weak_deps: true installroot: / list: null lock_timeout: 30 name: - krb5-workstation nobest: false releasever: null security: false skip_broken: false sslverify: true state: present update_cache: false update_only: false validate_certs: true msg: 'Failed to download metadata for repo ''codeready-builder-for-rhel-8-x86_64-rpms'': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried' rc: 1 results: []
2024-01-30T16:09:48.632 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab
2024-01-30T16:09:48.645 INFO:teuthology.task.ansible:Skipping ansible cleanup...
</pre>
Ceph - Bug #64254 (New): PR check builds fail because clang isn't available
https://tracker.ceph.com/issues/64254
2024-01-30T19:40:35Z
Casey Bodley
cbodley@redhat.com
<p>from <a class="external" href="https://jenkins.ceph.com/job/ceph-pull-requests/128387/consoleFull">https://jenkins.ceph.com/job/ceph-pull-requests/128387/consoleFull</a></p>
<p>during <code>src/script/run-make.sh</code>, <code>install-deps.sh</code> fails to install clang:<br /><pre>
The following packages have unmet dependencies:
libclang-common-14-dev : Depends: libllvm14 (= 1:14.0.0-1ubuntu1.1) but 1:14.0.6~++20230131082223+f28c006a5895-1~exp1~20230131082249.127 is to be installed
libclang-cpp14 : Depends: libllvm14 (= 1:14.0.0-1ubuntu1.1) but 1:14.0.6~++20230131082223+f28c006a5895-1~exp1~20230131082249.127 is to be installed
E: Unable to correct problems, you have held broken packages.
</pre></p>
<p><code>discover_compiler()</code> looks for the latest clang version, but doesn't find any so defaults to gcc:<br /><pre>
CI_DEBUG: Finding compiler for ci-build
CI_DEBUG: Our cmake_opts are: -DCMAKE_CXX_COMPILER=g++ -DCMAKE_C_COMPILER=gcc
</pre></p>
<p>the build ends up failing due to warnings from the cpp_redis submodule:<br /><pre>
[243/2719] Building CXX object src/cpp_redis/CMakeFiles/cpp_redis.dir/sources/core/client.cpp.o
FAILED: src/cpp_redis/CMakeFiles/cpp_redis.dir/sources/core/client.cpp.o
/usr/bin/ccache /usr/bin/g++ -DBOOST_ASIO_DISABLE_THREAD_KEYWORD_EXTENSION -DBOOST_ASIO_HAS_IO_URING -DBOOST_ASIO_NO_TS_EXECUTORS -DHAVE_CONFIG_H -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -D_REENTRANT -D_THREAD_SAFE -D__CEPH__ -D__STDC_FORMAT_MACROS -D__linux__ -I/home/jenkins-build/build/workspace/ceph-pull-requests/build/src/include -I/home/jenkins-build/build/workspace/ceph-pull-requests/src -I/home/jenkins-build/build/workspace/ceph-pull-requests/src/cpp_redis/includes -I/home/jenkins-build/build/workspace/ceph-pull-requests/src/cpp_redis/deps/include -I/home/jenkins-build/build/workspace/ceph-pull-requests/src/cpp_redis/tacopie/includes -isystem /opt/ceph/include -isystem /home/jenkins-build/build/workspace/ceph-pull-requests/build/include -std=c++11 -W -Wall -Wextra -O3 -Og -Werror -fPIC -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free -DBOOST_PHOENIX_STL_TUPLE_H_ -Wall -fno-strict-aliasing -fsigned-char -Wtype-limits -Wignored-qualifiers -Wpointer-arith -Werror=format-security -Winit-self -Wno-unknown-pragmas -Wnon-virtual-dtor -Wno-ignored-qualifiers -ftemplate-depth-1024 -Wpessimizing-move -Wredundant-move -Wstrict-null-sentinel -Woverloaded-virtual -DCEPH_DEBUG_MUTEX -fstack-protector-strong -D_GLIBCXX_ASSERTIONS -fdiagnostics-color=auto -std=c++20 -MD -MT src/cpp_redis/CMakeFiles/cpp_redis.dir/sources/core/client.cpp.o -MF src/cpp_redis/CMakeFiles/cpp_redis.dir/sources/core/client.cpp.o.d -o src/cpp_redis/CMakeFiles/cpp_redis.dir/sources/core/client.cpp.o -c /home/jenkins-build/build/workspace/ceph-pull-requests/src/cpp_redis/sources/core/client.cpp
In file included from /home/jenkins-build/build/workspace/ceph-pull-requests/src/cpp_redis/includes/cpp_redis/core/client.hpp:35,
from /home/jenkins-build/build/workspace/ceph-pull-requests/src/cpp_redis/sources/core/client.cpp:23:
/home/jenkins-build/build/workspace/ceph-pull-requests/src/cpp_redis/includes/cpp_redis/core/sentinel.hpp: In lambda function:
/home/jenkins-build/build/workspace/ceph-pull-requests/src/cpp_redis/includes/cpp_redis/core/sentinel.hpp:109:58: error: implicit capture of ‘this’ via ‘[=]’ is deprecated in C++20 [-Werror=deprecated]
109 | if (!m_sync_condvar.wait_for(lock_callback, timeout, [=] {
| ^
/home/jenkins-build/build/workspace/ceph-pull-requests/src/cpp_redis/includes/cpp_redis/core/sentinel.hpp:109:58: note: add explicit ‘this’ or ‘*this’ capture
In file included from /home/jenkins-build/build/workspace/ceph-pull-requests/src/cpp_redis/sources/core/client.cpp:23:
(snip)
/home/jenkins-build/build/workspace/ceph-pull-requests/src/cpp_redis/sources/core/client.cpp: In lambda function:
/home/jenkins-build/build/workspace/ceph-pull-requests/src/cpp_redis/sources/core/client.cpp:4031:19: error: implicit capture of ‘this’ via ‘[=]’ is deprecated in C++20 [-Werror=deprecated]
4031 | return exec_cmd([=](const reply_callback_t& cb) -> client& { return zunionstore(destination, numkeys, keys, weights, method, cb); });
| ^
/home/jenkins-build/build/workspace/ceph-pull-requests/src/cpp_redis/sources/core/client.cpp:4031:19: note: add explicit ‘this’ or ‘*this’ capture
cc1plus: all warnings being treated as errors
</pre></p>
<p>in earlier successful builds (ex. <a class="external" href="https://jenkins.ceph.com/job/ceph-pull-requests/128316/consoleFull">https://jenkins.ceph.com/job/ceph-pull-requests/128316/consoleFull</a>):<br /><pre>
clang is already the newest version (1:14.0-55~exp2).
...
CI_DEBUG: Finding compiler for ci-build
CI_DEBUG: Our cmake_opts are: -DCMAKE_CXX_COMPILER=clang++-14 -DCMAKE_C_COMPILER=clang-14
</pre><br />and cpp_redis compiles/links without warnings/errors<br /><pre>
[317/2719] Linking CXX static library lib/libcpp_redis.a
</pre></p>
Linux kernel client - Bug #64172 (Fix Under Review): Test failure: test_multiple_path_r (tasks.ce...
https://tracker.ceph.com/issues/64172
2024-01-25T05:55:58Z
Venky Shankar
vshankar@redhat.com
<p>/a/vshankar-2024-01-22_07:03:31-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi/7525717</p>
<p>The test setup involves "read" cap on a file system path (directory), remount the directory as file system root and read the created files.</p>
<p>MDS logs: ./remote/smithi157/log/ceph-mds.c.log.gz</p>
<pre>
2024-01-22T08:27:55.205+0000 7f81a7600640 1 -- [v2:172.21.15.157:6835/1231338113,v1:172.21.15.157:6837/1231338113] <== client.17628 v1:192.168.0.1:0/312855551 9 ==== client_request(client.17628:5 lookupino #0x1 2024-01-22T08:27:55.205939+0000 caller_uid=0, caller_gid=0{0,}) v6 ==== 176+0+0 (unknown 772642831 0 0) 0x55ab1d52bb00 con 0x55ab1d52e400
2024-01-22T08:27:55.205+0000 7f81a7600640 4 mds.0.server handle_client_request client_request(client.17628:5 lookupino #0x1 2024-01-22T08:27:55.205939+0000 caller_uid=0, caller_gid=0{0,}) v6
2024-01-22T08:27:55.205+0000 7f81a7600640 20 mds.0.356 get_session have 0x55ab1d202f00 client.17628 v1:192.168.0.1:0/312855551 state open
2024-01-22T08:27:55.205+0000 7f81a7600640 15 mds.0.server oldest_client_tid=5
2024-01-22T08:27:55.205+0000 7f81a7600640 7 mds.0.cache request_start request(client.17628:5 nref=2 cr=0x55ab1d52bb00)
2024-01-22T08:27:55.205+0000 7f81a7600640 7 mds.0.server dispatch_client_request client_request(client.17628:5 lookupino #0x1 2024-01-22T08:27:55.205939+0000 caller_uid=0, caller_gid=0{0,}) v6
2024-01-22T08:27:55.205+0000 7f81a7600640 20 Session check_access path
2024-01-22T08:27:55.205+0000 7f81a7600640 10 MDSAuthCap is_capable inode(path / owner 0:0 mode 041777) by caller 0:0 mask 0 new 0:0 cap: MDSAuthCaps[allow r fsname=cephfs path="/dir1/dir12", allow r fsname=cephfs path="/dir2/dir22"]
2024-01-22T08:27:55.205+0000 7f81a7600640 7 mds.0.server reply_client_request -13 ((13) Permission denied) client_request(client.17628:5 lookupino #0x1 2024-01-22T08:27:55.205939+0000 caller_uid=0, caller_gid=0{0,}) v6
2024-01-22T08:27:55.205+0000 7f81a7600640 10 mds.0.server apply_allocated_inos 0x0 / [] / 0x0
2024-01-22T08:27:55.205+0000 7f81a7600640 20 mds.0.server lat 0.000095
2024-01-22T08:27:55.205+0000 7f81a7600640 10 mds.0.356 send_message_client client.17628 v1:192.168.0.1:0/312855551 client_reply(???:5 = -13 (13) Permission denied) v1
2024-01-22T08:27:55.205+0000 7f81a7600640 1 -- [v2:172.21.15.157:6835/1231338113,v1:172.21.15.157:6837/1231338113] --> v1:192.168.0.1:0/312855551 -- client_reply(???:5 = -13 (13) Permission denied) v1 -- 0x55ab1d5b4700 con 0x55ab1d52e400
</pre>
Ceph - Bug #63617 (New): ceph-common: CommonSafeTimer<std::mutex>::timer_thread(): python3.12 kil...
https://tracker.ceph.com/issues/63617
2023-11-23T18:40:45Z
Kaleb KEITHLEY
<p><a class="external" href="https://bugzilla.redhat.com/show_bug.cgi?id=2251165">https://bugzilla.redhat.com/show_bug.cgi?id=2251165</a></p>
<p>Description of problem:</p>
<p>Version-Release number of selected component:<br />ceph-common-2:18.2.1-1.fc39</p>
<p>Additional info:<br />reporter: libreport-2.17.11<br />cmdline: /usr/bin/python3.12 /usr/bin/ceph -s<br />backtrace_rating: 4<br />runlevel: N 5<br />executable: /usr/bin/python3.12<br />journald_cursor: s=9f8a7a66b4194fdcbd75dcd3edf4da87;i=173e8c976;b=a08b8db920744522980a5387af245706;m=2743cc1c;t=60accf74a277f;x=cef1ac3a8dc81a9d<br />comment: <br />cgroup: 0::/user.slice/user-1000.slice/user/app.slice/app-org.kde.konsole-44b42a69b68946748c9899bd38ac8c6d.scope<br />kernel: 6.6.2-200.fc39.x86_64<br />uid: 1000<br />rootdir: /<br />crash_function: CommonSafeTimer<std::mutex>::timer_thread<br />type: CCpp<br />package: ceph-common-2:18.2.1-1.fc39<br />reason: python3.12 killed by SIGSEGV</p>
<p>Truncated backtrace:<br />Thread no. 1 (3 frames)<br /> #0 CommonSafeTimer<std::mutex>::timer_thread at /usr/src/debug/ceph-18.2.1-1.fc39.x86_64/src/common/Timer.cc:103<br /> <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: gpf in tcp_sendpage (Closed)" href="https://tracker.ceph.com/issues/1">#1</a> CommonSafeTimerThread<std::mutex>::entry at /usr/src/debug/ceph-18.2.1-1.fc39.x86_64/src/common/Timer.cc:33<br /> <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: leaked dentry ref on umount (Closed)" href="https://tracker.ceph.com/issues/3">#3</a> clone3 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78</p>
rbd - Bug #61891 (In Progress): Parent data is not copied up when cloned images are mirrored.
https://tracker.ceph.com/issues/61891
2023-07-04T11:30:27Z
Nithya Balachandran
<p>Steps :<br />Set up 2 ceph clusters for rbd mirroring.</p>
<p>$ bin/rbd --cluster site-a create -s 8M data/src1<br />$ bin/rbd --cluster site-a map -t nbd data/src1</p>
<p>$ xfs_io -d -c 'pwrite -S 0xadad -b 4M 0 5M' /dev/nbd0</p>
<p>$ bin/rbd --cluster site-a snap create data/src1@snap1<br />$ bin/rbd --cluster site-a snap protect data/src1@snap1<br />$ bin/rbd --cluster site-a clone data/src1@snap1 data/dst1</p>
<p>$ bin/rbd --cluster site-a unmap -t nbd data/src1<br />$ bin/rbd --cluster site-a map -t nbd data/dst1</p>
<p>#Enable mirroring on both parent and child images<br />$ bin/rbd --cluster site-a mirror image enable data/src1 snapshot <br />$ bin/rbd --cluster site-a mirror image enable data/dst1 snapshot</p>
<p>#Wait until synced</p>
<p>$ xfs_io -d -c 'pwrite -S 0x11 -b 4M 7M 1M' /dev/nbd0<br />$ xfs_io -d -c 'pread -v 4M 512' /dev/nbd0<br />00400000: ad ad ad ad ad ad ad ad ad ad ad ad ad ad ad ad ................<br />00400010: ad ad ad ad ad ad ad ad ad ad ad ad ad ad ad ad ................<br />00400020: ad ad ad ad ad ad ad ad ad ad ad ad ad ad ad ad ................<br />00400030: ad ad ad ad ad ad ad ad ad ad ad ad ad ad ad ad ................<br />00400040: ad ad ad ad ad ad ad ad ad ad ad ad ad ad ad ad ................<br />...</p>
<p>$ bin/rbd --cluster site-a mirror image snapshot data/dst1<br />$ bin/rbd --cluster site-a unmap -t nbd data/dst1<br />$ bin/rbd --cluster site-a mirror image demote data/dst1<br />$ bin/rbd --cluster site-b mirror image promote data/dst1<br />$ bin/rbd --cluster site-b map -t nbd data/dst1<br />/dev/nb0</p>
<p>#Data written directly to the clone has been copied<br />$ xfs_io -d -c 'pread -v 7M 512' /dev/nbd0<br />00700000: 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 ................<br />00700010: 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 ................<br />00700020: 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 ................<br />00700030: 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 ................<br />00700040: 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 ................<br />00700050: 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 ................<br />00700060: 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 ................<br />...</p>
<p>#The parent data has not been copied up to object 1:<br />$ xfs_io -d -c 'pread -v 4M 512' /dev/nbd0<br />00400000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................<br />00400010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................<br />00400020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................<br />00400030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................<br />00400040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................<br />00400050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................<br />00400060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................<br />...</p>
Ceph - Bug #61400 (New): valgrind+ceph-mon: segmentation fault in rocksdb+tcmalloc
https://tracker.ceph.com/issues/61400
2023-05-24T14:28:51Z
Patrick Donnelly
pdonnell@redhat.com
<pre>
0> 2023-05-24T02:54:54.546+0000 708e7c0 -1 *** Caught signal (Segmentation fault) **
in thread 708e7c0 thread_name:ceph-mon
ceph version 18.0.0-4167-gfa0e62c4 (fa0e62c4a1d8e4a737d9cbe50224f70009b79b28) reef (dev)
1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x5827420]
2: (tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**)+0x20) [0x55f50a0]
3: (tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**)+0x20) [0x55f5370]
4: (tcmalloc::CentralFreeList::RemoveRange(void**, void**, int)+0x80) [0x55f5430]
5: (tcmalloc::ThreadCache::FetchFromCentralCache(unsigned int, int, void* (*)(unsigned long))+0x76) [0x55f8e46]
6: (tcmalloc::allocate_full_cpp_throw_oom(unsigned long)+0x165) [0x5609015]
7: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool)+0x22b) [0x109b0dd]
8: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x65) [0x109a139]
9: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x760) [0xf56d80]
10: (MonitorDBStore::open(std::ostream&)+0xfd) [0xc5ca1d]
11: main()
12: __libc_start_main()
13: _start()
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
</pre>
<p>From: /ceph/teuthology-archive/pdonnell-2023-05-23_18:20:18-fs-wip-pdonnell-testing-20230523.134409-distro-default-smithi/7284230/remote/smithi007/log/ceph-mon.a.log.gz</p>
<p>This happened shortly after the ceph-mon starts.</p>
<p>Here's the error from valgrind as well:</p>
<pre>
<error>
<unique>0x1</unique>
<tid>1</tid>
<threadname>ceph-mon</threadname>
<kind>InvalidRead</kind>
<what>Invalid read of size 8</what>
<stack>
<frame>
<ip>0x55F50A0</ip>
<obj>/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4.5.3</obj>
<fn>tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**)</fn>
</frame>
<frame>
<ip>0x55F536F</ip>
<obj>/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4.5.3</obj>
<fn>tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**)</fn>
</frame>
<frame>
<ip>0x55F542F</ip>
<obj>/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4.5.3</obj>
<fn>tcmalloc::CentralFreeList::RemoveRange(void**, void**, int)</fn>
</frame>
<frame>
<ip>0x55F8E45</ip>
<obj>/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4.5.3</obj>
<fn>tcmalloc::ThreadCache::FetchFromCentralCache(unsigned int, int, void* (*)(unsigned long))</fn>
</frame>
<frame>
<ip>0x5609014</ip>
<obj>/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4.5.3</obj>
<fn>tcmalloc::allocate_full_cpp_throw_oom(unsigned long)</fn>
</frame>
<frame>
<ip>0x109B0DC</ip>
<obj>/usr/bin/ceph-mon</obj>
<fn>rocksdb::DBImpl::Open(rocksdb::DBOptions const&amp;, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const&amp;, std::vector&lt;rocksdb::ColumnFamilyDescriptor, std::allocator&lt;rocksdb::ColumnFamilyDescriptor&gt; &gt; const&amp;, std::vector&lt;rocksdb::ColumnFamilyHandle*, std::allocator&lt;rocksdb::ColumnFamilyHandle*&gt; &gt;*, rocksdb::DB**, bool, bool)</fn>
</frame>
<frame>
<ip>0x109A138</ip>
<obj>/usr/bin/ceph-mon</obj>
<fn>rocksdb::DB::Open(rocksdb::DBOptions const&amp;, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const&amp;, std::vector&lt;rocksdb::ColumnFamilyDescriptor, std::allocator&lt;rocksdb::ColumnFamilyDescriptor&gt; &gt; const&amp;, std::vector&lt;rocksdb::ColumnFamilyHandle*, std::allocator&lt;rocksdb::ColumnFamilyHandle*&gt; &gt;*, rocksdb::DB**)</fn>
</frame>
<frame>
<ip>0xF56D7F</ip>
<obj>/usr/bin/ceph-mon</obj>
<fn>RocksDBStore::do_open(std::ostream&amp;, bool, bool, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const&amp;)</fn>
<dir>./obj-x86_64-linux-gnu/src/kv/./src/kv</dir>
<file>RocksDBStore.cc</file>
<line>1193</line>
</frame>
<frame>
<ip>0xC5CA1C</ip>
<obj>/usr/bin/ceph-mon</obj>
<fn>MonitorDBStore::open(std::ostream&amp;)</fn>
<dir>./obj-x86_64-linux-gnu/src/./src/mon</dir>
<file>MonitorDBStore.h</file>
<line>674</line>
</frame>
<frame>
<ip>0xC368BF</ip>
<obj>/usr/bin/ceph-mon</obj>
<fn>main</fn>
<dir>./obj-x86_64-linux-gnu/src/./src</dir>
<file>ceph_mon.cc</file>
<line>639</line>
</frame>
</stack>
<auxwhat>Address 0x20 is not stack'd, malloc'd or (recently) free'd</auxwhat>
</error>
</pre>
<p>From: /ceph/teuthology-archive/pdonnell-2023-05-23_18:20:18-fs-wip-pdonnell-testing-20230523.134409-distro-default-smithi/7284230/remote/smithi007/log/valgrind/mon.a.log.gz</p>
Linux kernel client - Bug #55823 (New): NETDEV WATCHDOG: enp3s0f1 (ixgbe): transmit queue 3 timed...
https://tracker.ceph.com/issues/55823
2022-06-02T08:56:17Z
Xiubo Li
xiubli@redhat.com
<p><a class="external" href="http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-13_17:09:16-fs-wip-vshankar-testing-20220513-120051-testing-default-smithi/6832877/console_logs/smithi117.log">http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-13_17:09:16-fs-wip-vshankar-testing-20220513-120051-testing-default-smithi/6832877/console_logs/smithi117.log</a></p>
<p><a class="external" href="http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-31_02:47:51-fs-wip-vshankar-fscrypt-20220530-091336-testing-default-smithi/6853733/console_logs/smithi025.log">http://qa-proxy.ceph.com/teuthology/vshankar-2022-05-31_02:47:51-fs-wip-vshankar-fscrypt-20220530-091336-testing-default-smithi/6853733/console_logs/smithi025.log</a></p>
<p>The teuthology test failed without any logs in <strong><em>remote/</em></strong> direcotry and could see the following call traces:</p>
<pre>
[ 1081.462059] ------------[ cut here ]------------
[ 1081.466857] NETDEV WATCHDOG: enp3s0f1 (ixgbe): transmit queue 3 timed out
[ 1081.473785] WARNING: CPU: 3 PID: 0 at net/sched/sch_generic.c:530 dev_watchdog+0x20f/0x250
[ 1081.482199] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm iTCO_wdt ipmi_ssif iTCO_vendor_support irqbypass crct10dif_pclmul crc32_pclmul joydev ghash_clmulni_intel i2c_i801 lpc_ich wmi i2c_smbus mfd_core acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter acpi_pad mei_me mei ioatdma ixgbe igb mdio i2c_algo_bit crc32c_intel ptp nvme pps_core dca nvme_core fuse
[ 1081.537034] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G S 5.18.0-rc6-ceph-gb850759e370d #1
[ 1081.546475] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0 12/17/2015
[ 1081.553993] RIP: 0010:dev_watchdog+0x20f/0x250
[ 1081.558537] Code: 00 e9 4d ff ff ff 48 89 df c6 05 31 96 f6 00 01 e8 c6 fb f9 ff 44 89 e9 48 89 de 48 c7 c7 70 e2 41 82 48 89 c2 e8 8b 19 19 00 <0f> 0b e9 2f ff ff ff e8 f5 05 1e 00 85 c0 74 b5 80 3d f8 8e f6 00
[ 1081.577478] RSP: 0018:ffffc900001b4e80 EFLAGS: 00010282
[ 1081.582807] RAX: 0000000000000000 RBX: ffff88817b2c0000 RCX: 0000000000000103
[ 1081.590042] RDX: 0000000000000103 RSI: ffffffff823936a5 RDI: 00000000ffffffff
[ 1081.597290] RBP: ffff88817b2c0508 R08: 0000000000000000 R09: c0000000ffffdfff
[ 1081.604539] R10: 0000000000000001 R11: ffffc900001b4ca0 R12: ffff88817b2c0420
[ 1081.611786] R13: 0000000000000003 R14: ffff88817b2c0508 R15: ffff88885fcdd480
[ 1081.619027] FS: 0000000000000000(0000) GS:ffff88885fcc0000(0000) knlGS:0000000000000000
[ 1081.627257] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1081.633108] CR2: 000055ebf3d75000 CR3: 0000000002612004 CR4: 00000000003706e0
[ 1081.640347] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1081.647590] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1081.654828] Call Trace:
[ 1081.657366] <IRQ>
[ 1081.659474] ? mq_change_real_num_tx+0xd0/0xd0
[ 1081.664022] call_timer_fn+0xa1/0x2c0
[ 1081.667794] ? mq_change_real_num_tx+0xd0/0xd0
[ 1081.672345] run_timer_softirq+0x50d/0x530
[ 1081.676547] ? lock_is_held_type+0xe3/0x140
[ 1081.680842] __do_softirq+0xdc/0x483
[ 1081.684522] irq_exit_rcu+0xdc/0x120
[ 1081.688197] sysvec_apic_timer_interrupt+0x9e/0xc0
[ 1081.693093] </IRQ>
[ 1081.695281] <TASK>
[ 1081.697476] asm_sysvec_apic_timer_interrupt+0x12/0x20
[ 1081.702714] RIP: 0010:cpuidle_enter_state+0x10a/0x4b0
[ 1081.707868] Code: 6c dc 8f ff 45 84 ff 74 16 9c 58 0f 1f 40 00 f6 c4 02 0f 85 43 03 00 00 31 ff e8 c1 51 96 ff e8 5c f5 9d ff fb 0f 1f 44 00 00 <45> 85 f6 0f 88 12 01 00 00 49 63 d6 4c 2b 24 24 48 8d 04 52 48 8d
[ 1081.726813] RSP: 0018:ffffc9000010be88 EFLAGS: 00000206
[ 1081.732137] RAX: 000000000073c7f9 RBX: 0000000000000004 RCX: 0000000000000000
[ 1081.739373] RDX: 0000000000000000 RSI: ffffffff8240e25f RDI: ffffffff823a6b0f
[ 1081.746609] RBP: ffffe8ffffcc3a00 R08: 0000000000000001 R09: 0000000000000001
[ 1081.753847] R10: 000004ae9b8e4927 R11: ffff88885fcec1c4 R12: 000000fbcc2958a1
[ 1081.761084] R13: ffffffff827ed4c0 R14: 0000000000000004 R15: 0000000000000000
[ 1081.768339] ? cpuidle_enter_state+0x104/0x4b0
[ 1081.772890] cpuidle_enter+0x29/0x40
[ 1081.776568] do_idle+0x245/0x260
[ 1081.787555] cpu_startup_entry+0x19/0x20
[ 1081.791579] start_secondary+0x106/0x130
[ 1081.795599] secondary_startup_64_no_verify+0xd5/0xdb
[ 1081.800769] </TASK>
[ 1081.803045] irq event stamp: 7588692
[ 1081.806724] hardirqs last enabled at (7588702): [<ffffffff811607de>] __up_console_sem+0x5e/0x70
[ 1081.815643] hardirqs last disabled at (7588711): [<ffffffff811607c3>] __up_console_sem+0x43/0x70
[ 1081.824571] softirqs last enabled at (7587812): [<ffffffff81e00321>] __do_softirq+0x321/0x483
[ 1081.833317] softirqs last disabled at (7587835): [<ffffffff810e4a0c>] irq_exit_rcu+0xdc/0x120
[ 1081.841969] ---[ end trace 0000000000000000 ]---
[ 1081.846718] ixgbe 0000:03:00.1 enp3s0f1: initiating reset due to tx timeout
[ 1081.854216] ixgbe 0000:03:00.1 enp3s0f1: Reset adapter
[ 1085.558440] libceph: mon1 (1)172.21.15.117:6789 socket error on write
[ 1085.822639] libceph: mon1 (1)172.21.15.117:6789 socket error on write
[ 1086.334273] libceph: mon1 (1)172.21.15.117:6789 socket error on write
[ 1087.414306] libceph: mon1 (1)172.21.15.117:6789 socket error on write
[ 1089.398379] libceph: mon1 (1)172.21.15.117:6789 socket error on write
[ 1093.430734] libceph: mon1 (1)172.21.15.117:6789 socket error on write
[ 1093.954303] watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [fsstress:89394]
[ 1093.959305] watchdog: BUG: soft lockup - CPU#1 stuck for 26s! [fsstress:89395]
[ 1093.960304] watchdog: BUG: soft lockup - CPU#2 stuck for 26s! [fsstress:89392]
[ 1093.960308] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm iTCO_wdt ipmi_ssif iTCO_vendor_support irqbypass crct10dif_pclmul crc32_pclmul joydev ghash_clmulni_intel i2c_i801 lpc_ich wmi i2c_smbus mfd_core acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter acpi_pad mei_me mei ioatdma ixgbe igb mdio i2c_algo_bit crc32c_intel ptp nvme pps_core dca nvme_core fuse
[ 1093.960358] irq event stamp: 50920
[ 1093.960359] hardirqs last enabled at (50919): [<ffffffff81c00d42>] asm_sysvec_apic_timer_interrupt+0x12/0x20
[ 1093.960364] hardirqs last disabled at (50920): [<ffffffff81ae1e4a>] sysvec_apic_timer_interrupt+0xa/0xc0
[ 1093.960367] softirqs last enabled at (1840): [<ffffffff81e00321>] __do_softirq+0x321/0x483
[ 1093.960370] softirqs last disabled at (1807): [<ffffffff810e4a0c>] irq_exit_rcu+0xdc/0x120
[ 1093.960372] CPU: 2 PID: 89392 Comm: fsstress Tainted: G S W 5.18.0-rc6-ceph-gb850759e370d #1
[ 1093.960375] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0 12/17/2015
[ 1093.960376] RIP: 0010:queued_spin_lock_slowpath+0x1b3/0x1e0
[ 1093.960381] Code: 83 e0 03 83 e9 01 48 c1 e0 04 48 63 c9 48 05 40 e5 02 00 48 03 04 cd 20 0b 43 82 48 89 10 8b 42 08 85 c0 75 09 f3 90 8b 42 08 <85> c0 74 f7 48 8b 0a 48 85 c9 0f 84 6b ff ff ff 0f 0d 09 e9 63 ff
[ 1093.960382] RSP: 0018:ffffc900017f3e38 EFLAGS: 00000246
[ 1093.960384] RAX: 0000000000000000 RBX: ffff88817bbfc4b0 RCX: 0000000000000007
[ 1093.960385] RDX: ffff88885fcae540 RSI: ffffffff823936a5 RDI: ffffffff823a6afe
[ 1093.960386] RBP: ffff88885fcae540 R08: 00000000000052ef R09: 0000000000000000
[ 1093.960387] R10: 0000000000000001 R11: c359abbe014cc9bb R12: 00000000000c0000
[ 1093.960388] R13: ffff88817b7d3000 R14: ffff88817b7d3998 R15: ffffffffa089c2f3
[ 1093.960390] FS: 00007f4103475500(0000) GS:ffff88885fc80000(0000) knlGS:0000000000000000
[ 1093.960391] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1093.960392] CR2: 000055ebf31cd300 CR3: 00000001d35fc002 CR4: 00000000003706e0
[ 1093.960393] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1093.960394] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1093.960396] Call Trace:
[ 1093.960397] <TASK>
[ 1093.960400] do_raw_spin_lock+0xb1/0xc0
[ 1093.960403] flush_dirty_session_caps+0x82/0x150 [ceph]
[ 1093.960422] ? __ceph_put_cap_refs+0x440/0x440 [ceph]
[ 1093.960434] ceph_mdsc_iterate_sessions+0x65/0xa0 [ceph]
[ 1093.960449] ? __x64_sys_tee+0xc0/0xc0
[ 1093.960453] ceph_sync_fs+0x9b/0x100 [ceph]
[ 1093.960462] iterate_supers+0x93/0xe0
[ 1093.960468] ksys_sync+0x4f/0xa0
[ 1093.960471] __do_sys_sync+0xa/0x10
[ 1093.960474] do_syscall_64+0x34/0x80
[ 1093.960477] entry_SYSCALL_64_after_hwframe+0x44/0xae
[ 1093.960481] RIP: 0033:0x7f4102ef44fb
[ 1093.960483] Code: c3 48 8b 0d 8f a9 2c 00 f7 d8 64 89 01 b8 ff ff ff ff eb c2 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa b8 a2 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d a9 2c 00 f7 d8 64 89 01 48
[ 1093.960485] RSP: 002b:00007ffc90169858 EFLAGS: 00000202 ORIG_RAX: 00000000000000a2
[ 1093.960487] RAX: ffffffffffffffda RBX: 0000000000405c5a RCX: 00007f4102ef44fb
[ 1093.960488] RDX: 000000007f123e51 RSI: 000000007f123e51 RDI: 000000000000000f
[ 1093.960489] RBP: 00007ffc90169870 R08: 00007f41031bf200 R09: 00007f41031bf260
[ 1093.960490] R10: 0000000000000003 R11: 0000000000000202 R12: 0000000000401760
[ 1093.960491] R13: 00007ffc90169bb0 R14: 0000000000000000 R15: 0000000000000000
[ 1093.960502] </TASK>
[ 1093.961671] Modules linked in: ceph
[ 1093.962303] watchdog: BUG: soft lockup - CPU#4 stuck for 26s! [fsstress:89393]
[ 1093.962306] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm iTCO_wdt ipmi_ssif iTCO_vendor_support irqbypass crct10dif_pclmul crc32_pclmul joydev ghash_clmulni_intel i2c_i801 lpc_ich wmi i2c_smbus mfd_core acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter acpi_pad mei_me mei ioatdma ixgbe igb mdio i2c_algo_bit crc32c_intel ptp nvme pps_core dca nvme_core fuse
[ 1093.962374] irq event stamp: 54074
[ 1093.962375] hardirqs last enabled at (54073): [<ffffffff81c00d42>] asm_sysvec_apic_timer_interrupt+0x12/0x20
[ 1093.962379] hardirqs last disabled at (54074): [<ffffffff81ae1e4a>] sysvec_apic_timer_interrupt+0xa/0xc0
[ 1093.962383] softirqs last enabled at (5108): [<ffffffff81e00321>] __do_softirq+0x321/0x483
[ 1093.962386] softirqs last disabled at (5089): [<ffffffff810e4a0c>] irq_exit_rcu+0xdc/0x120
[ 1093.962389] CPU: 4 PID: 89393 Comm: fsstress Tainted: G S W L 5.18.0-rc6-ceph-gb850759e370d #1
[ 1093.962392] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0 12/17/2015
[ 1093.962394] RIP: 0010:queued_spin_lock_slowpath+0x1b3/0x1e0
[ 1093.962398] Code: 83 e0 03 83 e9 01 48 c1 e0 04 48 63 c9 48 05 40 e5 02 00 48 03 04 cd 20 0b 43 82 48 89 10 8b 42 08 85 c0 75 09 f3 90 8b 42 08 <85> c0 74 f7 48 8b 0a 48 85 c9 0f 84 6b ff ff ff 0f 0d 09 e9 63 ff
[ 1093.962400] RSP: 0018:ffffc900016b3e38 EFLAGS: 00000246
[ 1093.962403] RAX: 0000000000000000 RBX: ffff88817bbfc4b0 RCX: 0000000000000001
[ 1093.962405] RDX: ffff88885fd2e540 RSI: ffffffff823936a5 RDI: ffffffff823a6afe
[ 1093.962406] RBP: ffff88885fd2e540 R08: 00000000000052ef R09: 0000000000000000
[ 1093.962408] R10: 0000000000000001 R11: c359abbe014cc9bb R12: 0000000000140000
[ 1093.962410] R13: ffff88817b7d3000 R14: ffff88817b7d3998 R15: ffffffffa089c2f3
[ 1093.962412] FS: 00007f4103475500(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000
[ 1093.962414] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1093.962416] CR2: 000055efcbcd2000 CR3: 000000011a0b8005 CR4: 00000000003706e0
[ 1093.962418] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1093.962420] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1093.962421] Call Trace:
[ 1093.962423] <TASK>
[ 1093.962427] do_raw_spin_lock+0xb1/0xc0
[ 1093.962432] flush_dirty_session_caps+0x82/0x150 [ceph]
[ 1093.962453] ? __ceph_put_cap_refs+0x440/0x440 [ceph]
[ 1093.962467] ceph_mdsc_iterate_sessions+0x65/0xa0 [ceph]
[ 1093.962487] ? __x64_sys_tee+0xc0/0xc0
[ 1093.962490] ceph_sync_fs+0x9b/0x100 [ceph]
[ 1093.962502] iterate_supers+0x93/0xe0
[ 1093.962510] ksys_sync+0x4f/0xa0
[ 1093.962514] __do_sys_sync+0xa/0x10
[ 1093.962517] do_syscall_64+0x34/0x80
[ 1093.962521] entry_SYSCALL_64_after_hwframe+0x44/0xae
[ 1093.962525] RIP: 0033:0x7f4102ef44fb
[ 1093.962528] Code: c3 48 8b 0d 8f a9 2c 00 f7 d8 64 89 01 b8 ff ff ff ff eb c2 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa b8 a2 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d a9 2c 00 f7 d8 64 89 01 48
[ 1093.962530] RSP: 002b:00007ffc90169858 EFLAGS: 00000202 ORIG_RAX: 00000000000000a2
[ 1093.962533] RAX: ffffffffffffffda RBX: 0000000000405c5a RCX: 00007f4102ef44fb
[ 1093.962535] RDX: 000000000d0cb5a8 RSI: 000000000d0cb5a8 RDI: 000000000000000b
[ 1093.962537] RBP: 00007ffc90169870 R08: 00007f41031bf1e4 R09: 00007f41031bf260
[ 1093.962539] R10: 0000000000000003 R11: 0000000000000202 R12: 0000000000401760
[ 1093.962540] R13: 00007ffc90169bb0 R14: 0000000000000000 R15: 0000000000000000
[ 1093.962560] </TASK>
[ 1093.965303] watchdog: BUG: soft lockup - CPU#5 stuck for 26s! [fsstress:89400]
[ 1093.965306] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm iTCO_wdt ipmi_ssif iTCO_vendor_support irqbypass crct10dif_pclmul crc32_pclmul joydev ghash_clmulni_intel i2c_i801 lpc_ich wmi i2c_smbus mfd_core acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter acpi_pad mei_me mei ioatdma ixgbe igb mdio i2c_algo_bit crc32c_intel ptp nvme pps_core dca nvme_core fuse
[ 1093.965362] irq event stamp: 51000
[ 1093.965363] hardirqs last enabled at (50999): [<ffffffff81c00d42>] asm_sysvec_apic_timer_interrupt+0x12/0x20
[ 1093.965367] hardirqs last disabled at (51000): [<ffffffff81ae1e4a>] sysvec_apic_timer_interrupt+0xa/0xc0
[ 1093.965370] softirqs last enabled at (1608): [<ffffffff81e00321>] __do_softirq+0x321/0x483
[ 1093.965372] softirqs last disabled at (1589): [<ffffffff810e4a0c>] irq_exit_rcu+0xdc/0x120
[ 1093.965375] CPU: 5 PID: 89400 Comm: fsstress Tainted: G S W L 5.18.0-rc6-ceph-gb850759e370d #1
[ 1093.965377] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0 12/17/2015
[ 1093.965378] RIP: 0010:queued_spin_lock_slowpath+0x1b0/0x1e0
[ 1093.965383] Code: c1 e9 12 83 e0 03 83 e9 01 48 c1 e0 04 48 63 c9 48 05 40 e5 02 00 48 03 04 cd 20 0b 43 82 48 89 10 8b 42 08 85 c0 75 09 f3 90 <8b> 42 08 85 c0 74 f7 48 8b 0a 48 85 c9 0f 84 6b ff ff ff 0f 0d 09
[ 1093.965385] RSP: 0018:ffffc90001893e38 EFLAGS: 00000246
[ 1093.965387] RAX: 0000000000000000 RBX: ffff88817bbfc4b0 RCX: 0000000000000000
[ 1093.965389] RDX: ffff88885fd6e540 RSI: ffffffff823936a5 RDI: ffffffff823a6afe
[ 1093.965390] RBP: ffff88885fd6e540 R08: 00000000000052ef R09: 0000000000000000
[ 1093.965392] R10: 0000000000000001 R11: c359abbe014cc9bb R12: 0000000000180000
[ 1093.965393] R13: ffff88817b7d3000 R14: ffff88817b7d3998 R15: ffffffffa089c2f3
[ 1093.965395] FS: 00007f4103475500(0000) GS:ffff88885fd40000(0000) knlGS:0000000000000000
[ 1093.965397] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1093.965398] CR2: 000055a818759000 CR3: 00000001bb392004 CR4: 00000000003706e0
[ 1093.965400] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1093.965402] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1093.965403] Call Trace:
[ 1093.965404] <TASK>
[ 1093.965409] do_raw_spin_lock+0xb1/0xc0
[ 1093.965413] flush_dirty_session_caps+0x82/0x150 [ceph]
[ 1093.965431] ? __ceph_put_cap_refs+0x440/0x440 [ceph]
[ 1093.965443] ceph_mdsc_iterate_sessions+0x65/0xa0 [ceph]
[ 1093.965461] ? __x64_sys_tee+0xc0/0xc0
[ 1093.965464] ceph_sync_fs+0x9b/0x100 [ceph]
[ 1093.965475] iterate_supers+0x93/0xe0
[ 1093.965482] ksys_sync+0x4f/0xa0
[ 1093.965486] __do_sys_sync+0xa/0x10
[ 1093.965488] do_syscall_64+0x34/0x80
[ 1093.965492] entry_SYSCALL_64_after_hwframe+0x44/0xae
[ 1093.965495] RIP: 0033:0x7f4102ef44fb
[ 1093.965498] Code: c3 48 8b 0d 8f a9 2c 00 f7 d8 64 89 01 b8 ff ff ff ff eb c2 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa b8 a2 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d a9 2c 00 f7 d8 64 89 01 48
[ 1093.965500] RSP: 002b:00007ffc90169858 EFLAGS: 00000202 ORIG_RAX: 00000000000000a2
[ 1093.965503] RAX: ffffffffffffffda RBX: 0000000000405c5a RCX: 00007f4102ef44fb
[ 1093.965504] RDX: 0000000030ea7f1b RSI: 0000000030ea7f1b RDI: 000000000000000d
[ 1093.965505] RBP: 00007ffc90169870 R08: 00007f41031bf1f4 R09: 00007f41031bf260
[ 1093.965507] R10: 0000000000000003 R11: 0000000000000202 R12: 0000000000401760
[ 1093.965508] R13: 00007ffc90169bb0 R14: 0000000000000000 R15: 0000000000000000
[ 1093.965524] </TASK>
[ 1093.967304] watchdog: BUG: soft lockup - CPU#7 stuck for 26s! [fsstress:89397]
[ 1093.967308] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm iTCO_wdt ipmi_ssif iTCO_vendor_support irqbypass crct10dif_pclmul crc32_pclmul joydev ghash_clmulni_intel i2c_i801 lpc_ich wmi i2c_smbus mfd_core acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter acpi_pad mei_me mei ioatdma ixgbe igb mdio i2c_algo_bit crc32c_intel ptp nvme pps_core dca nvme_core fuse
[ 1093.967383] irq event stamp: 56042
[ 1093.967384] hardirqs last enabled at (56041): [<ffffffff81c00d42>] asm_sysvec_apic_timer_interrupt+0x12/0x20
[ 1093.967388] hardirqs last disabled at (56042): [<ffffffff81ae1e4a>] sysvec_apic_timer_interrupt+0xa/0xc0
[ 1093.967392] softirqs last enabled at (55992): [<ffffffff81e00321>] __do_softirq+0x321/0x483
[ 1093.967395] softirqs last disabled at (55987): [<ffffffff810e4a0c>] irq_exit_rcu+0xdc/0x120
[ 1093.967398] CPU: 7 PID: 89397 Comm: fsstress Tainted: G S W L 5.18.0-rc6-ceph-gb850759e370d #1
[ 1093.967401] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0 12/17/2015
[ 1093.967403] RIP: 0010:queued_spin_lock_slowpath+0x1b3/0x1e0
[ 1093.967408] Code: 83 e0 03 83 e9 01 48 c1 e0 04 48 63 c9 48 05 40 e5 02 00 48 03 04 cd 20 0b 43 82 48 89 10 8b 42 08 85 c0 75 09 f3 90 8b 42 08 <85> c0 74 f7 48 8b 0a 48 85 c9 0f 84 6b ff ff ff 0f 0d 09 e9 63 ff
[ 1093.967410] RSP: 0018:ffffc9000187be38 EFLAGS: 00000246
[ 1093.967414] RAX: 0000000000000000 RBX: ffff88817bbfc4b0 RCX: 0000000000000005
[ 1093.967415] RDX: ffff88885fdee540 RSI: ffffffff823936a5 RDI: ffffffff823a6afe
[ 1093.967417] RBP: ffff88885fdee540 R08: 00000000000052ef R09: 0000000000000000
[ 1093.967419] R10: 0000000000000001 R11: c359abbe014cc9bb R12: 0000000000200000
[ 1093.967421] R13: ffff88817b7d3000 R14: ffff88817b7d3998 R15: ffffffffa089c2f3
[ 1093.967423] FS: 00007f4103475500(0000) GS:ffff88885fdc0000(0000) knlGS:0000000000000000
[ 1093.967426] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1093.967428] CR2: 0000560366ba9320 CR3: 00000001bcad2004 CR4: 00000000003706e0
[ 1093.967430] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1093.967432] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1093.967434] Call Trace:
[ 1093.967435] <TASK>
[ 1093.967440] do_raw_spin_lock+0xb1/0xc0
[ 1093.967446] flush_dirty_session_caps+0x82/0x150 [ceph]
[ 1093.967469] ? __ceph_put_cap_refs+0x440/0x440 [ceph]
[ 1093.967484] ceph_mdsc_iterate_sessions+0x65/0xa0 [ceph]
[ 1093.967506] ? __x64_sys_tee+0xc0/0xc0
[ 1093.967509] ceph_sync_fs+0x9b/0x100 [ceph]
[ 1093.967524] iterate_supers+0x93/0xe0
[ 1093.967532] ksys_sync+0x4f/0xa0
[ 1093.967537] __do_sys_sync+0xa/0x10
[ 1093.967541] do_syscall_64+0x34/0x80
[ 1093.967545] entry_SYSCALL_64_after_hwframe+0x44/0xae
[ 1093.967549] RIP: 0033:0x7f4102ef44fb
[ 1093.967553] Code: c3 48 8b 0d 8f a9 2c 00 f7 d8 64 89 01 b8 ff ff ff ff eb c2 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa b8 a2 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d a9 2c 00 f7 d8 64 89 01 48
[ 1093.967555] RSP: 002b:00007ffc90169858 EFLAGS: 00000202 ORIG_RAX: 00000000000000a2
[ 1093.967558] RAX: ffffffffffffffda RBX: 0000000000405c5a RCX: 00007f4102ef44fb
[ 1093.967560] RDX: 000000003783312f RSI: 000000003783312f RDI: 000000000000000f
[ 1093.967562] RBP: 00007ffc90169870 R08: 00007f41031bf204 R09: 00007f41031bf260
[ 1093.967564] R10: 0000000000000003 R11: 0000000000000202 R12: 0000000000401760
[ 1093.967566] R13: 00007ffc90169bb0 R14: 0000000000000000 R15: 0000000000000000
[ 1093.967587] </TASK>
</pre>
<pre>
[13220.949256] watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [umount:203080]
[13220.956508] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc binfmt_misc overlay xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm iTCO_wdt iTCO_vendor_support irqbypass ipmi_ssif crct10dif_pclmul crc32_pclmul ghash_clmulni_intel joydev wmi mei_me lpc_ich i2c_i801 mei mfd_core i2c_smbus acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe crc32c_intel igb mdio ptp nvme i2c_algo_bit pps_core nvme_core dca fuse
[13221.012949] irq event stamp: 62390
[13221.016438] hardirqs last enabled at (62389): [<ffffffff81c00d42>] asm_sysvec_apic_timer_interrupt+0x12/0x20
[13221.026466] hardirqs last disabled at (62390): [<ffffffff81ae1e4a>] sysvec_apic_timer_interrupt+0xa/0xc0
[13221.036057] softirqs last enabled at (11798): [<ffffffff81e00321>] __do_softirq+0x321/0x483
[13221.044600] softirqs last disabled at (11741): [<ffffffff810e4a0c>] irq_exit_rcu+0xdc/0x120
[13221.053062] CPU: 0 PID: 203080 Comm: umount Tainted: G S 5.18.0-ceph-g342bda2e5287 #1
[13221.062302] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0b 04/10/2017
[13221.069890] RIP: 0010:do_raw_spin_lock+0x3e/0xc0
[13221.074595] Code: 8b 53 10 65 48 8b 04 25 00 be 01 00 48 39 c2 74 4e 8b 53 08 65 8b 05 f1 ef eb 7e 39 c2 74 5a 31 c0 ba 01 00 00 00 f0 0f b1 13 <75> 67 65 8b 05 d9 ef eb 7e 89 43 08 65 48 8b 04 25 00 be 01 00 48
[13221.093501] RSP: 0018:ffffc90001b77c30 EFLAGS: 00000246
[13221.098810] RAX: 0000000000000000 RBX: ffff88817f1b51a0 RCX: 0000000000000000
[13221.106033] RDX: 0000000000000001 RSI: ffffffff8233333d RDI: ffff88817f1b51a0
[13221.113252] RBP: 0000000000000002 R08: 00000000000037ab R09: 0000000000000000
[13221.120474] R10: 0000000000000001 R11: 917bf5a7cab73d6b R12: ffff88810bfb24b0
[13221.127692] R13: ffff888235830000 R14: ffff888235830998 R15: ffffffffa0982353
[13221.134914] FS: 00007f11c96bd080(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000
[13221.143107] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[13221.148939] CR2: 00007f11c7c501f0 CR3: 000000017a60e001 CR4: 00000000001706f0
[13221.156162] Call Trace:
[13221.158689] <TASK>
[13221.160875] ceph_check_caps+0x68/0x9e0 [ceph]
[13221.165427] ? __lock_acquire+0xa25/0x1d80
[13221.169611] ? lock_is_held_type+0xe3/0x140
[13221.173879] ? find_held_lock+0x2d/0x90
[13221.177804] ? flush_dirty_session_caps+0x63/0x150 [ceph]
[13221.183294] ? lock_release+0x13d/0x2b0
[13221.187220] flush_dirty_session_caps+0x72/0x150 [ceph]
[13221.192544] ? __ceph_put_cap_refs+0x440/0x440 [ceph]
[13221.197689] ceph_mdsc_iterate_sessions+0x65/0xa0 [ceph]
[13221.203100] ceph_mdsc_pre_umount+0x52/0x230 [ceph]
[13221.208074] ceph_kill_sb+0x1c/0x90 [ceph]
[13221.212263] deactivate_locked_super+0x29/0x60
[13221.216797] cleanup_mnt+0xb8/0x140
[13221.220375] task_work_run+0x6d/0xb0
[13221.224042] exit_to_user_mode_prepare+0x226/0x230
[13221.228922] syscall_exit_to_user_mode+0x25/0x60
[13221.233627] do_syscall_64+0x40/0x80
[13221.237292] entry_SYSCALL_64_after_hwframe+0x44/0xae
[13221.242435] RIP: 0033:0x7f11c7cfcdfb
[13221.246096] Code: 20 2c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa 31 f6 e9 05 00 00 00 0f 1f 44 00 00 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 20 2c 00 f7 d8 64 89 01 48
[13221.264998] RSP: 002b:00007ffe5921e968 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6
[13221.272672] RAX: 0000000000000000 RBX: 000055d1a94d15d0 RCX: 00007f11c7cfcdfb
[13221.279897] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055d1a94d17b0
[13221.287117] RBP: 0000000000000000 R08: 000055d1a94d17e0 R09: 00007f11c7d80620
[13221.294336] R10: 0000000000000000 R11: 0000000000000246 R12: 000055d1a94d17b0
[13221.301554] R13: 00007f11c9258184 R14: 0000000000000000 R15: 00000000ffffffff
[13221.308785] </TASK>
[13248.949436] watchdog: BUG: soft lockup - CPU#0 stuck for 53s! [umount:203080]
[13248.956669] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc binfmt_misc overlay xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm iTCO_wdt iTCO_vendor_support irqbypass ipmi_ssif crct10dif_pclmul crc32_pclmul ghash_clmulni_intel joydev wmi mei_me lpc_ich i2c_i801 mei mfd_core i2c_smbus acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe crc32c_intel igb mdio ptp nvme i2c_algo_bit pps_core nvme_core dca fuse
[13249.013104] irq event stamp: 118000
[13249.016674] hardirqs last enabled at (117999): [<ffffffff81c00d42>] asm_sysvec_apic_timer_interrupt+0x12/0x20
[13249.026788] hardirqs last disabled at (118000): [<ffffffff81ae1e4a>] sysvec_apic_timer_interrupt+0xa/0xc0
[13249.036460] softirqs last enabled at (11798): [<ffffffff81e00321>] __do_softirq+0x321/0x483
[13249.045001] softirqs last disabled at (11741): [<ffffffff810e4a0c>] irq_exit_rcu+0xdc/0x120
[13249.053465] CPU: 0 PID: 203080 Comm: umount Tainted: G S L 5.18.0-ceph-g342bda2e5287 #1
[13249.062704] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0b 04/10/2017
[13249.070290] RIP: 0010:rcu_is_watching+0x1/0x40
[13249.074824] Code: 92 00 e9 0d ff ff ff 66 90 0f 1f 44 00 00 48 c7 c7 45 e3 40 82 e8 9f ca 96 00 65 8a 05 7c 7d eb 7e 0f be c0 c3 0f 1f 40 00 53 <65> ff 05 98 50 ea 7e e8 63 ca 96 00 48 c7 c3 d0 e6 02 00 89 c0 48
[13249.093730] RSP: 0018:ffffc90001b77d20 EFLAGS: 00000202
[13249.099038] RAX: 0000000000000001 RBX: ffffc90001b77d47 RCX: 0000000000000000
[13249.106264] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffc90001b77d47
[13249.113484] RBP: ffff88810bfb24c8 R08: 0000000000000001 R09: 0000000000000000
[13249.120701] R10: 0000000000000001 R11: 917bf5a7cab73d6b R12: 0000000000000000
[13249.127921] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[13249.135140] FS: 00007f11c96bd080(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000
[13249.143333] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[13249.149167] CR2: 00007f11c7c501f0 CR3: 000000017a60e001 CR4: 00000000001706f0
[13249.156390] Call Trace:
[13249.158920] <TASK>
[13249.161104] rcu_read_lock_held_common+0x21/0x40
[13249.165810] rcu_read_lock_sched_held+0x23/0x90
[13249.170427] lock_acquire+0x19c/0x2f0
[13249.174176] ? flush_dirty_session_caps+0x63/0x150 [ceph]
[13249.179676] ? lock_release+0x13d/0x2b0
[13249.183600] _raw_spin_lock+0x2f/0x40
[13249.187348] ? flush_dirty_session_caps+0x82/0x150 [ceph]
[13249.192841] flush_dirty_session_caps+0x82/0x150 [ceph]
[13249.198167] ? __ceph_put_cap_refs+0x440/0x440 [ceph]
[13249.203319] ceph_mdsc_iterate_sessions+0x65/0xa0 [ceph]
[13249.208728] ceph_mdsc_pre_umount+0x52/0x230 [ceph]
[13249.213701] ceph_kill_sb+0x1c/0x90 [ceph]
[13249.217891] deactivate_locked_super+0x29/0x60
[13249.222418] cleanup_mnt+0xb8/0x140
[13249.225995] task_work_run+0x6d/0xb0
[13249.229663] exit_to_user_mode_prepare+0x226/0x230
[13249.234544] syscall_exit_to_user_mode+0x25/0x60
[13249.239249] do_syscall_64+0x40/0x80
[13249.242914] entry_SYSCALL_64_after_hwframe+0x44/0xae
[13249.248054] RIP: 0033:0x7f11c7cfcdfb
[13249.251718] Code: 20 2c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa 31 f6 e9 05 00 00 00 0f 1f 44 00 00 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 20 2c 00 f7 d8 64 89 01 48
[13249.270615] RSP: 002b:00007ffe5921e968 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6
[13249.278291] RAX: 0000000000000000 RBX: 000055d1a94d15d0 RCX: 00007f11c7cfcdfb
[13249.285517] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055d1a94d17b0
[13249.292736] RBP: 0000000000000000 R08: 000055d1a94d17e0 R09: 00007f11c7d80620
[13249.299955] R10: 0000000000000000 R11: 0000000000000246 R12: 000055d1a94d17b0
[13249.307175] R13: 00007f11c9258184 R14: 0000000000000000 R15: 00000000ffffffff
[13249.314403] </TASK>
[13260.774515] rcu: INFO: rcu_preempt self-detected stall on CPU
[13260.780352] rcu: 0-....: (64270 ticks this GP) idle=75d/1/0x4000000000000000 softirq=685569/685569 fqs=16199
[13260.790851] (t=65015 jiffies g=1444649 q=2645)
[13260.795471] NMI backtrace for cpu 0
[13260.799047] CPU: 0 PID: 203080 Comm: umount Tainted: G S L 5.18.0-ceph-g342bda2e5287 #1
[13260.808286] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0b 04/10/2017
[13260.823595] Call Trace:
[13260.826131] <IRQ>
[13260.828224] dump_stack_lvl+0x55/0x6d
[13260.831974] nmi_cpu_backtrace.cold.4+0x30/0x7c
[13260.836588] ? lapic_can_unplug_cpu+0x70/0x70
[13260.841032] nmi_trigger_cpumask_backtrace+0xc9/0xe0
[13260.846089] rcu_dump_cpu_stacks+0xce/0x150
[13260.850360] rcu_sched_clock_irq.cold.70+0x207/0x419
[13260.855415] ? lock_is_held_type+0xe3/0x140
[13260.859683] ? lock_is_held_type+0xe3/0x140
[13260.863960] ? tick_sched_do_timer+0xa0/0xa0
[13260.868313] update_process_times+0x93/0xc0
[13260.872586] tick_sched_handle.isra.14+0x1f/0x60
[13260.877286] tick_sched_timer+0x6f/0x80
[13260.881210] __hrtimer_run_queues+0x1bb/0x490
[13260.885662] hrtimer_interrupt+0x10a/0x220
[13260.889842] ? lock_is_held_type+0xe3/0x140
[13260.894110] __sysvec_apic_timer_interrupt+0x8a/0x250
[13260.899247] sysvec_apic_timer_interrupt+0x99/0xc0
[13260.904126] </IRQ>
[13260.906311] <TASK>
[13260.908497] asm_sysvec_apic_timer_interrupt+0x12/0x20
[13260.913723] RIP: 0010:debug_smp_processor_id+0xe/0x20
[13260.918861] Code: 00 01 0f 85 66 ff ff ff e9 3a ff ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 c7 c6 b5 37 39 82 48 c7 c7 0e 6c 3a 82 <e9> fd fe ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 fe 48
[13260.937769] RSP: 0018:ffffc90001b77d48 EFLAGS: 00000206
[13260.943083] RAX: 0000000000000001 RBX: ffffc90001b77d77 RCX: 0000000000000000
[13260.950308] RDX: 0000000000000001 RSI: ffffffff823937b5 RDI: ffffffff823a6c0e
[13260.957528] RBP: ffff88810bfb24c8 R08: 0000000000000622 R09: 0000000000000000
[13260.964747] R10: 0000000000000001 R11: 3fd23774e533f8b1 R12: ffffffffa0961443
[13260.971967] R13: ffff888235830000 R14: ffff888235830998 R15: ffffffffa0982353
[13260.979187] ? flush_dirty_session_caps+0x63/0x150 [ceph]
[13260.984694] rcu_is_watching+0xd/0x40
[13260.988441] rcu_read_lock_held_common+0x21/0x40
[13260.993151] rcu_read_lock_sched_held+0x23/0x90
[13260.997770] ? flush_dirty_session_caps+0x63/0x150 [ceph]
[13261.003268] lock_release+0x1f9/0x2b0
[13261.007018] _raw_spin_unlock+0x17/0x40
[13261.010941] flush_dirty_session_caps+0x63/0x150 [ceph]
[13261.016261] ? __ceph_put_cap_refs+0x440/0x440 [ceph]
[13261.021404] ceph_mdsc_iterate_sessions+0x65/0xa0 [ceph]
[13261.026816] ceph_mdsc_pre_umount+0x52/0x230 [ceph]
[13261.031790] ceph_kill_sb+0x1c/0x90 [ceph]
[13261.035982] deactivate_locked_super+0x29/0x60
[13261.040516] cleanup_mnt+0xb8/0x140
[13261.044093] task_work_run+0x6d/0xb0
[13261.047759] exit_to_user_mode_prepare+0x226/0x230
[13261.052639] syscall_exit_to_user_mode+0x25/0x60
[13261.057344] do_syscall_64+0x40/0x80
[13261.061010] entry_SYSCALL_64_after_hwframe+0x44/0xae
[13261.066151] RIP: 0033:0x7f11c7cfcdfb
[13261.069814] Code: 20 2c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa 31 f6 e9 05 00 00 00 0f 1f 44 00 00 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 20 2c 00 f7 d8 64 89 01 48
[13261.088716] RSP: 002b:00007ffe5921e968 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6
[13261.096389] RAX: 0000000000000000 RBX: 000055d1a94d15d0 RCX: 00007f11c7cfcdfb
[13261.103614] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055d1a94d17b0
[13261.110834] RBP: 0000000000000000 R08: 000055d1a94d17e0 R09: 00007f11c7d80620
[13261.118051] R10: 0000000000000000 R11: 0000000000000246 R12: 000055d1a94d17b0
[13261.125274] R13: 00007f11c9258184 R14: 0000000000000000 R15: 00000000ffffffff
[13261.132509] </TASK>
[13284.949670] watchdog: BUG: soft lockup - CPU#0 stuck for 86s! [umount:203080]
[13284.956902] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc binfmt_misc overlay xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm iTCO_wdt iTCO_vendor_support irqbypass ipmi_ssif crct10dif_pclmul crc32_pclmul ghash_clmulni_intel joydev wmi mei_me lpc_ich i2c_i801 mei mfd_core i2c_smbus acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe crc32c_intel igb mdio ptp nvme i2c_algo_bit pps_core nvme_core dca fuse
[13285.013335] irq event stamp: 188948
[13285.016914] hardirqs last enabled at (188947): [<ffffffff81c00d42>] asm_sysvec_apic_timer_interrupt+0x12/0x20
[13285.027034] hardirqs last disabled at (188948): [<ffffffff81ae1e4a>] sysvec_apic_timer_interrupt+0xa/0xc0
[13285.036708] softirqs last enabled at (11798): [<ffffffff81e00321>] __do_softirq+0x321/0x483
[13285.045252] softirqs last disabled at (11741): [<ffffffff810e4a0c>] irq_exit_rcu+0xdc/0x120
[13285.053712] CPU: 0 PID: 203080 Comm: umount Tainted: G S L 5.18.0-ceph-g342bda2e5287 #1
[13285.062952] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0b 04/10/2017
[13285.070538] RIP: 0010:ceph_check_caps+0x0/0x9e0 [ceph]
[13285.075774] Code: 97 50 08 00 00 b8 01 00 00 00 48 39 ca 76 1a 48 8b b7 58 08 00 00 31 c0 48 39 f2 76 0c 48 01 c9 48 01 f2 48 39 d1 0f 93 c0 c3 <0f> 1f 44 00 00 41 57 41 56 41 55 41 54 55 89 f5 53 48 89 fb 48 81
[13285.094680] RSP: 0018:ffffc90001b77dd8 EFLAGS: 00000246
[13285.099989] RAX: 0000000000000000 RBX: ffff88817f1b4d00 RCX: 0000000000000001
[13285.107212] RDX: 0000000000000000 RSI: 0000000000000002 RDI: ffff88817f1b4d00
[13285.114432] RBP: ffff88817f1b53b8 R08: 0000000000000622 R09: 0000000000000000
[13285.121652] R10: 0000000000000001 R11: 3fd23774e533f8b1 R12: ffff88810bfb24b0
[13285.128870] R13: ffff888235830000 R14: ffff888235830998 R15: ffffffffa0982353
[13285.136091] FS: 00007f11c96bd080(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000
[13285.144285] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[13285.150118] CR2: 00007f11c7c501f0 CR3: 000000017a60e001 CR4: 00000000001706f0
[13285.157343] Call Trace:
[13285.159869] <TASK>
[13285.162053] flush_dirty_session_caps+0x72/0x150 [ceph]
[13285.167375] ? __ceph_put_cap_refs+0x440/0x440 [ceph]
[13285.172519] ceph_mdsc_iterate_sessions+0x65/0xa0 [ceph]
[13285.177934] ceph_mdsc_pre_umount+0x52/0x230 [ceph]
[13285.182906] ceph_kill_sb+0x1c/0x90 [ceph]
[13285.187098] deactivate_locked_super+0x29/0x60
[13285.191631] cleanup_mnt+0xb8/0x140
[13285.195208] task_work_run+0x6d/0xb0
[13285.198876] exit_to_user_mode_prepare+0x226/0x230
[13285.203757] syscall_exit_to_user_mode+0x25/0x60
[13285.208460] do_syscall_64+0x40/0x80
[13285.212124] entry_SYSCALL_64_after_hwframe+0x44/0xae
[13285.217267] RIP: 0033:0x7f11c7cfcdfb
[13285.220931] Code: 20 2c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa 31 f6 e9 05 00 00 00 0f 1f 44 00 00 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 20 2c 00 f7 d8 64 89 01 48
[13285.239838] RSP: 002b:00007ffe5921e968 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6
[13285.247513] RAX: 0000000000000000 RBX: 000055d1a94d15d0 RCX: 00007f11c7cfcdfb
[13285.254744] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055d1a94d17b0
[13285.261966] RBP: 0000000000000000 R08: 000055d1a94d17e0 R09: 00007f11c7d80620
[13285.269186] R10: 0000000000000000 R11: 0000000000000246 R12: 000055d1a94d17b0
[13285.276404] R13: 00007f11c9258184 R14: 0000000000000000 R15: 00000000ffffffff
[13285.283633] </TASK>
[13312.949851] watchdog: BUG: soft lockup - CPU#0 stuck for 112s! [umount:203080]
[13312.957189] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc binfmt_misc overlay xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm iTCO_wdt iTCO_vendor_support irqbypass ipmi_ssif crct10dif_pclmul crc32_pclmul ghash_clmulni_intel joydev wmi mei_me lpc_ich i2c_i801 mei mfd_core i2c_smbus acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe crc32c_intel igb mdio ptp nvme i2c_algo_bit pps_core nvme_core dca fuse
[13313.013624] irq event stamp: 244580
[13313.017199] hardirqs last enabled at (244579): [<ffffffff81c00d42>] asm_sysvec_apic_timer_interrupt+0x12/0x20
[13313.027315] hardirqs last disabled at (244580): [<ffffffff81ae1e4a>] sysvec_apic_timer_interrupt+0xa/0xc0
[13313.036994] softirqs last enabled at (11798): [<ffffffff81e00321>] __do_softirq+0x321/0x483
[13313.045535] softirqs last disabled at (11741): [<ffffffff810e4a0c>] irq_exit_rcu+0xdc/0x120
[13313.053996] CPU: 0 PID: 203080 Comm: umount Tainted: G S L 5.18.0-ceph-g342bda2e5287 #1
[13313.063235] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0b 04/10/2017
[13313.070825] RIP: 0010:do_raw_spin_unlock+0x41/0xb0
[13313.075704] Code: 48 65 48 8b 04 25 00 be 01 00 48 39 43 10 75 53 65 8b 05 f2 ee eb 7e 39 43 08 75 61 48 c7 c0 ff ff ff ff 48 89 43 10 89 43 08 <c6> 03 00 5b c3 e8 75 cb 3f 00 85 c0 74 c3 48 c7 c6 a2 0b 35 82 48
[13313.094610] RSP: 0018:ffffc90001b77dc0 EFLAGS: 00000246
[13313.099919] RAX: ffffffffffffffff RBX: ffff88810bfb24b0 RCX: ffffc90001b77d94
[13313.107143] RDX: 0000000000000001 RSI: ffffffff8233333d RDI: ffff88810bfb24b0
[13313.114363] RBP: ffff88817f1b53b8 R08: 0000000000000622 R09: 0000000000000000
[13313.121581] R10: 0000000000000001 R11: 3fd23774e533f8b1 R12: ffff88810bfb24b0
[13313.128800] R13: ffff888235830000 R14: ffff888235830998 R15: ffffffffa0982353
[13313.136021] FS: 00007f11c96bd080(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000
[13313.144213] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[13313.150041] CR2: 00007f11c7c501f0 CR3: 000000017a60e001 CR4: 00000000001706f0
[13313.157263] Call Trace:
[13313.159791] <TASK>
[13313.161975] _raw_spin_unlock+0x1f/0x40
[13313.165898] flush_dirty_session_caps+0x63/0x150 [ceph]
[13313.171224] ? __ceph_put_cap_refs+0x440/0x440 [ceph]
[13313.176369] ceph_mdsc_iterate_sessions+0x65/0xa0 [ceph]
[13313.181780] ceph_mdsc_pre_umount+0x52/0x230 [ceph]
[13313.186753] ceph_kill_sb+0x1c/0x90 [ceph]
[13313.190946] deactivate_locked_super+0x29/0x60
[13313.195477] cleanup_mnt+0xb8/0x140
[13313.199056] task_work_run+0x6d/0xb0
[13313.202722] exit_to_user_mode_prepare+0x226/0x230
[13313.207605] syscall_exit_to_user_mode+0x25/0x60
[13313.212308] do_syscall_64+0x40/0x80
[13313.215973] entry_SYSCALL_64_after_hwframe+0x44/0xae
[13313.221115] RIP: 0033:0x7f11c7cfcdfb
[13313.232528] Code: 20 2c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa 31 f6 e9 05 00 00 00 0f 1f 44 00 00 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 20 2c 00 f7 d8 64 89 01 48
[13313.251435] RSP: 002b:00007ffe5921e968 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6
[13313.259109] RAX: 0000000000000000 RBX: 000055d1a94d15d0 RCX: 00007f11c7cfcdfb
[13313.266333] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055d1a94d17b0
[13313.273553] RBP: 0000000000000000 R08: 000055d1a94d17e0 R09: 00007f11c7d80620
[13313.280772] R10: 0000000000000000 R11: 0000000000000246 R12: 000055d1a94d17b0
[13313.287991] R13: 00007f11c9258184 R14: 0000000000000000 R15: 00000000ffffffff
[13313.295221] </TASK>
[13340.950034] watchdog: BUG: soft lockup - CPU#0 stuck for 138s! [umount:203080]
[13340.957368] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc binfmt_misc overlay xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm iTCO_wdt iTCO_vendor_support irqbypass ipmi_ssif crct10dif_pclmul crc32_pclmul ghash_clmulni_intel joydev wmi mei_me lpc_ich i2c_i801 mei mfd_core i2c_smbus acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe crc32c_intel igb mdio ptp nvme i2c_algo_bit pps_core nvme_core dca fuse
[13341.013803] irq event stamp: 300200
[13341.017371] hardirqs last enabled at (300199): [<ffffffff81c00d42>] asm_sysvec_apic_timer_interrupt+0x12/0x20
[13341.027486] hardirqs last disabled at (300200): [<ffffffff81ae1e4a>] sysvec_apic_timer_interrupt+0xa/0xc0
[13341.037165] softirqs last enabled at (11798): [<ffffffff81e00321>] __do_softirq+0x321/0x483
[13341.045709] softirqs last disabled at (11741): [<ffffffff810e4a0c>] irq_exit_rcu+0xdc/0x120
[13341.054167] CPU: 0 PID: 203080 Comm: umount Tainted: G S L 5.18.0-ceph-g342bda2e5287 #1
[13341.063406] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0b 04/10/2017
[13341.070997] RIP: 0010:rcu_lockdep_current_cpu_online+0xc/0x80
[13341.076831] Code: 01 65 ff 0d 76 50 ea 7e 74 02 5b c3 0f 1f 44 00 00 5b c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 bd 01 00 00 00 53 <65> 8b 05 4d 50 ea 7e a9 00 00 f0 00 75 0a 8b 05 1c c2 72 01 85 c0
[13341.095734] RSP: 0018:ffffc90001b77d48 EFLAGS: 00000202
[13341.101043] RAX: 0000000000000001 RBX: ffffc90001b77d77 RCX: 0000000000000000
[13341.108268] RDX: 0000000000000001 RSI: ffffffff823937b5 RDI: ffffffff823a6c0e
[13341.115487] RBP: 0000000000000001 R08: 0000000000000622 R09: 0000000000000000
[13341.122706] R10: 0000000000000001 R11: 3fd23774e533f8b1 R12: ffffffffa0961443
[13341.129926] R13: ffff888235830000 R14: ffff888235830998 R15: ffffffffa0982353
[13341.137146] FS: 00007f11c96bd080(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000
[13341.145338] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[13341.151164] CR2: 00007f11c7c501f0 CR3: 000000017a60e001 CR4: 00000000001706f0
[13341.158387] Call Trace:
[13341.160916] <TASK>
[13341.163100] rcu_read_lock_held_common+0x2a/0x40
[13341.167806] rcu_read_lock_sched_held+0x23/0x90
[13341.172422] ? flush_dirty_session_caps+0x63/0x150 [ceph]
[13341.177918] lock_release+0x1f9/0x2b0
[13341.181669] _raw_spin_unlock+0x17/0x40
[13341.185596] flush_dirty_session_caps+0x63/0x150 [ceph]
[13341.190920] ? __ceph_put_cap_refs+0x440/0x440 [ceph]
[13341.196066] ceph_mdsc_iterate_sessions+0x65/0xa0 [ceph]
[13341.201478] ceph_mdsc_pre_umount+0x52/0x230 [ceph]
[13341.206460] ceph_kill_sb+0x1c/0x90 [ceph]
[13341.210650] deactivate_locked_super+0x29/0x60
[13341.215182] cleanup_mnt+0xb8/0x140
[13341.218762] task_work_run+0x6d/0xb0
[13341.222428] exit_to_user_mode_prepare+0x226/0x230
[13341.227310] syscall_exit_to_user_mode+0x25/0x60
[13341.232014] do_syscall_64+0x40/0x80
[13341.235677] entry_SYSCALL_64_after_hwframe+0x44/0xae
[13341.240811] RIP: 0033:0x7f11c7cfcdfb
[13341.244474] Code: 20 2c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa 31 f6 e9 05 00 00 00 0f 1f 44 00 00 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 20 2c 00 f7 d8 64 89 01 48
[13341.263375] RSP: 002b:00007ffe5921e968 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6
[13341.271048] RAX: 0000000000000000 RBX: 000055d1a94d15d0 RCX: 00007f11c7cfcdfb
[13341.278264] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055d1a94d17b0
[13341.285486] RBP: 0000000000000000 R08: 000055d1a94d17e0 R09: 00007f11c7d80620
[13341.292704] R10: 0000000000000000 R11: 0000000000000246 R12: 000055d1a94d17b0
[13341.299924] R13: 00007f11c9258184 R14: 0000000000000000 R15: 00000000ffffffff
[13341.307153] </TASK>
[13368.950214] watchdog: BUG: soft lockup - CPU#0 stuck for 164s! [umount:203080]
[13368.957547] Modules linked in: ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc binfmt_misc overlay xfs libcrc32c sunrpc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm iTCO_wdt iTCO_vendor_support irqbypass ipmi_ssif crct10dif_pclmul crc32_pclmul ghash_clmulni_intel joydev wmi mei_me lpc_ich i2c_i801 mei mfd_core i2c_smbus acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler ioatdma acpi_pad acpi_power_meter ixgbe crc32c_intel igb mdio ptp nvme i2c_algo_bit pps_core nvme_core dca fuse
[13369.013981] irq event stamp: 355794
[13369.017551] hardirqs last enabled at (355793): [<ffffffff81c00d42>] asm_sysvec_apic_timer_interrupt+0x12/0x20
[13369.027666] hardirqs last disabled at (355794): [<ffffffff81ae1e4a>] sysvec_apic_timer_interrupt+0xa/0xc0
[13369.037349] softirqs last enabled at (11798): [<ffffffff81e00321>] __do_softirq+0x321/0x483
[13369.045896] softirqs last disabled at (11741): [<ffffffff810e4a0c>] irq_exit_rcu+0xdc/0x120
[13369.054356] CPU: 0 PID: 203080 Comm: umount Tainted: G S L 5.18.0-ceph-g342bda2e5287 #1
[13369.063594] Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0b 04/10/2017
[13369.071183] RIP: 0010:lock_acquire+0x107/0x2f0
[13369.075717] Code: c4 20 e8 ec f3 98 00 b8 ff ff ff ff 65 0f c1 05 d7 84 ec 7e 83 f8 01 0f 85 d3 01 00 00 48 83 3c 24 00 74 06 fb 0f 1f 44 00 00 <48> 8b 44 24 30 65 48 33 04 25 28 00 00 00 0f 85 d0 01 00 00 48 83
[13369.094623] RSP: 0018:ffffc90001b77bb8 EFLAGS: 00000206
[13369.099930] RAX: 0000000000000001 RBX: 0000000000000001 RCX: 0000000000000000
[13369.107155] RDX: 917bf5a7cab73d6b RSI: ffffffff8233333d RDI: ffffffff823a6c1f
[13369.114375] RBP: ffff88817f1b51b8 R08: 00000000000037ab R09: 0000000000000000
[13369.121593] R10: 0000000000000001 R11: 917bf5a7cab73d6b R12: 0000000000000000
[13369.128813] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[13369.136034] FS: 00007f11c96bd080(0000) GS:ffff88885fc00000(0000) knlGS:0000000000000000
[13369.144228] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[13369.150061] CR2: 00007f11c7c501f0 CR3: 000000017a60e001 CR4: 00000000001706f0
[13369.157286] Call Trace:
[13369.159822] <TASK>
[13369.162005] ? ceph_check_caps+0x828/0x9e0 [ceph]
[13369.166804] ? lock_release+0x13d/0x2b0
[13369.170730] _raw_spin_lock+0x2f/0x40
[13369.174480] ? ceph_check_caps+0x68/0x9e0 [ceph]
[13369.179188] ceph_check_caps+0x68/0x9e0 [ceph]
[13369.183734] ? __lock_acquire+0xa25/0x1d80
[13369.187920] ? lock_is_held_type+0xe3/0x140
[13369.192186] ? find_held_lock+0x2d/0x90
[13369.196112] ? flush_dirty_session_caps+0x63/0x150 [ceph]
[13369.201600] ? lock_release+0x13d/0x2b0
[13369.205526] flush_dirty_session_caps+0x72/0x150 [ceph]
[13369.210850] ? __ceph_put_cap_refs+0x440/0x440 [ceph]
[13369.215997] ceph_mdsc_iterate_sessions+0x65/0xa0 [ceph]
[13369.221409] ceph_mdsc_pre_umount+0x52/0x230 [ceph]
[13369.226381] ceph_kill_sb+0x1c/0x90 [ceph]
[13369.230573] deactivate_locked_super+0x29/0x60
[13369.235104] cleanup_mnt+0xb8/0x140
[13369.238675] task_work_run+0x6d/0xb0
[13369.242342] exit_to_user_mode_prepare+0x226/0x230
[13369.247224] syscall_exit_to_user_mode+0x25/0x60
[13369.251929] do_syscall_64+0x40/0x80
[13369.255590] entry_SYSCALL_64_after_hwframe+0x44/0xae
[13369.260726] RIP: 0033:0x7f11c7cfcdfb
[13369.264387] Code: 20 2c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa 31 f6 e9 05 00 00 00 0f 1f 44 00 00 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 20 2c 00 f7 d8 64 89 01 48
[13369.283288] RSP: 002b:00007ffe5921e968 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6
[13369.290961] RAX: 0000000000000000 RBX: 000055d1a94d15d0 RCX: 00007f11c7cfcdfb
[13369.298188] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055d1a94d17b0
[13369.305406] RBP: 0000000000000000 R08: 000055d1a94d17e0 R09: 00007f11c7d80620
[13369.312624] R10: 0000000000000000 R11: 0000000000000246 R12: 000055d1a94d17b0
[13369.319846] R13: 00007f11c9258184 R14: 0000000000000000 R15: 00000000ffffffff
[13369.327073] </TASK>
</pre>
Ceph - Bug #49717 (New): "SELinux denials found" in upgrade:luminous-x-nautilus-distro-basic-gibba
https://tracker.ceph.com/issues/49717
2021-03-10T16:44:57Z
Yuri Weinstein
yweinste@redhat.com
<p>This is for 14.2.17 release</p>
<p>Run: <a class="external" href="https://pulpito.ceph.com/teuthology-2021-03-10_00:07:04-upgrade:luminous-x-nautilus-distro-basic-gibba/">https://pulpito.ceph.com/teuthology-2021-03-10_00:07:04-upgrade:luminous-x-nautilus-distro-basic-gibba/</a><br />Jos: '5951264', '5951252', '5951248'<br />Logs: /a/teuthology-2021-03-10_00:07:04-upgrade:luminous-x-nautilus-distro-basic-gibba/5951248/teuthology.log</p>
<pre>
Failure: SELinux denials found on ubuntu@gibba006.front.sepia.ceph.com: ['type=AVC msg=audit(1615371444.456:8400): avc: denied { getattr } for pid=30173 comm="ntpd" path="/proc/30173/net/if_inet6" dev="proc" ino=4026532164 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1615378344.451:10970): avc: denied { getattr } for pid=30173 comm="ntpd" path="/proc/30173/net/if_inet6" dev="proc" ino=4026532164 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1615375344.477:10117): avc: denied { read } for pid=30173 comm="ntpd" name="if_inet6" dev="proc" ino=4026532164 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1615372944.450:10052): avc: denied { read } for pid=30173 comm="ntpd" name="if_inet6" dev="proc" ino=4026532164 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1615380001.162:12440): avc: denied { read } for pid=77383 comm="sadc" name="dev" dev="proc" ino=4026531982 scontext=system_u:system_r:sysstat_t:s0-s0:c0.c1023 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1615372802.160:10025): avc: denied { getattr } for pid=60640 comm="sadc" path="/proc/60640/net/dev" dev="proc" ino=4026531982 scontext=system_u:system_r:sysstat_t:s0-s0:c0.c1023 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1615382544.480:18256): avc: denied { getattr } for pid=30173 comm="ntpd" path="/proc/30173/net/if_inet6" dev="proc" ino=4026532164 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1615382544.480:18255): avc: denied { open } for pid=30173 comm="ntpd" path="/proc/30173/net/if_inet6" dev="proc" ino=4026532164 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1615371444.455:8399): avc: denied { read } for pid=30173 comm="ntpd" name="if_inet6" dev="proc" ino=4026532164 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1615383001.368:19269): avc: denied { read } for pid=126378 comm="sadc" name="dev" dev="proc" ino=4026531982 scontext=system_u:system_r:sysstat_t:s0-s0:c0.c1023 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1615383001.368:19270): avc: denied { getattr } for pid=126378 comm="sadc" path="/proc/126378/net/dev" dev="proc" ino=4026531982 scontext=system_u:system_r:sysstat_t:s0-s0:c0.c1023 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1615384344.505:22333): avc: denied { getattr } for pid=30173 comm="ntpd" path="/proc/30173/net/if_inet6" dev="proc" ino=4026532164 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1615373401.408:10059): avc: denied { open } for pid=60834 comm="sadc" path="/proc/60834/net/dev" dev="proc" ino=4026531982 scontext=system_u:system_r:sysstat_t:s0-s0:c0.c1023 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1615384344.504:22332): avc: denied { open } for pid=30173 comm="ntpd" path="/proc/30173/net/if_inet6" dev="proc" ino=4026532164 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1615380001.163:12441): avc: denied { getattr } for pid=77383 comm="sadc" path="/proc/77383/net/dev" dev="proc" ino=4026531982 scontext=system_u:system_r:sysstat_t:s0-s0:c0.c1023 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1615380001.162:12440): avc: denied { open } for pid=77383 comm="sadc" path="/proc/77383/net/dev" dev="proc" ino=4026531982 scontext=system_u:system_r:sysstat_t:s0-s0:c0.c1023 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1615375344.477:10118): avc: denied { getattr } for pid=30173 comm="ntpd" path="/proc/30173/net/if_inet6" dev="proc" ino=4026532164 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1615376401.631:10920): avc: denied { getattr } for pid=66824 comm="sadc" path="/proc/66824/net/dev" dev="proc" ino=4026531982 scontext=system_u:system_r:sysstat_t:s0-s0:c0.c1023 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1615378344.451:10969): avc: denied { read } for pid=30173 comm="ntpd" name="if_inet6" dev="proc" ino=4026532164 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1615372944.450:10053): avc: denied { getattr } for pid=30173 comm="ntpd" path="/proc/30173/net/if_inet6" dev="proc" ino=4026532164 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1615373401.409:10060): avc: denied { getattr } for pid=60834 comm="sadc" path="/proc/60834/net/dev" dev="proc" ino=4026531982 scontext=system_u:system_r:sysstat_t:s0-s0:c0.c1023 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1615382544.480:18255): avc: denied { read } for pid=30173 comm="ntpd" name="if_inet6" dev="proc" ino=4026532164 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1615371444.455:8399): avc: denied { open } for pid=30173 comm="ntpd" path="/proc/30173/net/if_inet6" dev="proc" ino=4026532164 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1615372802.160:10024): avc: denied { open } for pid=60640 comm="sadc" path="/proc/60640/net/dev" dev="proc" ino=4026531982 scontext=system_u:system_r:sysstat_t:s0-s0:c0.c1023 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1615372344.481:10017): avc: denied { read } for pid=30173 comm="ntpd" name="if_inet6" dev="proc" ino=4026532164 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1615375344.477:10117): avc: denied { open } for pid=30173 comm="ntpd" path="/proc/30173/net/if_inet6" dev="proc" ino=4026532164 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1615376401.630:10919): avc: denied { read } for pid=66824 comm="sadc" name="dev" dev="proc" ino=4026531982 scontext=system_u:system_r:sysstat_t:s0-s0:c0.c1023 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1615372802.160:10024): avc: denied { read } for pid=60640 comm="sadc" name="dev" dev="proc" ino=4026531982 scontext=system_u:system_r:sysstat_t:s0-s0:c0.c1023 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1615371601.609:8933): avc: denied { read } for pid=50577 comm="sadc" name="dev" dev="proc" ino=4026531982 scontext=system_u:system_r:sysstat_t:s0-s0:c0.c1023 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1615372344.482:10018): avc: denied { getattr } for pid=30173 comm="ntpd" path="/proc/30173/net/if_inet6" dev="proc" ino=4026532164 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1615371601.609:8933): avc: denied { open } for pid=50577 comm="sadc" path="/proc/50577/net/dev" dev="proc" ino=4026531982 scontext=system_u:system_r:sysstat_t:s0-s0:c0.c1023 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1615383001.368:19269): avc: denied { open } for pid=126378 comm="sadc" path="/proc/126378/net/dev" dev="proc" ino=4026531982 scontext=system_u:system_r:sysstat_t:s0-s0:c0.c1023 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1615378344.451:10969): avc: denied { open } for pid=30173 comm="ntpd" path="/proc/30173/net/if_inet6" dev="proc" ino=4026532164 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1615372344.481:10017): avc: denied { open } for pid=30173 comm="ntpd" path="/proc/30173/net/if_inet6" dev="proc" ino=4026532164 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1615371601.610:8934): avc: denied { getattr } for pid=50577 comm="sadc" path="/proc/50577/net/dev" dev="proc" ino=4026531982 scontext=system_u:system_r:sysstat_t:s0-s0:c0.c1023 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1615376401.630:10919): avc: denied { open } for pid=66824 comm="sadc" path="/proc/66824/net/dev" dev="proc" ino=4026531982 scontext=system_u:system_r:sysstat_t:s0-s0:c0.c1023 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1615372944.450:10052): avc: denied { open } for pid=30173 comm="ntpd" path="/proc/30173/net/if_inet6" dev="proc" ino=4026532164 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1615373401.408:10059): avc: denied { read } for pid=60834 comm="sadc" name="dev" dev="proc" ino=4026531982 scontext=system_u:system_r:sysstat_t:s0-s0:c0.c1023 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1615384344.504:22332): avc: denied { read } for pid=30173 comm="ntpd" name="if_inet6" dev="proc" ino=4026532164 scontext=system_u:system_r:ntpd_t:s0 tcontext=system_u:object_r:devtty_t:s0 tclass=file permissive=1']
</pre>
Ceph - Bug #49673 (New): "Failed to send signal 1: None" in upgrade:nautilus-p2p-nautilus
https://tracker.ceph.com/issues/49673
2021-03-09T16:52:19Z
Yuri Weinstein
yweinste@redhat.com
<p>This is for 14.2.17 release</p>
<p>Run: <a class="external" href="https://pulpito.ceph.com/yuriw-2021-03-08_19:14:19-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/">https://pulpito.ceph.com/yuriw-2021-03-08_19:14:19-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/</a><br />Jobs: all<br />Logs: <a class="external" href="http://qa-proxy.ceph.com/teuthology/yuriw-2021-03-08_19:14:19-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5947104/teuthology.log">http://qa-proxy.ceph.com/teuthology/yuriw-2021-03-08_19:14:19-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/5947104/teuthology.log</a></p>
<pre>
2021-03-08T20:45:52.183 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/orchestra/daemon/state.py", line 108, in signal
self.proc.stdin.write(struct.pack('!b', sig))
File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 405, in write
self._write_all(data)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 522, in _write_all
count = self._write(data)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1364, in _write
self.channel.sendall(data)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 846, in sendall
sent = self.send(s)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 801, in send
return self._send(s, m)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1198, in _send
raise socket.error("Socket is closed")
OSError: Socket is closed
2021-03-08T20:45:52.286 INFO:tasks.ceph.osd.1.smithi078.stderr:2021-03-08 20:45:52.280 7f197d065700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 39693) UID: 0
2021-03-08T20:45:52.325 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd unset noscrub
2021-03-08T20:45:52.388 INFO:tasks.ceph.osd.7.smithi190.stderr:2021-03-08 20:45:52.388 7fc2df47b700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 129834) UID: 0
2021-03-08T20:45:52.487 ERROR:teuthology.orchestra.daemon.state:Failed to send signal 1: None
</pre>
CephFS - Bug #43748 (Fix Under Review): client: improve wanted handling so we don't request unuse...
https://tracker.ceph.com/issues/43748
2020-01-21T18:45:23Z
Patrick Donnelly
pdonnell@redhat.com
<p>In an active/standby configuration of two clients managed by file locks, the standby client causes unbuffered I/O on the active client. Additionally, and more importantly, failure of the standby client causes the active client I/O to freeze because the standby client receives Frw caps it cannot release.</p>
<p>This can be reproduced by starting two ceph-fuse clients. Then in the first "active" client, do:</p>
<pre>
flock -x bar1 -c 'date; echo enter; while sleep 1; do date; echo foobar | tee -a bar1; done'
</pre>
<p>Then in the "standby" client:</p>
<pre>
flock -x bar1 -c 'date; echo 2 enter; sleep 30; date; echo 2 bye;'
</pre>
<p>Note: the second client should never print output because it cannot obtain the exclusive lock on bar1.</p>
<p>Now, send SIGKILL to the standby ceph-fuse client. This will cause I/O to halt for the first client until the MDS autocloses the "standby" ceph-fuse client session after 60s.</p>
<p>Here's the log from the standby client:</p>
<pre>
2020-01-21T17:44:33.248+0000 7f1d0b7fe700 20 client.4278 trim_cache size 2 max 16384
2020-01-21T17:44:33.343+0000 7f1d0a7fc700 1 -- 127.0.0.1:0/361750000 <== mds.0 v2:127.0.0.1:6854/1480111679 45 ==== client_caps(revoke ino 0x10000000001 11 seq 34 caps=pAsLsXs dirty=- wanted=pFscr follows 0 size 294/0 ts 1/18446744073709551615 mtime 2020-01-21T17:44:32.297260+0000) v11 ==== 252+0+0 (crc 0 0 0) 0x7f1d1c00a1e0 con 0x557c4d018dc0
2020-01-21T17:44:33.343+0000 7f1d0a7fc700 10 client.4278 mds.0 seq now 31
2020-01-21T17:44:33.343+0000 7f1d0a7fc700 5 client.4278 handle_cap_grant on in 0x10000000001 mds.0 seq 34 caps now pAsLsXs was pAsLsXsFr
2020-01-21T17:44:33.343+0000 7f1d0a7fc700 10 client.4278 update_inode_file_time 0x10000000001.head(faked_ino=0 ref=4 ll_ref=2 cap_refs={} open={1=1} mode=100640 size=287/0 nlink=1 btime=0.000000 mtime=2020-01-21T17:44:31.232141+0000 ctime=2020-01-21T17:44:31.232141+0000 caps=pAsLsXsFr(0=pAsLsXsFr) objectset[0x10000000001 ts 0/0 objects 0 dirty_or_tx 0] parents=0x10000000000.head["bar1"] 0x7f1cf4005a10) pAsLsXsFr ctime 2020-01-21T17:44:32.297260+0000 mtime 2020-01-21T17:44:32.297260+0000
2020-01-21T17:44:33.343+0000 7f1d0a7fc700 10 client.4278 revocation of Fr
2020-01-21T17:44:33.343+0000 7f1d0a7fc700 10 client.4278 check_caps on 0x10000000001.head(faked_ino=0 ref=4 ll_ref=2 cap_refs={} open={1=1} mode=100640 size=287/0 nlink=1 btime=0.000000 mtime=2020-01-21T17:44:32.297260+0000 ctime=2020-01-21T17:44:32.297260+0000 caps=pAsLsXs(0=pAsLsXs) objectset[0x10000000001 ts 0/0 objects 0 dirty_or_tx 0] parents=0x10000000000.head["bar1"] 0x7f1cf4005a10) wanted pFscr used - issued pAsLsXs revoking Fr flags=0
2020-01-21T17:44:33.343+0000 7f1d0a7fc700 10 client.4278 cap mds.0 issued pAsLsXs implemented pAsLsXsFr revoking Fr
2020-01-21T17:44:33.343+0000 7f1d0a7fc700 10 client.4278 completed revocation of Fr
2020-01-21T17:44:33.343+0000 7f1d0a7fc700 10 client.4278 send_cap 0x10000000001.head(faked_ino=0 ref=4 ll_ref=2 cap_refs={} open={1=1} mode=100640 size=287/0 nlink=1 btime=0.000000 mtime=2020-01-21T17:44:32.297260+0000 ctime=2020-01-21T17:44:32.297260+0000 caps=pAsLsXs(0=pAsLsXs) objectset[0x10000000001 ts 0/0 objects 0 dirty_or_tx 0] parents=0x10000000000.head["bar1"] 0x7f1cf4005a10) mds.0 seq 34 used - want pFscr flush - retain pAsxLsxXsxFsxcwbl held pAsLsXsFr revoking Fr dropping -
2020-01-21T17:44:33.343+0000 7f1d0a7fc700 15 client.4278 auth cap, setting max_size = 0
2020-01-21T17:44:33.343+0000 7f1d0a7fc700 1 -- 127.0.0.1:0/361750000 --> [v2:127.0.0.1:6854/1480111679,v1:127.0.0.1:6855/1480111679] -- client_caps(update ino 0x10000000001 11 seq 34 caps=pAsLsXs dirty=- wanted=pFscr follows 0 size 287/0 ts 1/18446744073709551615 mtime 2020-01-21T17:44:32.297260+0000) v11 -- 0x7f1d1c009850 con 0x557c4d018dc0
2020-01-21T17:44:33.343+0000 7f1d0a7fc700 1 -- 127.0.0.1:0/361750000 <== mds.0 v2:127.0.0.1:6854/1480111679 46 ==== client_caps(grant ino 0x10000000001 11 seq 35 caps=pAsLsXsFc dirty=- wanted=pFscr follows 0 size 294/0 ts 1/18446744073709551615 mtime 2020-01-21T17:44:32.297260+0000) v11 ==== 252+0+0 (crc 0 0 0) 0x7f1d1c00a1e0 con 0x557c4d018dc0
2020-01-21T17:44:33.343+0000 7f1d0a7fc700 10 client.4278 mds.0 seq now 32
2020-01-21T17:44:33.343+0000 7f1d0a7fc700 5 client.4278 handle_cap_grant on in 0x10000000001 mds.0 seq 35 caps now pAsLsXsFc was pAsLsXs
2020-01-21T17:44:33.343+0000 7f1d0a7fc700 10 client.4278 update_inode_file_time 0x10000000001.head(faked_ino=0 ref=4 ll_ref=2 cap_refs={} open={1=1} mode=100640 size=287/0 nlink=1 btime=0.000000 mtime=2020-01-21T17:44:32.297260+0000 ctime=2020-01-21T17:44:32.297260+0000 caps=pAsLsXs(0=pAsLsXs) objectset[0x10000000001 ts 0/0 objects 0 dirty_or_tx 0] parents=0x10000000000.head["bar1"] 0x7f1cf4005a10) pAsLsXs ctime 2020-01-21T17:44:32.297260+0000 mtime 2020-01-21T17:44:32.297260+0000
2020-01-21T17:44:33.343+0000 7f1d0a7fc700 10 client.4278 size 287 -> 294
2020-01-21T17:44:33.343+0000 7f1d0a7fc700 10 client.4278 grant, new caps are Fc
2020-01-21T17:44:33.347+0000 7f1d0a7fc700 1 -- 127.0.0.1:0/361750000 <== mds.0 v2:127.0.0.1:6854/1480111679 47 ==== client_caps(revoke ino 0x10000000001 11 seq 36 caps=pAsLsXs dirty=- wanted=pFscr follows 0 size 294/0 ts 1/18446744073709551615 mtime 2020-01-21T17:44:32.297260+0000) v11 ==== 252+0+0 (crc 0 0 0) 0x7f1d1c009850 con 0x557c4d018dc0
2020-01-21T17:44:33.347+0000 7f1d0a7fc700 10 client.4278 mds.0 seq now 33
2020-01-21T17:44:33.347+0000 7f1d0a7fc700 5 client.4278 handle_cap_grant on in 0x10000000001 mds.0 seq 36 caps now pAsLsXs was pAsLsXsFc
2020-01-21T17:44:33.347+0000 7f1d0a7fc700 10 client.4278 update_inode_file_time 0x10000000001.head(faked_ino=0 ref=4 ll_ref=2 cap_refs={} open={1=1} mode=100640 size=294/0 nlink=1 btime=0.000000 mtime=2020-01-21T17:44:32.297260+0000 ctime=2020-01-21T17:44:32.297260+0000 caps=pAsLsXsFc(0=pAsLsXsFc) objectset[0x10000000001 ts 0/0 objects 0 dirty_or_tx 0] parents=0x10000000000.head["bar1"] 0x7f1cf4005a10) pAsLsXsFc ctime 2020-01-21T17:44:32.297260+0000 mtime 2020-01-21T17:44:32.297260+0000
2020-01-21T17:44:33.347+0000 7f1d0a7fc700 10 client.4278 revocation of Fc
2020-01-21T17:44:33.347+0000 7f1d0a7fc700 10 client.4278 check_caps on 0x10000000001.head(faked_ino=0 ref=4 ll_ref=2 cap_refs={} open={1=1} mode=100640 size=294/0 nlink=1 btime=0.000000 mtime=2020-01-21T17:44:32.297260+0000 ctime=2020-01-21T17:44:32.297260+0000 caps=pAsLsXs(0=pAsLsXs) objectset[0x10000000001 ts 0/0 objects 0 dirty_or_tx 0] parents=0x10000000000.head["bar1"] 0x7f1cf4005a10) wanted pFscr used - issued pAsLsXs revoking Fc flags=0
2020-01-21T17:44:33.347+0000 7f1d0a7fc700 10 client.4278 cap mds.0 issued pAsLsXs implemented pAsLsXsFc revoking Fc
2020-01-21T17:44:33.347+0000 7f1d0a7fc700 10 client.4278 completed revocation of Fc
2020-01-21T17:44:33.347+0000 7f1d0a7fc700 10 client.4278 send_cap 0x10000000001.head(faked_ino=0 ref=4 ll_ref=2 cap_refs={} open={1=1} mode=100640 size=294/0 nlink=1 btime=0.000000 mtime=2020-01-21T17:44:32.297260+0000 ctime=2020-01-21T17:44:32.297260+0000 caps=pAsLsXs(0=pAsLsXs) objectset[0x10000000001 ts 0/0 objects 0 dirty_or_tx 0] parents=0x10000000000.head["bar1"] 0x7f1cf4005a10) mds.0 seq 36 used - want pFscr flush - retain pAsxLsxXsxFsxrwbl held pAsLsXsFc revoking Fc dropping -
2020-01-21T17:44:33.347+0000 7f1d0a7fc700 15 client.4278 auth cap, setting max_size = 0
2020-01-21T17:44:33.347+0000 7f1d0a7fc700 1 -- 127.0.0.1:0/361750000 --> [v2:127.0.0.1:6854/1480111679,v1:127.0.0.1:6855/1480111679] -- client_caps(update ino 0x10000000001 11 seq 36 caps=pAsLsXs dirty=- wanted=pFscr follows 0 size 294/0 ts 1/18446744073709551615 mtime 2020-01-21T17:44:32.297260+0000) v11 -- 0x7f1d1c009dd0 con 0x557c4d018dc0
2020-01-21T17:44:33.349+0000 7f1d0a7fc700 1 -- 127.0.0.1:0/361750000 <== mds.0 v2:127.0.0.1:6854/1480111679 48 ==== client_caps(grant ino 0x10000000001 11 seq 37 caps=pAsLsXsFrw dirty=- wanted=pFscr follows 0 size 294/0 ts 1/18446744073709551615 mtime 2020-01-21T17:44:32.297260+0000) v11 ==== 252+0+0 (crc 0 0 0) 0x7f1d1c009dd0 con 0x557c4d018dc0
2020-01-21T17:44:33.349+0000 7f1d0a7fc700 10 client.4278 mds.0 seq now 34
2020-01-21T17:44:33.349+0000 7f1d0a7fc700 5 client.4278 handle_cap_grant on in 0x10000000001 mds.0 seq 37 caps now pAsLsXsFrw was pAsLsXs
2020-01-21T17:44:33.349+0000 7f1d0a7fc700 10 client.4278 update_inode_file_time 0x10000000001.head(faked_ino=0 ref=4 ll_ref=2 cap_refs={} open={1=1} mode=100640 size=294/0 nlink=1 btime=0.000000 mtime=2020-01-21T17:44:32.297260+0000 ctime=2020-01-21T17:44:32.297260+0000 caps=pAsLsXs(0=pAsLsXs) objectset[0x10000000001 ts 0/0 objects 0 dirty_or_tx 0] parents=0x10000000000.head["bar1"] 0x7f1cf4005a10) pAsLsXs ctime 2020-01-21T17:44:32.297260+0000 mtime 2020-01-21T17:44:32.297260+0000
2020-01-21T17:44:33.349+0000 7f1d0a7fc700 10 client.4278 grant, new caps are Frw
2020-01-21T17:44:33.350+0000 7f1d0a7fc700 1 -- 127.0.0.1:0/361750000 <== mds.0 v2:127.0.0.1:6854/1480111679 49 ==== client_caps(revoke ino 0x10000000001 11 seq 38 caps=pAsLsXsFr dirty=- wanted=pFscr follows 0 size 294/0 ts 1/18446744073709551615 mtime 2020-01-21T17:44:32.297260+0000) v11 ==== 252+0+0 (crc 0 0 0) 0x7f1d1c004260 con 0x557c4d018dc0
2020-01-21T17:44:33.350+0000 7f1d0a7fc700 10 client.4278 mds.0 seq now 35
2020-01-21T17:44:33.350+0000 7f1d0a7fc700 5 client.4278 handle_cap_grant on in 0x10000000001 mds.0 seq 38 caps now pAsLsXsFr was pAsLsXsFrw
2020-01-21T17:44:33.350+0000 7f1d0a7fc700 10 client.4278 update_inode_file_time 0x10000000001.head(faked_ino=0 ref=4 ll_ref=2 cap_refs={} open={1=1} mode=100640 size=294/0 nlink=1 btime=0.000000 mtime=2020-01-21T17:44:32.297260+0000 ctime=2020-01-21T17:44:32.297260+0000 caps=pAsLsXsFrw(0=pAsLsXsFrw) objectset[0x10000000001 ts 0/0 objects 0 dirty_or_tx 0] parents=0x10000000000.head["bar1"] 0x7f1cf4005a10) pAsLsXsFrw ctime 2020-01-21T17:44:32.297260+0000 mtime 2020-01-21T17:44:32.297260+0000
2020-01-21T17:44:33.350+0000 7f1d0a7fc700 10 client.4278 revocation of Fw
2020-01-21T17:44:33.350+0000 7f1d0a7fc700 10 client.4278 check_caps on 0x10000000001.head(faked_ino=0 ref=4 ll_ref=2 cap_refs={} open={1=1} mode=100640 size=294/0 nlink=1 btime=0.000000 mtime=2020-01-21T17:44:32.297260+0000 ctime=2020-01-21T17:44:32.297260+0000 caps=pAsLsXsFr(0=pAsLsXsFr) objectset[0x10000000001 ts 0/0 objects 0 dirty_or_tx 0] parents=0x10000000000.head["bar1"] 0x7f1cf4005a10) wanted pFscr used - issued pAsLsXsFr revoking Fw flags=0
2020-01-21T17:44:33.350+0000 7f1d0a7fc700 10 client.4278 cap mds.0 issued pAsLsXsFr implemented pAsLsXsFrw revoking Fw
2020-01-21T17:44:33.350+0000 7f1d0a7fc700 10 client.4278 completed revocation of Fw
2020-01-21T17:44:33.350+0000 7f1d0a7fc700 10 client.4278 send_cap 0x10000000001.head(faked_ino=0 ref=4 ll_ref=2 cap_refs={} open={1=1} mode=100640 size=294/0 nlink=1 btime=0.000000 mtime=2020-01-21T17:44:32.297260+0000 ctime=2020-01-21T17:44:32.297260+0000 caps=pAsLsXsFr(0=pAsLsXsFr) objectset[0x10000000001 ts 0/0 objects 0 dirty_or_tx 0] parents=0x10000000000.head["bar1"] 0x7f1cf4005a10) mds.0 seq 38 used - want pFscr flush - retain pAsxLsxXsxFsxcrbl held pAsLsXsFrw revoking Fw dropping -
2020-01-21T17:44:33.350+0000 7f1d0a7fc700 15 client.4278 auth cap, setting max_size = 0
2020-01-21T17:44:33.350+0000 7f1d0a7fc700 1 -- 127.0.0.1:0/361750000 --> [v2:127.0.0.1:6854/1480111679,v1:127.0.0.1:6855/1480111679] -- client_caps(update ino 0x10000000001 11 seq 38 caps=pAsLsXsFr dirty=- wanted=pFscr follows 0 size 294/0 ts 1/18446744073709551615 mtime 2020-01-21T17:44:32.297260+0000) v11 -- 0x7f1d1c009dd0 con 0x557c4d018dc0
</pre>
<p>There's a lot of back-and-forth on the caps but the highlight here to me is "wanted=pFscr". The standby client first notes it wants those caps as part of opening the file just prior to setfilelock, on the MDS:</p>
<pre>
2020-01-21T17:44:25.829+0000 7f651abf0700 7 mds.0.server dispatch_client_request client_request(client.4278:7 open #0x10000000001 2020-01-21T17:44:25.829383+0000 caller_uid=1156, caller_gid=1156{1001,1156,}) v4
...
2020-01-21T17:44:25.830+0000 7f651abf0700 20 mds.0.locker client.4278 pending pAsLsXs allowed pAsLsXs wanted pFscr
2020-01-21T17:44:25.830+0000 7f651abf0700 10 mds.0.locker eval done
2020-01-21T17:44:25.830+0000 7f651abf0700 12 mds.0.server open issued caps pAsLsXs for client.4278 on [inode 0x10000000001 [2,head] /foo/bar1 auth v96 ap=1 dirtyparent s=252 n(v0 rc2020-01-21T17:44:25.531606+0000 b252 1=1+0) (ifile excl->sync) (iversion lock) cr={4274=0-4194304@1} caps={4274=pAsLsXsFscr/pAsLsXsFsxcrwb/pFscr@9,4278=pAsLsXs/pFscr@3},l=4274(-1) | ptrwaiter=0 request=1 lock=1 caps=1 dirtyparent=1 dirty=1 waiter=0 authpin=1 0x55f2eda8c800]
2020-01-21T17:44:25.830+0000 7f651abf0700 20 mds.0.bal hit_dir 0 pop is 1.99945, frag * size 1 [pop IRD:[C 2.00e+00] IWR:[C 9.31e+00] RDR:[C 0.00e+00] FET:[C 0.00e+00] STR:[C 0.00e+00] *LOAD:20.6]
2020-01-21T17:44:25.830+0000 7f651abf0700 7 mds.0.server reply_client_request 0 ((0) Success) client_request(client.4278:7 open #0x10000000001 2020-01-21T17:44:25.829383+0000 caller_uid=1156, caller_gid=1156{1001,1156,}) v4
...
2020-01-21T17:44:25.833+0000 7f651abf0700 1 -- [v2:127.0.0.1:6854/1480111679,v1:127.0.0.1:6855/1480111679] <== client.4278 127.0.0.1:0/361750000 16 ==== client_request(client.4278:8 setfilelock rule 2, type 2, owner 17160986462880593306, pid 0, start 0, length 0, wait 1 #0x10000000001 2020-01-21T17:44:25.832830+0000 caller_uid=1156, caller_gid=1156{1001,1156,}) v4 ==== 152+0+0 (crc 0 0 0) 0x55f2edb8ec00 con 0x55f2edb31680
</pre>
<p>So, I see two issues here:</p>
<p>- The client is saying it "wants" Fscr with no internal requests actually requiring those caps. In general, it's no problem for the client to start by asking for those caps as part of opening a file for the first time (indeed, it's an optimization!) but in this case it's hurting performance for the lifetime of the client. Question: does the kernel client behave the same way?</p>
<p>- The MDS is issuing caps based on "wanted" to the standby client blocking on setfilelock. I'm not sure if this is easily fixed in a way that doesn't break other things. I think ideally the MDS would ignore "wanted" for clients which do not hold the exclusive lock.</p>
CephFS - Bug #36389 (New): untar encounters unexpected EPERM on kclient/multimds cluster with thr...
https://tracker.ceph.com/issues/36389
2018-10-10T20:23:51Z
Patrick Donnelly
pdonnell@redhat.com
<pre>
2018-10-10T11:22:58.275 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e-s has gained rank=0, replacing gid=4215
2018-10-10T11:22:58.275 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.b
2018-10-10T11:22:59.800 INFO:tasks.workunit.client.0.smithi177.stderr:tar: ltp-full-20091231/testcases/open_hpi_testsuite/scripts/test: Cannot mkdir: Permission denied
2018-10-10T11:22:59.804 INFO:tasks.workunit.client.0.smithi177.stderr:tar: ltp-full-20091231/testcases/open_hpi_testsuite/scripts/test/generate_index.pl: Cannot open: Permission denied
2018-10-10T11:23:00.811 INFO:tasks.workunit.client.0.smithi177.stderr:tar: ltp-full-20091231/testcases/open_hpi_testsuite/scripts/test/coverage_report.pl: Cannot open: Permission denied
2018-10-10T11:23:00.811 INFO:tasks.workunit.client.0.smithi177.stderr:tar: ltp-full-20091231/testcases/open_hpi_testsuite/scripts/test/testconformance.pl: Cannot open: Permission denied
2018-10-10T11:23:00.812 INFO:tasks.workunit.client.0.smithi177.stderr:tar: ltp-full-20091231/testcases/open_hpi_testsuite/scripts/test/Makefile.in: Cannot open: Permission denied
2018-10-10T11:23:00.812 INFO:tasks.workunit.client.0.smithi177.stderr:tar: ltp-full-20091231/testcases/open_hpi_testsuite/scripts/test/testcoverage.pl: Cannot open: Permission denied
2018-10-10T11:23:00.812 INFO:tasks.workunit.client.0.smithi177.stderr:tar: ltp-full-20091231/testcases/open_hpi_testsuite/scripts/test/gsum2html.pl: Cannot open: Permission denied
2018-10-10T11:23:00.812 INFO:tasks.workunit.client.0.smithi177.stderr:tar: ltp-full-20091231/testcases/open_hpi_testsuite/scripts/test/conformance_report.pl: Cannot open: Permission denied
2018-10-10T11:23:00.812 INFO:tasks.workunit.client.0.smithi177.stderr:tar: ltp-full-20091231/testcases/open_hpi_testsuite/scripts/test/Makefile.am: Cannot open: Permission denied
2018-10-10T11:23:00.813 INFO:tasks.workunit.client.0.smithi177.stderr:tar: ltp-full-20091231/testcases/open_hpi_testsuite/scripts/test/gcov2html.pl: Cannot open: Permission denied
2018-10-10T11:23:00.813 INFO:tasks.workunit.client.0.smithi177.stderr:tar: ltp-full-20091231/testcases/open_hpi_testsuite/scripts/Makefile.am: Cannot open: Permission denied
2018-10-10T11:23:00.813 INFO:tasks.workunit.client.0.smithi177.stderr:tar: ltp-full-20091231/testcases/open_hpi_testsuite/scripts: Cannot utime: Permission denied
2018-10-10T11:23:00.813 INFO:tasks.workunit.client.0.smithi177.stderr:tar: ltp-full-20091231/testcases/open_hpi_testsuite/plugins: Cannot mkdir: Permission denied
2018-10-10T11:23:00.813 INFO:tasks.workunit.client.0.smithi177.stderr:tar: ltp-full-20091231/testcases/open_hpi_testsuite/plugins/ipmi: Cannot mkdir: Permission denied
2018-10-10T11:23:00.813 INFO:tasks.workunit.client.0.smithi177.stderr:tar: ltp-full-20091231/testcases/open_hpi_testsuite/plugins/ipmi/ipmi_entity_event.c: Cannot open: Permission denied
2018-10-10T11:23:00.813 INFO:tasks.workunit.client.0.smithi177.stderr:tar: ltp-full-20091231/testcases/open_hpi_testsuite/plugins/ipmi/atca_vshm_rdrs.c: Cannot open: Permission denied
2018-10-10T11:23:00.813 INFO:tasks.workunit.client.0.smithi177.stderr:tar: ltp-full-20091231/testcases/open_hpi_testsuite/plugins/ipmi/ipmi_sensor_event.c: Cannot open: Permission denied
2018-10-10T11:23:00.813 INFO:tasks.workunit.client.0.smithi177.stderr:tar: ltp-full-20091231/testcases/open_hpi_testsuite/plugins/ipmi/ipmi_close.c: Cannot open: Permission denied
2018-10-10T11:23:00.814 INFO:tasks.workunit.client.0.smithi177.stderr:tar: ltp-full-20091231/testcases/open_hpi_testsuite/plugins/ipmi/hotswap.c: Cannot open: Permission denied
2018-10-10T11:23:00.814 INFO:tasks.workunit.client.0.smithi177.stderr:tar: ltp-full-20091231/testcases/open_hpi_testsuite/plugins/ipmi/ipmi_control_event.c: Cannot open: Permission denied
2018-10-10T11:23:00.814 INFO:tasks.workunit.client.0.smithi177.stderr:tar: ltp-full-20091231/testcases/open_hpi_testsuite/plugins/ipmi/ipmi_connection.c: Cannot open: Permission denied
2018-10-10T11:23:00.814 INFO:tasks.workunit.client.0.smithi177.stderr:tar: ltp-full-20091231/testcases/open_hpi_testsuite/plugins/ipmi/ekeyfru.h: Cannot open: Permission denied
2018-10-10T11:23:00.814 INFO:tasks.workunit.client.0.smithi177.stderr:tar: ltp-full-20091231/testcases/open_hpi_testsuite/plugins/ipmi/ipmi_controls.c: Cannot open: Permission denied
2018-10-10T11:23:00.814 INFO:tasks.workunit.client.0.smithi177.stderr:
2018-10-10T11:23:00.814 INFO:tasks.workunit.client.0.smithi177.stderr:gzip: stdin: Permission denied
2018-10-10T11:23:00.815 INFO:tasks.workunit.client.0.smithi177.stderr:tar: Unexpected EOF in archive
2018-10-10T11:23:00.815 INFO:tasks.workunit.client.0.smithi177.stderr:tar: ltp-full-20091231/testcases/open_hpi_testsuite: Cannot utime: Permission denied
2018-10-10T11:23:00.815 INFO:tasks.workunit.client.0.smithi177.stderr:tar: ltp-full-20091231/testcases: Cannot utime: Permission denied
2018-10-10T11:23:00.815 INFO:tasks.workunit.client.0.smithi177.stderr:tar: ltp-full-20091231: Cannot utime: Permission denied
2018-10-10T11:23:00.815 INFO:tasks.workunit.client.0.smithi177.stderr:tar: Error is not recoverable: exiting now
2018-10-10T11:23:00.815 DEBUG:teuthology.orchestra.run:got remote process result: 2
2018-10-10T11:23:00.815 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0...
2018-10-10T11:23:00.816 INFO:teuthology.orchestra.run.smithi177:Running: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0'
2018-10-10T11:23:00.999 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-pdonnell-testing/teuthology/run_tasks.py", line 86, in run_tasks
manager = run_one_task(taskname, ctx=ctx, config=config)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-pdonnell-testing/teuthology/run_tasks.py", line 65, in run_one_task
return task(**kwargs)
File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20181008.224656/qa/tasks/workunit.py", line 136, in task
cleanup=cleanup)
File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20181008.224656/qa/tasks/workunit.py", line 286, in _spawn_on_all_clients
timeout=timeout)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-pdonnell-testing/teuthology/parallel.py", line 85, in __exit__
for result in self:
File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-pdonnell-testing/teuthology/parallel.py", line 99, in next
resurrect_traceback(result)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-pdonnell-testing/teuthology/parallel.py", line 22, in capture_traceback
return func(*args, **kwargs)
File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20181008.224656/qa/tasks/workunit.py", line 409, in _run_tests
label="workunit test {workunit}".format(workunit=workunit)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-pdonnell-testing/teuthology/orchestra/remote.py", line 193, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-pdonnell-testing/teuthology/orchestra/run.py", line 429, in run
r.wait()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-pdonnell-testing/teuthology/orchestra/run.py", line 161, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_wip-pdonnell-testing/teuthology/orchestra/run.py", line 183, in _raise_for_status
node=self.hostname, label=self.label
CommandFailedError: Command failed (workunit test suites/fsstress.sh) on smithi177 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=wip-pdonnell-testing-20181008.224656 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'
</pre>
<p>From: /ceph/teuthology-archive/pdonnell-2018-10-09_01:07:48-multimds-wip-pdonnell-testing-20181008.224656-distro-basic-smithi/3119273/teuthology.log</p>
<p>Branch: <a class="external" href="https://github.com/ceph/ceph-ci/tree/wip-pdonnell-testing-20181008.224656">https://github.com/ceph/ceph-ci/tree/wip-pdonnell-testing-20181008.224656</a></p>
CephFS - Bug #26969 (Need More Info): kclient: mount unexpectedly gets osdmap updates causing tes...
https://tracker.ceph.com/issues/26969
2018-08-20T22:07:52Z
Patrick Donnelly
pdonnell@redhat.com
<pre>
2018-08-19T12:25:21.152 INFO:tasks.cephfs_test_runner:======================================================================
2018-08-19T12:25:21.152 INFO:tasks.cephfs_test_runner:FAIL: test_barrier (tasks.cephfs.test_full.TestQuotaFull)
2018-08-19T12:25:21.152 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2018-08-19T12:25:21.152 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2018-08-19T12:25:21.152 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20180817.222213/qa/tasks/cephfs/test_full.py", line 116, in test_barrier
2018-08-19T12:25:21.152 INFO:tasks.cephfs_test_runner: self.assertEqual(mount_a_epoch, mount_a_initial_epoch)
2018-08-19T12:25:21.152 INFO:tasks.cephfs_test_runner:AssertionError: 102 != 99
</pre>
<p>I think we could just remove this part of the test.</p>
Ceph-deploy - Bug #22675 (Need More Info): cleanup ceph-deploy docs preflight documentation
https://tracker.ceph.com/issues/22675
2018-01-12T21:02:05Z
Vasu Kulkarni
vasu.kulkarni@gmail.com
<p>1) Cleanup all the unnecessary manual edits of repo mentioned in pre-flight docs<br /> <a class="external" href="http://docs.ceph.com/docs/master/start/quick-start-preflight/#ceph-deploy-setup">http://docs.ceph.com/docs/master/start/quick-start-preflight/#ceph-deploy-setup</a></p>
<pre><code>one can use --stable=jewel or --stable=luminous option from ceph-deploy</code></pre>
<p>2) edit selinux section, run in enforcing mode</p>
<p>3) Do we need yum-plugin-priorities? seems unnecessary</p>
<p>4) In the 'osd create', explain how to use lvm's</p>
CephFS - Bug #17847 (New): "Fuse mount failed to populate /sys/ after 31 seconds" in jewel 10.2.4
https://tracker.ceph.com/issues/17847
2016-11-09T16:39:03Z
Yuri Weinstein
yweinste@redhat.com
<p>Run: <a class="external" href="http://pulpito.front.sepia.ceph.com/yuriw-2016-11-09_15:03:11-upgrade:hammer-x-jewel-distro-basic-vps/">http://pulpito.front.sepia.ceph.com/yuriw-2016-11-09_15:03:11-upgrade:hammer-x-jewel-distro-basic-vps/</a><br />Job: 534539<br />Logs: <a class="external" href="http://qa-proxy.ceph.com/teuthology/yuriw-2016-11-09_15:03:11-upgrade:hammer-x-jewel-distro-basic-vps/534539/teuthology.log">http://qa-proxy.ceph.com/teuthology/yuriw-2016-11-09_15:03:11-upgrade:hammer-x-jewel-distro-basic-vps/534539/teuthology.log</a></p>
<pre>
016-11-09T15:34:25.496 INFO:teuthology.task.sequential:In sequential, running task print...
2016-11-09T15:34:25.496 INFO:teuthology.task.print:**** done ceph.healthy
2016-11-09T15:34:25.496 ERROR:teuthology.parallel:Exception in parallel execution
Traceback (most recent call last):
File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 83, in __exit__
for result in self:
File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 101, in next
resurrect_traceback(result)
File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 19, in capture_traceback
return func(*args, **kwargs)
File "/home/teuthworker/src/teuthology_master/teuthology/task/parallel.py", line 63, in _run_spawned
mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config)
File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 65, in run_one_task
return task(**kwargs)
File "/home/teuthworker/src/teuthology_master/teuthology/task/full_sequential.py", line 30, in task
mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg)
File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 65, in run_one_task
return task(**kwargs)
File "/home/teuthworker/src/teuthology_master/teuthology/task/sequential.py", line 48, in task
mgr.__enter__()
File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/home/teuthworker/src/ceph-qa-suite_wip-17734-jewel/tasks/ceph_fuse.py", line 126, in task
mount.mount()
File "/home/teuthworker/src/ceph-qa-suite_wip-17734-jewel/tasks/cephfs/fuse_mount.py", line 123, in mount
waited
RuntimeError: Fuse mount failed to populate /sys/ after 31 seconds
</pre>