Project

General

Profile

Actions
Copy link

Bug #45342

closed

qa/tasks/vstart_runner.py: RuntimeError: Fuse mount failed to populate /sys/ after 31 seconds

Added by Xiubo Li almost 4 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Xiubo Li
Category:
-
Target version:
Ceph - v16.0.0
% Done:

0%

Source:
Development
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
qa-suite
Labels (FS):
qa
Pull request ID:
34842
Crash signature (v1):
Crash signature (v2):

Description

2020-04-29 19:44:16,687.687 INFO:__main__:Running ['sudo', 'bash', '-c', 'nmcli connection up brx.0']
2020-04-29 19:44:16,974.974 INFO:__main__:Running ['mkdir', '-p', '/tmp/tmpp1sn2sjz/mnt.0']
2020-04-29 19:44:17,009.009 INFO:__main__:Running ['mount', '-t', 'fusectl', '/sys/fs/fuse/connections', '/sys/fs/fuse/connections']
mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy
2020-04-29 19:44:17,042.042 INFO:__main__:Running ['ls', '/sys/fs/fuse/connections']
2020-04-29 19:44:17,062.062 INFO:__main__:Pre-mount connections: [100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 125, 126, 127, 45, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99]
2020-04-29 19:44:17,063.063 INFO:__main__:Running ['nsenter', '--net=/var/run/netns/ceph-ns--tmp-tmpp1sn2sjz-mnt.0', './bin/ceph-fuse', '-f', '--name', 'client.0', '/tmp/tmpp1sn2sjz/mnt.0']
2020-04-29 19:44:17,084.084 INFO:__main__:Mounting client.0 with pid 16148
2020-04-29 19:44:17,085.085 INFO:__main__:Running ['mount', '-t', 'fusectl', '/sys/fs/fuse/connections', '/sys/fs/fuse/connections']
mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy
2020-04-29 19:44:17,119.119 INFO:__main__:Running ['ls', '/sys/fs/fuse/connections']
2020-04-29 19:44:18,156.156 INFO:__main__:Running ['mount', '-t', 'fusectl', '/sys/fs/fuse/connections', '/sys/fs/fuse/connections']
mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy
2020-04-29 19:44:18,193.193 INFO:__main__:Running ['ls', '/sys/fs/fuse/connections']
2020-04-29 19:44:19,227.227 INFO:__main__:Running ['mount', '-t', 'fusectl', '/sys/fs/fuse/connections', '/sys/fs/fuse/connections']
mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy
2020-04-29 19:44:19,263.263 INFO:__main__:Running ['ls', '/sys/fs/fuse/connections']
2020-04-29 19:44:20,296.296 INFO:__main__:Running ['mount', '-t', 'fusectl', '/sys/fs/fuse/connections', '/sys/fs/fuse/connections']
mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy
2020-04-29 19:44:20,335.335 INFO:__main__:Running ['ls', '/sys/fs/fuse/connections']
2020-04-29 19:44:21,360.360 INFO:__main__:Running ['mount', '-t', 'fusectl', '/sys/fs/fuse/connections', '/sys/fs/fuse/connections']
mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy
2020-04-29 19:44:21,388.388 INFO:__main__:Running ['ls', '/sys/fs/fuse/connections']
2020-04-29 19:44:22,411.411 INFO:__main__:Running ['mount', '-t', 'fusectl', '/sys/fs/fuse/connections', '/sys/fs/fuse/connections']
mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy
2020-04-29 19:44:22,438.438 INFO:__main__:Running ['ls', '/sys/fs/fuse/connections']
2020-04-29 19:44:23,461.461 INFO:__main__:Running ['mount', '-t', 'fusectl', '/sys/fs/fuse/connections', '/sys/fs/fuse/connections']
mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy
2020-04-29 19:44:23,495.495 INFO:__main__:Running ['ls', '/sys/fs/fuse/connections']
2020-04-29 19:44:24,522.522 INFO:__main__:Running ['mount', '-t', 'fusectl', '/sys/fs/fuse/connections', '/sys/fs/fuse/connections']
mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy
2020-04-29 19:44:24,551.551 INFO:__main__:Running ['ls', '/sys/fs/fuse/connections']
2020-04-29 19:44:25,572.572 INFO:__main__:Running ['mount', '-t', 'fusectl', '/sys/fs/fuse/connections', '/sys/fs/fuse/connections']
mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy
2020-04-29 19:44:25,606.606 INFO:__main__:Running ['ls', '/sys/fs/fuse/connections']
2020-04-29 19:44:26,630.630 INFO:__main__:Running ['mount', '-t', 'fusectl', '/sys/fs/fuse/connections', '/sys/fs/fuse/connections']
mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy
2020-04-29 19:44:26,668.668 INFO:__main__:Running ['ls', '/sys/fs/fuse/connections']
2020-04-29 19:44:27,695.695 INFO:__main__:Running ['mount', '-t', 'fusectl', '/sys/fs/fuse/connections', '/sys/fs/fuse/connections']
mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy
2020-04-29 19:44:27,733.733 INFO:__main__:Running ['ls', '/sys/fs/fuse/connections']
2020-04-29 19:44:28,762.762 INFO:__main__:Running ['mount', '-t', 'fusectl', '/sys/fs/fuse/connections', '/sys/fs/fuse/connections']
mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy
2020-04-29 19:44:28,799.799 INFO:__main__:Running ['ls', '/sys/fs/fuse/connections']
2020-04-29 19:44:29,829.829 INFO:__main__:Running ['mount', '-t', 'fusectl', '/sys/fs/fuse/connections', '/sys/fs/fuse/connections']
mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy
2020-04-29 19:44:29,859.859 INFO:__main__:Running ['ls', '/sys/fs/fuse/connections']
2020-04-29 19:44:30,884.884 INFO:__main__:Running ['mount', '-t', 'fusectl', '/sys/fs/fuse/connections', '/sys/fs/fuse/connections']
mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy
2020-04-29 19:44:30,926.926 INFO:__main__:Running ['ls', '/sys/fs/fuse/connections']
2020-04-29 19:44:31,961.961 INFO:__main__:Running ['mount', '-t', 'fusectl', '/sys/fs/fuse/connections', '/sys/fs/fuse/connections']
mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy
2020-04-29 19:44:31,997.997 INFO:__main__:Running ['ls', '/sys/fs/fuse/connections']
2020-04-29 19:44:33,025.025 INFO:__main__:Running ['mount', '-t', 'fusectl', '/sys/fs/fuse/connections', '/sys/fs/fuse/connections']
mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy
2020-04-29 19:44:33,061.061 INFO:__main__:Running ['ls', '/sys/fs/fuse/connections']
2020-04-29 19:44:34,093.093 INFO:__main__:Running ['mount', '-t', 'fusectl', '/sys/fs/fuse/connections', '/sys/fs/fuse/connections']
mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy
2020-04-29 19:44:34,137.137 INFO:__main__:Running ['ls', '/sys/fs/fuse/connections']
2020-04-29 19:44:35,162.162 INFO:__main__:Running ['mount', '-t', 'fusectl', '/sys/fs/fuse/connections', '/sys/fs/fuse/connections']
mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy
2020-04-29 19:44:35,199.199 INFO:__main__:Running ['ls', '/sys/fs/fuse/connections']
2020-04-29 19:44:36,226.226 INFO:__main__:Running ['mount', '-t', 'fusectl', '/sys/fs/fuse/connections', '/sys/fs/fuse/connections']
mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy
2020-04-29 19:44:36,261.261 INFO:__main__:Running ['ls', '/sys/fs/fuse/connections']
2020-04-29 19:44:37,289.289 INFO:__main__:Running ['mount', '-t', 'fusectl', '/sys/fs/fuse/connections', '/sys/fs/fuse/connections']
mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy
2020-04-29 19:44:37,326.326 INFO:__main__:Running ['ls', '/sys/fs/fuse/connections']
2020-04-29 19:44:38,370.370 INFO:__main__:Running ['mount', '-t', 'fusectl', '/sys/fs/fuse/connections', '/sys/fs/fuse/connections']
mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy
2020-04-29 19:44:38,399.399 INFO:__main__:Running ['ls', '/sys/fs/fuse/connections']
2020-04-29 19:44:39,426.426 INFO:__main__:Running ['mount', '-t', 'fusectl', '/sys/fs/fuse/connections', '/sys/fs/fuse/connections']
mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy
2020-04-29 19:44:39,461.461 INFO:__main__:Running ['ls', '/sys/fs/fuse/connections']
2020-04-29 19:44:40,489.489 INFO:__main__:Running ['mount', '-t', 'fusectl', '/sys/fs/fuse/connections', '/sys/fs/fuse/connections']
mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy
2020-04-29 19:44:40,529.529 INFO:__main__:Running ['ls', '/sys/fs/fuse/connections']
2020-04-29 19:44:41,555.555 INFO:__main__:Running ['mount', '-t', 'fusectl', '/sys/fs/fuse/connections', '/sys/fs/fuse/connections']
mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy
2020-04-29 19:44:41,595.595 INFO:__main__:Running ['ls', '/sys/fs/fuse/connections']
2020-04-29 19:44:42,626.626 INFO:__main__:Running ['mount', '-t', 'fusectl', '/sys/fs/fuse/connections', '/sys/fs/fuse/connections']
mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy
2020-04-29 19:44:42,662.662 INFO:__main__:Running ['ls', '/sys/fs/fuse/connections']
2020-04-29 19:44:43,690.690 INFO:__main__:Running ['mount', '-t', 'fusectl', '/sys/fs/fuse/connections', '/sys/fs/fuse/connections']
mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy
2020-04-29 19:44:43,726.726 INFO:__main__:Running ['ls', '/sys/fs/fuse/connections']
2020-04-29 19:44:44,754.754 INFO:__main__:Running ['mount', '-t', 'fusectl', '/sys/fs/fuse/connections', '/sys/fs/fuse/connections']
mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy
2020-04-29 19:44:44,789.789 INFO:__main__:Running ['ls', '/sys/fs/fuse/connections']
2020-04-29 19:44:45,816.816 INFO:__main__:Running ['mount', '-t', 'fusectl', '/sys/fs/fuse/connections', '/sys/fs/fuse/connections']
mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy
2020-04-29 19:44:45,853.853 INFO:__main__:Running ['ls', '/sys/fs/fuse/connections']
2020-04-29 19:44:46,874.874 INFO:__main__:Running ['mount', '-t', 'fusectl', '/sys/fs/fuse/connections', '/sys/fs/fuse/connections']
mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy
2020-04-29 19:44:46,913.913 INFO:__main__:Running ['ls', '/sys/fs/fuse/connections']
2020-04-29 19:44:47,936.936 INFO:__main__:Running ['mount', '-t', 'fusectl', '/sys/fs/fuse/connections', '/sys/fs/fuse/connections']
mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy
2020-04-29 19:44:47,971.971 INFO:__main__:Running ['ls', '/sys/fs/fuse/connections']
2020-04-29 19:44:48,993.993 INFO:__main__:Running ['mount', '-t', 'fusectl', '/sys/fs/fuse/connections', '/sys/fs/fuse/connections']
mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy
2020-04-29 19:44:49,031.031 INFO:__main__:Running ['ls', '/sys/fs/fuse/connections']
2020-04-29 19:44:50,068.068 INFO:__main__:test_readonly_authorization (tasks.cephfs.test_volume_client.TestVolumeClient) ... ERROR
2020-04-29 19:44:50,069.069 INFO:__main__:Stopped test: test_readonly_authorization (tasks.cephfs.test_volume_client.TestVolumeClient) in 6||     __init__ [LocalMDSCluster]  | 740         # /sys/fs/fuse/connections
2.418003s
2020-04-29 19:44:50,070.070 INFO:__main__:
2020-04-29 19:44:50,070.070 INFO:__main__:======================================================================
2020-04-29 19:44:50,071.071 INFO:__main__:ERROR: test_readonly_authorization (tasks.cephfs.test_volume_client.TestVolumeClient)
2020-04-29 19:44:50,071.071 INFO:__main__:----------------------------------------------------------------------
2020-04-29 19:44:50,071.071 INFO:__main__:Traceback (most recent call last):
2020-04-29 19:44:50,072.072 INFO:__main__:  File "/data/ceph/qa/tasks/cephfs/test_volume_client.py", line 19, in setUp
2020-04-29 19:44:50,072.072 INFO:__main__:    CephFSTestCase.setUp(self)
2020-04-29 19:44:50,073.073 INFO:__main__:  File "/data/ceph/qa/tasks/cephfs/cephfs_test_case.py", line 145, in setUp
2020-04-29 19:44:50,073.073 INFO:__main__:    self.mounts[i].mount_wait()
2020-04-29 19:44:50,074.074 INFO:__main__:  File "/data/ceph/qa/tasks/cephfs/mount.py", line 420, in mount_wait
2020-04-29 19:44:50,074.074 INFO:__main__:    mount_options=mount_options)
2020-04-29 19:44:50,074.074 INFO:__main__:  File "../qa/tasks/vstart_runner.py", line 780, in mount
2020-04-29 19:44:50,075.075 INFO:__main__:    waited
2020-04-29 19:44:50,075.075 INFO:__main__:RuntimeError: Fuse mount failed to populate /sys/ after 31 seconds
2020-04-29 19:44:50,075.075 INFO:__main__:
2020-04-29 19:44:50,075.075 INFO:__main__:----------------------------------------------------------------------
2020-04-29 19:44:50,076.076 INFO:__main__:Ran 1 test in 62.419s
2020-04-29 19:44:50,076.076 INFO:__main__:
2020-04-29 19:44:50,076.076 INFO:__main__:FAILED (errors=1)
2020-04-29 19:44:50,076.076 INFO:__main__:
2020-04-29 19:44:50,076.076 INFO:__main__:======================================================================
2020-04-29 19:44:50,076.076 INFO:__main__:ERROR: test_readonly_authorization (tasks.cephfs.test_volume_client.TestVolumeClient)
2020-04-29 19:44:50,076.076 INFO:__main__:----------------------------------------------------------------------
2020-04-29 19:44:50,076.076 INFO:__main__:Traceback (most recent call last):
2020-04-29 19:44:50,077.077 INFO:__main__:  File "/data/ceph/qa/tasks/cephfs/test_volume_client.py", line 19, in setUp
2020-04-29 19:44:50,077.077 INFO:__main__:    CephFSTestCase.setUp(self)
2020-04-29 19:44:50,077.077 INFO:__main__:  File "/data/ceph/qa/tasks/cephfs/cephfs_test_case.py", line 145, in setUp
2020-04-29 19:44:50,077.077 INFO:__main__:    self.mounts[i].mount_wait()
2020-04-29 19:44:50,077.077 INFO:__main__:  File "/data/ceph/qa/tasks/cephfs/mount.py", line 420, in mount_wait
2020-04-29 19:44:50,077.077 INFO:__main__:    mount_options=mount_options)
2020-04-29 19:44:50,077.077 INFO:__main__:  File "../qa/tasks/vstart_runner.py", line 780, in mount
2020-04-29 19:44:50,077.077 INFO:__main__:    waited
2020-04-29 19:44:50,077.077 INFO:__main__:RuntimeError: Fuse mount failed to populate /sys/ after 31 seconds
2020-04-29 19:44:50,077.077 INFO:__main__:
Actions
Copy link
 #1

Updated by Xiubo Li almost 4 years ago

  • Status changed from New to In Progress

If the -l option is specified in :

# MDS=3 MON=3 MGR=1 OSD=3 ../src/vstart.sh -n -l

The ceph cluster will use the localhost, which in the network namespace container the ceph-fuse daemon couldn't connect to it.

Actions
Copy link
 #2

Updated by Xiubo Li almost 4 years ago

  • Pull request ID set to 34842
Actions
Copy link
 #3

Updated by Xiubo Li almost 4 years ago

  • Status changed from In Progress to Fix Under Review
Actions
Copy link
 #4

Updated by Patrick Donnelly almost 4 years ago

  • Target version set to v16.0.0
  • Source set to Development
  • ceph-qa-suite deleted (fs)
  • Component(FS) qa-suite added
Actions
Copy link
 #5

Updated by Xiubo Li almost 4 years ago

Rebased and revolved the conflicts.

Actions
Copy link
 #6

Updated by Patrick Donnelly over 3 years ago

  • Status changed from Fix Under Review to Resolved
Actions
Copy link

Also available in: Atom PDF