Actions
Bug #45145
closedqa/test_full: failed to open 'large_file_a': No space left on device
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
fs
Component(FS):
qa-suite
Labels (FS):
qa, qa-failure
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
2020-04-20T03:49:15.369 INFO:tasks.cephfs.test_full:Writing 373MB should fill this cluster
2020-04-20T03:49:15.369 INFO:teuthology.orchestra.run.smithi102:> true
2020-04-20T03:49:15.386 INFO:teuthology.orchestra.run:Running command with timeout 900
2020-04-20T03:49:15.386 INFO:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T
2020-04-20T03:49:15.386 INFO:teuthology.orchestra.run.smithi102:> ' -- /home/ubuntu/cephtest/mnt.0)
2020-04-20T03:49:15.446 INFO:teuthology.orchestra.run.smithi102.stdout:fuseblk
2020-04-20T03:49:15.446 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0
2020-04-20T03:49:15.446 INFO:teuthology.orchestra.run.smithi102:> true
2020-04-20T03:49:15.496 INFO:teuthology.orchestra.run.smithi102:> cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=large_file_a bs=1M conv=fdatasync count=186 seek=0
2020-04-20T03:49:15.565 INFO:teuthology.orchestra.run.smithi102.stderr:dd: failed to open 'large_file_a': No space left on device
2020-04-20T03:49:15.566 DEBUG:teuthology.orchestra.run:got remote process result: 1
2020-04-20T03:49:15.570 INFO:tasks.cephfs_test_runner:test_full_same_file (tasks.cephfs.test_full.TestClusterFull) ... ERROR
2020-04-20T03:49:15.570 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules...
2020-04-20T03:49:15.570 INFO:teuthology.orchestra.run.smithi100:> true
2020-04-20T03:49:15.588 INFO:teuthology.orchestra.run.smithi100:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore'
2020-04-20T03:49:15.659 INFO:teuthology.orchestra.run.smithi102:> true
2020-04-20T03:49:15.673 INFO:teuthology.orchestra.run.smithi102:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore'
2020-04-20T03:49:15.744 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules.
2020-04-20T03:49:15.744 INFO:teuthology.orchestra.run.smithi102:> true
2020-04-20T03:49:15.797 INFO:teuthology.orchestra.run:Running command with timeout 900
2020-04-20T03:49:15.798 INFO:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T
2020-04-20T03:49:15.798 INFO:teuthology.orchestra.run.smithi102:> ' -- /home/ubuntu/cephtest/mnt.0)
2020-04-20T03:49:15.856 INFO:teuthology.orchestra.run.smithi102.stdout:fuseblk
2020-04-20T03:49:15.856 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0
2020-04-20T03:49:15.856 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi102.front.sepia.ceph.com...
2020-04-20T03:49:15.856 INFO:teuthology.orchestra.run.smithi102:> true
2020-04-20T03:49:15.906 INFO:teuthology.orchestra.run:Running command with timeout 1800
2020-04-20T03:49:15.907 INFO:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0)
2020-04-20T03:49:15.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi102.stderr:ceph-fuse[58521]: fuse finished with error 0 and tester_r 0
2020-04-20T03:49:15.983 INFO:teuthology.orchestra.run.smithi102:> true
2020-04-20T03:49:16.035 INFO:teuthology.orchestra.run:Running command with timeout 900
2020-04-20T03:49:16.035 INFO:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T
2020-04-20T03:49:16.035 INFO:teuthology.orchestra.run.smithi102:> ' -- /home/ubuntu/cephtest/mnt.0)
2020-04-20T03:49:16.095 INFO:teuthology.orchestra.run.smithi102.stdout:ext2/ext3
2020-04-20T03:49:16.095 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3'
2020-04-20T03:49:16.137 INFO:teuthology.orchestra.run.smithi102:> true
2020-04-20T03:49:16.150 INFO:teuthology.orchestra.run:Running command with timeout 300
2020-04-20T03:49:16.150 INFO:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.0)
2020-04-20T03:49:16.210 INFO:teuthology.orchestra.run.smithi102:> true
2020-04-20T03:49:16.260 INFO:teuthology.orchestra.run:Running command with timeout 900
2020-04-20T03:49:16.261 INFO:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T
2020-04-20T03:49:16.261 INFO:teuthology.orchestra.run.smithi102:> ' -- /home/ubuntu/cephtest/mnt.1)
2020-04-20T03:49:16.316 INFO:teuthology.orchestra.run.smithi102.stdout:fuseblk
2020-04-20T03:49:16.317 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1
2020-04-20T03:49:16.317 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi102.front.sepia.ceph.com...
2020-04-20T03:49:16.317 INFO:teuthology.orchestra.run.smithi102:> true
2020-04-20T03:49:16.376 INFO:teuthology.orchestra.run:Running command with timeout 1800
2020-04-20T03:49:16.376 INFO:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1)
2020-04-20T03:49:16.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi102.stderr:ceph-fuse[59253]: fuse finished with error 0 and tester_r 0
2020-04-20T03:49:16.461 INFO:teuthology.orchestra.run.smithi102:> true
2020-04-20T03:49:16.506 INFO:teuthology.orchestra.run:Running command with timeout 900
2020-04-20T03:49:16.506 INFO:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T
2020-04-20T03:49:16.506 INFO:teuthology.orchestra.run.smithi102:> ' -- /home/ubuntu/cephtest/mnt.1)
2020-04-20T03:49:16.578 INFO:teuthology.orchestra.run.smithi102.stdout:ext2/ext3
2020-04-20T03:49:16.579 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3'
2020-04-20T03:49:16.581 INFO:teuthology.orchestra.run.smithi102:> true
2020-04-20T03:49:16.623 INFO:teuthology.orchestra.run:Running command with timeout 300
2020-04-20T03:49:16.623 INFO:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.1)
2020-04-20T03:49:16.685 INFO:teuthology.orchestra.run.smithi100:> true
2020-04-20T03:49:16.701 INFO:teuthology.orchestra.run:Running command with timeout 900
2020-04-20T03:49:16.701 INFO:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T
2020-04-20T03:49:16.702 INFO:teuthology.orchestra.run.smithi100:> ' -- /home/ubuntu/cephtest/mnt.2)
2020-04-20T03:49:16.759 DEBUG:teuthology.orchestra.run:got remote process result: 1
2020-04-20T03:49:16.763 INFO:teuthology.orchestra.run.smithi100.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory
2020-04-20T03:49:16.764 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2
2020-04-20T03:49:16.764 INFO:teuthology.orchestra.run.smithi100:> true
2020-04-20T03:49:16.814 INFO:teuthology.orchestra.run:Running command with timeout 300
2020-04-20T03:49:16.815 INFO:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.2)
2020-04-20T03:49:16.881 INFO:teuthology.orchestra.run.smithi100:> true
2020-04-20T03:49:16.929 INFO:teuthology.orchestra.run:Running command with timeout 900
2020-04-20T03:49:16.929 INFO:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T
2020-04-20T03:49:16.929 INFO:teuthology.orchestra.run.smithi100:> ' -- /home/ubuntu/cephtest/mnt.3)
2020-04-20T03:49:16.985 DEBUG:teuthology.orchestra.run:got remote process result: 1
2020-04-20T03:49:16.987 INFO:teuthology.orchestra.run.smithi100.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.3': No such file or directory
2020-04-20T03:49:16.987 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.3
2020-04-20T03:49:16.988 INFO:teuthology.orchestra.run.smithi100:> true
2020-04-20T03:49:17.040 INFO:teuthology.orchestra.run:Running command with timeout 300
2020-04-20T03:49:17.041 INFO:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec rm -rf /home/ubuntu/cephtest/mnt.3)
2020-04-20T03:49:17.101 INFO:teuthology.orchestra.run.smithi102:> true
2020-04-20T03:49:17.115 INFO:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early log 'Ended test tasks.cephfs.test_full.TestClusterFull.test_full_same_file'
2020-04-20T03:49:17.264 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.260+0000 7fe3b0c9a700 1 Processor -- start
2020-04-20T03:49:17.264 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.260+0000 7fe3b0c9a700 1 -- start start
2020-04-20T03:49:17.272 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.261+0000 7fe3b0c9a700 1 --2- >> v2:172.21.15.100:3300/0 conn(0x7fe3ac1441a0 0x7fe3ac146d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect
2020-04-20T03:49:17.272 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.261+0000 7fe3b0c9a700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3ac0af790 con 0x7fe3ac1471e0
2020-04-20T03:49:17.272 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.261+0000 7fe3b0c9a700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3ac12c700 con 0x7fe3ac0b2c30
2020-04-20T03:49:17.272 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.261+0000 7fe3b0c9a700 1 -- --> v2:172.21.15.100:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3ac12c9a0 con 0x7fe3ac1441a0
2020-04-20T03:49:17.273 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.261+0000 7fe3ab7fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe3ac0b2c30 0x7fe3ac0b5710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60274/0 (socket says 172.21.15.102:60274)
2020-04-20T03:49:17.273 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.261+0000 7fe3ab7fe700 1 -- 172.21.15.102:0/735168108 learned_addr learned my addr 172.21.15.102:0/735168108 (peer_addr_for_me v1:172.21.15.102:0/0)
2020-04-20T03:49:17.273 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.261+0000 7fe3aa7fc700 1 -- 172.21.15.102:0/735168108 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2263373645 0 0) 0x7fe3ac12c700 con 0x7fe3ac0b2c30
2020-04-20T03:49:17.273 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.261+0000 7fe3aa7fc700 1 -- 172.21.15.102:0/735168108 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe398003230 con 0x7fe3ac0b2c30
2020-04-20T03:49:17.274 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.261+0000 7fe3aaffd700 1 --2- 172.21.15.102:0/735168108 >> v2:172.21.15.100:3300/0 conn(0x7fe3ac1441a0 0x7fe3ac146d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0
2020-04-20T03:49:17.282 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.261+0000 7fe3aa7fc700 1 -- 172.21.15.102:0/735168108 <== mon.1 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1908025047 0 0) 0x7fe398003230 con 0x7fe3ac0b2c30
2020-04-20T03:49:17.282 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.261+0000 7fe3aa7fc700 1 -- 172.21.15.102:0/735168108 >> v2:172.21.15.100:3300/0 conn(0x7fe3ac1441a0 msgr2=0x7fe3ac146d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down
2020-04-20T03:49:17.282 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.261+0000 7fe3aa7fc700 1 --2- 172.21.15.102:0/735168108 >> v2:172.21.15.100:3300/0 conn(0x7fe3ac1441a0 0x7fe3ac146d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop
2020-04-20T03:49:17.282 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.261+0000 7fe3aa7fc700 1 -- 172.21.15.102:0/735168108 >> v1:172.21.15.100:6789/0 conn(0x7fe3ac1471e0 legacy=0x7fe3ac147630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down
2020-04-20T03:49:17.282 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.261+0000 7fe3aa7fc700 1 -- 172.21.15.102:0/735168108 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3ac12cc40 con 0x7fe3ac0b2c30
2020-04-20T03:49:17.283 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.261+0000 7fe3aa7fc700 1 -- 172.21.15.102:0/735168108 <== mon.1 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1136816987 0 0) 0x7fe39c001710 con 0x7fe3ac0b2c30
2020-04-20T03:49:17.283 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.261+0000 7fe3aa7fc700 1 -- 172.21.15.102:0/735168108 >> v1:172.21.15.102:6789/0 conn(0x7fe3ac0b2c30 legacy=0x7fe3ac0b5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down
2020-04-20T03:49:17.283 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.261+0000 7fe3aa7fc700 1 --2- 172.21.15.102:0/735168108 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe398003720 0x7fe398003b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect
2020-04-20T03:49:17.283 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.261+0000 7fe3aa7fc700 1 --2- 172.21.15.102:0/735168108 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3980044d0 0x7fe398004950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect
2020-04-20T03:49:17.283 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.261+0000 7fe3aa7fc700 1 --2- 172.21.15.102:0/735168108 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe398004db0 0x7fe39800b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect
2020-04-20T03:49:17.283 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.261+0000 7fe3aa7fc700 1 -- 172.21.15.102:0/735168108 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ac12c9a0 con 0x7fe398003720
2020-04-20T03:49:17.284 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.261+0000 7fe3aa7fc700 1 -- 172.21.15.102:0/735168108 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3980041c0 con 0x7fe3980044d0
2020-04-20T03:49:17.284 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.261+0000 7fe3aa7fc700 1 -- 172.21.15.102:0/735168108 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe39800bb80 con 0x7fe398004db0
2020-04-20T03:49:17.285 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.262+0000 7fe3aaffd700 1 --2- 172.21.15.102:0/735168108 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3980044d0 0x7fe398004950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0
2020-04-20T03:49:17.285 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.262+0000 7fe3ab7fe700 1 --2- 172.21.15.102:0/735168108 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe398003720 0x7fe398003b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0
2020-04-20T03:49:17.285 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.262+0000 7fe3abfff700 1 --2- 172.21.15.102:0/735168108 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe398004db0 0x7fe39800b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0
2020-04-20T03:49:17.285 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.262+0000 7fe3aaffd700 1 -- 172.21.15.102:0/735168108 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe398004db0 msgr2=0x7fe39800b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down
2020-04-20T03:49:17.285 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.262+0000 7fe3aaffd700 1 --2- 172.21.15.102:0/735168108 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe398004db0 0x7fe39800b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop
2020-04-20T03:49:17.286 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.262+0000 7fe3aaffd700 1 -- 172.21.15.102:0/735168108 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe398003720 msgr2=0x7fe398003b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down
2020-04-20T03:49:17.286 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.262+0000 7fe3aaffd700 1 --2- 172.21.15.102:0/735168108 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe398003720 0x7fe398003b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop
2020-04-20T03:49:17.286 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.262+0000 7fe3abfff700 1 -- 172.21.15.102:0/735168108 reap_dead start
2020-04-20T03:49:17.286 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.262+0000 7fe3aaffd700 1 -- 172.21.15.102:0/735168108 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe398003230 con 0x7fe3980044d0
2020-04-20T03:49:17.287 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.262+0000 7fe3aaffd700 1 --2- 172.21.15.102:0/735168108 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3980044d0 0x7fe398004950 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe3980054a0 tx=0x7fe3a0010050).handle_read_frame_epilogue_main read frame epilogue bytes=32
2020-04-20T03:49:17.287 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.262+0000 7fe3aaffd700 1 --2- 172.21.15.102:0/735168108 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3980044d0 0x7fe398004950 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe3980054a0 tx=0x7fe3a0010050).handle_read_frame_epilogue_main read frame epilogue bytes=32
2020-04-20T03:49:17.287 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.262+0000 7fe3aaffd700 1 --2- 172.21.15.102:0/735168108 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3980044d0 0x7fe398004950 secure :-1 s=READY pgs=944 cs=0 l=1 rx=0x7fe3980054a0 tx=0x7fe3a0010050).ready entity=mon.1 client_cookie=7d4fc10708d629ce server_cookie=0 in_seq=0 out_seq=0
2020-04-20T03:49:17.287 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.262+0000 7fe3aaffd700 1 --2- 172.21.15.102:0/735168108 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3980044d0 0x7fe398004950 secure :-1 s=THROTTLE_DONE pgs=944 cs=0 l=1 rx=0x7fe3980054a0 tx=0x7fe3a0010050).handle_read_frame_epilogue_main read frame epilogue bytes=32
2020-04-20T03:49:17.287 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.262+0000 7fe3aa7fc700 1 -- 172.21.15.102:0/735168108 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe3a0008d30 con 0x7fe3980044d0
2020-04-20T03:49:17.287 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.262+0000 7fe3aaffd700 1 --2- 172.21.15.102:0/735168108 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3980044d0 0x7fe398004950 secure :-1 s=THROTTLE_DONE pgs=944 cs=0 l=1 rx=0x7fe3980054a0 tx=0x7fe3a0010050).handle_read_frame_epilogue_main read frame epilogue bytes=32
2020-04-20T03:49:17.287 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.262+0000 7fe3aa7fc700 1 -- 172.21.15.102:0/735168108 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3a0008e90 con 0x7fe3980044d0
2020-04-20T03:49:17.288 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.262+0000 7fe3aaffd700 1 --2- 172.21.15.102:0/735168108 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3980044d0 0x7fe398004950 secure :-1 s=THROTTLE_DONE pgs=944 cs=0 l=1 rx=0x7fe3980054a0 tx=0x7fe3a0010050).handle_read_frame_epilogue_main read frame epilogue bytes=32
2020-04-20T03:49:17.288 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.262+0000 7fe3aa7fc700 1 -- 172.21.15.102:0/735168108 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe3a0010a40 con 0x7fe3980044d0
2020-04-20T03:49:17.288 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.262+0000 7fe3b0c9a700 1 -- 172.21.15.102:0/735168108 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3980044d0 msgr2=0x7fe398004950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down
2020-04-20T03:49:17.289 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.262+0000 7fe3b0c9a700 1 --2- 172.21.15.102:0/735168108 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3980044d0 0x7fe398004950 secure :-1 s=READY pgs=944 cs=0 l=1 rx=0x7fe3980054a0 tx=0x7fe3a0010050).stop
2020-04-20T03:49:17.289 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.263+0000 7fe3b0c9a700 1 -- 172.21.15.102:0/735168108 shutdown_connections
2020-04-20T03:49:17.290 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.263+0000 7fe3b0c9a700 1 --2- 172.21.15.102:0/735168108 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3980044d0 0x7fe398004950 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rx=0 tx=0).stop
2020-04-20T03:49:17.290 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.263+0000 7fe3b0c9a700 1 -- 172.21.15.102:0/735168108 >> 172.21.15.102:0/735168108 conn(0x7fe3ac06a1e0 msgr2=0x7fe3ac06a5f0 unknown :-1 s=STATE_NONE l=0).mark_down
2020-04-20T03:49:17.290 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.263+0000 7fe3b0c9a700 1 -- 172.21.15.102:0/735168108 shutdown_connections
2020-04-20T03:49:17.290 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.263+0000 7fe3b0c9a700 1 -- 172.21.15.102:0/735168108 wait complete.
2020-04-20T03:49:17.290 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.263+0000 7fe3b0c9a700 1 Processor -- start
2020-04-20T03:49:17.290 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.264+0000 7fe3b0c9a700 1 -- start start
2020-04-20T03:49:17.290 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.264+0000 7fe3b0c9a700 1 --2- >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3ac0b4f90 0x7fe3ac146380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect
2020-04-20T03:49:17.291 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.264+0000 7fe3b0c9a700 1 -- --> v1:172.21.15.100:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3ac0b7060 con 0x7fe3ac1277a0
2020-04-20T03:49:17.291 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.264+0000 7fe3b0c9a700 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3ac0b7300 con 0x7fe3980044d0
2020-04-20T03:49:17.291 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.264+0000 7fe3b0c9a700 1 -- --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ac127c30 con 0x7fe3ac0b4f90
2020-04-20T03:49:17.291 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.264+0000 7fe3ab7fe700 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe3980044d0 0x7fe3ac0b4880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60286/0 (socket says 172.21.15.102:60286)
2020-04-20T03:49:17.291 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.264+0000 7fe3ab7fe700 1 -- 172.21.15.102:0/3964334314 learned_addr learned my addr 172.21.15.102:0/3964334314 (peer_addr_for_me v1:172.21.15.102:0/0)
2020-04-20T03:49:17.292 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.264+0000 7fe3aaffd700 1 --2- 172.21.15.102:0/3964334314 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3ac0b4f90 0x7fe3ac146380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0
2020-04-20T03:49:17.292 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.265+0000 7fe3a8ff9700 1 -- 172.21.15.102:0/3964334314 <== mon.1 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1817772371 0 0) 0x7fe3ac0b7300 con 0x7fe3980044d0
2020-04-20T03:49:17.292 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.265+0000 7fe3a8ff9700 1 -- 172.21.15.102:0/3964334314 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe38c002fe0 con 0x7fe3980044d0
2020-04-20T03:49:17.292 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.265+0000 7fe3a8ff9700 1 -- 172.21.15.102:0/3964334314 <== mon.0 v1:172.21.15.100:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2677493022 0 0) 0x7fe3ac0b7060 con 0x7fe3ac1277a0
2020-04-20T03:49:17.292 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.265+0000 7fe3a8ff9700 1 -- 172.21.15.102:0/3964334314 --> v1:172.21.15.100:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3ac0b7300 con 0x7fe3ac1277a0
2020-04-20T03:49:17.292 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.265+0000 7fe3a8ff9700 1 -- 172.21.15.102:0/3964334314 <== mon.1 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3096428054 0 0) 0x7fe38c002fe0 con 0x7fe3980044d0
2020-04-20T03:49:17.292 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.265+0000 7fe3aaffd700 1 -- 172.21.15.102:0/3964334314 >> v1:172.21.15.102:6789/0 conn(0x7fe3980044d0 legacy=0x7fe3ac0b4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down
2020-04-20T03:49:17.293 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.265+0000 7fe3aaffd700 1 -- 172.21.15.102:0/3964334314 >> v1:172.21.15.100:6789/0 conn(0x7fe3ac1277a0 legacy=0x7fe3ac0b3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down
2020-04-20T03:49:17.293 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.265+0000 7fe3aaffd700 1 -- 172.21.15.102:0/3964334314 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3ac0b75a0 con 0x7fe3ac0b4f90
2020-04-20T03:49:17.293 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.265+0000 7fe3aaffd700 1 --2- 172.21.15.102:0/3964334314 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3ac0b4f90 0x7fe3ac146380 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe39c0010d0 tx=0x7fe3a0010050).handle_read_frame_epilogue_main read frame epilogue bytes=32
2020-04-20T03:49:17.293 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.265+0000 7fe3aaffd700 1 --2- 172.21.15.102:0/3964334314 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3ac0b4f90 0x7fe3ac146380 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe39c0010d0 tx=0x7fe3a0010050).handle_read_frame_epilogue_main read frame epilogue bytes=32
2020-04-20T03:49:17.293 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.265+0000 7fe3aaffd700 1 --2- 172.21.15.102:0/3964334314 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3ac0b4f90 0x7fe3ac146380 secure :-1 s=READY pgs=137 cs=0 l=1 rx=0x7fe39c0010d0 tx=0x7fe3a0010050).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0
2020-04-20T03:49:17.293 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.265+0000 7fe3aaffd700 1 --2- 172.21.15.102:0/3964334314 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3ac0b4f90 0x7fe3ac146380 secure :-1 s=THROTTLE_DONE pgs=137 cs=0 l=1 rx=0x7fe39c0010d0 tx=0x7fe3a0010050).handle_read_frame_epilogue_main read frame epilogue bytes=32
2020-04-20T03:49:17.293 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.265+0000 7fe3a8ff9700 1 -- 172.21.15.102:0/3964334314 <== mon.2 v2:172.21.15.100:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe39c001710 con 0x7fe3ac0b4f90
2020-04-20T03:49:17.293 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.265+0000 7fe3aaffd700 1 --2- 172.21.15.102:0/3964334314 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3ac0b4f90 0x7fe3ac146380 secure :-1 s=THROTTLE_DONE pgs=137 cs=0 l=1 rx=0x7fe39c0010d0 tx=0x7fe3a0010050).handle_read_frame_epilogue_main read frame epilogue bytes=32
2020-04-20T03:49:17.294 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.265+0000 7fe3b0c9a700 1 -- 172.21.15.102:0/3964334314 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3ac0b7a40 con 0x7fe3ac0b4f90
2020-04-20T03:49:17.294 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.265+0000 7fe3a8ff9700 1 -- 172.21.15.102:0/3964334314 <== mon.2 v2:172.21.15.100:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3a000f760 con 0x7fe3ac0b4f90
2020-04-20T03:49:17.294 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.265+0000 7fe3aaffd700 1 --2- 172.21.15.102:0/3964334314 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3ac0b4f90 0x7fe3ac146380 secure :-1 s=THROTTLE_DONE pgs=137 cs=0 l=1 rx=0x7fe39c0010d0 tx=0x7fe3a0010050).handle_read_frame_epilogue_main read frame epilogue bytes=32
2020-04-20T03:49:17.294 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.265+0000 7fe3a8ff9700 1 -- 172.21.15.102:0/3964334314 <== mon.2 v2:172.21.15.100:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe3a0010ea0 con 0x7fe3ac0b4f90
2020-04-20T03:49:17.294 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.265+0000 7fe3b0c9a700 1 -- 172.21.15.102:0/3964334314 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3ac0b6c30 con 0x7fe3ac0b4f90
2020-04-20T03:49:17.294 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.266+0000 7fe3aaffd700 1 --2- 172.21.15.102:0/3964334314 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3ac0b4f90 0x7fe3ac146380 secure :-1 s=THROTTLE_DONE pgs=137 cs=0 l=1 rx=0x7fe39c0010d0 tx=0x7fe3a0010050).handle_read_frame_epilogue_main read frame epilogue bytes=32
2020-04-20T03:49:17.295 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.266+0000 7fe3b0c9a700 1 -- 172.21.15.102:0/3964334314 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3ac0b7730 con 0x7fe3ac0b4f90
2020-04-20T03:49:17.295 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.266+0000 7fe3a8ff9700 1 -- 172.21.15.102:0/3964334314 <== mon.2 v2:172.21.15.100:3301/0 4 ==== mgrmap(e 4) v1 ==== 59679+0+0 (secure 0 0 0) 0x7fe3a00058b0 con 0x7fe3ac0b4f90
2020-04-20T03:49:17.295 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.266+0000 7fe3aaffd700 1 --2- 172.21.15.102:0/3964334314 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3ac0b4f90 0x7fe3ac146380 secure :-1 s=THROTTLE_DONE pgs=137 cs=0 l=1 rx=0x7fe39c0010d0 tx=0x7fe3a0010050).handle_read_frame_epilogue_main read frame epilogue bytes=32
2020-04-20T03:49:17.295 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.267+0000 7fe3a8ff9700 1 --2- 172.21.15.102:0/3964334314 >> [v2:172.21.15.100:6800/30786,v1:172.21.15.100:6801/30786] conn(0x7fe38c04ddc0 0x7fe38c050280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect
2020-04-20T03:49:17.295 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.267+0000 7fe3a8ff9700 1 -- 172.21.15.102:0/3964334314 <== mon.2 v2:172.21.15.100:3301/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe3a00610a0 con 0x7fe3ac0b4f90
2020-04-20T03:49:17.296 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.267+0000 7fe3a8ff9700 1 -- 172.21.15.102:0/3964334314 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_subscribe({osdmap=90}) v3 -- 0x7fe38c002fe0 con 0x7fe3ac0b4f90
2020-04-20T03:49:17.296 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.267+0000 7fe3ab7fe700 1 --2- 172.21.15.102:0/3964334314 >> [v2:172.21.15.100:6800/30786,v1:172.21.15.100:6801/30786] conn(0x7fe38c04ddc0 0x7fe38c050280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0
2020-04-20T03:49:17.296 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.267+0000 7fe3ab7fe700 1 --2- 172.21.15.102:0/3964334314 >> [v2:172.21.15.100:6800/30786,v1:172.21.15.100:6801/30786] conn(0x7fe38c04ddc0 0x7fe38c050280 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe39c001a00 tx=0x7fe3ac060e10).handle_read_frame_epilogue_main read frame epilogue bytes=32
2020-04-20T03:49:17.296 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.267+0000 7fe3ab7fe700 1 --2- 172.21.15.102:0/3964334314 >> [v2:172.21.15.100:6800/30786,v1:172.21.15.100:6801/30786] conn(0x7fe38c04ddc0 0x7fe38c050280 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe39c001a00 tx=0x7fe3ac060e10).handle_read_frame_epilogue_main read frame epilogue bytes=32
2020-04-20T03:49:17.296 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.267+0000 7fe3ab7fe700 1 --2- 172.21.15.102:0/3964334314 >> [v2:172.21.15.100:6800/30786,v1:172.21.15.100:6801/30786] conn(0x7fe38c04ddc0 0x7fe38c050280 secure :-1 s=READY pgs=374 cs=0 l=1 rx=0x7fe39c001a00 tx=0x7fe3ac060e10).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0
2020-04-20T03:49:17.297 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.270+0000 7fe3aaffd700 1 --2- 172.21.15.102:0/3964334314 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3ac0b4f90 0x7fe3ac146380 secure :-1 s=THROTTLE_DONE pgs=137 cs=0 l=1 rx=0x7fe39c0010d0 tx=0x7fe3a0010050).handle_read_frame_epilogue_main read frame epilogue bytes=32
2020-04-20T03:49:17.297 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.270+0000 7fe3a8ff9700 1 -- 172.21.15.102:0/3964334314 <== mon.2 v2:172.21.15.100:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+128047 (secure 0 0 0) 0x7fe3a003c540 con 0x7fe3ac0b4f90
2020-04-20T03:49:17.465 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.462+0000 7fe3b0c9a700 1 -- 172.21.15.102:0/3964334314 --> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_full.TestClusterFull.test_full_same_file"]} v 0) v1 -- 0x7fe3ac056c90 con 0x7fe3ac0b4f90
2020-04-20T03:49:17.590 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.587+0000 7fe3aaffd700 1 --2- 172.21.15.102:0/3964334314 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3ac0b4f90 0x7fe3ac146380 secure :-1 s=THROTTLE_DONE pgs=137 cs=0 l=1 rx=0x7fe39c0010d0 tx=0x7fe3a0010050).handle_read_frame_epilogue_main read frame epilogue bytes=32
2020-04-20T03:49:17.591 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.587+0000 7fe3a8ff9700 1 -- 172.21.15.102:0/3964334314 <== mon.2 v2:172.21.15.100:3301/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_full.TestClusterFull.test_full_same_file"]}]=0 v255) v1 ==== 137+0+0 (secure 0 0 0) 0x7fe3a003fb60 con 0x7fe3ac0b4f90
2020-04-20T03:49:17.592 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.588+0000 7fe3b0c9a700 1 -- 172.21.15.102:0/3964334314 >> [v2:172.21.15.100:6800/30786,v1:172.21.15.100:6801/30786] conn(0x7fe38c04ddc0 msgr2=0x7fe38c050280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down
2020-04-20T03:49:17.593 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.588+0000 7fe3b0c9a700 1 --2- 172.21.15.102:0/3964334314 >> [v2:172.21.15.100:6800/30786,v1:172.21.15.100:6801/30786] conn(0x7fe38c04ddc0 0x7fe38c050280 secure :-1 s=READY pgs=374 cs=0 l=1 rx=0x7fe39c001a00 tx=0x7fe3ac060e10).stop
2020-04-20T03:49:17.593 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.588+0000 7fe3b0c9a700 1 -- 172.21.15.102:0/3964334314 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3ac0b4f90 msgr2=0x7fe3ac146380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down
2020-04-20T03:49:17.593 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.588+0000 7fe3b0c9a700 1 --2- 172.21.15.102:0/3964334314 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3ac0b4f90 0x7fe3ac146380 secure :-1 s=READY pgs=137 cs=0 l=1 rx=0x7fe39c0010d0 tx=0x7fe3a0010050).stop
2020-04-20T03:49:17.593 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.588+0000 7fe3b0c9a700 1 -- 172.21.15.102:0/3964334314 shutdown_connections
2020-04-20T03:49:17.594 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.588+0000 7fe3b0c9a700 1 --2- 172.21.15.102:0/3964334314 >> [v2:172.21.15.100:6800/30786,v1:172.21.15.100:6801/30786] conn(0x7fe38c04ddc0 0x7fe38c050280 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rx=0 tx=0).stop
2020-04-20T03:49:17.594 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.588+0000 7fe3b0c9a700 1 --2- 172.21.15.102:0/3964334314 >> [v2:172.21.15.100:3301/0,v1:172.21.15.100:6790/0] conn(0x7fe3ac0b4f90 0x7fe3ac146380 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rx=0 tx=0).stop
2020-04-20T03:49:17.594 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.588+0000 7fe3b0c9a700 1 -- 172.21.15.102:0/3964334314 >> 172.21.15.102:0/3964334314 conn(0x7fe3ac06a1e0 msgr2=0x7fe3ac12a7c0 unknown :-1 s=STATE_NONE l=0).mark_down
2020-04-20T03:49:17.594 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.588+0000 7fe3b0c9a700 1 -- 172.21.15.102:0/3964334314 shutdown_connections
2020-04-20T03:49:17.594 INFO:teuthology.orchestra.run.smithi102.stderr:2020-04-20T03:49:17.588+0000 7fe3b0c9a700 1 -- 172.21.15.102:0/3964334314 wait complete.
2020-04-20T03:49:17.605 INFO:tasks.cephfs_test_runner:
2020-04-20T03:49:17.605 INFO:tasks.cephfs_test_runner:======================================================================
2020-04-20T03:49:17.605 INFO:tasks.cephfs_test_runner:ERROR: test_full_same_file (tasks.cephfs.test_full.TestClusterFull)
2020-04-20T03:49:17.605 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-04-20T03:49:17.605 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-04-20T03:49:17.606 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-lxb-libfuse3-2020-04-18-2/qa/tasks/cephfs/test_full.py", line 197, in test_full_same_file
2020-04-20T03:49:17.606 INFO:tasks.cephfs_test_runner: self._test_full(False)
2020-04-20T03:49:17.606 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-lxb-libfuse3-2020-04-18-2/qa/tasks/cephfs/test_full.py", line 135, in _test_full
2020-04-20T03:49:17.606 INFO:tasks.cephfs_test_runner: self.mount_a.write_n_mb("large_file_a", self.fill_mb // 2)
2020-04-20T03:49:17.606 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-lxb-libfuse3-2020-04-18-2/qa/tasks/cephfs/mount.py", line 425, in write_n_mb
2020-04-20T03:49:17.606 INFO:tasks.cephfs_test_runner: ], wait=wait)
2020-04-20T03:49:17.606 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-lxb-libfuse3-2020-04-18-2/qa/tasks/cephfs/mount.py", line 205, in run_shell
2020-04-20T03:49:17.606 INFO:tasks.cephfs_test_runner: omit_sudo=omit_sudo)
2020-04-20T03:49:17.607 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 203, in run
2020-04-20T03:49:17.607 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
2020-04-20T03:49:17.607 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 473, in run
2020-04-20T03:49:17.607 INFO:tasks.cephfs_test_runner: r.wait()
2020-04-20T03:49:17.607 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 162, in wait
2020-04-20T03:49:17.607 INFO:tasks.cephfs_test_runner: self._raise_for_status()
2020-04-20T03:49:17.607 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 184, in _raise_for_status
2020-04-20T03:49:17.607 INFO:tasks.cephfs_test_runner: node=self.hostname, label=self.label
2020-04-20T03:49:17.607 INFO:tasks.cephfs_test_runner:CommandFailedError: Command failed on smithi102 with status 1: 'cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=large_file_a bs=1M conv=fdatasync count=186 seek=0'
2020-04-20T03:49:17.608 INFO:tasks.cephfs_test_runner:
2020-04-20T03:49:17.608 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-04-20T03:49:17.608 INFO:tasks.cephfs_test_runner:Ran 5 tests in 259.215s
2020-04-20T03:49:17.608 INFO:tasks.cephfs_test_runner:
2020-04-20T03:49:17.608 INFO:tasks.cephfs_test_runner:FAILED (errors=1)
2020-04-20T03:49:17.608 INFO:tasks.cephfs_test_runner:
2020-04-20T03:49:17.608 INFO:tasks.cephfs_test_runner:======================================================================
2020-04-20T03:49:17.609 INFO:tasks.cephfs_test_runner:ERROR: test_full_same_file (tasks.cephfs.test_full.TestClusterFull)
2020-04-20T03:49:17.609 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2020-04-20T03:49:17.609 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2020-04-20T03:49:17.609 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-lxb-libfuse3-2020-04-18-2/qa/tasks/cephfs/test_full.py", line 197, in test_full_same_file
2020-04-20T03:49:17.609 INFO:tasks.cephfs_test_runner: self._test_full(False)
2020-04-20T03:49:17.609 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-lxb-libfuse3-2020-04-18-2/qa/tasks/cephfs/test_full.py", line 135, in _test_full
2020-04-20T03:49:17.609 INFO:tasks.cephfs_test_runner: self.mount_a.write_n_mb("large_file_a", self.fill_mb // 2)
2020-04-20T03:49:17.609 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-lxb-libfuse3-2020-04-18-2/qa/tasks/cephfs/mount.py", line 425, in write_n_mb
2020-04-20T03:49:17.610 INFO:tasks.cephfs_test_runner: ], wait=wait)
2020-04-20T03:49:17.610 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-lxb-libfuse3-2020-04-18-2/qa/tasks/cephfs/mount.py", line 205, in run_shell
2020-04-20T03:49:17.610 INFO:tasks.cephfs_test_runner: omit_sudo=omit_sudo)
2020-04-20T03:49:17.610 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 203, in run
2020-04-20T03:49:17.610 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
2020-04-20T03:49:17.610 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 473, in run
2020-04-20T03:49:17.610 INFO:tasks.cephfs_test_runner: r.wait()
2020-04-20T03:49:17.610 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 162, in wait
2020-04-20T03:49:17.610 INFO:tasks.cephfs_test_runner: self._raise_for_status()
2020-04-20T03:49:17.611 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 184, in _raise_for_status
2020-04-20T03:49:17.611 INFO:tasks.cephfs_test_runner: node=self.hostname, label=self.label
2020-04-20T03:49:17.611 INFO:tasks.cephfs_test_runner:CommandFailedError: Command failed on smithi102 with status 1: 'cd /home/ubuntu/cephtest/mnt.0 && sudo dd if=/dev/urandom of=large_file_a bs=1M conv=fdatasync count=186 seek=0'
2020-04-20T03:49:17.611 INFO:tasks.cephfs_test_runner:
2020-04-20T03:49:17.611 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 90, in run_tasks
manager.__enter__()
File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-lxb-libfuse3-2020-04-18-2/qa/tasks/cephfs_test_runner.py", line 211, in task
raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests)))
RuntimeError: Test failure: test_full_same_file (tasks.cephfs.test_full.TestClusterFull)
2020-04-20T03:49:17.641 ERROR:teuthology.run_tasks: Sentry event: http://sentry.ceph.com/sepia/teuthology/?q=4ae8de9dcd7441b09da554683d6525e9
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 90, in run_tasks
manager.__enter__()
File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-lxb-libfuse3-2020-04-18-2/qa/tasks/cephfs_test_runner.py", line 211, in task
raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests)))
RuntimeError: Test failure: test_full_same_file (tasks.cephfs.test_full.TestClusterFull)
2020-04-20T03:49:17.641 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner
2020-04-20T03:49:17.650 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse
2020-04-20T03:49:17.664 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients...
2020-04-20T03:49:17.664 INFO:teuthology.orchestra.run.smithi102:> true
Updated by Xiubo Li about 4 years ago
- Status changed from New to Fix Under Review
Updated by Patrick Donnelly about 3 years ago
- Project changed from 23 to CephFS
- Priority changed from Normal to Urgent
- Target version set to v17.0.0
- Source set to Q/A
- Component(FS) qa-suite added
- Labels (FS) qa, qa-failure added
Updated by Patrick Donnelly about 3 years ago
- Is duplicate of Bug #45434: qa: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) failed added
Updated by Patrick Donnelly about 3 years ago
- Status changed from Fix Under Review to Duplicate
Actions