Ceph » CephFS

2020-05-16T13:01:44.540 INFO:tasks.cephfs.mount:Removing the 'ceph-brx'
2020-05-16T13:01:44.540 INFO:teuthology.orchestra.run:Running command with timeout 300
2020-05-16T13:01:44.540 INFO:teuthology.orchestra.run.smithi114:> sudo bash -c 'ip link set ceph-brx down'
2020-05-16T13:01:44.605 INFO:teuthology.orchestra.run:Running command with timeout 300
2020-05-16T13:01:44.606 INFO:teuthology.orchestra.run.smithi114:> sudo bash -c 'ip link delete ceph-brx'
2020-05-16T13:01:44.606 INFO:tasks.ceph.osd.0.smithi114.stderr:2020-05-16T13:01:44.601+0000 7f2768605700 -1 osd.0 28 heartbeat_check: no reply from 172.21.15.38:6820 osd.4 ever on either front or back, first ping sent 2020-05-16T12:59:59.405725+0000 (oldest deadline 2020-05-16T13:00:19.405725+0000)
2020-05-16T13:01:44.607 INFO:tasks.ceph.osd.0.smithi114.stderr:2020-05-16T13:01:44.601+0000 7f2768605700 -1 osd.0 28 heartbeat_check: no reply from 172.21.15.38:6812 osd.5 ever on either front or back, first ping sent 2020-05-16T12:59:59.405725+0000 (oldest deadline 2020-05-16T13:00:19.405725+0000)
2020-05-16T13:01:44.607 INFO:tasks.ceph.osd.0.smithi114.stderr:2020-05-16T13:01:44.601+0000 7f2768605700 -1 osd.0 28 heartbeat_check: no reply from 172.21.15.38:6804 osd.6 ever on either front or back, first ping sent 2020-05-16T12:59:59.405725+0000 (oldest deadline 2020-05-16T13:00:19.405725+0000)
2020-05-16T13:01:44.607 INFO:tasks.ceph.osd.0.smithi114.stderr:2020-05-16T13:01:44.601+0000 7f2768605700 -1 osd.0 28 heartbeat_check: no reply from 172.21.15.38:6828 osd.7 ever on either front or back, first ping sent 2020-05-16T12:59:59.405725+0000 (oldest deadline 2020-05-16T13:00:19.405725+0000)
2020-05-16T13:01:44.614 INFO:tasks.ceph.osd.2.smithi114.stderr:2020-05-16T13:01:44.609+0000 7f72cece0700 -1 osd.2 28 heartbeat_check: no reply from 172.21.15.38:6820 osd.4 ever on either front or back, first ping sent 2020-05-16T12:59:55.305823+0000 (oldest deadline 2020-05-16T13:00:15.305823+0000)
2020-05-16T13:01:44.614 INFO:tasks.ceph.osd.2.smithi114.stderr:2020-05-16T13:01:44.609+0000 7f72cece0700 -1 osd.2 28 heartbeat_check: no reply from 172.21.15.38:6812 osd.5 ever on either front or back, first ping sent 2020-05-16T12:59:55.305823+0000 (oldest deadline 2020-05-16T13:00:15.305823+0000)
2020-05-16T13:01:44.615 INFO:tasks.ceph.osd.2.smithi114.stderr:2020-05-16T13:01:44.609+0000 7f72cece0700 -1 osd.2 28 heartbeat_check: no reply from 172.21.15.38:6804 osd.6 ever on either front or back, first ping sent 2020-05-16T12:59:55.305823+0000 (oldest deadline 2020-05-16T13:00:15.305823+0000)
2020-05-16T13:01:44.615 INFO:tasks.ceph.osd.2.smithi114.stderr:2020-05-16T13:01:44.609+0000 7f72cece0700 -1 osd.2 28 heartbeat_check: no reply from 172.21.15.38:6828 osd.7 ever on either front or back, first ping sent 2020-05-16T12:59:55.305823+0000 (oldest deadline 2020-05-16T13:00:15.305823+0000)
2020-05-16T13:01:44.615 INFO:tasks.ceph.osd.2.smithi114.stderr:2020-05-16T13:01:44.613+0000 7f72cece0700 -1 osd.2 28 get_health_metrics reporting 104 slow ops, oldest is osd_op(mds.0.5:115966 3.5 3:a6630910:::1000000c882.00000000:head [omap-set-header in=274b,omap-set-vals in=5968b,omap-rm-keys in=472b] snapc 0=[] ondisk+write+known_if_redirected+full_force e28)
2020-05-16T13:01:44.713 INFO:teuthology.orchestra.run:Running command with timeout 300
2020-05-16T13:01:44.714 INFO:teuthology.orchestra.run.smithi114:> route
2020-05-16T13:01:44.758 INFO:teuthology.orchestra.run.smithi114.stdout:Kernel IP routing table
2020-05-16T13:01:44.758 INFO:teuthology.orchestra.run.smithi114.stdout:Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
2020-05-16T13:01:44.759 INFO:teuthology.orchestra.run.smithi114.stdout:default         _gateway        0.0.0.0         UG    100    0        0 enp3s0f1
2020-05-16T13:01:44.759 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.0.0      0.0.0.0         255.255.240.0   U     0      0        0 enp3s0f1
2020-05-16T13:01:44.759 INFO:teuthology.orchestra.run.smithi114.stdout:_gateway        0.0.0.0         255.255.255.255 UH    100    0        0 enp3s0f1
2020-05-16T13:01:44.759 INFO:teuthology.orchestra.run:Running command with timeout 300
2020-05-16T13:01:44.760 INFO:teuthology.orchestra.run.smithi114:> sudo bash -c 'iptables -D FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT'
2020-05-16T13:01:44.857 INFO:teuthology.orchestra.run:Running command with timeout 300
2020-05-16T13:01:44.857 INFO:teuthology.orchestra.run.smithi114:> sudo bash -c 'iptables -D FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT'
2020-05-16T13:01:44.876 INFO:teuthology.orchestra.run:Running command with timeout 300
2020-05-16T13:01:44.877 INFO:teuthology.orchestra.run.smithi114:> sudo bash -c 'iptables -t nat -D POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE'
2020-05-16T13:01:45.125 INFO:tasks.ceph.osd.3.smithi114.stderr:2020-05-16T13:01:45.121+0000 7f7ae1214700 -1 osd.3 28 heartbeat_check: no reply from 172.21.15.38:6820 osd.4 ever on either front or back, first ping sent 2020-05-16T12:59:55.310340+0000 (oldest deadline 2020-05-16T13:00:15.310340+0000)
2020-05-16T13:01:45.126 INFO:tasks.ceph.osd.3.smithi114.stderr:2020-05-16T13:01:45.121+0000 7f7ae1214700 -1 osd.3 28 heartbeat_check: no reply from 172.21.15.38:6812 osd.5 ever on either front or back, first ping sent 2020-05-16T12:59:55.310340+0000 (oldest deadline 2020-05-16T13:00:15.310340+0000)
2020-05-16T13:01:45.126 INFO:tasks.ceph.osd.3.smithi114.stderr:2020-05-16T13:01:45.121+0000 7f7ae1214700 -1 osd.3 28 heartbeat_check: no reply from 172.21.15.38:6804 osd.6 ever on either front or back, first ping sent 2020-05-16T12:59:55.310340+0000 (oldest deadline 2020-05-16T13:00:15.310340+0000)
2020-05-16T13:01:45.126 INFO:tasks.ceph.osd.3.smithi114.stderr:2020-05-16T13:01:45.121+0000 7f7ae1214700 -1 osd.3 28 heartbeat_check: no reply from 172.21.15.38:6828 osd.7 ever on either front or back, first ping sent 2020-05-16T12:59:55.310340+0000 (oldest deadline 2020-05-16T13:00:15.310340+0000)
2020-05-16T13:01:45.127 INFO:tasks.ceph.osd.3.smithi114.stderr:2020-05-16T13:01:45.121+0000 7f7ae1214700 -1 osd.3 28 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4160.0:29 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected e28)
2020-05-16T13:01:45.352 INFO:tasks.ceph.mon.a.smithi114.stderr:2020-05-16T13:01:45.349+0000 7f65f7f05700 -1 mon.a@1(probing) e1 get_health_metrics reporting 8638 slow ops, oldest is osd_failure(failed timeout osd.4 [v2:172.21.15.38:6816/12946,v1:172.21.15.38:6817/12946] for 25sec e28 v28)
2020-05-16T13:01:45.408 INFO:tasks.ceph.osd.1.smithi114.stderr:2020-05-16T13:01:45.405+0000 7f473bcfe700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.38:6820 osd.4 ever on either front or back, first ping sent 2020-05-16T12:59:35.123796+0000 (oldest deadline 2020-05-16T12:59:55.123796+0000)
2020-05-16T13:01:45.409 INFO:tasks.ceph.osd.1.smithi114.stderr:2020-05-16T13:01:45.405+0000 7f473bcfe700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.38:6812 osd.5 ever on either front or back, first ping sent 2020-05-16T12:59:35.123796+0000 (oldest deadline 2020-05-16T12:59:55.123796+0000)
2020-05-16T13:01:45.409 INFO:tasks.ceph.osd.1.smithi114.stderr:2020-05-16T13:01:45.405+0000 7f473bcfe700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.38:6804 osd.6 ever on either front or back, first ping sent 2020-05-16T12:59:35.123796+0000 (oldest deadline 2020-05-16T12:59:55.123796+0000)
2020-05-16T13:01:45.409 INFO:tasks.ceph.osd.1.smithi114.stderr:2020-05-16T13:01:45.405+0000 7f473bcfe700 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.38:6828 osd.7 ever on either front or back, first ping sent 2020-05-16T12:59:35.123796+0000 (oldest deadline 2020-05-16T12:59:55.123796+0000)
2020-05-16T13:01:45.561 INFO:tasks.ceph.osd.0.smithi114.stderr:2020-05-16T13:01:45.557+0000 7f2768605700 -1 osd.0 28 heartbeat_check: no reply from 172.21.15.38:6820 osd.4 ever on either front or back, first ping sent 2020-05-16T12:59:59.405725+0000 (oldest deadline 2020-05-16T13:00:19.405725+0000)
2020-05-16T13:01:45.561 INFO:tasks.ceph.osd.0.smithi114.stderr:2020-05-16T13:01:45.557+0000 7f2768605700 -1 osd.0 28 heartbeat_check: no reply from 172.21.15.38:6812 osd.5 ever on either front or back, first ping sent 2020-05-16T12:59:59.405725+0000 (oldest deadline 2020-05-16T13:00:19.405725+0000)
2020-05-16T13:01:45.561 INFO:tasks.ceph.osd.0.smithi114.stderr:2020-05-16T13:01:45.557+0000 7f2768605700 -1 osd.0 28 heartbeat_check: no reply from 172.21.15.38:6804 osd.6 ever on either front or back, first ping sent 2020-05-16T12:59:59.405725+0000 (oldest deadline 2020-05-16T13:00:19.405725+0000)
2020-05-16T13:01:45.561 INFO:tasks.ceph.osd.0.smithi114.stderr:2020-05-16T13:01:45.557+0000 7f2768605700 -1 osd.0 28 heartbeat_check: no reply from 172.21.15.38:6828 osd.7 ever on either front or back, first ping sent 2020-05-16T12:59:59.405725+0000 (oldest deadline 2020-05-16T13:00:19.405725+0000)

From: /ceph/teuthology-archive/pdonnell-2020-05-16_06:07:05-fs-wip-pdonnell-testing-20200516.030215-distro-basic-smithi/5060521/teuthology.log