Tools » teuthology

http://pulpito.ceph.com/yuriw-2020-05-13_14:51:27-powercycle-wip-yuri-octopus_15.2.2_RC0-distro-basic-smithi/

Looking at http://qa-proxy.ceph.com/teuthology/yuriw-2020-05-13_14:51:27-powercycle-wip-yuri-octopus_15.2.2_RC0-distro-basic-smithi/5052176/teuthology.log specifically

I don't know when the job gives up completely but I suspect something's broken either in teuthology or the powercycle jobs.

2020-05-13T16:43:13.311 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1
2020-05-13T16:43:13.950 INFO:tasks.ceph.osd.1.smithi204.stderr:2020-05-13T16:43:13.949+0000 7f7e34ddbec0 -1 Falling back to public interface
2020-05-13T16:43:15.013 INFO:tasks.ceph.osd.1.smithi204.stderr:2020-05-13T16:43:15.011+0000 7f7e34ddbec0 -1 osd.1 0 log_to_monitors {default=true}
2020-05-13T16:43:16.866 INFO:tasks.ceph.osd.1.smithi204.stderr:2020-05-13T16:43:16.863+0000 7f7e1d5a3700 -1 osd.1 0 waiting for initial osdmap
2020-05-13T16:43:39.863 INFO:teuthology.orchestra.run.smithi204:> true
2020-05-13T16:43:39.879 INFO:teuthology.orchestra.run.smithi204:> sync
2020-05-13T16:43:40.939 DEBUG:tasks.thrashosds:checking console status of smithi204
2020-05-13T16:43:40.940 DEBUG:teuthology.orchestra.console:Waiting for login prompt on smithi204
2020-05-13T16:43:40.940 DEBUG:teuthology.orchestra.console:pexpect command: console -M conserver.front.sepia.ceph.com -p 3109 -f smithi204
2020-05-13T16:43:41.007 DEBUG:teuthology.orchestra.console:expect: smithi204 login
2020-05-13T16:43:41.227 DEBUG:teuthology.orchestra.console:expect after: smithi204 login: 
2020-05-13T16:43:43.745 INFO:tasks.ceph.ceph_manager.ceph:kill_osd on osd.1 doing powercycle of ubuntu@smithi204.front.sepia.ceph.com
2020-05-13T16:43:43.746 INFO:teuthology.orchestra.console:Power off smithi204
2020-05-13T16:43:43.746 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2020-05-13T16:43:43.775 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-05-13T16:43:47.752 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi204.front.sepia.ceph.com', 'timeout': 60}
NoValidConnectionsError: [Errno None] Unable to connect to port 22 on 172.21.15.204
NoValidConnectionsError: [Errno None] Unable to connect to port 22 on 172.21.15.204
NoValidConnectionsError: [Errno None] Unable to connect to port 22 on 172.21.15.204
2020-05-13T16:43:47.785 DEBUG:teuthology.run_tasks:Unwinding manager install
2020-05-13T16:43:47.798 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse
2020-05-13T16:43:47.815 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-05-13T16:43:48.678 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds
2020-05-13T16:43:51.932 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-05-13T16:43:52.149 INFO:teuthology.orchestra.console:Power off for smithi204 completed
2020-05-13T16:43:57.252 INFO:tasks.ceph.ceph_manager.ceph:kill_osd on osd.1 doing powercycle of ubuntu@smithi204.front.sepia.ceph.com
2020-05-13T16:43:57.252 INFO:teuthology.orchestra.console:Power on smithi204
2020-05-13T16:43:57.252 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2020-05-13T16:43:57.281 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-05-13T16:44:01.310 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-05-13T16:44:05.427 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2020-05-13T16:44:05.722 INFO:teuthology.orchestra.console:Power on for smithi204 completed
2020-05-13T16:44:05.823 DEBUG:teuthology.orchestra.console:Waiting for login prompt on smithi204
2020-05-13T16:44:05.824 DEBUG:teuthology.orchestra.console:pexpect command: console -M conserver.front.sepia.ceph.com -p 3109 -f smithi204
2020-05-13T16:44:05.891 DEBUG:teuthology.orchestra.console:expect: smithi204 login
2020-05-13T16:45:57.999 DEBUG:teuthology.orchestra.console:expect after: smithi204 login: 
2020-05-13T16:45:58.155 INFO:teuthology.misc:trying to connect to ubuntu@smithi204.front.sepia.ceph.com
2020-05-13T16:45:58.158 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi204.front.sepia.ceph.com', 'timeout': 60}
2020-05-13T16:45:58.608 INFO:teuthology.orchestra.run.smithi204:> true
2020-05-13T16:46:00.140 DEBUG:tasks.ceph_manager:Mounting data for osd.1 on ubuntu@smithi204.front.sepia.ceph.com
2020-05-13T16:46:00.140 INFO:tasks.ceph_manager:Mounting osd.1: dev: ubuntu@smithi204.front.sepia.ceph.com, cluster: cephmountpoint: /var/lib/ceph/osd/ceph-1, type: xfs, options: ['noatime']
2020-05-13T16:46:00.140 INFO:teuthology.orchestra.run.smithi204:> true
2020-05-13T16:46:00.387 INFO:teuthology.orchestra.run.smithi204:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-1
2020-05-13T16:46:00.889 INFO:teuthology.orchestra.run.smithi204:> true
2020-05-13T16:46:01.133 INFO:teuthology.orchestra.run.smithi204:> sudo install -d -m0777 -- /var/run/ceph
2020-05-13T16:46:01.210 INFO:teuthology.orchestra.run.smithi204:> true
2020-05-13T16:46:01.223 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1
2020-05-13T16:46:01.278 INFO:teuthology.orchestra.run.smithi204:> true
2020-05-13T16:46:01.324 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight
2020-05-13T16:46:02.595 INFO:teuthology.orchestra.run.smithi204.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory
2020-05-13T16:46:03.231 INFO:tasks.ceph.osd.1.smithi204.stderr:2020-05-13T16:46:03.228+0000 7fa9529d6ec0 -1 Falling back to public interface
2020-05-13T16:46:04.517 INFO:tasks.ceph.osd.1.smithi204.stderr:2020-05-13T16:46:04.514+0000 7fa9529d6ec0 -1 osd.1 19 log_to_monitors {default=true}
2020-05-13T16:46:07.599 INFO:teuthology.orchestra.run.smithi204:> true
2020-05-13T16:46:07.618 INFO:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight
2020-05-13T16:46:07.786 INFO:teuthology.orchestra.run.smithi204.stdout:{
2020-05-13T16:46:07.787 INFO:teuthology.orchestra.run.smithi204.stdout:    "ops": [],
2020-05-13T16:46:07.787 INFO:teuthology.orchestra.run.smithi204.stdout:    "num_ops": 0
2020-05-13T16:46:07.787 INFO:teuthology.orchestra.run.smithi204.stdout:}
2020-05-13T16:46:19.527 DEBUG:teuthology.run_tasks:Unwinding manager ceph
NoValidConnectionsError: [Errno None] Unable to connect to port 22 on 172.21.15.204
2020-05-13T16:47:04.991 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi204.front.sepia.ceph.com