Bug #37681
closed
qa: power off still resulted in client sending session close
Added by Patrick Donnelly over 5 years ago.
Updated over 4 years ago.
Component(FS):
kceph, qa-suite
Description
2018-12-14T23:24:05.467 DEBUG:tasks.cephfs.mount:File background_file became visible from 1 after 0s
2018-12-14T23:24:05.494 INFO:teuthology.orchestra.console:Power off smithi059
2018-12-14T23:24:05.495 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2018-12-14T23:24:05.524 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2018-12-14T23:24:09.553 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2018-12-14T23:24:13.672 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2018-12-14T23:24:17.789 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
From: /ceph/teuthology-archive/pdonnell-2018-12-13_01:18:47-kcephfs-wip-pdonnell-testing-20181212.231100-distro-basic-smithi/3335083/teuthology.log
2018-12-14 23:24:09.334 7fd1b1036700 1 -- 172.21.15.86:6816/2023644697 <== client.6072 172.21.15.59:0/3399749026 15 ==== client_session(request_close seq 15) ==== 28+0+0 (1415024270 0 0) 0x56550375c900 con 0x5655029f7800
2018-12-14 23:24:09.334 7fd1b1036700 20 mds.0.69 get_session have 0x5655036e0380 client.6072 172.21.15.59:0/3399749026 state open
2018-12-14 23:24:09.334 7fd1b1036700 3 mds.0.server handle_client_session client_session(request_close seq 15) from client.6072
2018-12-14 23:24:09.334 7fd1b1036700 20 mds.0.sessionmap mark_projected s=0x5655036e0380 name=client.6072 pv=3 -> 4
From: /ceph/teuthology-archive/pdonnell-2018-12-13_01:18:47-kcephfs-wip-pdonnell-testing-20181212.231100-distro-basic-smithi/3335083/remote/smithi086/log/ceph-mds.y-s.log.gz
- Category set to Testing
- Assignee deleted (
Zheng Yan)
- Start date deleted (
12/17/2018)
- Severity changed from 3 - minor to 4 - irritation
- Target version changed from v14.0.0 to v15.0.0
- Target version deleted (
v15.0.0)
- Assignee set to Zheng Yan
- Priority changed from Normal to High
- Target version set to v15.0.0
- Backport changed from mimic,luminous to nautilus,mimic
Still happening:
2019-06-22T02:53:59.709+0000 7f2c0168d700 1 -- [v2:172.21.15.175:6832/1453658918,v1:172.21.15.175:6833/1453658918] <== client.7633 v1:172.21.15.144:0/3786346239 15 ==== client_session(request_close seq 15) ==== 28+0+0 (unknown 3619525983 0 0) 0x55abf56918c0 con 0x55abf565b000
2019-06-22T02:53:59.709+0000 7f2c0168d700 20 mds.0.75 get_session have 0x55abf6359400 client.7633 v1:172.21.15.144:0/3786346239 state open
2019-06-22T02:53:59.709+0000 7f2c0168d700 3 mds.0.server handle_client_session client_session(request_close seq 15) from client.7633
From: /ceph/teuthology-archive/pdonnell-2019-06-21_01:44:51-kcephfs-wip-pdonnell-testing-20190620.220400-distro-basic-smithi/4053099/remote/smithi175/log/ceph-mds.c.log.gz
Took a while to find this ticket. Here's the symptom of the problem for easy google lookup:
2019-06-22T02:56:35.425 INFO:tasks.cephfs_test_runner:======================================================================
2019-06-22T02:56:35.425 INFO:tasks.cephfs_test_runner:FAIL: test_evicted_caps (tasks.cephfs.test_client_recovery.TestClientRecovery)
2019-06-22T02:56:35.425 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2019-06-22T02:56:35.425 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2019-06-22T02:56:35.425 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20190620.220400/qa/tasks/cephfs/test_client_recovery.py", line 305, in test_evicted_caps
2019-06-22T02:56:35.425 INFO:tasks.cephfs_test_runner: self.assertFalse(cap_waiter.finished)
2019-06-22T02:56:35.425 INFO:tasks.cephfs_test_runner:AssertionError: True is not false
- Subject changed from kclient: power off still resulted in client sending session close to qa: power off still resulted in client sending session close
- Status changed from New to In Progress
- Assignee changed from Zheng Yan to Patrick Donnelly
- Backport changed from nautilus,mimic to nautilus,mimic,luminous
The correct ipmitool command to simulate pulling the power plug is "power reset". "power off" will permit graceful shutdown. Thanks to David Galloway for pointing this out.
- Status changed from In Progress to Fix Under Review
- Pull request ID set to 28825
- Status changed from Fix Under Review to Pending Backport
- Backport changed from nautilus,mimic,luminous to nautilus,mimic
- Related to Bug #40999: qa: AssertionError: u'open' != 'stale' added
- Copied to Backport #41128: nautilus: qa: power off still resulted in client sending session close added
- Copied to Backport #41129: mimic: qa: power off still resulted in client sending session close added
- Status changed from Pending Backport to Resolved
While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".
Also available in: Atom
PDF