Project

General

Profile

Bug #37681

qa: power off still resulted in client sending session close

Added by Patrick Donnelly 8 months ago. Updated 24 days ago.

Status:
Pending Backport
Priority:
High
Category:
Testing
Target version:
Start date:
Due date:
% Done:

0%

Source:
Q/A
Tags:
Backport:
nautilus,mimic
Regression:
No
Severity:
4 - irritation
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
kceph, qa-suite
Labels (FS):
Pull request ID:

Description

2018-12-14T23:24:05.467 DEBUG:tasks.cephfs.mount:File background_file became visible from 1 after 0s
2018-12-14T23:24:05.494 INFO:teuthology.orchestra.console:Power off smithi059
2018-12-14T23:24:05.495 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2018-12-14T23:24:05.524 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2018-12-14T23:24:09.553 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2018-12-14T23:24:13.672 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2018-12-14T23:24:17.789 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status

From: /ceph/teuthology-archive/pdonnell-2018-12-13_01:18:47-kcephfs-wip-pdonnell-testing-20181212.231100-distro-basic-smithi/3335083/teuthology.log

2018-12-14 23:24:09.334 7fd1b1036700  1 -- 172.21.15.86:6816/2023644697 <== client.6072 172.21.15.59:0/3399749026 15 ==== client_session(request_close seq 15) ==== 28+0+0 (1415024270 0 0) 0x56550375c900 con 0x5655029f7800
2018-12-14 23:24:09.334 7fd1b1036700 20 mds.0.69 get_session have 0x5655036e0380 client.6072 172.21.15.59:0/3399749026 state open
2018-12-14 23:24:09.334 7fd1b1036700  3 mds.0.server handle_client_session client_session(request_close seq 15) from client.6072
2018-12-14 23:24:09.334 7fd1b1036700 20 mds.0.sessionmap mark_projected s=0x5655036e0380 name=client.6072 pv=3 -> 4

From: /ceph/teuthology-archive/pdonnell-2018-12-13_01:18:47-kcephfs-wip-pdonnell-testing-20181212.231100-distro-basic-smithi/3335083/remote/smithi086/log/ceph-mds.y-s.log.gz


Related issues

Related to fs - Bug #40999: qa: AssertionError: u'open' != 'stale' Pending Backport
Copied to fs - Backport #41128: nautilus: qa: power off still resulted in client sending session close New
Copied to fs - Backport #41129: mimic: qa: power off still resulted in client sending session close New

History

#1 Updated by Patrick Donnelly 8 months ago

  • Category set to Testing
  • Assignee deleted (Zheng Yan)
  • Start date deleted (12/17/2018)
  • Severity changed from 3 - minor to 4 - irritation

#2 Updated by Patrick Donnelly 6 months ago

  • Target version changed from v14.0.0 to v15.0.0

#3 Updated by Patrick Donnelly 6 months ago

  • Target version deleted (v15.0.0)

#4 Updated by Patrick Donnelly about 2 months ago

  • Assignee set to Zheng Yan
  • Priority changed from Normal to High
  • Target version set to v15.0.0
  • Backport changed from mimic,luminous to nautilus,mimic

Still happening:

2019-06-22T02:53:59.709+0000 7f2c0168d700  1 -- [v2:172.21.15.175:6832/1453658918,v1:172.21.15.175:6833/1453658918] <== client.7633 v1:172.21.15.144:0/3786346239 15 ==== client_session(request_close seq 15) ==== 28+0+0 (unknown 3619525983 0 0) 0x55abf56918c0 con 0x55abf565b000
2019-06-22T02:53:59.709+0000 7f2c0168d700 20 mds.0.75 get_session have 0x55abf6359400 client.7633 v1:172.21.15.144:0/3786346239 state open
2019-06-22T02:53:59.709+0000 7f2c0168d700  3 mds.0.server handle_client_session client_session(request_close seq 15) from client.7633

From: /ceph/teuthology-archive/pdonnell-2019-06-21_01:44:51-kcephfs-wip-pdonnell-testing-20190620.220400-distro-basic-smithi/4053099/remote/smithi175/log/ceph-mds.c.log.gz

Took a while to find this ticket. Here's the symptom of the problem for easy google lookup:

2019-06-22T02:56:35.425 INFO:tasks.cephfs_test_runner:======================================================================
2019-06-22T02:56:35.425 INFO:tasks.cephfs_test_runner:FAIL: test_evicted_caps (tasks.cephfs.test_client_recovery.TestClientRecovery)
2019-06-22T02:56:35.425 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2019-06-22T02:56:35.425 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2019-06-22T02:56:35.425 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20190620.220400/qa/tasks/cephfs/test_client_recovery.py", line 305, in test_evicted_caps
2019-06-22T02:56:35.425 INFO:tasks.cephfs_test_runner:    self.assertFalse(cap_waiter.finished)
2019-06-22T02:56:35.425 INFO:tasks.cephfs_test_runner:AssertionError: True is not false

#5 Updated by Patrick Donnelly about 2 months ago

  • Subject changed from kclient: power off still resulted in client sending session close to qa: power off still resulted in client sending session close
  • Status changed from New to In Progress
  • Assignee changed from Zheng Yan to Patrick Donnelly
  • Backport changed from nautilus,mimic to nautilus,mimic,luminous

The correct ipmitool command to simulate pulling the power plug is "power reset". "power off" will permit graceful shutdown. Thanks to David Galloway for pointing this out.

#6 Updated by Patrick Donnelly about 2 months ago

  • Status changed from In Progress to Need Review
  • Pull request ID set to 28825

#7 Updated by Patrick Donnelly 24 days ago

  • Status changed from Need Review to Pending Backport
  • Backport changed from nautilus,mimic,luminous to nautilus,mimic

#8 Updated by Patrick Donnelly 24 days ago

  • Related to Bug #40999: qa: AssertionError: u'open' != 'stale' added

#9 Updated by Patrick Donnelly 24 days ago

backport note: also need fix for https://tracker.ceph.com/issues/40999

#10 Updated by Patrick Donnelly 16 days ago

  • Copied to Backport #41128: nautilus: qa: power off still resulted in client sending session close added

#11 Updated by Patrick Donnelly 16 days ago

  • Copied to Backport #41129: mimic: qa: power off still resulted in client sending session close added

Also available in: Atom PDF