Project

General

Profile

Bug #4298

misdirected op in ffsb test

Added by Tamilarasi muthamizhan about 11 years ago. Updated about 11 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Logs: ubuntu@teuthology:/a/teuthology-2013-02-27_20:00:05-regression-last-master-basic/13866

ubuntu@teuthology:/a/teuthology-2013-02-27_20:00:05-regression-last-master-basic/13866$ cat config.yaml 
kernel: &id001
  kdb: true
  sha1: 83ca14fdd35821554058e5fd4fa7b118ee504a33
nuke-on-error: true
overrides:
  ceph:
    conf:
      mds:
        debug mds: 1/20
      osd:
        osd op thread timeout: 60
    fs: btrfs
    log-whitelist:
    - slow request
    sha1: 9a7a9d06c0623ccc116a1d3b71c765c20a17e98e
  s3tests:
    branch: last
  workunit:
    sha1: 9a7a9d06c0623ccc116a1d3b71c765c20a17e98e
roles:
- - mon.a
  - mon.c
  - osd.0
  - osd.1
  - osd.2
- - mon.b
  - mds.a
  - osd.3
  - osd.4
  - osd.5
- - client.0
targets:
  ubuntu@plana54.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC60+3L8IN2WBpWHY94YAuCOMVdKs3xUqYQpO1ie127fBomk7fiEhR0RovhmDHWIzNr3qvvNkIp9Y+MHUcpZ7C4MFFMYsy2+zq026Ag3XLEOyZWDSyPfMapd5+nmuvxJqEvAx4wAWBhYVEB3aPFmDmz4mayZ9aSYoA1lhsClxfYpAHZ0zRWX3kY1KxXlk6UrZy0igYGvKIvmubkYcmFzOPsI3aWpgWU1rEXGWsFHOlwaor0KJPnpEsZYTrlPyLZqJcKbI/EcHgti0ak22vsDT7LVMKoyPXeUFL5ZGUEpuqQ+IMiECCMKa8X8vPG2MN9V6DK3gQezF+lo5CRCAu7DYdn
  ubuntu@plana57.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCuMOcu2XPQovy/Qzmwyvc9tvGP9JZVJ6cqiJ3RPOSGgAifKLTxe2ramHpD8AKcdthu8VAfouFpZK4CtBWKJowurR+4yZKgEugzvYuZ/nK/np56vreBQmRBWD1vLPtxPsTT3YGu5qx+ixdSwrSxexxc0/7+EW9x1D6knL+OGUNWksoGIRlXxjh9qafbw/1XKeQQF28vxBXHofXUFY8USMUcq5HDuaFfmgKzufH6vk84oqyr/jtGej6b4g6tbGiHPYR+o5tmTQHyxpOxqLZP2RFFqHlQ/QaOmRvSNIoOo+1UbqdcWsLk16/lXIS1mI+BZsZouk1H+fGeMTEUDGktiPW7
  ubuntu@plana58.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSrEdvZzLOAbSb6/bmBSAazRmmiN+zGTF6PoS2K83VPDk5CkrpMjxOnHhOC+FnOBo9tss/XZ/UaBx6BOmzjMqVvCZaS81LQ2hM7fR4qrBOBjFATab2XgaU0sbS1T4ihFrEgN+j5FTBSAiR+F6mEJhxiLaf3IJSDarp9aD5c7pij29pKjzp8FcoKsmLyMbdW+6AzAVwKcdEsawcXEQL8P6CE04hj5r2NdLGKOohSDZYNrXWozQ0sq3l+MMFjQKJKGPzSKvQCVklngex6XgfNTssNI1Se0WqpPlpSJNeajbUegGVEEbYFn5au3eQvrttl78zghEtENwCBCij063zNlkf
tasks:
- internal.lock_machines:
  - 3
  - plana
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.check_conflict: null
- kernel: *id001
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock: null
- ceph:
    log-whitelist:
    - wrongly marked me down
    - objects unfound and apparently lost
- thrashosds: null
- kclient: null
- workunit:
    clients:
      all:
      - suites/ffsb.sh
ubuntu@teuthology:/a/teuthology-2013-02-27_20:00:05-regression-last-master-basic/13866$ cat summary.yaml 
ceph-sha1: 9a7a9d06c0623ccc116a1d3b71c765c20a17e98e
client.0-kernel-sha1: 83ca14fdd35821554058e5fd4fa7b118ee504a33
description: collection:kernel-thrash clusters:fixed-3.yaml fs:btrfs.yaml thrashers:default.yaml
  workloads:kclient_workunit_suites_ffsb.yaml
duration: 1552.3628141880035
failure_reason: '"2013-02-27 21:23:55.491567 osd.5 10.214.132.20:6805/8521 1 : [WRN]
  client.4120 10.214.132.24:0/2690850591 misdirected client.4120.1:3260 pg 0.e5281787
  to osd.5 not [0,1] in e67/67" in cluster log'
flavor: basic
mon.a-kernel-sha1: 83ca14fdd35821554058e5fd4fa7b118ee504a33
mon.b-kernel-sha1: 83ca14fdd35821554058e5fd4fa7b118ee504a33
owner: scheduled_teuthology@teuthology
success: false

History

#1 Updated by Sage Weil about 11 years ago

  • Status changed from New to 12
  • Priority changed from Normal to Urgent

#2 Updated by Tamilarasi muthamizhan about 11 years ago

recent log: ubuntu@teuthology:/a/teuthology-2013-03-04_20:00:05-regression-bobtail-master-basic/16316

2013-03-04 20:49:35.837524 7fb4bffd5700 0 log [WRN] : slow request 70.125732 seconds old, received at 2013-03-04 20:48:25.711645: osd_op(client.4125.1:63166 100000001b7.0000000c [write 573440~4096 [1@4194
304]] 0.c74fd2f7 RETRY snapc 1=[]) v4 currently reached pg
2013-03-04 20:49:35.837535 7fb4bffd5700 0 log [WRN] : slow request 70.125150 seconds old, received at 2013-03-04 20:48:25.712227: osd_op(client.4125.1:63167 100000001b7.0000000c [write 585728~8192 [1@4194
304]] 0.c74fd2f7 RETRY snapc 1=[]) v4 currently reached pg
2013-03-04 20:49:36.361811 7fb4b37bc700 0 log [WRN] : client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:290058 pg 0.cd320aba to osd.2 not [4,2,0] in e60/62
2013-03-04 20:49:36.362399 7fb4b2fbb700 0 log [WRN] : client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:290057 pg 0.cd320aba to osd.2 not [4,2,0] in e60/62
2013-03-04 20:49:36.362880 7fb4b37bc700 0 log [WRN] : client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:290056 pg 0.cd320aba to osd.2 not [4,2,0] in e60/62
2013-03-04 20:49:36.363472 7fb4b2fbb700 0 log [WRN] : client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:290055 pg 0.cd320aba to osd.2 not [4,2,0] in e60/62
2013-03-04 20:49:36.363621 7fb4b37bc700 0 log [WRN] : client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:290054 pg 0.cd320aba to osd.2 not [4,2,0] in e60/62
2013-03-04 20:49:36.363817 7fb4b37bc700 0 log [WRN] : client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:290053 pg 0.cd320aba to osd.2 not [4,2,0] in e60/62
2013-03-04 20:49:36.364409 7fb4b37bc700 0 log [WRN] : client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:290052 pg 0.cd320aba to osd.2 not [4,2,0] in e60/62
2013-03-04 20:49:36.364725 7fb4b2fbb700 0 log [WRN] : client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:290051 pg 0.cd320aba to osd.2 not [4,2,0] in e60/62
2013-03-04 20:49:36.365048 7fb4b2fbb700 0 log [WRN] : client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:290050 pg 0.cd320aba to osd.2 not [4,2,0] in e60/62
2013-03-04 20:49:36.365328 7fb4b37bc700 0 log [WRN] : client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:290049 pg 0.cd320aba to osd.2 not [4,2,0] in e60/62
2013-03-04 20:49:36.365660 7fb4b2fbb700 0 log [WRN] : client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:290048 pg 0.cd320aba to osd.2 not [4,2,0] in e60/62
2013-03-04 20:49:36.365869 7fb4b2fbb700 0 log [WRN] : client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:290047 pg 0.cd320aba to osd.2 not [4,2,0] in e60/62
2013-03-04 20:49:36.519306 7fb4b37bc700 0 log [WRN] : client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:288020 pg 0.e62aca8a to osd.2 not [4,2,0] in e60/62
2013-03-04 20:49:36.519506 7fb4b37bc700 0 log [WRN] : client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:288019 pg 0.e62aca8a to osd.2 not [4,2,0] in e60/62
2013-03-04 20:49:36.519884 7fb4b2fbb700 0 log [WRN] : client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:288018 pg 0.e62aca8a to osd.2 not [4,2,0] in e60/62
2013-03-04 20:49:36.522037 7fb4b2fbb700 0 log [WRN] : client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:288017 pg 0.e62aca8a to osd.2 not [4,2,0] in e60/62
2013-03-04 20:49:36.522224 7fb4b2fbb700 0 log [WRN] : client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:288016 pg 0.31c5c550 to osd.2 not [4,2,3] in e60/62
2013-03-04 20:49:36.522604 7fb4b37bc700 0 log [WRN] : client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:288015 pg 0.31c5c550 to osd.2 not [4,2,3] in e60/62
2013-03-04 20:49:36.522798 7fb4b37bc700 0 log [WRN] : client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:288014 pg 0.31c5c550 to osd.2 not [4,2,3] in e60/62
2013-03-04 20:49:36.524948 7fb4b2fbb700 0 log [WRN] : client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:288013 pg 0.31c5c550 to osd.2 not [4,2,3] in e60/62

ubuntu@teuthology:/a/teuthology-2013-03-04_20:00:05-regression-bobtail-master-basic/16316$ cat config.yaml 
kernel: &id001
  kdb: true
  sha1: 83ca14fdd35821554058e5fd4fa7b118ee504a33
nuke-on-error: true
overrides:
  ceph:
    conf:
      mds:
        debug mds: 1/20
    fs: btrfs
    log-whitelist:
    - slow request
    sha1: ee943c8bcf36f1e2218d8e25edfa38ec5fe4bec2
  s3tests:
    branch: bobtail
  workunit:
    sha1: ee943c8bcf36f1e2218d8e25edfa38ec5fe4bec2
roles:
- - mon.a
  - mon.c
  - osd.0
  - osd.1
  - osd.2
- - mon.b
  - mds.a
  - osd.3
  - osd.4
  - osd.5
- - client.0
targets:
  ubuntu@plana28.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnMB6Wmqywp9d2Ns8PYeySKtDaQCkkfldLWV57qoMTDC9+ZoIJcSUE9OshK8pwO7C14cU+IiFIjY4a/fctylCuopSnneI7tsTq2er/Te2XOA+4Q1L4K2twa0pTwTAPyqqsZODqM4O/QG/voJjKMvrazp85VsmQ7mM1bOdR4wCNbPZoydg1/iO7cxSx8iSb2LdWTx+mRFSrWzHWhqvxwfHDZXtim3U/dej+7dFDZrM8Cq1PsvOXkE+zEkmSAMU64JUPZBTSdYl8Dh1aMMz+BWfIn5ZUFzcKt/7LLims85X4rBGyp1CAmhyTtTDBmNrvemZI6BEl+xA1kamhG+Cp/Z+n
  ubuntu@plana51.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLTsxC+nR+xtTXMbOtazCh7MOzgBKjX/oCLMP16k0AtH8Ui92tlqsfNxcHczUol0DNzxCITgrhF8FTvgM3EgkbOUVAxGj+xLqfxsdlf58nTVXbm/pOGYnvOI8CvA4DgISHDbkzuFH4FKtR8qNTTFVmtEXaZ+jpSvn7vrYuI/Uu9XZOQh73phYW8zvVB1x8770czM0Gy2wgxdNguKy6L/Q9ShsLcFfm8Uvxf6aXb3qmuxwGhqYsMlNl0X3AjoOwmow74rodlcMvQP/pAQdjMZfe1lBPqsjmU518BE5eo7zV3O9iF6ahOrm8igOu9bfki0G52R22pA3hE9BPKPfzA0hL
  ubuntu@plana52.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9kswBp2g5ZV1Qrvlee8MvUOCNdubQFqUBr5WSsmFBODqEuiitWbhuBu2Ucz0lBMf41DpMKLeYDN0lIC94GZmGaiCN+Ak9Ia05d/uRvesT2nDgHB3Z9J/zEFlY8RVxL3xhD+hq4u8dbASlqqoMDiBP+7efZMxt4Ndnzr/yOxge3KenxyQImBUS+OV+BqnfCOHf6BqM33U1leXz2kng7ocxoE91DAMslKD/2DPRSYEhfucUJZk6IYevr/g0JVhbfvjSlZzwUEfTyVmPeqNyls/U+azhKlvQbqpb+ttc02RNydQ1YgOgHFCaqd9Vm8XjUU6vYGlkFHZ+BMJuEwA9AH/D
tasks:
- internal.lock_machines: 3
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.check_conflict: null
- kernel: *id001
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock: null
- ceph:
    log-whitelist:
    - wrongly marked me down
    - objects unfound and apparently lost
- thrashosds: null
- kclient: null
- workunit:
    clients:
      all:
      - suites/ffsb.sh
ubuntu@teuthology:/a/teuthology-2013-03-04_20:00:05-regression-bobtail-master-basic/16316$ cat summary.yaml 
ceph-sha1: ee943c8bcf36f1e2218d8e25edfa38ec5fe4bec2
client.0-kernel-sha1: 83ca14fdd35821554058e5fd4fa7b118ee504a33
description: collection:kernel-thrash clusters:fixed-3.yaml fs:btrfs.yaml thrashers:default.yaml
  workloads:kclient_workunit_suites_ffsb.yaml
duration: 6913.5912799835205
failure_reason: '"2013-03-04 20:49:36.361825 osd.2 10.214.132.27:6806/4701 331 : [WRN]
  client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:290058 pg 0.cd320aba
  to osd.2 not [4,2,0] in e60/62" in cluster log'
flavor: basic
mon.a-kernel-sha1: 83ca14fdd35821554058e5fd4fa7b118ee504a33
mon.b-kernel-sha1: 83ca14fdd35821554058e5fd4fa7b118ee504a33
owner: scheduled_teuthology@teuthology
success: false

#3 Updated by Sage Weil about 11 years ago

  • Status changed from 12 to 7

I screwed up the decoding of the pgid in the recent kernel client changes. Testing a fix now, wip-osdmap.

#4 Updated by Sage Weil about 11 years ago

  • Status changed from 7 to Fix Under Review

#5 Updated by Sage Weil about 11 years ago

  • Status changed from Fix Under Review to Resolved

commit 2f60d3028438dd1fef122d37786ee685d727e8a7
Author: Sage Weil <>
Date: Wed Mar 6 14:57:03 2013 -0800

libceph: fix decoding of pgids

Also available in: Atom PDF