Bug #4298
misdirected op in ffsb test
0%
Description
Logs: ubuntu@teuthology:/a/teuthology-2013-02-27_20:00:05-regression-last-master-basic/13866
ubuntu@teuthology:/a/teuthology-2013-02-27_20:00:05-regression-last-master-basic/13866$ cat config.yaml kernel: &id001 kdb: true sha1: 83ca14fdd35821554058e5fd4fa7b118ee504a33 nuke-on-error: true overrides: ceph: conf: mds: debug mds: 1/20 osd: osd op thread timeout: 60 fs: btrfs log-whitelist: - slow request sha1: 9a7a9d06c0623ccc116a1d3b71c765c20a17e98e s3tests: branch: last workunit: sha1: 9a7a9d06c0623ccc116a1d3b71c765c20a17e98e roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mds.a - osd.3 - osd.4 - osd.5 - - client.0 targets: ubuntu@plana54.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC60+3L8IN2WBpWHY94YAuCOMVdKs3xUqYQpO1ie127fBomk7fiEhR0RovhmDHWIzNr3qvvNkIp9Y+MHUcpZ7C4MFFMYsy2+zq026Ag3XLEOyZWDSyPfMapd5+nmuvxJqEvAx4wAWBhYVEB3aPFmDmz4mayZ9aSYoA1lhsClxfYpAHZ0zRWX3kY1KxXlk6UrZy0igYGvKIvmubkYcmFzOPsI3aWpgWU1rEXGWsFHOlwaor0KJPnpEsZYTrlPyLZqJcKbI/EcHgti0ak22vsDT7LVMKoyPXeUFL5ZGUEpuqQ+IMiECCMKa8X8vPG2MN9V6DK3gQezF+lo5CRCAu7DYdn ubuntu@plana57.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCuMOcu2XPQovy/Qzmwyvc9tvGP9JZVJ6cqiJ3RPOSGgAifKLTxe2ramHpD8AKcdthu8VAfouFpZK4CtBWKJowurR+4yZKgEugzvYuZ/nK/np56vreBQmRBWD1vLPtxPsTT3YGu5qx+ixdSwrSxexxc0/7+EW9x1D6knL+OGUNWksoGIRlXxjh9qafbw/1XKeQQF28vxBXHofXUFY8USMUcq5HDuaFfmgKzufH6vk84oqyr/jtGej6b4g6tbGiHPYR+o5tmTQHyxpOxqLZP2RFFqHlQ/QaOmRvSNIoOo+1UbqdcWsLk16/lXIS1mI+BZsZouk1H+fGeMTEUDGktiPW7 ubuntu@plana58.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSrEdvZzLOAbSb6/bmBSAazRmmiN+zGTF6PoS2K83VPDk5CkrpMjxOnHhOC+FnOBo9tss/XZ/UaBx6BOmzjMqVvCZaS81LQ2hM7fR4qrBOBjFATab2XgaU0sbS1T4ihFrEgN+j5FTBSAiR+F6mEJhxiLaf3IJSDarp9aD5c7pij29pKjzp8FcoKsmLyMbdW+6AzAVwKcdEsawcXEQL8P6CE04hj5r2NdLGKOohSDZYNrXWozQ0sq3l+MMFjQKJKGPzSKvQCVklngex6XgfNTssNI1Se0WqpPlpSJNeajbUegGVEEbYFn5au3eQvrttl78zghEtENwCBCij063zNlkf tasks: - internal.lock_machines: - 3 - plana - internal.save_config: null - internal.check_lock: null - internal.connect: null - internal.check_conflict: null - kernel: *id001 - internal.base: null - internal.archive: null - internal.coredump: null - internal.syslog: null - internal.timer: null - chef: null - clock: null - ceph: log-whitelist: - wrongly marked me down - objects unfound and apparently lost - thrashosds: null - kclient: null - workunit: clients: all: - suites/ffsb.sh ubuntu@teuthology:/a/teuthology-2013-02-27_20:00:05-regression-last-master-basic/13866$ cat summary.yaml ceph-sha1: 9a7a9d06c0623ccc116a1d3b71c765c20a17e98e client.0-kernel-sha1: 83ca14fdd35821554058e5fd4fa7b118ee504a33 description: collection:kernel-thrash clusters:fixed-3.yaml fs:btrfs.yaml thrashers:default.yaml workloads:kclient_workunit_suites_ffsb.yaml duration: 1552.3628141880035 failure_reason: '"2013-02-27 21:23:55.491567 osd.5 10.214.132.20:6805/8521 1 : [WRN] client.4120 10.214.132.24:0/2690850591 misdirected client.4120.1:3260 pg 0.e5281787 to osd.5 not [0,1] in e67/67" in cluster log' flavor: basic mon.a-kernel-sha1: 83ca14fdd35821554058e5fd4fa7b118ee504a33 mon.b-kernel-sha1: 83ca14fdd35821554058e5fd4fa7b118ee504a33 owner: scheduled_teuthology@teuthology success: false
History
#1 Updated by Sage Weil about 11 years ago
- Status changed from New to 12
- Priority changed from Normal to Urgent
#2 Updated by Tamilarasi muthamizhan about 11 years ago
recent log: ubuntu@teuthology:/a/teuthology-2013-03-04_20:00:05-regression-bobtail-master-basic/16316
2013-03-04 20:49:35.837524 7fb4bffd5700 0 log [WRN] : slow request 70.125732 seconds old, received at 2013-03-04 20:48:25.711645: osd_op(client.4125.1:63166 100000001b7.0000000c [write 573440~4096 [1@4194
304]] 0.c74fd2f7 RETRY snapc 1=[]) v4 currently reached pg
2013-03-04 20:49:35.837535 7fb4bffd5700 0 log [WRN] : slow request 70.125150 seconds old, received at 2013-03-04 20:48:25.712227: osd_op(client.4125.1:63167 100000001b7.0000000c [write 585728~8192 [1@4194
304]] 0.c74fd2f7 RETRY snapc 1=[]) v4 currently reached pg
2013-03-04 20:49:36.361811 7fb4b37bc700 0 log [WRN] : client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:290058 pg 0.cd320aba to osd.2 not [4,2,0] in e60/62
2013-03-04 20:49:36.362399 7fb4b2fbb700 0 log [WRN] : client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:290057 pg 0.cd320aba to osd.2 not [4,2,0] in e60/62
2013-03-04 20:49:36.362880 7fb4b37bc700 0 log [WRN] : client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:290056 pg 0.cd320aba to osd.2 not [4,2,0] in e60/62
2013-03-04 20:49:36.363472 7fb4b2fbb700 0 log [WRN] : client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:290055 pg 0.cd320aba to osd.2 not [4,2,0] in e60/62
2013-03-04 20:49:36.363621 7fb4b37bc700 0 log [WRN] : client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:290054 pg 0.cd320aba to osd.2 not [4,2,0] in e60/62
2013-03-04 20:49:36.363817 7fb4b37bc700 0 log [WRN] : client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:290053 pg 0.cd320aba to osd.2 not [4,2,0] in e60/62
2013-03-04 20:49:36.364409 7fb4b37bc700 0 log [WRN] : client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:290052 pg 0.cd320aba to osd.2 not [4,2,0] in e60/62
2013-03-04 20:49:36.364725 7fb4b2fbb700 0 log [WRN] : client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:290051 pg 0.cd320aba to osd.2 not [4,2,0] in e60/62
2013-03-04 20:49:36.365048 7fb4b2fbb700 0 log [WRN] : client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:290050 pg 0.cd320aba to osd.2 not [4,2,0] in e60/62
2013-03-04 20:49:36.365328 7fb4b37bc700 0 log [WRN] : client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:290049 pg 0.cd320aba to osd.2 not [4,2,0] in e60/62
2013-03-04 20:49:36.365660 7fb4b2fbb700 0 log [WRN] : client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:290048 pg 0.cd320aba to osd.2 not [4,2,0] in e60/62
2013-03-04 20:49:36.365869 7fb4b2fbb700 0 log [WRN] : client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:290047 pg 0.cd320aba to osd.2 not [4,2,0] in e60/62
2013-03-04 20:49:36.519306 7fb4b37bc700 0 log [WRN] : client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:288020 pg 0.e62aca8a to osd.2 not [4,2,0] in e60/62
2013-03-04 20:49:36.519506 7fb4b37bc700 0 log [WRN] : client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:288019 pg 0.e62aca8a to osd.2 not [4,2,0] in e60/62
2013-03-04 20:49:36.519884 7fb4b2fbb700 0 log [WRN] : client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:288018 pg 0.e62aca8a to osd.2 not [4,2,0] in e60/62
2013-03-04 20:49:36.522037 7fb4b2fbb700 0 log [WRN] : client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:288017 pg 0.e62aca8a to osd.2 not [4,2,0] in e60/62
2013-03-04 20:49:36.522224 7fb4b2fbb700 0 log [WRN] : client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:288016 pg 0.31c5c550 to osd.2 not [4,2,3] in e60/62
2013-03-04 20:49:36.522604 7fb4b37bc700 0 log [WRN] : client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:288015 pg 0.31c5c550 to osd.2 not [4,2,3] in e60/62
2013-03-04 20:49:36.522798 7fb4b37bc700 0 log [WRN] : client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:288014 pg 0.31c5c550 to osd.2 not [4,2,3] in e60/62
2013-03-04 20:49:36.524948 7fb4b2fbb700 0 log [WRN] : client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:288013 pg 0.31c5c550 to osd.2 not [4,2,3] in e60/62
ubuntu@teuthology:/a/teuthology-2013-03-04_20:00:05-regression-bobtail-master-basic/16316$ cat config.yaml kernel: &id001 kdb: true sha1: 83ca14fdd35821554058e5fd4fa7b118ee504a33 nuke-on-error: true overrides: ceph: conf: mds: debug mds: 1/20 fs: btrfs log-whitelist: - slow request sha1: ee943c8bcf36f1e2218d8e25edfa38ec5fe4bec2 s3tests: branch: bobtail workunit: sha1: ee943c8bcf36f1e2218d8e25edfa38ec5fe4bec2 roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mds.a - osd.3 - osd.4 - osd.5 - - client.0 targets: ubuntu@plana28.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnMB6Wmqywp9d2Ns8PYeySKtDaQCkkfldLWV57qoMTDC9+ZoIJcSUE9OshK8pwO7C14cU+IiFIjY4a/fctylCuopSnneI7tsTq2er/Te2XOA+4Q1L4K2twa0pTwTAPyqqsZODqM4O/QG/voJjKMvrazp85VsmQ7mM1bOdR4wCNbPZoydg1/iO7cxSx8iSb2LdWTx+mRFSrWzHWhqvxwfHDZXtim3U/dej+7dFDZrM8Cq1PsvOXkE+zEkmSAMU64JUPZBTSdYl8Dh1aMMz+BWfIn5ZUFzcKt/7LLims85X4rBGyp1CAmhyTtTDBmNrvemZI6BEl+xA1kamhG+Cp/Z+n ubuntu@plana51.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLTsxC+nR+xtTXMbOtazCh7MOzgBKjX/oCLMP16k0AtH8Ui92tlqsfNxcHczUol0DNzxCITgrhF8FTvgM3EgkbOUVAxGj+xLqfxsdlf58nTVXbm/pOGYnvOI8CvA4DgISHDbkzuFH4FKtR8qNTTFVmtEXaZ+jpSvn7vrYuI/Uu9XZOQh73phYW8zvVB1x8770czM0Gy2wgxdNguKy6L/Q9ShsLcFfm8Uvxf6aXb3qmuxwGhqYsMlNl0X3AjoOwmow74rodlcMvQP/pAQdjMZfe1lBPqsjmU518BE5eo7zV3O9iF6ahOrm8igOu9bfki0G52R22pA3hE9BPKPfzA0hL ubuntu@plana52.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9kswBp2g5ZV1Qrvlee8MvUOCNdubQFqUBr5WSsmFBODqEuiitWbhuBu2Ucz0lBMf41DpMKLeYDN0lIC94GZmGaiCN+Ak9Ia05d/uRvesT2nDgHB3Z9J/zEFlY8RVxL3xhD+hq4u8dbASlqqoMDiBP+7efZMxt4Ndnzr/yOxge3KenxyQImBUS+OV+BqnfCOHf6BqM33U1leXz2kng7ocxoE91DAMslKD/2DPRSYEhfucUJZk6IYevr/g0JVhbfvjSlZzwUEfTyVmPeqNyls/U+azhKlvQbqpb+ttc02RNydQ1YgOgHFCaqd9Vm8XjUU6vYGlkFHZ+BMJuEwA9AH/D tasks: - internal.lock_machines: 3 - internal.save_config: null - internal.check_lock: null - internal.connect: null - internal.check_conflict: null - kernel: *id001 - internal.base: null - internal.archive: null - internal.coredump: null - internal.syslog: null - internal.timer: null - chef: null - clock: null - ceph: log-whitelist: - wrongly marked me down - objects unfound and apparently lost - thrashosds: null - kclient: null - workunit: clients: all: - suites/ffsb.sh ubuntu@teuthology:/a/teuthology-2013-03-04_20:00:05-regression-bobtail-master-basic/16316$ cat summary.yaml ceph-sha1: ee943c8bcf36f1e2218d8e25edfa38ec5fe4bec2 client.0-kernel-sha1: 83ca14fdd35821554058e5fd4fa7b118ee504a33 description: collection:kernel-thrash clusters:fixed-3.yaml fs:btrfs.yaml thrashers:default.yaml workloads:kclient_workunit_suites_ffsb.yaml duration: 6913.5912799835205 failure_reason: '"2013-03-04 20:49:36.361825 osd.2 10.214.132.27:6806/4701 331 : [WRN] client.4125 10.214.131.12:0/3805183745 misdirected client.4125.1:290058 pg 0.cd320aba to osd.2 not [4,2,0] in e60/62" in cluster log' flavor: basic mon.a-kernel-sha1: 83ca14fdd35821554058e5fd4fa7b118ee504a33 mon.b-kernel-sha1: 83ca14fdd35821554058e5fd4fa7b118ee504a33 owner: scheduled_teuthology@teuthology success: false
#3 Updated by Sage Weil about 11 years ago
- Status changed from 12 to 7
I screwed up the decoding of the pgid in the recent kernel client changes. Testing a fix now, wip-osdmap.
#4 Updated by Sage Weil about 11 years ago
- Status changed from 7 to Fix Under Review
#5 Updated by Sage Weil about 11 years ago
- Status changed from Fix Under Review to Resolved
commit 2f60d3028438dd1fef122d37786ee685d727e8a7
Author: Sage Weil <sage@inktank.com>
Date: Wed Mar 6 14:57:03 2013 -0800
libceph: fix decoding of pgids