Actions
Bug #6057
closedosd: log bound mismatch after bobtail -> dumpling -> next upgrade
% Done:
100%
Source:
Q/A
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
2013-08-19 06:24:27.814763 osd.2 0.0.0.0:6808/24619 1 : [ERR] 3.4 log bound mismatch, info (0''0,30''164] actual [21''1,23''162]" in cluster log
ubuntu@teuthology:/a/teuthology-2013-08-19_01:31:11-upgrade-next-testing-basic-plana/1453$ cat orig.config.yaml
kernel:
kdb: true
sha1: 546140dd51e9ec7e34fe0b0a5814240828f68f7d
machine_type: plana
nuke-on-error: true
os_type: ubuntu
overrides:
admin_socket:
branch: next
ceph:
conf:
mon:
debug mon: 20
debug ms: 1
debug paxos: 20
log-whitelist:
- slow request
sha1: b007b3304c2020aa9f122ec6fef83a909053db3a
ceph-deploy:
branch:
dev: next
conf:
client:
log file: /var/log/ceph/ceph-$name.$pid.log
mon:
debug mon: 1
debug ms: 20
debug paxos: 20
install:
ceph:
sha1: b007b3304c2020aa9f122ec6fef83a909053db3a
s3tests:
branch: next
workunit:
sha1: b007b3304c2020aa9f122ec6fef83a909053db3a
roles:
- - mon.a
- mds.a
- osd.0
- osd.1
- - mon.b
- mon.c
- osd.2
- osd.3
- - client.0
tasks:
- chef: null
- clock.check: null
- install:
branch: bobtail
- ceph: null
- rgw: null
- s3tests:
client.0:
force-branch: bobtail
rgw_server: client.0
- install.upgrade:
all:
branch: dumpling
- ceph.restart:
- mon.a
- mon.b
- mon.c
- mds.a
- osd.0
- osd.1
- osd.2
- osd.3
- rgw.client.0
- s3readwrite:
client.0:
readwrite:
bucket: rwtest
duration: 300
files:
num: 10
size: 2000
stddev: 500
readers: 10
writers: 3
rgw_server: client.0
- install.upgrade:
all:
branch: next
- ceph.restart:
- mon.a
- mon.b
- mon.c
- mds.a
- osd.0
- osd.1
- osd.2
- osd.3
- rgw.client.0
- s3readwrite:
client.0:
readwrite:
bucket: rwtest
duration: 300
files:
num: 10
size: 2000
stddev: 500
readers: 10
writers: 3
rgw_server: client.0
teuthology_branch: next
there are about ~30 identical failures
Updated by Tamilarasi muthamizhan over 10 years ago
- Subject changed from osd: log bound mismatch after bobtail -> dumpling upgrade to osd: log bound mismatch after bobtail -> dumpling -> next upgrade
Updated by Sage Weil over 10 years ago
<initial start> as bobtail ... 2013-08-19 16:12:06.255051 7f835a7f8700 10 osd.0 pg_epoch: 27 pg[4.0( v 27'7 (0'0,27'7] local-les=12 n=3 ec=10 les/c 12/12 10/10/10) [1,0] r=1 lpr=12 luod=0'0 lcod 15'6 active] sub_op_modify_applied on 0x327ea80 op osd_sub_op(client.4110.0:5294 4.0 a45046b8/gc.12/head//4 [] v 27'7 snapset=0=[]:[] snapc=0=[$ ... <restart> at 0.67.1-7-g96d719e ... 2013-08-19 16:18:59.394638 7fcce4066780 10 osd.0 27 load_pgs loaded pg[4.0( v 27'7 (0'0,27'7] local-les=12 n=3 ec=10 les/c 12/12 10/10/10) [1,0] r=1 lpr=27 lcod 0'0 inactive NOTIFY] log(0'0,27'7] .... 2013-08-19 16:25:16.516233 7fccd90b6700 10 osd.0 pg_epoch: 34 pg[4.0( v 34'538 (0'0,34'538] local-les=32 n=3 ec=10 les/c 32/32 31/31/31) [1,0] r=1 lpr=31 pi=10-30/4 luod=0'0 lcod 34'537 active] sub_op_modify_applied on 0x2629200 op osd_sub_op(client.4716.0:38049 4.0 990e66d8/gc.0/head//4 [] v 34'538 snapse$ ... <restart> as 0.67-238-g68c1c70 ... 2013-08-19 16:38:42.403570 7f0b8b909780 10 osd.0 34 load_pgs loaded pg[4.0( v 34'538 (0'0,34'538] local-les=32 n=3 ec=10 les/c 32/32 31/31/31) [1,0] r=1 lpr=34 pi=10-30/4 (log bound mismatch, actual=[13'1,27'7]) lcod 0'0 inactive NOTIFY] log(0'0,34'538]
Updated by Sage Weil over 10 years ago
fatty:/home/sage/tmp/6057/ceph-osd.0.log for the full log
Updated by Samuel Just over 10 years ago
Ran the above yaml on wip-6057, seems to work.
Updated by Sage Weil over 10 years ago
- Status changed from New to Resolved
yay, tested ok for me too. merged and backported
Actions