Project

General

Profile

Bug #19067

missing set not persisted

Added by Sage Weil about 7 years ago. Updated over 6 years ago.

Status:
Need More Info
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
OSD
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2017-02-23 17:25:07.129003 7fb7843fa700 10 osd.5 pg_epoch: 2486 pg[1.2d( empty local-les=0 n=0 ec=1089 les/c/f 2479/2479/0 2484/2486/2482) [5,3] r=0 lpr=2486 pi=2478-2485/3 crt=0'0 mlcod 0'0 peering] proc_master_log for osd.2: log((0'0,6'5], crt=6'5) missing(0)
2017-02-23 17:25:07.129010 7fb7843fa700 10 merge_log log((0'0,6'5], crt=6'5) from osd.2 into log((0'0,0'0], crt=0'0)
2017-02-23 17:25:07.129014 7fb7843fa700 10 merge_log extending head to 6'5
2017-02-23 17:25:07.129015 7fb7843fa700 20   ? 6'4 (0'0) modify   1:b50e409b:::mds_snaptable:head by mds.0.7:29 2017-02-23 16:19:29.556067
2017-02-23 17:25:07.129023 7fb7843fa700 20 update missing, append 6'4 (0'0) modify   1:b50e409b:::mds_snaptable:head by mds.0.7:29 2017-02-23 16:19:29.556067
2017-02-23 17:25:07.129032 7fb7843fa700 10 merge_log result log((0'0,6'5], crt=6'5) missing(1) changed=1
...
2017-02-23 17:25:07.129286 7fb7935b1700 10 filestore(/var/lib/ceph/osd/ceph-5) _do_transaction on 0x7fb7b172ad80
2017-02-23 17:25:07.129297 7fb7935b1700 15 filestore(/var/lib/ceph/osd/ceph-5) _omap_rmkeyrange 1.2d_head/#1:b4000000::::head# [0000000000.00000000000000000000,4294967295.18446744073709551615]
2017-02-23 17:25:07.129307 7fb7935b1700 15 filestore(/var/lib/ceph/osd/ceph-5) get_omap_iterator 1.2d_head/#1:b4000000::::head#
2017-02-23 17:25:07.129397 7fb7935b1700 15 filestore(/var/lib/ceph/osd/ceph-5) _omap_rmkeys 1.2d_head/#1:b4000000::::head#
2017-02-23 17:25:07.129413 7fb7935b1700 10 filestore oid: #1:b4000000::::head# not skipping op, *spos 96182.0.1
2017-02-23 17:25:07.129416 7fb7935b1700 10 filestore  > header.spos 0.0.0
2017-02-23 17:25:07.129432 7fb7935b1700 15 filestore(/var/lib/ceph/osd/ceph-5) _omap_setkeys 1.2d_head/#1:b4000000::::head#
2017-02-23 17:25:07.129437 7fb7935b1700 10 filestore oid: #1:b4000000::::head# not skipping op, *spos 96182.0.2
2017-02-23 17:25:07.129439 7fb7935b1700 10 filestore  > header.spos 0.0.0
2017-02-23 17:25:07.129488 7fb7935b1700 20 filestore(/var/lib/ceph/osd/ceph-5) _omap_setkeys 1.2d_head/#1:b4000000::::head# = 0

but then later, when we restart,
2017-02-23 17:25:14.399629 7f952edb3980 15 filestore(/var/lib/ceph/osd/ceph-5) omap_get_values 1.2d_head/#1:b4000000::::head#
2017-02-23 17:25:14.399670 7f952edb3980 15 filestore(/var/lib/ceph/osd/ceph-5) omap_get_values 1.2d_head/#1:b4000000::::head# = 0()
2017-02-23 17:25:14.399676 7f952edb3980 10 osd.5 2487 pgid 1.2d coll 1.2d_head
2017-02-23 17:25:14.399678 7f952edb3980 15 filestore(/var/lib/ceph/osd/ceph-5) omap_get_values 1.2d_head/#1:b4000000::::head#
2017-02-23 17:25:14.399704 7f952edb3980 15 filestore(/var/lib/ceph/osd/ceph-5) omap_get_values 1.2d_head/#1:b4000000::::head# = 0()
2017-02-23 17:25:14.399710 7f952edb3980 10 osd.5 2487 _open_lock_pg 1.2d
2017-02-23 17:25:14.399713 7f952edb3980 10 osd.5 2487 _get_pool 1
2017-02-23 17:25:14.399728 7f952edb3980  5 osd.5 pg_epoch: 2486 pg[1.2d(unlocked)] enter Initial
2017-02-23 17:25:14.399734 7f952edb3980 20 osd.5 pg_epoch: 2486 pg[1.2d(unlocked)] enter NotTrimming
2017-02-23 17:25:14.399741 7f952edb3980 15 filestore(/var/lib/ceph/osd/ceph-5) omap_get_values 1.2d_head/#1:b4000000::::head#
2017-02-23 17:25:14.399776 7f952edb3980 15 filestore(/var/lib/ceph/osd/ceph-5) omap_get_values 1.2d_head/#1:b4000000::::head# = 0()
2017-02-23 17:25:14.399790 7f952edb3980 20 read_log_and_missing coll 1.2d_head log_oid #1:b4000000::::head#
2017-02-23 17:25:14.399803 7f952edb3980 10 filestore(/var/lib/ceph/osd/ceph-5) stat 1.2d_head/#1:b4000000::::head# = 0 (size 0)
2017-02-23 17:25:14.399806 7f952edb3980 15 filestore(/var/lib/ceph/osd/ceph-5) get_omap_iterator 1.2d_head/#1:b4000000::::head#
2017-02-23 17:25:14.399827 7f952edb3980 20 read_log_and_missing 6'4 (0'0) modify   1:b50e409b:::mds_snaptable:head by mds.0.7:29 2017-02-23 16:19:29.556067 0
2017-02-23 17:25:14.399845 7f952edb3980 10 read_log_and_missing done
2017-02-23 17:25:14.399851 7f952edb3980 10 osd.5 2487 PG 1.2d must upgrade...
2017-02-23 17:25:14.399862 7f952edb3980  5 write_log_and_missing with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, writeout_from: 4294967295'18446744073709551615, trimmed: , clear_divergent_priors: 0
...
2017-02-23 17:25:14.400166 7f952edb3980 10 osd.5 pg_epoch: 2486 pg[1.2d( v 6'5 lc 0'0 (0'0,6'5] local-les=2479 n=0 ec=1089 les/c/f 2479/2479/0 2484/2486/2482) [5,3] r=0 lpr=0 pi=2478-2485/3 crt=6'5 mlcod 0'0 inactive] handle_loaded

and there is no more missing (no m=1 in pg prefix)!

/a/sage-2017-02-23_15:15:06-rados-master---basic-smithi/852213

This manifests as an error like

"2017-02-23 17:59:27.872305 osd.5 172.21.15.143:6818/22155 94 : cluster [ERR] 1.2d shard 5 missing 1:b50e409b:::mds_snaptable:head" in cluster log

History

#1 Updated by Sage Weil about 7 years ago

  • Description updated (diff)

#2 Updated by Sage Weil almost 7 years ago

  • Priority changed from Immediate to Urgent

#3 Updated by Greg Farnum almost 7 years ago

  • Project changed from Ceph to RADOS
  • Due date set to 06/13/2017
  • Component(RADOS) OSD added

#4 Updated by Sage Weil over 6 years ago

  • Status changed from 12 to Need More Info
  • Priority changed from Urgent to Normal

Also available in: Atom PDF