Project

General

Profile

Actions
Copy link

Bug #19440

open

osd: trims maps taht pgs haven't consumed yet when there are gaps

Added by Sage Weil about 7 years ago. Updated over 4 years ago.

Status:
New
Priority:
Normal
Assignee:
Sage Weil
Category:
Correctness/Safety
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Monitor, OSD
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2017-03-31 12:03:22.514217 7fac11bb3700 20 osd.1 pg_epoch: 250 pg[0.7( empty local-les=83 n=0 ec=1 les/c/f 83/83/0 82/82/5) [1,5,4] r=0 lpr=191 crt=0'0 mlcod 0'0 peering] handle_activate_map: Not dirtying info: last_persisted is 191 while current is 250
2017-03-31 12:03:22.514223 7fac11bb3700 10 osd.1 pg_epoch: 250 pg[0.7( empty local-les=83 n=0 ec=1 les/c/f 83/83/0 82/82/5) [1,5,4] r=0 lpr=191 crt=0'0 mlcod 0'0 peering] handle_peering_event: epoch_sent: 250 epoch_requested: 250 NullEvt
2017-03-31 12:03:22.514229 7fac11bb3700 10 log is not dirty
...note epoch is 250...
...
we start removing maps...
2017-03-31 12:03:22.515745 7fac1d45e700  1 -- 172.21.15.181:6804/3478 <== mon.0 172.21.15.111:6789/0 10 ==== osd_map(253..253 src has 253..288) v3 ==== 2900+0+0 (1484330653 0 0) 0x562069a8e000 con 0x56206d3a8000
2017-03-31 12:03:22.515758 7fac1d45e700 20 osd.1 250 OSD::ms_dispatch: osd_map(253..253 src has 253..288) v3
2017-03-31 12:03:22.515762 7fac1d45e700 10 osd.1 250 do_waiters -- start
2017-03-31 12:03:22.515764 7fac1d45e700 10 osd.1 250 do_waiters -- finish
2017-03-31 12:03:22.515766 7fac1d45e700 20 osd.1 250 _dispatch 0x562069a8e000 osd_map(253..253 src has 253..288) v3
2017-03-31 12:03:22.515771 7fac1d45e700  3 osd.1 250 handle_osd_map epochs [253,253], i have 250, src has [253,288]
2017-03-31 12:03:22.515775 7fac1d45e700 10 osd.1 250 handle_osd_map message skips epochs 251..252
2017-03-31 12:03:22.515778 7fac1d45e700 10 osd.1 250 handle_osd_map  got full map for epoch 253
2017-03-31 12:03:22.515905 7fac1d45e700 20 osd.1 250 got_full_map 253, nothing requested
2017-03-31 12:03:22.515914 7fac1d45e700 20 osd.1 250  removing old osdmap epoch 1
2017-03-31 12:03:22.515924 7fac1d45e700 20 osd.1 250  removing old osdmap epoch 2
...
and shortly after, the next event on that pg thread above is:
2017-03-31 12:03:22.566389 7fac11bb3700 20 osd.1 253 get_map 251 - loading and decoding 0x562069d2f440
...
2017-03-31 12:03:22.566431 7fac11bb3700 20 bluestore(/var/lib/ceph/osd/ceph-1).collection(meta 0x562068e72200)  r -2 v.len 0
Actions
Copy link
 #1

Updated by Sage Weil about 7 years ago

  • Subject changed from osd: trims maps taht pgs haven't consumed ye to osd: trims maps taht pgs haven't consumed yet
  • Description updated (diff)
  • Status changed from New to Fix Under Review
  • Assignee set to Sage Weil
  • Priority changed from Normal to Urgent
Actions
Copy link
 #3

Updated by Sage Weil about 7 years ago

/a/sage-2017-03-31_02:07:33-rados:thrash-wip-kill-subop-reordered---basic-smithi/968193

Actions
Copy link
 #4

Updated by Sage Weil about 7 years ago

Totally misdiagnosed this one; closing the PR.

The problem looks like it's related to map skipping. Here:

2017-03-31 12:03:22.515745 7fac1d45e700  1 -- 172.21.15.181:6804/3478 <== mon.0 172.21.15.111:6789/0 10 ==== osd_map(253..253 src has 253..288) v3 ==== 2900+0+0 (1484330653 0 0) 0x562069a8e000 con 0x56206d3a8000
2017-03-31 12:03:22.515758 7fac1d45e700 20 osd.1 250 OSD::ms_dispatch: osd_map(253..253 src has 253..288) v3
2017-03-31 12:03:22.515762 7fac1d45e700 10 osd.1 250 do_waiters -- start
2017-03-31 12:03:22.515764 7fac1d45e700 10 osd.1 250 do_waiters -- finish
2017-03-31 12:03:22.515766 7fac1d45e700 20 osd.1 250 _dispatch 0x562069a8e000 osd_map(253..253 src has 253..288) v3
2017-03-31 12:03:22.515771 7fac1d45e700  3 osd.1 250 handle_osd_map epochs [253,253], i have 250, src has [253,288]
2017-03-31 12:03:22.515775 7fac1d45e700 10 osd.1 250 handle_osd_map message skips epochs 251..252
2017-03-31 12:03:22.515778 7fac1d45e700 10 osd.1 250 handle_osd_map  got full map for epoch 253

we are on 250, the oldest the mon has is 253, so we are skipping.

the pgs get a ton of ENOENTs as they try to consume the maps.

Actions
Copy link
 #5

Updated by Sage Weil almost 7 years ago

  • Status changed from Fix Under Review to 12
Actions
Copy link
 #6

Updated by Greg Farnum almost 7 years ago

  • Project changed from Ceph to RADOS
  • Category set to Correctness/Safety
  • Component(RADOS) Monitor, OSD added
Actions
Copy link
 #7

Updated by Sage Weil almost 7 years ago

  • Subject changed from osd: trims maps taht pgs haven't consumed yet to osd: trims maps taht pgs haven't consumed yet when there are gaps
  • Priority changed from Urgent to Normal
Actions
Copy link
 #8

Updated by Patrick Donnelly over 4 years ago

  • Status changed from 12 to New
Actions
Copy link

Also available in: Atom PDF