Project

General

Profile

Bug #8505

OSD osd/OSD.cc: 6222: FAILED assert(p->second.empty())

Added by Greg Farnum about 5 years ago. Updated about 5 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
OSD
Target version:
-
Start date:
06/02/2014
Due date:
% Done:

0%

Source:
Development
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:

Description

ubuntu@teuthology:/a/gregf-2014-05-29_16:19:17-rados-wip-client-fast-dispatch-testing-basic-plana/280831/remote/plana47/log/ceph-osd.4.log.gz

2014-05-30 16:18:52.985619 7fabf9c18700 10 osd.4 263  discarding waiting ops for 3.4s2
2014-05-30 16:18:52.985658 7fabf9c18700 10 osd.4 263  discarding waiting ops for 3.6s2
2014-05-30 16:18:52.985684 7fabf9c18700 10 osd.4 263  discarding waiting ops for 3.7s0
2014-05-30 16:18:52.985752 7fabe98f5700  1 -- 10.214.132.31:6802/7429 <== osd.5 10.214.132.31:6812/7433 2868 ==== MOSDECSubOpRead(3.4s2 262 ECSubRead(tid=1192, to_read={193d8934/plana4023310-290/head//3=526336,526336}, attrs_to_read=)) v1 ==== 136+0+0 (2632774283 0 0) 0x4265b40 con 0x34656e0
2014-05-30 16:18:52.985862 7fabe98f5700 10 osd.4 263 handle_replica_op MOSDECSubOpRead(3.4s2 262 ECSubRead(tid=1192, to_read={193d8934/plana4023310-290/head//3=526336,526336}, attrs_to_read=)) v1 epoch 262
2014-05-30 16:18:52.985875 7fabe98f5700 20 osd.4 263 should_share_map osd.5 10.214.132.31:6812/7433 262
2014-05-30 16:18:52.987841 7fabf9c18700 -1 osd/OSD.cc: In function 'void OSD::consume_map()' thread 7fabf9c18700 time 2014-05-30 16:18:52.985751
osd/OSD.cc: 6222: FAILED assert(p->second.empty())

Looks like a straightforward race that I missed with the fast dispatch changes (the OSD clears out the queue, then a fast dispatch thread delivers another op to the queue, then the OSD asserts that the queue is empty). Hopefully it will be straightforward to fix.


Related issues

Blocked by Ceph - Bug #8396: osd: message delayed in Session misdirected after split Resolved 05/19/2014

Associated revisions

Revision 25466839
Added by Gregory Farnum about 5 years ago

Merge pull request #1944 from athanatos/wip-8396

Move the waiting_for_pg list down into the Sessions, and coordinate the updates so that we don't get races
between the Pipe and handle_map threads.

Fixes: #8396
Fixes: #8505

Reviewed-by: Greg Farnum <>

History

#1 Updated by Samuel Just about 5 years ago

  • Priority changed from High to Urgent

#2 Updated by Samuel Just about 5 years ago

Actually, this should be fixed by the patch fixing 8396

#3 Updated by Samuel Just about 5 years ago

  • Status changed from New to Testing

#4 Updated by Samuel Just about 5 years ago

  • Assignee set to Samuel Just

#5 Updated by Greg Farnum about 5 years ago

  • Status changed from Testing to Resolved

Merged into master as of commit: 25466839589813047c975e44e67e14f34e32139e

#6 Updated by Samuel Just about 5 years ago

  • Status changed from Resolved to Verified

Sage is reverting it, borken

#7 Updated by Samuel Just about 5 years ago

  • Status changed from Verified to Resolved

Also available in: Atom PDF