Project

General

Profile

Bug #325

writes starve reads

Added by Sage Weil over 13 years ago. Updated over 12 years ago.

Status:
Can't reproduce
Priority:
Normal
Assignee:
-
Category:
-
Target version:
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

From: Henry C Chang <>
To:
Subject: writes-starving-reads problem

Hi,

I ran into "writes-starving-reads" problem on a machine with 48G RAM.

I have two processes running concurrently:
Process 1 copies an 8 GB file from local disk to CEPH.
Process 2 copies an 8 GB file from CEPH to local disk.

Process 1 finished quickly since the data were cached in memory first.
Process 2 was blocked until all the cached data have been written to OSD.
(Maybe not all the cached data, but I waited for a long time.)

I don't have this problem on a machine with 8G RAM.

Has anyone experienced this before or any idea why?

Thanks,
Henry

History

#1 Updated by Sage Weil over 13 years ago

  • Subject changed from write starve reads to writes starve reads

#2 Updated by Sage Weil over 13 years ago

  • Target version set to v0.22

#3 Updated by Sage Weil over 13 years ago

  • Target version changed from v0.22 to v0.21.4

#4 Updated by Greg Farnum over 13 years ago

Hmm, I think I'm seeing this problem or an alternative instance of it, running a multi-mds cluster locally. bonnie++ (in the kclient, uml) was taking a long while to finish cleaning up so I wanted to list the bonnie directory and see how much stuff was left. But the ls is just hanging.
This looks to be a problem with the client -- there are no requests in-flight for the ls, just for bonnie cleanup.
And mounting the system via cfuse I was able to get a listing, although it was a little slow to respond.

This isn't very clean, though -- cfuse has crashed several times trying to repeat and after accidentally canceling my ls and starting again I have seen a getattr request on the root that took a while -- but then when it left the uncompleted-requests listing the ls didn't finish.

#5 Updated by Sage Weil over 13 years ago

  • Target version changed from v0.21.4 to v0.23

#6 Updated by Sage Weil over 13 years ago

  • Target version changed from v0.23 to 12

#7 Updated by Sage Weil almost 13 years ago

  • Target version changed from 12 to v0.27
  • translation missing: en.field_position deleted (344)
  • translation missing: en.field_position set to 326

#8 Updated by Sage Weil almost 13 years ago

  • translation missing: en.field_story_points changed from 0 to 5
  • translation missing: en.field_position deleted (326)
  • translation missing: en.field_position set to 326

#9 Updated by Sage Weil almost 13 years ago

  • translation missing: en.field_position deleted (327)
  • translation missing: en.field_position set to 341

#10 Updated by Sage Weil almost 13 years ago

  • translation missing: en.field_position deleted (337)
  • translation missing: en.field_position set to 597

#11 Updated by Sage Weil almost 13 years ago

  • Target version changed from v0.27 to v0.28
  • translation missing: en.field_position deleted (600)
  • translation missing: en.field_position set to 2

#12 Updated by Sage Weil almost 13 years ago

  • translation missing: en.field_position deleted (16)
  • translation missing: en.field_position set to 629

#13 Updated by Greg Farnum almost 13 years ago

  • Target version changed from v0.28 to v0.29
  • translation missing: en.field_position deleted (627)
  • translation missing: en.field_position set to 1

#14 Updated by Sage Weil almost 13 years ago

  • translation missing: en.field_position deleted (3)
  • translation missing: en.field_position set to 2

#15 Updated by Sage Weil almost 13 years ago

  • translation missing: en.field_position deleted (12)
  • translation missing: en.field_position set to 1
  • translation missing: en.field_position changed from 1 to 657

#16 Updated by Sage Weil almost 13 years ago

  • translation missing: en.field_position deleted (660)
  • translation missing: en.field_position set to 1
  • translation missing: en.field_position changed from 1 to 663

#17 Updated by Sage Weil almost 13 years ago

  • translation missing: en.field_position deleted (663)
  • translation missing: en.field_position set to 661

#18 Updated by Sage Weil almost 13 years ago

  • Target version changed from v0.29 to v0.30
  • translation missing: en.field_position deleted (665)
  • translation missing: en.field_position set to 16

#19 Updated by Sage Weil almost 13 years ago

  • translation missing: en.field_position deleted (26)
  • translation missing: en.field_position set to 682

#20 Updated by Sage Weil almost 13 years ago

  • Target version changed from v0.30 to v0.31
  • translation missing: en.field_position deleted (684)
  • translation missing: en.field_position set to 6

#21 Updated by Sage Weil almost 13 years ago

  • translation missing: en.field_position deleted (12)
  • translation missing: en.field_position set to 699

#22 Updated by Sage Weil almost 13 years ago

  • translation missing: en.field_position deleted (701)
  • translation missing: en.field_position set to 696

#23 Updated by Henry Chang almost 13 years ago

This problem seems to be fixed with the latest kclient and server. Cannot reproduce it now.

#24 Updated by Sage Weil over 12 years ago

  • Status changed from New to Can't reproduce

Also available in: Atom PDF