Project

General

Profile

Actions
Copy link

Bug #15019

closed

hammer: fs test fails with log [ERR] : OSD full dropping all updates 100% full

Added by Wei-Chung Cheng about 8 years ago. Updated about 8 years ago.

Status:
Duplicate
Priority:
Normal
Assignee:
Wei-Chung Cheng
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
Severity:
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

https://github.com/ceph/ceph/pull/7977

Related issues 1 (0 open — 1 closed)

Related to Ceph - Backport #13335: hammer: OSD crashed when reached pool's max_bytes quotaResolvedAlexey SheplyakovActions
Actions
Copy link
 #1

Updated by Wei-Chung Cheng about 8 years ago

  • Tracker changed from Bug to Backport
  • Release set to hammer
Actions
Copy link
 #2

Updated by Wei-Chung Cheng about 8 years ago

  • Status changed from New to In Progress
  • Assignee set to Wei-Chung Cheng
Actions
Copy link
 #3

Updated by Loïc Dachary about 8 years ago

  • Related to Backport #13335: hammer: OSD crashed when reached pool's max_bytes quota added
Actions
Copy link
 #4

Updated by Wei-Chung Cheng about 8 years ago

  • Description updated (diff)
Actions
Copy link
 #5

Updated by Loïc Dachary about 8 years ago

  • Subject changed from check osd full mechanism will cause failure when osd is too full to hammer: revert "OSD crashed when reached pool's max_bytes quota"
Actions
Copy link
 #6

Updated by Loïc Dachary about 8 years ago

Trying on hammer

teuthology-suite --priority 1000 --suite fs --filter="fs/recovery/{clusters/2-remote-clients.yaml debug/mds_client.yaml mounts/ceph-fuse.yaml tasks/mds-full.yaml}" --suite-branch hammer --email loic@dachary.org --ceph hammer --machine-type smithi,mira teuthology-suite --priority 1000 --suite fs --filter="fs/recovery/{clusters/2-remote-clients.yaml debug/mds_client.yaml mounts/ceph-fuse.yaml tasks/mds-full.yaml}" --suite-branch hammer --email loic@dachary.org --ceph v0.94.6 --machine-type smithi,mira

The same test failed February 8th, 2016 http://pulpito.ceph.com/loic-2016-02-08_23:43:36-fs-hammer-backports---basic-multi/1087/
The same test passed January 29th, 2016 http://pulpito.ceph.com/loic-2016-01-29_03:02:05-fs-hammer---basic-multi/49423/

git log --merges --since 2016-01-29 --until 2016-02-07 --format='%H' ceph/hammer | \
while read sha1 ; do \
  echo ; git log --format='** %aD "%s":https://github.com/ceph/ceph/commit/%H' ${sha1}^1..${sha1} ; \
done | perl -p -e 'print "* \"PR $1\":https://github.com/ceph/ceph/pull/$1\n" if(/Merge pull request #(\d+)/)'

The most likely candidate for this regression is : https://github.com/ceph/ceph/pull/6918 hammer: osd: check for full before changing the cached obc (hammer)

Trying on v0.94.5

Actions
Copy link
 #7

Updated by Loïc Dachary about 8 years ago

  • Tracker changed from Backport to Bug
  • Subject changed from hammer: revert "OSD crashed when reached pool's max_bytes quota" to hammer: fs test fails with log [ERR] : OSD full dropping all updates 100% full
  • Status changed from In Progress to Duplicate
  • % Done set to 0

This is a manifestation of an incomplete backport for which an the followup has been scheduled at http://tracker.ceph.com/issues/14824#note-5 . The discussion and rationale for keeping the incomplete backport and failing this test can be found in the QE validation hammer mail thread. The issue http://tracker.ceph.com/issues/14716 is referenced and http://tracker.ceph.com/issues/14716#note-11 states that it is related to http://tracker.ceph.com/issues/14824.

Closing https://github.com/ceph/ceph/pull/7977 and marking this as a duplicate. The title is changed to match the error message so that it's easier to find when future tests fail.

Actions
Copy link
 #8

Updated by Loïc Dachary about 8 years ago

  • Related to Backport #14824: hammer: rbd and pool quota do not go well together added
Actions
Copy link

Also available in: Atom PDF