Project

General

Profile

Bug #15019

hammer: fs test fails with log [ERR] : OSD full dropping all updates 100% full

Added by Wei-Chung Cheng over 4 years ago. Updated over 4 years ago.

Status:
Duplicate
Priority:
Normal
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
Severity:
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:


Related issues

Related to Ceph - Backport #13335: hammer: OSD crashed when reached pool's max_bytes quota Resolved

History

#1 Updated by Wei-Chung Cheng over 4 years ago

  • Tracker changed from Bug to Backport
  • Release set to hammer

#2 Updated by Wei-Chung Cheng over 4 years ago

  • Status changed from New to In Progress
  • Assignee set to Wei-Chung Cheng

#3 Updated by Loic Dachary over 4 years ago

  • Related to Backport #13335: hammer: OSD crashed when reached pool's max_bytes quota added

#4 Updated by Wei-Chung Cheng over 4 years ago

  • Description updated (diff)

#5 Updated by Loic Dachary over 4 years ago

  • Subject changed from check osd full mechanism will cause failure when osd is too full to hammer: revert "OSD crashed when reached pool's max_bytes quota"

#6 Updated by Loic Dachary over 4 years ago

Trying on hammer

teuthology-suite --priority 1000 --suite fs --filter="fs/recovery/{clusters/2-remote-clients.yaml debug/mds_client.yaml mounts/ceph-fuse.yaml tasks/mds-full.yaml}" --suite-branch hammer --email loic@dachary.org --ceph hammer --machine-type smithi,mira teuthology-suite --priority 1000 --suite fs --filter="fs/recovery/{clusters/2-remote-clients.yaml debug/mds_client.yaml mounts/ceph-fuse.yaml tasks/mds-full.yaml}" --suite-branch hammer --email loic@dachary.org --ceph v0.94.6 --machine-type smithi,mira

The same test failed February 8th, 2016 http://pulpito.ceph.com/loic-2016-02-08_23:43:36-fs-hammer-backports---basic-multi/1087/
The same test passed January 29th, 2016 http://pulpito.ceph.com/loic-2016-01-29_03:02:05-fs-hammer---basic-multi/49423/

git log --merges --since 2016-01-29 --until 2016-02-07 --format='%H' ceph/hammer | \
while read sha1 ; do \
  echo ; git log --format='** %aD "%s":https://github.com/ceph/ceph/commit/%H' ${sha1}^1..${sha1} ; \
done | perl -p -e 'print "* \"PR $1\":https://github.com/ceph/ceph/pull/$1\n" if(/Merge pull request #(\d+)/)'

The most likely candidate for this regression is : https://github.com/ceph/ceph/pull/6918 hammer: osd: check for full before changing the cached obc (hammer)

Trying on v0.94.5

#7 Updated by Loic Dachary over 4 years ago

  • Tracker changed from Backport to Bug
  • Subject changed from hammer: revert "OSD crashed when reached pool's max_bytes quota" to hammer: fs test fails with log [ERR] : OSD full dropping all updates 100% full
  • Status changed from In Progress to Duplicate
  • % Done set to 0

This is a manifestation of an incomplete backport for which an the followup has been scheduled at http://tracker.ceph.com/issues/14824#note-5 . The discussion and rationale for keeping the incomplete backport and failing this test can be found in the QE validation hammer mail thread. The issue http://tracker.ceph.com/issues/14716 is referenced and http://tracker.ceph.com/issues/14716#note-11 states that it is related to http://tracker.ceph.com/issues/14824.

Closing https://github.com/ceph/ceph/pull/7977 and marking this as a duplicate. The title is changed to match the error message so that it's easier to find when future tests fail.

#8 Updated by Loic Dachary over 4 years ago

  • Related to Backport #14824: hammer: rbd and pool quota do not go well together added

Also available in: Atom PDF