Project

General

Profile

Actions

Tasks #63669

open

Feature #63663: mds,client: add crash-consistent snapshot support

qa: add teuthology tests for quiescing a group of subvolumes

Added by Patrick Donnelly 5 months ago. Updated 23 days ago.

Status:
Fix Under Review
Priority:
Normal
Assignee:
Category:
Testing
Target version:
% Done:

0%

Tags:
squid,reef
Reviewed:
Affected Versions:
Component(FS):
mgr/volumes, qa-suite
Labels (FS):
qa
Pull request ID:

Description

Add a "Thrasher" task that executes in the background for fs:workload jobs like the current "fwd_scrub" Thrasher [4]. (See also mds_thrash.py [5])

What this is basically doing is periodically running a forward scrub on the file system. This is introduced into the suite and tuned using [6]. This thrasher should:

  • quiesce a subset of the current subvolumes ; right now fs:workload creates only two subvolumes but only one is used (IIRC) depending on the mount type. It would be good to automatically support multiple clients/subvolumes with future updates to fs:workload.
  • take a snapshot on each subvolume after quiesced
  • (make optional with an option like [1]) verify the quiesce: copy the _verify_quiesce method from test_quiesce.py [2] (eventually it will be merged into a common library)
  • Use teuthology postmerge filters to filter out the snap-schedule task when we are testing the quiesce+snapshot thrasher.

[1] https://github.com/ceph/ceph/blob/8ffdab763374c29194121bdc9c4f48e10cae0da1/qa/suites/fs/workload/tasks/2-scrub/yes.yaml#L9-L10
[2] https://github.com/ceph/ceph/pull/54581
[3] https://github.com/ceph/ceph/blob/8ffdab763374c29194121bdc9c4f48e10cae0da1/qa/suites/fs/workload/tasks/3-snaps/yes.yaml
[4] https://github.com/ceph/ceph/blob/8ffdab763374c29194121bdc9c4f48e10cae0da1/qa/tasks/fwd_scrub.py
[5] https://github.com/ceph/ceph/blob/8ffdab763374c29194121bdc9c4f48e10cae0da1/qa/tasks/mds_thrash.py
[6] https://github.com/ceph/ceph/blob/8ffdab763374c29194121bdc9c4f48e10cae0da1/qa/suites/fs/workload/tasks/2-scrub/yes.yaml

Actions #1

Updated by Patrick Donnelly 2 months ago

  • Description updated (diff)
  • Category set to Testing
  • Status changed from New to In Progress
  • Assignee set to Leonid Usov
  • Target version changed from v19.0.0 to v19.1.0
  • Tags set to squid,reef
  • Labels (FS) qa added
Actions #2

Updated by Patrick Donnelly 2 months ago

  • Description updated (diff)
Actions #3

Updated by Leonid Usov 2 months ago

@Patrick, I have several discussion points wrt the approach

1. Should we add more clients and/or more mountpoints to the workloads? This would affect all running tasks and may have a non-linear impact on the test runtime
2. Since this is not a functional test of the quiesce feature, should we just issue a quiesce of the fs root? If that is the case, we could cope with a background script config
3. Having a separate snapshot workload, we could have that perform the snapshot for us while we are doing background quiesces.

Actions #4

Updated by Patrick Donnelly 2 months ago

Leonid Usov wrote:

@Patrick, I have several discussion points wrt the approach

1. Should we add more clients and/or more mountpoints to the workloads? This would affect all running tasks and may have a non-linear impact on the test runtime

Not at this time. Make the thrasher capable of handling multiple subvolumes (probably a list as part of its task config, i.e. yaml).

2. Since this is not a functional test of the quiesce feature, should we just issue a quiesce of the fs root?

I want us to exercise the `ceph fs subvolume quiesce` interface as well.

If that is the case, we could cope with a background script config

Didn't understand this question.

3. Having a separate snapshot workload, we could have that perform the snapshot for us while we are doing background quiesces.

I think having the snapshot part of the quiesce thrasher makes sense. It's not a big lift.

Actions #5

Updated by Leonid Usov 2 months ago

If that is the case, we could cope with a background script config

Didn't understand this question.

I meant that if we could suffice with a simple periodic quiesce to the root, then we didn't even need a dedicated trasher script, just a yaml config file with the shell loop that issues the quiesce and sleeps for some time.
This approach will cover multiple subvolumes since it would be quiescing root

Actions #6

Updated by Leonid Usov 2 months ago

then we didn't even need a dedicated trasher script, just a yaml config file with the shell loop that issues the quiesce and sleeps for some time.

disregarding the way it'll be implemented, I would like to focus on the suggestion to quiesce the root. I appreciate that we'd like to test the quiesce subvolume commands, but I don't think this is the main goal of this background trasher. I think that the main goal is to see how different workloads respond to quiesces, and for that we don't need to do anything else but just quiescing everything periodically.

As for the quiesce subvolume, we should have dedicated functional tests that focus on those aspects, creating multiple clients, mountpoints etc.

Actions #7

Updated by Patrick Donnelly 2 months ago

Leonid Usov wrote:

If that is the case, we could cope with a background script config

Didn't understand this question.

I meant that if we could suffice with a simple periodic quiesce to the root, then we didn't even need a dedicated trasher script, just a yaml config file with the shell loop that issues the quiesce and sleeps for some time.
This approach will cover multiple subvolumes since it would be quiescing root

But then there's no verification. The quiesce thrasher gives us the opportunity to do e2e testing and verification.

Actions #8

Updated by Leonid Usov 2 months ago

We shouldn't overload. The background quiesce is an important test, but it is not functional e2e testing. For that we need a separate dedicated suite, like the one you have in functional for the quiesce path. In that dedicated suite we'll have more control over the number of clients, mount points, subvolumes etc.

Here I will use the quiesce database to install the pause so it will be an e2e test in that regard. However, the type of workloads we're running gives us very little control over advanced quiesce scenarios. There are just 2 mount points, 1 for fuse and 1 for kmount, and just two subvolumes one per each mount. Changing that for the sake of quiesce doesn't make sense, since the workloads are very different and aren't that flexible.

Actions #9

Updated by Leonid Usov 23 days ago

  • Status changed from In Progress to Fix Under Review
  • Pull request ID set to 55823
Actions

Also available in: Atom PDF