Project

General

Profile

Actions
Copy link

Bug #62171

closed

All OSD shards should use the same scheduler type when osd_op_queue=debug_random.

Added by Aishwarya Mathuria 9 months ago. Updated about 1 month ago.

Status:
Resolved
Priority:
Normal
Assignee:
Sridhar Seshasayee
Category:
Correctness/Safety
Target version:
-
% Done:

0%

Source:
Tags:
backport_processed
Backport:
quincy, reef
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
rados
Component(RADOS):
OSD
Pull request ID:
53524
Crash signature (v1):
Crash signature (v2):

Description

In many teuthology tests, osd_op_queue is set to debug_random in order to perform testing with WPQ or mClock scheduler.
However, in an OSD, each shard is getting assigned a scheduler randomly. This randomness should be at OSD level not OSDShard level. 


2023-07-22T19:15:20.824+0000 7fefa9083640  0 osd.4:1.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=64)
2023-07-22T19:15:20.824+0000 7fefa9083640  1 bdev(0x55e9fe1c8e00 /var/lib/ceph/osd/ceph-4/block) open path /var/lib/ceph/osd/ceph-4/block
2023-07-22T19:15:20.824+0000 7fefa9083640  1 bdev(0x55e9fe1c8e00 /var/lib/ceph/osd/ceph-4/block) open size 96636764160 (0x1680000000, 90 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported
2023-07-22T19:15:20.824+0000 7fefa9083640 10 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label
2023-07-22T19:15:20.824+0000 7fefa9083640 10 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label got bdev(osd_uuid 32dbe31f-b837-4bf6-9505-1ffa6e2c7016, size 0x1680000000, btime 2023-07-22T19:14:50.886561+0000, desc main, 13 meta)
2023-07-22T19:15:20.824+0000 7fefa9083640  1 bluestore(/var/lib/ceph/osd/ceph-4) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06
2023-07-22T19:15:20.824+0000 7fefa9083640  1 bdev(0x55e9fe1c8e00 /var/lib/ceph/osd/ceph-4/block) close
2023-07-22T19:15:21.120+0000 7fefa9083640  0 osd.4:2.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=64)
2023-07-22T19:15:21.120+0000 7fefa9083640  1 bdev(0x55e9fe1c8e00 /var/lib/ceph/osd/ceph-4/block) open path /var/lib/ceph/osd/ceph-4/block
2023-07-22T19:15:21.120+0000 7fefa9083640  1 bdev(0x55e9fe1c8e00 /var/lib/ceph/osd/ceph-4/block) open size 96636764160 (0x1680000000, 90 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported
2023-07-22T19:15:21.120+0000 7fefa9083640 10 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label
2023-07-22T19:15:21.120+0000 7fefa9083640 10 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label got bdev(osd_uuid 32dbe31f-b837-4bf6-9505-1ffa6e2c7016, size 0x1680000000, btime 2023-07-22T19:14:50.886561+0000, desc main, 13 meta)
2023-07-22T19:15:21.120+0000 7fefa9083640  1 bluestore(/var/lib/ceph/osd/ceph-4) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06
2023-07-22T19:15:21.120+0000 7fefa9083640  1 bdev(0x55e9fe1c8e00 /var/lib/ceph/osd/ceph-4/block) close
2023-07-22T19:15:21.408+0000 7fefa9083640  0 osd.4:3.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=64)
2023-07-22T19:15:21.408+0000 7fefa9083640  1 bdev(0x55e9fe1c8e00 /var/lib/ceph/osd/ceph-4/block) open path /var/lib/ceph/osd/ceph-4/block
2023-07-22T19:15:21.408+0000 7fefa9083640  1 bdev(0x55e9fe1c8e00 /var/lib/ceph/osd/ceph-4/block) open size 96636764160 (0x1680000000, 90 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported
2023-07-22T19:15:21.408+0000 7fefa9083640 10 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label
2023-07-22T19:15:21.408+0000 7fefa9083640 10 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label got bdev(osd_uuid 32dbe31f-b837-4bf6-9505-1ffa6e2c7016, size 0x1680000000, btime 2023-07-22T19:14:50.886561+0000, desc main, 13 meta)
2023-07-22T19:15:21.408+0000 7fefa9083640  1 bluestore(/var/lib/ceph/osd/ceph-4) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06
2023-07-22T19:15:21.408+0000 7fefa9083640  1 bdev(0x55e9fe1c8e00 /var/lib/ceph/osd/ceph-4/block) close
2023-07-22T19:15:21.704+0000 7fefa9083640  0 osd.4:4.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=64)
2023-07-22T19:15:21.704+0000 7fefa9083640  1 bdev(0x55e9fe1c8e00 /var/lib/ceph/osd/ceph-4/block) open path /var/lib/ceph/osd/ceph-4/block
2023-07-22T19:15:21.704+0000 7fefa9083640  1 bdev(0x55e9fe1c8e00 /var/lib/ceph/osd/ceph-4/block) open size 96636764160 (0x1680000000, 90 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported
2023-07-22T19:15:21.704+0000 7fefa9083640 10 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label
2023-07-22T19:15:21.704+0000 7fefa9083640 10 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label got bdev(osd_uuid 32dbe31f-b837-4bf6-9505-1ffa6e2c7016, size 0x1680000000, btime 2023-07-22T19:14:50.886561+0000, desc main, 13 meta)
2023-07-22T19:15:21.704+0000 7fefa9083640  1 bluestore(/var/lib/ceph/osd/ceph-4) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06
2023-07-22T19:15:21.704+0000 7fefa9083640  1 bdev(0x55e9fe1c8e00 /var/lib/ceph/osd/ceph-4/block) close
2023-07-22T19:15:22.008+0000 7fefa9083640  1 mClockScheduler: set_osd_capacity_params_from_config: osd_bandwidth_cost_per_io: 58525.17 bytes/io, osd_bandwidth_capacity_per_shard 157286400.00 bytes/second
2023-07-22T19:15:22.008+0000 7fefa9083640  0 osd.4:5.OSDShard using op scheduler mClockScheduler
2023-07-22T19:15:22.008+0000 7fefa9083640  1 bdev(0x55e9fe1c8e00 /var/lib/ceph/osd/ceph-4/block) open path /var/lib/ceph/osd/ceph-4/block
2023-07-22T19:15:22.008+0000 7fefa9083640  1 bdev(0x55e9fe1c8e00 /var/lib/ceph/osd/ceph-4/block) open size 96636764160 (0x1680000000, 90 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported
2023-07-22T19:15:22.008+0000 7fefa9083640 10 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label
2023-07-22T19:15:22.008+0000 7fefa9083640 10 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label got bdev(osd_uuid 32dbe31f-b837-4bf6-9505-1ffa6e2c7016, size 0x1680000000, btime 2023-07-22T19:14:50.886561+0000, desc main, 13 meta)
2023-07-22T19:15:22.008+0000 7fefa9083640  1 bluestore(/var/lib/ceph/osd/ceph-4) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06
2023-07-22T19:15:22.008+0000 7fefa9083640  1 bdev(0x55e9fe1c8e00 /var/lib/ceph/osd/ceph-4/block) close
2023-07-22T19:15:22.332+0000 7fefa9083640  1 mClockScheduler: set_osd_capacity_params_from_config: osd_bandwidth_cost_per_io: 58525.17 bytes/io, osd_bandwidth_capacity_per_shard 157286400.00 bytes/second
2023-07-22T19:15:22.332+0000 7fefa9083640  0 osd.4:6.OSDShard using op scheduler mClockScheduler
2023-07-22T19:15:22.332+0000 7fefa9083640  1 bdev(0x55e9fe1c8e00 /var/lib/ceph/osd/ceph-4/block) open path /var/lib/ceph/osd/ceph-4/block
2023-07-22T19:15:22.332+0000 7fefa9083640  1 bdev(0x55e9fe1c8e00 /var/lib/ceph/osd/ceph-4/block) open size 96636764160 (0x1680000000, 90 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported
2023-07-22T19:15:22.332+0000 7fefa9083640 10 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label
2023-07-22T19:15:22.332+0000 7fefa9083640 10 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label got bdev(osd_uuid 32dbe31f-b837-4bf6-9505-1ffa6e2c7016, size 0x1680000000, btime 2023-07-22T19:14:50.886561+0000, desc main, 13 meta)
2023-07-22T19:15:22.332+0000 7fefa9083640  1 bluestore(/var/lib/ceph/osd/ceph-4) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06
2023-07-22T19:15:22.332+0000 7fefa9083640  1 bdev(0x55e9fe1c8e00 /var/lib/ceph/osd/ceph-4/block) close
2023-07-22T19:15:22.612+0000 7fefa9083640  1 mClockScheduler: set_osd_capacity_params_from_config: osd_bandwidth_cost_per_io: 58525.17 bytes/io, osd_bandwidth_capacity_per_shard 157286400.00 bytes/second
2023-07-22T19:15:22.612+0000 7fefa9083640  0 osd.4:7.OSDShard using op scheduler mClockScheduler

from: /a/rfriedma-2023-07-22_17:48:57-rados:thrash-wip-rf-cephx-init-distro-default-smithi/7348140/ but can be reproduced easily

Related issues 2 (0 open — 2 closed)

Copied to RADOS - Backport #63873: quincy: All OSD shards should use the same scheduler type when osd_op_queue=debug_random.RejectedSridhar SeshasayeeActions
Copied to RADOS - Backport #63874: reef: All OSD shards should use the same scheduler type when osd_op_queue=debug_random.ResolvedSridhar SeshasayeeActions
Actions
Copy link
 #1

Updated by Radoslaw Zarzynski 9 months ago

  • Status changed from New to In Progress
Actions
Copy link
 #2

Updated by Sridhar Seshasayee 8 months ago

  • Subject changed from All OSD shards should use the same scheduler when osd_op_queue is set to debug_random to All OSD shards should use the same scheduler (osd_op_queue) when testing using teuthology.
  • Category set to Tests
  • ceph-qa-suite rados added
Actions
Copy link
 #3

Updated by Sridhar Seshasayee 8 months ago

  • Subject changed from All OSD shards should use the same scheduler (osd_op_queue) when testing using teuthology. to All OSD shards should use the same scheduler (osd_op_queue) when using osd_op_queue=debug_random.
Actions
Copy link
 #4

Updated by Sridhar Seshasayee 8 months ago

  • Subject changed from All OSD shards should use the same scheduler (osd_op_queue) when using osd_op_queue=debug_random. to All OSD shards should use the same scheduler type when osd_op_queue=debug_random.
Actions
Copy link
 #5

Updated by Sridhar Seshasayee 8 months ago

  • Category changed from Tests to Correctness/Safety
  • Pull request ID set to 53524
  • Component(RADOS) OSD added
Actions
Copy link
 #6

Updated by Sridhar Seshasayee 8 months ago

Tests with fix shows same scheduler type applied on all OSD shards.

OSD logs from vstart cluster with osd_op_queue = debug_random

osd.0

$ grep "op scheduler" out/osd.0.log 
2023-09-21T13:49:45.793+0530 7f71f762e580  0 osd.0:0.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
2023-09-21T13:49:46.061+0530 7f71f762e580  0 osd.0:1.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
2023-09-21T13:49:46.328+0530 7f71f762e580  0 osd.0:2.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
2023-09-21T13:49:46.593+0530 7f71f762e580  0 osd.0:3.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
2023-09-21T13:49:46.620+0530 7f71f762e580  0 osd.0:4.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
2023-09-21T13:49:46.888+0530 7f71f762e580  0 osd.0:5.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
2023-09-21T13:49:47.159+0530 7f71f762e580  0 osd.0:6.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
2023-09-21T13:49:47.434+0530 7f71f762e580  0 osd.0:7.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)

osd.1

$ grep "op scheduler" out/osd.1.log 
2023-09-21T13:49:50.576+0530 7fce7e6b7580  0 osd.1:0.OSDShard using op scheduler mClockScheduler
2023-09-21T13:49:50.853+0530 7fce7e6b7580  0 osd.1:1.OSDShard using op scheduler mClockScheduler
2023-09-21T13:49:51.122+0530 7fce7e6b7580  0 osd.1:2.OSDShard using op scheduler mClockScheduler
2023-09-21T13:49:51.150+0530 7fce7e6b7580  0 osd.1:3.OSDShard using op scheduler mClockScheduler
2023-09-21T13:49:51.417+0530 7fce7e6b7580  0 osd.1:4.OSDShard using op scheduler mClockScheduler
2023-09-21T13:49:51.688+0530 7fce7e6b7580  0 osd.1:5.OSDShard using op scheduler mClockScheduler
2023-09-21T13:49:51.955+0530 7fce7e6b7580  0 osd.1:6.OSDShard using op scheduler mClockScheduler
2023-09-21T13:49:51.974+0530 7fce7e6b7580  0 osd.1:7.OSDShard using op scheduler mClockScheduler
Actions
Copy link
 #7

Updated by Sridhar Seshasayee 5 months ago

  • Status changed from In Progress to Fix Under Review
Actions
Copy link
 #8

Updated by Sridhar Seshasayee 4 months ago

  • Status changed from Fix Under Review to Resolved
  • Backport set to quincy, reef
Actions
Copy link
 #9

Updated by Sridhar Seshasayee 4 months ago

  • Status changed from Resolved to Pending Backport
Actions
Copy link
 #10

Updated by Backport Bot 4 months ago

  • Copied to Backport #63873: quincy: All OSD shards should use the same scheduler type when osd_op_queue=debug_random. added
Actions
Copy link
 #11

Updated by Backport Bot 4 months ago

  • Copied to Backport #63874: reef: All OSD shards should use the same scheduler type when osd_op_queue=debug_random. added
Actions
Copy link
 #12

Updated by Backport Bot 4 months ago

  • Tags set to backport_processed
Actions
Copy link
 #13

Updated by Sridhar Seshasayee about 1 month ago

  • Status changed from Pending Backport to Resolved
Actions
Copy link

Also available in: Atom PDF