Project

General

Profile

Bug #47700

during OSD deletion: Module 'cephadm' has failed: Set changed size during iteration

Added by Joshua Schmid 4 months ago. Updated 10 days ago.

Status:
Fix Under Review
Priority:
Normal
Category:
cephadm/osd
Target version:
% Done:

0%

Source:
Development
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

Deleted all OSDs on one node with this command:

for OSD in $(ceph osd ls-tree sesnode2); do ceph orch osd rm $OSD --force; done

--> now ceph -shows this:

sesnode1:~ # ceph -s
  cluster:
    id:     88c8695a-f29d-11ea-ab5b-40a6b7164708
    health: HEALTH_ERR
            Module 'cephadm' has failed: Set changed size during iteration
            3 slow ops, oldest one blocked for 4358 sec, daemons [mon.sesnode2,mon.sesnode3] have slow ops.

  services:
    mon: 3 daemons, quorum sesnode1,sesnode2,sesnode3 (age 11h)
    mgr: sesnode3.hpyadv(active, since 16m), standbys: sesnode1.zzsnjy, sesnode2.lwcuju
    mds: cephfs:1 {0=cephfs.sesnode6.rjpieg=up:active} 1 up:standby
    osd: 152 osds: 152 up (since 4m), 152 in (since 4m)
    rgw: 3 daemons active (default.default.sesnode4.xqbctf, default.default.sesnode5.qktkld, default.default.sesnode6.ffagxh)

  task status:
    scrub status:
        mds.cephfs.sesnode6.rjpieg: idle

  data:
    pools:   9 pools, 2225 pgs
    objects: 5.44k objects, 20 GiB
    usage:   5.2 TiB used, 371 TiB / 377 TiB avail
    pgs:     2225 active+clean

  io:
    client:   254 B/s rd, 0 op/s rd, 0 op/s wr

sesnode1:~ # ceph health detail
HEALTH_ERR Module 'cephadm' has failed: Set changed size during iteration; 3 slow ops, oldest one blocked for 4478 sec, daemons [mon.sesnode2,mon.sesnode3] have slow ops.
[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: Set changed size during iteration
    Module 'cephadm' has failed: Set changed size during iteration
[WRN] SLOW_OPS: 3 slow ops, oldest one blocked for 4478 sec, daemons [mon.sesnode2,mon.sesnode3] have slow ops.

From the log of the manager:

debug 2020-09-14T09:42:27.590+0000 7f1e38634700  0 log_channel(audit) log [DBG] : from='client.62407 -' entity='client.admin' cmd=[{"prefix": "orch osd rm", "svc_id": ["106"], "force": true, "targe
t": ["mon-mgr", ""]}]: dispatch
debug 2020-09-14T09:42:27.694+0000 7f1e3daff700 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'cephadm' while running on mgr.sesnode3.hpyadv: Set changed size during iteration
debug 2020-09-14T09:42:27.694+0000 7f1e3daff700 -1 cephadm.serve:
debug 2020-09-14T09:42:27.694+0000 7f1e3daff700 -1 Traceback (most recent call last):
  File "/usr/share/ceph/mgr/cephadm/module.py", line 502, in serve
    self.rm_util.process_removal_queue()
  File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 295, in process_removal_queue
    self.cleanup()
  File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 352, in cleanup
    not_in_cluster_osds = self.mgr.to_remove_osds.not_in_cluster()
  File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 671, in not_in_cluster
    return [osd for osd in self if not osd.exists]
  File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 671, in <listcomp>
    return [osd for osd in self if not osd.exists]
RuntimeError: Set changed size during iteration

History

#1 Updated by Joshua Schmid 4 months ago

  • Pull request ID set to 37492

#2 Updated by Joshua Schmid 4 months ago

  • Status changed from New to Fix Under Review

#3 Updated by Sebastian Wagner 2 months ago

  • Assignee changed from Joshua Schmid to Sebastian Wagner

#4 Updated by Michael Fritch 10 days ago

  • Pull request ID changed from 37492 to 38815

Also available in: Atom PDF