Actions
Bug #54238
opencephadm upgrade pacifc to quincy -> causing osd's FULL/cascading failure
Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
- Upgrade was started at 2022-02-08T01:54:28
ceph-mgr.e24-h01-000-r640.rdu2.scalelab.redhat.com.mrqbjw.log-20220208.gz:150705:2022-02-08T01:54:28.816+0000 7efca85d0700 0 log_channel(audit) log [DBG] : from='client.194120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a00e8b315af02865380634f8100dc7d18a18af4f", "target": ["mon-mgr", ""]}]: dispatch
- ceph status
# ceph -s
cluster:
id: dac59560-8316-11ec-b627-bc97e17ab990
health: HEALTH_ERR
3356 failed cephadm daemon(s)
1 hosts fail cephadm check
1756 osds down
627 full osd(s)
Reduced data availability: 233 pgs inactive, 1 pg down, 3 pgs peering
Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull
Degraded data redundancy: 5318/10536 objects degraded (50.475%), 64 pgs degraded, 64 pgs undersized
2 pool(s) full
15 slow ops, oldest one blocked for 101279 sec, daemons [osd.158,mon.e24-h01-000-r640.rdu2.scalelab.redhat.com] have slow ops.
services:
mon: 3 daemons, quorum e24-h01-000-r640.rdu2.scalelab.redhat.com,e24-h02-000-r640,e24-h03-000-r640 (age 24h)
mgr: e24-h01-000-r640.rdu2.scalelab.redhat.com.mrqbjw(active, since 24h), standbys: e24-h02-000-r640.kxdcop
osd: 3904 osds: 1171 up (since 24h), 2927 in (since 28h); 16 remapped pgs
data:
pools: 2 pools, 257 pgs
objects: 3.51k objects, 14 GiB
usage: 4.7 TiB used, 3.3 TiB / 8.0 TiB avail
pgs: 71.984% pgs unknown
18.677% pgs not active
5318/10536 objects degraded (50.475%)
185 unknown
42 undersized+degraded+peered
19 active+undersized+degraded
3 peering
2 undersized+degraded+remapped+backfill_wait+backfill_toofull+peered
2 active+clean+scrubbing
1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull
1 active+clean+scrubbing+deep
1 down
1 active+clean
- Version
# ceph versions
{
"mon": {
"ceph version 17.0.0-10315-ga00e8b31 (a00e8b315af02865380634f8100dc7d18a18af4f) quincy (dev)": 3
},
"mgr": {
"ceph version 17.0.0-10315-ga00e8b31 (a00e8b315af02865380634f8100dc7d18a18af4f) quincy (dev)": 2
},
"osd": {
"ceph version 16.2.7-34.el8cp (70c4491bd537223bac6d19fdba941d452d4641c2) pacific (stable)": 681,
"ceph version 17.0.0-10315-ga00e8b31 (a00e8b315af02865380634f8100dc7d18a18af4f) quincy (dev)": 490
},
"mds": {},
"overall": {
"ceph version 16.2.7-34.el8cp (70c4491bd537223bac6d19fdba941d452d4641c2) pacific (stable)": 681,
"ceph version 17.0.0-10315-ga00e8b31 (a00e8b315af02865380634f8100dc7d18a18af4f) quincy (dev)": 495
}
}
- From cluster logs
2022-02-08T09:20:00.000394+0000 mon.e24-h01-000-r640.rdu2.scalelab.redhat.com (mon.0) 22414 : cluster [ERR] [ERR] OSD_FULL: 1 full osd(s) 2022-02-08T09:20:00.000411+0000 mon.e24-h01-000-r640.rdu2.scalelab.redhat.com (mon.0) 22415 : cluster [ERR] osd.2080 is full
From OSD logs:
2022-02-08T09:17:16.223+0000 7f1583974700 1 osd.2080 44163 advance_pg 4.69a3 is merge source, target is 4.29a3
2022-02-08T09:17:16.223+0000 7f1583974700 1 osd.2080 44163 advance_pg merging 4.29a3
2022-02-08T09:17:16.223+0000 7f1583974700 1 osd.2080 pg_epoch: 44163 pg[4.29a3( v 22136'1361 (22136'1361,22136'1361] local-lis/les=44161/44162 n=127 ec=8418/7032 lis/c=44161/44161 les/c/f=44162/44162/43997 sis=44163 pruub=14.786740303s) [1347,2080,3393] r=1 lpr=44163 pi=[44161,44163)/1 luod=0'0 lua=22134'1340 crt=22136'1361 lcod 22133'1339 mlcod 0'0 active pruub 534281.437500000s@ mbc={}] start_peering_interval up [1347,2080,3393] -> [1347,2080,3393], acting [1347,2080,3393] -> [1347,2080,3393], acting_primary 1347 -> 1347, up_primary 1347 -> 1347, role 1 -> 1, features acting 4540138297136906239 upacting 4540138297136906239
2022-02-08T09:17:16.224+0000 7f1583974700 1 osd.2080 pg_epoch: 44163 pg[4.29a3( v 22136'1361 (22136'1361,22136'1361] local-lis/les=44161/44162 n=127 ec=8418/7032 lis/c=44161/44161 les/c/f=44162/44162/43997 sis=44163 pruub=14.786669731s) [1347,2080,3393] r=1 lpr=44163 pi=[44161,44163)/1 crt=22136'1361 lcod 22133'1339 mlcod 0'0 unknown NOTIFY pruub 534281.437500000s@ mbc={}] state<Start>: transitioning to Stray
2022-02-08T09:22:50.200+0000 7f1594fd1700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS -------
2022-02-08T09:22:50.200+0000 7f1594fd1700 4 rocksdb: [db_impl/db_impl.cc:851]
2022-02-08T12:47:13.007+0000 7f15967d4700 -1 bluefs _allocate allocation failed, needed 0x2236
2022-02-08T12:47:13.007+0000 7f15967d4700 -1 bluefs _flush_range allocated: 0xede0000 offset: 0xedded2d length: 0x3509
2022-02-08T12:47:13.015+0000 7f15967d4700 -1 /builddir/build/BUILD/ceph-16.2.7/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f15967d4700 time 2022-02-08T12:47:13.009096+0000
/builddir/build/BUILD/ceph-16.2.7/src/os/bluestore/BlueFS.cc: 2768: ceph_abort_msg("bluefs enospc")
ceph version 16.2.7-34.el8cp (70c4491bd537223bac6d19fdba941d452d4641c2) pacific (stable)
1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe5) [0x556186fc545e]
2: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1131) [0x5561876c0f41]
3: (BlueFS::_flush(BlueFS::FileWriter*, bool, bool*)+0x90) [0x5561876c1220]
4: (BlueFS::_flush(BlueFS::FileWriter*, bool, std::unique_lock<std::mutex>&)+0x32) [0x5561876d2332]
5: (BlueRocksWritableFile::Append(rocksdb::Slice const&)+0x11b) [0x5561876eac6b]
6: (rocksdb::LegacyWritableFileWrapper::Append(rocksdb::Slice const&, rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x1f) [0x556187b850df]
7: (rocksdb::WritableFileWriter::WriteBuffered(char const*, unsigned long)+0x58a) [0x556187c96e2a]
8: (rocksdb::WritableFileWriter::Append(rocksdb::Slice const&)+0x2d0) [0x556187c98280]
9: (rocksdb::BlockBasedTableBuilder::WriteRawBlock(rocksdb::Slice const&, rocksdb::CompressionType, rocksdb::BlockHandle*, bool)+0xb6) [0x556187db3a06]
10: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*, bool)+0x26c) [0x556187db434c]
11: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::BlockBuilder*, rocksdb::BlockHandle*, bool)+0x3c) [0x556187db4a4c]
12: (rocksdb::BlockBasedTableBuilder::Flush()+0x6d) [0x556187db4add]
13: (rocksdb::BlockBasedTableBuilder::Add(rocksdb::Slice const&, rocksdb::Slice const&)+0x2b8) [0x556187db7f48]
14: (rocksdb::BuildTable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, rocksdb::FileSystem*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase<rocksdb::Slice>*, std::vector<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> >, std::allocator<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> > > >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, unsigned long, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint, unsigned long)+0xa45) [0x556187d629c5]
15: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xcf5) [0x556187bc7a25]
16: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool, bool*)+0x1c2e) [0x556187bca15e]
17: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool, unsigned long*)+0xae8) [0x556187bcb4b8]
18: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool)+0x59d) [0x556187bc51dd]
19: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x15) [0x556187bc6575]
20: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x10c1) [0x556187b3e4b1]
21: (BlueStore::_open_db(bool, bool, bool)+0x8c7) [0x5561875bb0b7]
22: (BlueStore::_open_db_and_around(bool, bool)+0x2f7) [0x556187625b47]
23: (BlueStore::_mount()+0x204) [0x556187628a04]
24: (OSD::init()+0x380) [0x5561870fc5d0]
25: main()
26: __libc_start_main()
27: _start()
Updated by Vikhyat Umrao about 2 years ago
The node e24-h01-000-r640 has a file - upgrade.txt from the following command:
Path - /var/log/ceph/dac59560-8316-11ec-b627-bc97e17ab990/upgrade.txt zgrep -n "upgrade" ceph-mgr.e24-h01-000-r640.rdu2.scalelab.redhat.com.mrqbjw.log* > upgrade.txt
This file has upgrade details from the MGR logs.
Updated by Vikhyat Umrao about 2 years ago
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 446.6G 0 disk
├─sda1 8:1 0 512M 0 part /boot
└─sda2 8:2 0 446.1G 0 part
├─vg_e24--h01--000--r640-lv_root 253:0 0 438.1G 0 lvm /
└─vg_e24--h01--000--r640-lv_swap 253:1 0 8G 0 lvm [SWAP]
sdb 8:16 0 1.8T 0 disk
├─fakedevs-thinpool_tmeta 253:2 0 112M 0 lvm
│ └─fakedevs-thinpool-tpool 253:9 0 1.7T 0 lvm
│ ├─fakedevs-thinpool 253:10 0 1.7T 1 lvm
│ ├─fakedevs-disk1 253:11 0 15G 0 lvm
│ ├─fakedevs-disk2 253:12 0 15G 0 lvm
│ ├─fakedevs-disk3 253:13 0 15G 0 lvm
│ ├─fakedevs-disk4 253:14 0 15G 0 lvm
│ ├─fakedevs-disk5 253:15 0 15G 0 lvm
│ ├─fakedevs-disk6 253:16 0 15G 0 lvm
│ ├─fakedevs-disk7 253:17 0 15G 0 lvm
│ ├─fakedevs-disk8 253:18 0 15G 0 lvm
│ ├─fakedevs-disk9 253:19 0 15G 0 lvm
│ ├─fakedevs-disk10 253:20 0 15G 0 lvm
│ ├─fakedevs-disk11 253:21 0 15G 0 lvm
│ ├─fakedevs-disk12 253:22 0 15G 0 lvm
│ ├─fakedevs-disk13 253:23 0 15G 0 lvm
│ ├─fakedevs-disk14 253:24 0 15G 0 lvm
│ ├─fakedevs-disk15 253:25 0 15G 0 lvm
│ ├─fakedevs-disk16 253:26 0 15G 0 lvm
│ ├─fakedevs-disk17 253:27 0 15G 0 lvm
│ ├─fakedevs-disk18 253:28 0 15G 0 lvm
│ ├─fakedevs-disk19 253:29 0 15G 0 lvm
│ ├─fakedevs-disk20 253:30 0 15G 0 lvm
│ ├─fakedevs-disk21 253:31 0 15G 0 lvm
│ ├─fakedevs-disk22 253:32 0 15G 0 lvm
│ ├─fakedevs-disk23 253:33 0 15G 0 lvm
│ ├─fakedevs-disk24 253:34 0 15G 0 lvm
│ ├─fakedevs-disk25 253:35 0 15G 0 lvm
│ ├─fakedevs-disk26 253:36 0 15G 0 lvm
│ ├─fakedevs-disk27 253:37 0 15G 0 lvm
│ ├─fakedevs-disk28 253:38 0 15G 0 lvm
│ ├─fakedevs-disk29 253:39 0 15G 0 lvm
│ ├─fakedevs-disk30 253:40 0 15G 0 lvm
│ ├─fakedevs-disk31 253:41 0 15G 0 lvm
│ ├─fakedevs-disk32 253:42 0 15G 0 lvm
│ ├─fakedevs-disk33 253:43 0 15G 0 lvm
│ ├─fakedevs-disk34 253:44 0 15G 0 lvm
│ ├─fakedevs-disk35 253:45 0 15G 0 lvm
│ ├─fakedevs-disk36 253:46 0 15G 0 lvm
│ ├─fakedevs-disk37 253:47 0 15G 0 lvm
│ ├─fakedevs-disk38 253:48 0 15G 0 lvm
│ ├─fakedevs-disk39 253:49 0 15G 0 lvm
│ ├─fakedevs-disk40 253:50 0 15G 0 lvm
│ ├─fakedevs-disk41 253:51 0 15G 0 lvm
│ ├─fakedevs-disk42 253:52 0 15G 0 lvm
│ ├─fakedevs-disk43 253:53 0 15G 0 lvm
│ ├─fakedevs-disk44 253:54 0 15G 0 lvm
│ ├─fakedevs-disk45 253:55 0 15G 0 lvm
│ ├─fakedevs-disk46 253:56 0 15G 0 lvm
│ ├─fakedevs-disk47 253:57 0 15G 0 lvm
│ ├─fakedevs-disk48 253:58 0 15G 0 lvm
│ ├─fakedevs-disk49 253:59 0 15G 0 lvm
│ ├─fakedevs-disk50 253:60 0 15G 0 lvm
│ ├─fakedevs-disk51 253:61 0 15G 0 lvm
│ ├─fakedevs-disk52 253:62 0 15G 0 lvm
│ ├─fakedevs-disk53 253:63 0 15G 0 lvm
│ ├─fakedevs-disk54 253:64 0 15G 0 lvm
│ ├─fakedevs-disk55 253:65 0 15G 0 lvm
│ ├─fakedevs-disk56 253:66 0 15G 0 lvm
│ ├─fakedevs-disk57 253:67 0 15G 0 lvm
│ ├─fakedevs-disk58 253:68 0 15G 0 lvm
│ ├─fakedevs-disk59 253:69 0 15G 0 lvm
│ ├─fakedevs-disk60 253:70 0 15G 0 lvm
│ ├─fakedevs-disk61 253:71 0 15G 0 lvm
│ ├─fakedevs-disk62 253:72 0 15G 0 lvm
│ ├─fakedevs-disk63 253:73 0 15G 0 lvm
│ └─fakedevs-disk64 253:74 0 15G 0 lvm
└─fakedevs-thinpool_tdata 253:5 0 1.7T 0 lvm
└─fakedevs-thinpool-tpool 253:9 0 1.7T 0 lvm
├─fakedevs-thinpool 253:10 0 1.7T 1 lvm
├─fakedevs-disk1 253:11 0 15G 0 lvm
├─fakedevs-disk2 253:12 0 15G 0 lvm
├─fakedevs-disk3 253:13 0 15G 0 lvm
├─fakedevs-disk4 253:14 0 15G 0 lvm
├─fakedevs-disk5 253:15 0 15G 0 lvm
├─fakedevs-disk6 253:16 0 15G 0 lvm
├─fakedevs-disk7 253:17 0 15G 0 lvm
├─fakedevs-disk8 253:18 0 15G 0 lvm
├─fakedevs-disk9 253:19 0 15G 0 lvm
├─fakedevs-disk10 253:20 0 15G 0 lvm
├─fakedevs-disk11 253:21 0 15G 0 lvm
├─fakedevs-disk12 253:22 0 15G 0 lvm
├─fakedevs-disk13 253:23 0 15G 0 lvm
├─fakedevs-disk14 253:24 0 15G 0 lvm
├─fakedevs-disk15 253:25 0 15G 0 lvm
├─fakedevs-disk16 253:26 0 15G 0 lvm
├─fakedevs-disk17 253:27 0 15G 0 lvm
├─fakedevs-disk18 253:28 0 15G 0 lvm
├─fakedevs-disk19 253:29 0 15G 0 lvm
├─fakedevs-disk20 253:30 0 15G 0 lvm
├─fakedevs-disk21 253:31 0 15G 0 lvm
├─fakedevs-disk22 253:32 0 15G 0 lvm
├─fakedevs-disk23 253:33 0 15G 0 lvm
├─fakedevs-disk24 253:34 0 15G 0 lvm
├─fakedevs-disk25 253:35 0 15G 0 lvm
├─fakedevs-disk26 253:36 0 15G 0 lvm
├─fakedevs-disk27 253:37 0 15G 0 lvm
├─fakedevs-disk28 253:38 0 15G 0 lvm
├─fakedevs-disk29 253:39 0 15G 0 lvm
├─fakedevs-disk30 253:40 0 15G 0 lvm
├─fakedevs-disk31 253:41 0 15G 0 lvm
├─fakedevs-disk32 253:42 0 15G 0 lvm
├─fakedevs-disk33 253:43 0 15G 0 lvm
├─fakedevs-disk34 253:44 0 15G 0 lvm
├─fakedevs-disk35 253:45 0 15G 0 lvm
├─fakedevs-disk36 253:46 0 15G 0 lvm
├─fakedevs-disk37 253:47 0 15G 0 lvm
├─fakedevs-disk38 253:48 0 15G 0 lvm
├─fakedevs-disk39 253:49 0 15G 0 lvm
├─fakedevs-disk40 253:50 0 15G 0 lvm
├─fakedevs-disk41 253:51 0 15G 0 lvm
├─fakedevs-disk42 253:52 0 15G 0 lvm
├─fakedevs-disk43 253:53 0 15G 0 lvm
├─fakedevs-disk44 253:54 0 15G 0 lvm
├─fakedevs-disk45 253:55 0 15G 0 lvm
├─fakedevs-disk46 253:56 0 15G 0 lvm
├─fakedevs-disk47 253:57 0 15G 0 lvm
├─fakedevs-disk48 253:58 0 15G 0 lvm
├─fakedevs-disk49 253:59 0 15G 0 lvm
├─fakedevs-disk50 253:60 0 15G 0 lvm
├─fakedevs-disk51 253:61 0 15G 0 lvm
├─fakedevs-disk52 253:62 0 15G 0 lvm
├─fakedevs-disk53 253:63 0 15G 0 lvm
├─fakedevs-disk54 253:64 0 15G 0 lvm
├─fakedevs-disk55 253:65 0 15G 0 lvm
├─fakedevs-disk56 253:66 0 15G 0 lvm
├─fakedevs-disk57 253:67 0 15G 0 lvm
├─fakedevs-disk58 253:68 0 15G 0 lvm
├─fakedevs-disk59 253:69 0 15G 0 lvm
├─fakedevs-disk60 253:70 0 15G 0 lvm
├─fakedevs-disk61 253:71 0 15G 0 lvm
├─fakedevs-disk62 253:72 0 15G 0 lvm
├─fakedevs-disk63 253:73 0 15G 0 lvm
└─fakedevs-disk64 253:74 0 15G 0 lvm
sdc 8:32 0 1.8T 0 disk
├─data-thinpool_tmeta 253:3 0 112M 0 lvm
│ └─data-thinpool-tpool 253:6 0 1.7T 0 lvm
│ ├─data-thinpool 253:7 0 1.7T 1 lvm
│ ├─data-prometheus 253:8 0 1T 0 lvm
│ └─data-prometheus--pacific 253:139 0 60G 0 lvm
└─data-thinpool_tdata 253:4 0 1.7T 0 lvm
└─data-thinpool-tpool 253:6 0 1.7T 0 lvm
├─data-thinpool 253:7 0 1.7T 1 lvm
├─data-prometheus 253:8 0 1T 0 lvm
└─data-prometheus--pacific 253:139 0 60G 0 lvm
nbd1 43:32 0 15G 0 disk
└─ceph--ed9435ce--f5af--4fc8--b2e4--a314fcac3854-osd--block--e8c73e72--765c--41aa--895c--98146b6a107d 253:75 0 15G 0 lvm
nbd2 43:64 0 15G 0 disk
└─ceph--17919713--dc6c--4c9c--b22b--49a1417cc580-osd--block--c8765267--c933--4e55--b3b1--5769780f6c46 253:86 0 15G 0 lvm
nbd3 43:96 0 15G 0 disk
└─ceph--ce68c898--9a3c--4005--8cdc--4c6cd1d40023-osd--block--e2ec785f--a44e--4ca3--800d--22769eeb352d 253:97 0 15G 0 lvm
nbd4 43:128 0 15G 0 disk
└─ceph--3aadf1e1--9f74--4d2a--baad--f608bc01a3b4-osd--block--f9e4fef7--b910--489e--b1db--60abfc769380 253:108 0 15G 0 lvm
nbd5 43:160 0 15G 0 disk
└─ceph--0c08747d--852a--4d4d--9725--baccdf0a01ce-osd--block--88295aed--2446--4c28--bcfa--3e0239c30501 253:119 0 15G 0 lvm
nbd6 43:192 0 15G 0 disk
└─ceph--c30f856d--fd98--4ea2--936f--6640a4fbf50f-osd--block--e0a36fd4--86ff--44e7--8e17--a3097093f852 253:130 0 15G 0 lvm
nbd7 43:224 0 15G 0 disk
└─ceph--a5792f13--e16a--421d--8262--bc23c1067d16-osd--block--ee77ac95--24ae--4fc2--a145--752204d5f1f5 253:136 0 15G 0 lvm
nbd8 43:256 0 15G 0 disk
└─ceph--998d0a92--1651--42f5--87c9--6ec7945f7e3c-osd--block--dc2bf347--f929--4320--ad42--f7450df67581 253:137 0 15G 0 lvm
nbd9 43:288 0 15G 0 disk
└─ceph--999a60b9--9720--4d43--9768--a4f7b578b28b-osd--block--1e64f7ad--a8ea--4cf0--ba1d--165a07aeab95 253:138 0 15G 0 lvm
nbd10 43:320 0 15G 0 disk
└─ceph--060ba7d6--baca--4fc8--8d93--0b98d6a67504-osd--block--66c937af--f02e--48df--86c6--f6e6920b8843 253:76 0 15G 0 lvm
nbd11 43:352 0 15G 0 disk
└─ceph--e18cdca0--bedd--4b7e--9899--bcd06a2b9638-osd--block--0f28580d--82e1--4122--b854--ad78b58a9e35 253:77 0 15G 0 lvm
nbd12 43:384 0 15G 0 disk
└─ceph--918d0965--d8db--453e--b016--8f481d04702a-osd--block--9925c9b2--66ee--4a6b--accc--56f0781868a0 253:78 0 15G 0 lvm
nbd13 43:416 0 15G 0 disk
└─ceph--1ad7d592--3fc8--48ee--8d6e--2146c35e0a22-osd--block--4edc5663--eac6--4197--a029--9aca49b1416c 253:79 0 15G 0 lvm
nbd14 43:448 0 15G 0 disk
└─ceph--77db4696--b703--4ae2--90a5--031b4e55cd27-osd--block--cb4bcb9e--c9cf--430a--ba43--20d7796cd2be 253:80 0 15G 0 lvm
nbd15 43:480 0 15G 0 disk
└─ceph--94b83544--d48d--484e--b2a9--2061f9a66d15-osd--block--9fa1d9ae--0d64--4b56--b0d6--7ae29cd81125 253:81 0 15G 0 lvm
nbd16 43:512 0 15G 0 disk
└─ceph--dea0d1dc--dd65--4d4e--ac19--2a735f005baf-osd--block--2eef97f0--1eb7--4a79--b931--cd643e4c7884 253:82 0 15G 0 lvm
nbd17 43:544 0 15G 0 disk
└─ceph--17c14312--7faa--482b--9518--ebc1be9269f2-osd--block--5a5cf883--d4a4--4ca5--af48--a41f1c04c009 253:83 0 15G 0 lvm
nbd18 43:576 0 15G 0 disk
└─ceph--a4a330cd--a7a5--4e80--a5e3--bdb3ee236f93-osd--block--a47743fd--fe5a--4791--a186--9fbf6bffe7f2 253:84 0 15G 0 lvm
nbd19 43:608 0 15G 0 disk
└─ceph--677d302d--2f13--485d--8371--f0b4129bf7e1-osd--block--e31bcef6--1446--4f17--8417--26c632edb767 253:85 0 15G 0 lvm
nbd20 43:640 0 15G 0 disk
└─ceph--de479b9b--4172--4831--b1b3--d571bdb5ecaf-osd--block--5f1fdec0--04b1--4b68--a530--93788980ad3e 253:87 0 15G 0 lvm
nbd21 43:672 0 15G 0 disk
└─ceph--fb98939b--a944--41c1--a742--437d04dd1c82-osd--block--0abe419c--4dd0--417f--80cc--c21d4b7176ca 253:88 0 15G 0 lvm
nbd22 43:704 0 15G 0 disk
└─ceph--18de1a02--78a8--4436--aa0d--6e36def24d96-osd--block--662416bb--364e--4a24--bbb2--d9cf27b09f2e 253:89 0 15G 0 lvm
nbd23 43:736 0 15G 0 disk
└─ceph--2f3c7386--0688--4bec--85ca--74b5f0fea561-osd--block--15083a72--a422--4b6e--a75d--4cfac2d0521c 253:90 0 15G 0 lvm
nbd24 43:768 0 15G 0 disk
└─ceph--f4573d4c--6ca6--4b25--ab2e--4953a20f2bb0-osd--block--9675d1fd--c047--45ce--b05d--db0600db243d 253:91 0 15G 0 lvm
nbd25 43:800 0 15G 0 disk
└─ceph--362b901c--6396--4b0a--9494--ebad8421f725-osd--block--b70fe4d3--21a8--491a--a015--7cf81d0aafa9 253:92 0 15G 0 lvm
nbd26 43:832 0 15G 0 disk
└─ceph--01b6f2f9--176a--4d5e--8c50--d68f861ba53d-osd--block--abbf33d4--ef87--4339--92ea--4fa4a5309ebe 253:93 0 15G 0 lvm
nbd27 43:864 0 15G 0 disk
└─ceph--77ba91ce--330d--44d4--bcbb--c4055a244d4e-osd--block--e0e8f695--a288--4101--9577--3708215d46c9 253:94 0 15G 0 lvm
nbd28 43:896 0 15G 0 disk
└─ceph--2395986d--d1e5--499a--b3f8--9e564f950562-osd--block--b920377c--f347--4a7a--84ea--d08e79460594 253:95 0 15G 0 lvm
nbd29 43:928 0 15G 0 disk
└─ceph--27de290c--a59d--469c--b14f--08ab2a585546-osd--block--518970de--677b--43cf--ac29--ce36d113b9a6 253:96 0 15G 0 lvm
nbd30 43:960 0 15G 0 disk
└─ceph--15e4ca29--a38c--4781--89b3--e10881b03ed4-osd--block--955d1460--dff5--46ba--9c3f--7dd3769a750c 253:98 0 15G 0 lvm
nbd31 43:992 0 15G 0 disk
└─ceph--c013ca44--da65--4443--b4ff--0fa1a4fcdd9f-osd--block--bfcff753--bf06--42f4--8eec--40de0f9565cc 253:99 0 15G 0 lvm
nbd32 43:1024 0 15G 0 disk
└─ceph--58af7bed--f7d6--41b8--9ae1--9f7a96794379-osd--block--4ad1f99f--a138--4560--bb85--9a500f8982a2 253:100 0 15G 0 lvm
nbd33 43:1056 0 15G 0 disk
└─ceph--dc8e8e2b--b307--43d7--a0bf--7946584f42c2-osd--block--3adbbd98--dbb1--4694--9fc9--681ef908f8ba 253:101 0 15G 0 lvm
nbd34 43:1088 0 15G 0 disk
└─ceph--ef006baf--5bd8--4a82--bfa1--31ba3a1a86fd-osd--block--08bc5eb4--dedd--4702--950c--306de0d57503 253:102 0 15G 0 lvm
nbd35 43:1120 0 15G 0 disk
└─ceph--43701b40--a241--4bad--91b9--1b6d6c7a5ffa-osd--block--3646f554--6bb3--4bbd--92f4--adb5c9da71e2 253:103 0 15G 0 lvm
nbd36 43:1152 0 15G 0 disk
└─ceph--042574fd--0a92--453a--913a--2e58599bbff8-osd--block--637fd6c6--df74--4af9--ae5e--c2fa344e5179 253:104 0 15G 0 lvm
nbd37 43:1184 0 15G 0 disk
└─ceph--38a03704--73df--4490--9704--3ca7987b2b6b-osd--block--869b5f63--f818--4801--9d37--7a336b056e44 253:105 0 15G 0 lvm
nbd38 43:1216 0 15G 0 disk
└─ceph--64fb3718--a340--4841--af0e--3ed65a319859-osd--block--47959160--a067--4714--b0e3--853b5cce208b 253:106 0 15G 0 lvm
nbd39 43:1248 0 15G 0 disk
└─ceph--f652500c--fe39--474b--baa8--ca20d6a9c90d-osd--block--913df422--01ce--44a8--b68d--706d511859d5 253:107 0 15G 0 lvm
nbd40 43:1280 0 15G 0 disk
└─ceph--ec0fed29--21f0--4e17--911e--583647deb93e-osd--block--15db49ff--aad7--458a--a656--0a403e19f0a9 253:109 0 15G 0 lvm
nbd41 43:1312 0 15G 0 disk
└─ceph--58edd60a--6f8e--4457--af15--0e358c31eff0-osd--block--725a1860--4da1--4857--8d78--e7494c255b0d 253:110 0 15G 0 lvm
nbd42 43:1344 0 15G 0 disk
└─ceph--8f7d4875--9950--46ba--964c--c0133fd0a4dd-osd--block--0e711027--deda--496a--b4a3--662d14c20db6 253:111 0 15G 0 lvm
nbd43 43:1376 0 15G 0 disk
└─ceph--29af190e--1f75--4a7c--acfe--db499ddcf4c4-osd--block--45d885d4--d5c5--4fac--b949--7e24765654eb 253:112 0 15G 0 lvm
nbd44 43:1408 0 15G 0 disk
└─ceph--ea3bfacc--f1bf--44b3--bf40--1740d2814ab0-osd--block--8488e659--cb8e--4a1a--9f8e--aea0a509d09d 253:113 0 15G 0 lvm
nbd45 43:1440 0 15G 0 disk
└─ceph--869c250f--6d79--419b--833e--df7d92ab32d9-osd--block--43a13e23--7ef3--4e2f--b24f--2151350a4b8f 253:114 0 15G 0 lvm
nbd46 43:1472 0 15G 0 disk
└─ceph--6bcf9c3f--f7cf--48a0--91fa--89b7c2389942-osd--block--ef0f707e--ad1d--44f3--bda0--8a24f6e4188d 253:115 0 15G 0 lvm
nbd47 43:1504 0 15G 0 disk
└─ceph--b06d1fab--a99d--4758--869d--ec0c805fe598-osd--block--d56bb7a3--6d59--4d82--a791--6797fbf77950 253:116 0 15G 0 lvm
nbd48 43:1536 0 15G 0 disk
└─ceph--14491d13--a715--4eb2--bf98--5d2e5c9bdd8a-osd--block--074fc8bf--088a--4a39--a3ed--283a560bd688 253:117 0 15G 0 lvm
nbd49 43:1568 0 15G 0 disk
└─ceph--e38be3e4--4c3a--4b58--8b28--09a295c0d137-osd--block--50af9301--eeca--492c--9d18--ed6861e68dbd 253:118 0 15G 0 lvm
nbd50 43:1600 0 15G 0 disk
└─ceph--7118af7c--13b2--4215--a540--347a53c49538-osd--block--dae5cff0--19fd--4578--93dd--da1a9c348b36 253:120 0 15G 0 lvm
nbd51 43:1632 0 15G 0 disk
└─ceph--45d99b8d--02a1--41a8--b272--473270369675-osd--block--6239ccd8--12e4--483e--9efd--2796a64bd337 253:121 0 15G 0 lvm
nbd52 43:1664 0 15G 0 disk
└─ceph--a2ee4182--9b64--412f--bedc--5fb8336bed4b-osd--block--082e7459--229d--4c67--8f24--b2b6a24e256f 253:122 0 15G 0 lvm
nbd53 43:1696 0 15G 0 disk
└─ceph--da606365--e257--4a7c--b8b1--92ae021d18f0-osd--block--5f651116--06a6--4113--b75d--b102a3fe8087 253:123 0 15G 0 lvm
nbd54 43:1728 0 15G 0 disk
└─ceph--a0b7a000--34d5--4c3a--8464--fa9f16cdfa2d-osd--block--e8ee1d99--a87f--4ad6--90ec--2d4c7d6bc668 253:124 0 15G 0 lvm
nbd55 43:1760 0 15G 0 disk
└─ceph--70809034--67f9--4530--96b5--287c17723955-osd--block--43f08dc4--18b4--49bd--ba3e--13b69266e38c 253:125 0 15G 0 lvm
nbd56 43:1792 0 15G 0 disk
└─ceph--cf57c6e1--cbce--428e--9718--42a3c9fe3f92-osd--block--d29ee6ce--546d--4346--a792--ba2bfff5b1f0 253:126 0 15G 0 lvm
nbd57 43:1824 0 15G 0 disk
└─ceph--48cf2cd1--d570--4208--954a--9487acca6211-osd--block--7cb2976c--14ef--4aac--b760--f67b0b324bf4 253:127 0 15G 0 lvm
nbd58 43:1856 0 15G 0 disk
└─ceph--f0013eed--033b--43e6--b73e--98b086d2d756-osd--block--d0e94d2c--d54d--43b2--a41b--c73ed395a5e0 253:128 0 15G 0 lvm
nbd59 43:1888 0 15G 0 disk
└─ceph--dbc54a27--e88f--4cd8--b9c8--1369887511ae-osd--block--897d9c1b--4490--4906--a831--f4835b49fc37 253:129 0 15G 0 lvm
nbd60 43:1920 0 15G 0 disk
└─ceph--ae2d5c10--3127--45b0--a329--a8f87a215571-osd--block--2aae29b7--f1c8--4ab3--b9c4--e4c217130755 253:131 0 15G 0 lvm
nbd61 43:1952 0 15G 0 disk
└─ceph--5ee15ce6--3bbb--4d0d--8afc--11ca96672e74-osd--block--a53003f7--75a8--466a--8f40--5839f9ac930d 253:132 0 15G 0 lvm
nbd62 43:1984 0 15G 0 disk
└─ceph--c82d068d--63b0--4f44--8712--6f30e3b6157c-osd--block--ce7cb138--0e82--422a--842a--637f19642b30 253:133 0 15G 0 lvm
nbd63 43:2016 0 15G 0 disk
└─ceph--e11387f2--b358--48df--a881--3b3228d395ae-osd--block--bfadf223--c694--4d7e--8a73--99217f09ea90 253:134 0 15G 0 lvm
nbd64 43:2048 0 15G 0 disk
└─ceph--de6c61b5--b0ef--424e--bc06--8d01d1cc1797-osd--block--a466d236--53e1--4469--8548--a88745857d40 253:135 0 15G 0 lvm
Updated by Vikhyat Umrao about 2 years ago
- Related to Bug #54263: cephadm upgrade pacific to quincy autoscaler is scaling pgs from 32 -> 32768 for cephfs meta pool added
Actions