Project

General

Profile

Bug #54238

cephadm upgrade pacifc to quincy -> causing osd's FULL/cascading failure

Added by Vikhyat Umrao 10 months ago. Updated 10 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

- Upgrade was started at 2022-02-08T01:54:28

ceph-mgr.e24-h01-000-r640.rdu2.scalelab.redhat.com.mrqbjw.log-20220208.gz:150705:2022-02-08T01:54:28.816+0000 7efca85d0700  0 log_channel(audit) log [DBG] : from='client.194120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a00e8b315af02865380634f8100dc7d18a18af4f", "target": ["mon-mgr", ""]}]: dispatch

- ceph status

# ceph -s
  cluster:
    id:     dac59560-8316-11ec-b627-bc97e17ab990
    health: HEALTH_ERR
            3356 failed cephadm daemon(s)
            1 hosts fail cephadm check
            1756 osds down
            627 full osd(s)
            Reduced data availability: 233 pgs inactive, 1 pg down, 3 pgs peering
            Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull
            Degraded data redundancy: 5318/10536 objects degraded (50.475%), 64 pgs degraded, 64 pgs undersized
            2 pool(s) full
            15 slow ops, oldest one blocked for 101279 sec, daemons [osd.158,mon.e24-h01-000-r640.rdu2.scalelab.redhat.com] have slow ops.

  services:
    mon: 3 daemons, quorum e24-h01-000-r640.rdu2.scalelab.redhat.com,e24-h02-000-r640,e24-h03-000-r640 (age 24h)
    mgr: e24-h01-000-r640.rdu2.scalelab.redhat.com.mrqbjw(active, since 24h), standbys: e24-h02-000-r640.kxdcop
    osd: 3904 osds: 1171 up (since 24h), 2927 in (since 28h); 16 remapped pgs

  data:
    pools:   2 pools, 257 pgs
    objects: 3.51k objects, 14 GiB
    usage:   4.7 TiB used, 3.3 TiB / 8.0 TiB avail
    pgs:     71.984% pgs unknown
             18.677% pgs not active
             5318/10536 objects degraded (50.475%)
             185 unknown
             42  undersized+degraded+peered
             19  active+undersized+degraded
             3   peering
             2   undersized+degraded+remapped+backfill_wait+backfill_toofull+peered
             2   active+clean+scrubbing
             1   active+undersized+degraded+remapped+backfill_wait+backfill_toofull
             1   active+clean+scrubbing+deep
             1   down
             1   active+clean

- Version

# ceph versions
{
    "mon": {
        "ceph version 17.0.0-10315-ga00e8b31 (a00e8b315af02865380634f8100dc7d18a18af4f) quincy (dev)": 3
    },
    "mgr": {
        "ceph version 17.0.0-10315-ga00e8b31 (a00e8b315af02865380634f8100dc7d18a18af4f) quincy (dev)": 2
    },
    "osd": {
        "ceph version 16.2.7-34.el8cp (70c4491bd537223bac6d19fdba941d452d4641c2) pacific (stable)": 681,
        "ceph version 17.0.0-10315-ga00e8b31 (a00e8b315af02865380634f8100dc7d18a18af4f) quincy (dev)": 490
    },
    "mds": {},
    "overall": {
        "ceph version 16.2.7-34.el8cp (70c4491bd537223bac6d19fdba941d452d4641c2) pacific (stable)": 681,
        "ceph version 17.0.0-10315-ga00e8b31 (a00e8b315af02865380634f8100dc7d18a18af4f) quincy (dev)": 495
    }
}

- From cluster logs

2022-02-08T09:20:00.000394+0000 mon.e24-h01-000-r640.rdu2.scalelab.redhat.com (mon.0) 22414 : cluster [ERR] [ERR] OSD_FULL: 1 full osd(s)
2022-02-08T09:20:00.000411+0000 mon.e24-h01-000-r640.rdu2.scalelab.redhat.com (mon.0) 22415 : cluster [ERR]     osd.2080 is full

From OSD logs:

2022-02-08T09:17:16.223+0000 7f1583974700  1 osd.2080 44163 advance_pg 4.69a3 is merge source, target is 4.29a3
2022-02-08T09:17:16.223+0000 7f1583974700  1 osd.2080 44163 advance_pg merging 4.29a3
2022-02-08T09:17:16.223+0000 7f1583974700  1 osd.2080 pg_epoch: 44163 pg[4.29a3( v 22136'1361 (22136'1361,22136'1361] local-lis/les=44161/44162 n=127 ec=8418/7032 lis/c=44161/44161 les/c/f=44162/44162/43997 sis=44163 pruub=14.786740303s) [1347,2080,3393] r=1 lpr=44163 pi=[44161,44163)/1 luod=0'0 lua=22134'1340 crt=22136'1361 lcod 22133'1339 mlcod 0'0 active pruub 534281.437500000s@ mbc={}] start_peering_interval up [1347,2080,3393] -> [1347,2080,3393], acting [1347,2080,3393] -> [1347,2080,3393], acting_primary 1347 -> 1347, up_primary 1347 -> 1347, role 1 -> 1, features acting 4540138297136906239 upacting 4540138297136906239
2022-02-08T09:17:16.224+0000 7f1583974700  1 osd.2080 pg_epoch: 44163 pg[4.29a3( v 22136'1361 (22136'1361,22136'1361] local-lis/les=44161/44162 n=127 ec=8418/7032 lis/c=44161/44161 les/c/f=44162/44162/43997 sis=44163 pruub=14.786669731s) [1347,2080,3393] r=1 lpr=44163 pi=[44161,44163)/1 crt=22136'1361 lcod 22133'1339 mlcod 0'0 unknown NOTIFY pruub 534281.437500000s@ mbc={}] state<Start>: transitioning to Stray
2022-02-08T09:22:50.200+0000 7f1594fd1700  4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS -------
2022-02-08T09:22:50.200+0000 7f1594fd1700  4 rocksdb: [db_impl/db_impl.cc:851]
2022-02-08T12:47:13.007+0000 7f15967d4700 -1 bluefs _allocate allocation failed, needed 0x2236
2022-02-08T12:47:13.007+0000 7f15967d4700 -1 bluefs _flush_range allocated: 0xede0000 offset: 0xedded2d length: 0x3509
2022-02-08T12:47:13.015+0000 7f15967d4700 -1 /builddir/build/BUILD/ceph-16.2.7/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f15967d4700 time 2022-02-08T12:47:13.009096+0000
/builddir/build/BUILD/ceph-16.2.7/src/os/bluestore/BlueFS.cc: 2768: ceph_abort_msg("bluefs enospc")

 ceph version 16.2.7-34.el8cp (70c4491bd537223bac6d19fdba941d452d4641c2) pacific (stable)

1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe5) [0x556186fc545e]
 2: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1131) [0x5561876c0f41]
 3: (BlueFS::_flush(BlueFS::FileWriter*, bool, bool*)+0x90) [0x5561876c1220]
 4: (BlueFS::_flush(BlueFS::FileWriter*, bool, std::unique_lock<std::mutex>&)+0x32) [0x5561876d2332]
 5: (BlueRocksWritableFile::Append(rocksdb::Slice const&)+0x11b) [0x5561876eac6b]
 6: (rocksdb::LegacyWritableFileWrapper::Append(rocksdb::Slice const&, rocksdb::IOOptions const&, rocksdb::IODebugContext*)+0x1f) [0x556187b850df]
 7: (rocksdb::WritableFileWriter::WriteBuffered(char const*, unsigned long)+0x58a) [0x556187c96e2a]
 8: (rocksdb::WritableFileWriter::Append(rocksdb::Slice const&)+0x2d0) [0x556187c98280]
 9: (rocksdb::BlockBasedTableBuilder::WriteRawBlock(rocksdb::Slice const&, rocksdb::CompressionType, rocksdb::BlockHandle*, bool)+0xb6) [0x556187db3a06]
 10: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*, bool)+0x26c) [0x556187db434c]
 11: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::BlockBuilder*, rocksdb::BlockHandle*, bool)+0x3c) [0x556187db4a4c]
 12: (rocksdb::BlockBasedTableBuilder::Flush()+0x6d) [0x556187db4add]
 13: (rocksdb::BlockBasedTableBuilder::Add(rocksdb::Slice const&, rocksdb::Slice const&)+0x2b8) [0x556187db7f48]
 14: (rocksdb::BuildTable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, rocksdb::FileSystem*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase<rocksdb::Slice>*, std::vector<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> >, std::allocator<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> > > >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, unsigned long, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint, unsigned long)+0xa45) [0x556187d629c5]
 15: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xcf5) [0x556187bc7a25]
 16: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool, bool*)+0x1c2e) [0x556187bca15e]
 17: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool, unsigned long*)+0xae8) [0x556187bcb4b8]
 18: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool)+0x59d) [0x556187bc51dd]
 19: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x15) [0x556187bc6575]
 20: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x10c1) [0x556187b3e4b1]
 21: (BlueStore::_open_db(bool, bool, bool)+0x8c7) [0x5561875bb0b7]
 22: (BlueStore::_open_db_and_around(bool, bool)+0x2f7) [0x556187625b47]
 23: (BlueStore::_mount()+0x204) [0x556187628a04]
 24: (OSD::init()+0x380) [0x5561870fc5d0]
 25: main()
 26: __libc_start_main()
 27: _start()


Related issues

Related to RADOS - Bug #54263: cephadm upgrade pacific to quincy autoscaler is scaling pgs from 32 -> 32768 for cephfs meta pool Resolved

History

#1 Updated by Vikhyat Umrao 10 months ago

The node e24-h01-000-r640 has a file - upgrade.txt from the following command:

Path - /var/log/ceph/dac59560-8316-11ec-b627-bc97e17ab990/upgrade.txt
zgrep -n "upgrade" ceph-mgr.e24-h01-000-r640.rdu2.scalelab.redhat.com.mrqbjw.log* > upgrade.txt

This file has upgrade details from the MGR logs.

#2 Updated by Vikhyat Umrao 10 months ago

# lsblk 
NAME                                                                                                  MAJ:MIN  RM   SIZE RO TYPE MOUNTPOINT
sda                                                                                                     8:0     0 446.6G  0 disk 
├─sda1                                                                                                  8:1     0   512M  0 part /boot
└─sda2                                                                                                  8:2     0 446.1G  0 part 
  ├─vg_e24--h01--000--r640-lv_root                                                                    253:0     0 438.1G  0 lvm  /
  └─vg_e24--h01--000--r640-lv_swap                                                                    253:1     0     8G  0 lvm  [SWAP]
sdb                                                                                                     8:16    0   1.8T  0 disk 
├─fakedevs-thinpool_tmeta                                                                             253:2     0   112M  0 lvm  
│ └─fakedevs-thinpool-tpool                                                                           253:9     0   1.7T  0 lvm  
│   ├─fakedevs-thinpool                                                                               253:10    0   1.7T  1 lvm  
│   ├─fakedevs-disk1                                                                                  253:11    0    15G  0 lvm  
│   ├─fakedevs-disk2                                                                                  253:12    0    15G  0 lvm  
│   ├─fakedevs-disk3                                                                                  253:13    0    15G  0 lvm  
│   ├─fakedevs-disk4                                                                                  253:14    0    15G  0 lvm  
│   ├─fakedevs-disk5                                                                                  253:15    0    15G  0 lvm  
│   ├─fakedevs-disk6                                                                                  253:16    0    15G  0 lvm  
│   ├─fakedevs-disk7                                                                                  253:17    0    15G  0 lvm  
│   ├─fakedevs-disk8                                                                                  253:18    0    15G  0 lvm  
│   ├─fakedevs-disk9                                                                                  253:19    0    15G  0 lvm  
│   ├─fakedevs-disk10                                                                                 253:20    0    15G  0 lvm  
│   ├─fakedevs-disk11                                                                                 253:21    0    15G  0 lvm  
│   ├─fakedevs-disk12                                                                                 253:22    0    15G  0 lvm  
│   ├─fakedevs-disk13                                                                                 253:23    0    15G  0 lvm  
│   ├─fakedevs-disk14                                                                                 253:24    0    15G  0 lvm  
│   ├─fakedevs-disk15                                                                                 253:25    0    15G  0 lvm  
│   ├─fakedevs-disk16                                                                                 253:26    0    15G  0 lvm  
│   ├─fakedevs-disk17                                                                                 253:27    0    15G  0 lvm  
│   ├─fakedevs-disk18                                                                                 253:28    0    15G  0 lvm  
│   ├─fakedevs-disk19                                                                                 253:29    0    15G  0 lvm  
│   ├─fakedevs-disk20                                                                                 253:30    0    15G  0 lvm  
│   ├─fakedevs-disk21                                                                                 253:31    0    15G  0 lvm  
│   ├─fakedevs-disk22                                                                                 253:32    0    15G  0 lvm  
│   ├─fakedevs-disk23                                                                                 253:33    0    15G  0 lvm  
│   ├─fakedevs-disk24                                                                                 253:34    0    15G  0 lvm  
│   ├─fakedevs-disk25                                                                                 253:35    0    15G  0 lvm  
│   ├─fakedevs-disk26                                                                                 253:36    0    15G  0 lvm  
│   ├─fakedevs-disk27                                                                                 253:37    0    15G  0 lvm  
│   ├─fakedevs-disk28                                                                                 253:38    0    15G  0 lvm  
│   ├─fakedevs-disk29                                                                                 253:39    0    15G  0 lvm  
│   ├─fakedevs-disk30                                                                                 253:40    0    15G  0 lvm  
│   ├─fakedevs-disk31                                                                                 253:41    0    15G  0 lvm  
│   ├─fakedevs-disk32                                                                                 253:42    0    15G  0 lvm  
│   ├─fakedevs-disk33                                                                                 253:43    0    15G  0 lvm  
│   ├─fakedevs-disk34                                                                                 253:44    0    15G  0 lvm  
│   ├─fakedevs-disk35                                                                                 253:45    0    15G  0 lvm  
│   ├─fakedevs-disk36                                                                                 253:46    0    15G  0 lvm  
│   ├─fakedevs-disk37                                                                                 253:47    0    15G  0 lvm  
│   ├─fakedevs-disk38                                                                                 253:48    0    15G  0 lvm  
│   ├─fakedevs-disk39                                                                                 253:49    0    15G  0 lvm  
│   ├─fakedevs-disk40                                                                                 253:50    0    15G  0 lvm  
│   ├─fakedevs-disk41                                                                                 253:51    0    15G  0 lvm  
│   ├─fakedevs-disk42                                                                                 253:52    0    15G  0 lvm  
│   ├─fakedevs-disk43                                                                                 253:53    0    15G  0 lvm  
│   ├─fakedevs-disk44                                                                                 253:54    0    15G  0 lvm  
│   ├─fakedevs-disk45                                                                                 253:55    0    15G  0 lvm  
│   ├─fakedevs-disk46                                                                                 253:56    0    15G  0 lvm  
│   ├─fakedevs-disk47                                                                                 253:57    0    15G  0 lvm  
│   ├─fakedevs-disk48                                                                                 253:58    0    15G  0 lvm  
│   ├─fakedevs-disk49                                                                                 253:59    0    15G  0 lvm  
│   ├─fakedevs-disk50                                                                                 253:60    0    15G  0 lvm  
│   ├─fakedevs-disk51                                                                                 253:61    0    15G  0 lvm  
│   ├─fakedevs-disk52                                                                                 253:62    0    15G  0 lvm  
│   ├─fakedevs-disk53                                                                                 253:63    0    15G  0 lvm  
│   ├─fakedevs-disk54                                                                                 253:64    0    15G  0 lvm  
│   ├─fakedevs-disk55                                                                                 253:65    0    15G  0 lvm  
│   ├─fakedevs-disk56                                                                                 253:66    0    15G  0 lvm  
│   ├─fakedevs-disk57                                                                                 253:67    0    15G  0 lvm  
│   ├─fakedevs-disk58                                                                                 253:68    0    15G  0 lvm  
│   ├─fakedevs-disk59                                                                                 253:69    0    15G  0 lvm  
│   ├─fakedevs-disk60                                                                                 253:70    0    15G  0 lvm  
│   ├─fakedevs-disk61                                                                                 253:71    0    15G  0 lvm  
│   ├─fakedevs-disk62                                                                                 253:72    0    15G  0 lvm  
│   ├─fakedevs-disk63                                                                                 253:73    0    15G  0 lvm  
│   └─fakedevs-disk64                                                                                 253:74    0    15G  0 lvm  
└─fakedevs-thinpool_tdata                                                                             253:5     0   1.7T  0 lvm  
  └─fakedevs-thinpool-tpool                                                                           253:9     0   1.7T  0 lvm  
    ├─fakedevs-thinpool                                                                               253:10    0   1.7T  1 lvm  
    ├─fakedevs-disk1                                                                                  253:11    0    15G  0 lvm  
    ├─fakedevs-disk2                                                                                  253:12    0    15G  0 lvm  
    ├─fakedevs-disk3                                                                                  253:13    0    15G  0 lvm  
    ├─fakedevs-disk4                                                                                  253:14    0    15G  0 lvm  
    ├─fakedevs-disk5                                                                                  253:15    0    15G  0 lvm  
    ├─fakedevs-disk6                                                                                  253:16    0    15G  0 lvm  
    ├─fakedevs-disk7                                                                                  253:17    0    15G  0 lvm  
    ├─fakedevs-disk8                                                                                  253:18    0    15G  0 lvm  
    ├─fakedevs-disk9                                                                                  253:19    0    15G  0 lvm  
    ├─fakedevs-disk10                                                                                 253:20    0    15G  0 lvm  
    ├─fakedevs-disk11                                                                                 253:21    0    15G  0 lvm  
    ├─fakedevs-disk12                                                                                 253:22    0    15G  0 lvm  
    ├─fakedevs-disk13                                                                                 253:23    0    15G  0 lvm  
    ├─fakedevs-disk14                                                                                 253:24    0    15G  0 lvm  
    ├─fakedevs-disk15                                                                                 253:25    0    15G  0 lvm  
    ├─fakedevs-disk16                                                                                 253:26    0    15G  0 lvm  
    ├─fakedevs-disk17                                                                                 253:27    0    15G  0 lvm  
    ├─fakedevs-disk18                                                                                 253:28    0    15G  0 lvm  
    ├─fakedevs-disk19                                                                                 253:29    0    15G  0 lvm  
    ├─fakedevs-disk20                                                                                 253:30    0    15G  0 lvm  
    ├─fakedevs-disk21                                                                                 253:31    0    15G  0 lvm  
    ├─fakedevs-disk22                                                                                 253:32    0    15G  0 lvm  
    ├─fakedevs-disk23                                                                                 253:33    0    15G  0 lvm  
    ├─fakedevs-disk24                                                                                 253:34    0    15G  0 lvm  
    ├─fakedevs-disk25                                                                                 253:35    0    15G  0 lvm  
    ├─fakedevs-disk26                                                                                 253:36    0    15G  0 lvm  
    ├─fakedevs-disk27                                                                                 253:37    0    15G  0 lvm  
    ├─fakedevs-disk28                                                                                 253:38    0    15G  0 lvm  
    ├─fakedevs-disk29                                                                                 253:39    0    15G  0 lvm  
    ├─fakedevs-disk30                                                                                 253:40    0    15G  0 lvm  
    ├─fakedevs-disk31                                                                                 253:41    0    15G  0 lvm  
    ├─fakedevs-disk32                                                                                 253:42    0    15G  0 lvm  
    ├─fakedevs-disk33                                                                                 253:43    0    15G  0 lvm  
    ├─fakedevs-disk34                                                                                 253:44    0    15G  0 lvm  
    ├─fakedevs-disk35                                                                                 253:45    0    15G  0 lvm  
    ├─fakedevs-disk36                                                                                 253:46    0    15G  0 lvm  
    ├─fakedevs-disk37                                                                                 253:47    0    15G  0 lvm  
    ├─fakedevs-disk38                                                                                 253:48    0    15G  0 lvm  
    ├─fakedevs-disk39                                                                                 253:49    0    15G  0 lvm  
    ├─fakedevs-disk40                                                                                 253:50    0    15G  0 lvm  
    ├─fakedevs-disk41                                                                                 253:51    0    15G  0 lvm  
    ├─fakedevs-disk42                                                                                 253:52    0    15G  0 lvm  
    ├─fakedevs-disk43                                                                                 253:53    0    15G  0 lvm  
    ├─fakedevs-disk44                                                                                 253:54    0    15G  0 lvm  
    ├─fakedevs-disk45                                                                                 253:55    0    15G  0 lvm  
    ├─fakedevs-disk46                                                                                 253:56    0    15G  0 lvm  
    ├─fakedevs-disk47                                                                                 253:57    0    15G  0 lvm  
    ├─fakedevs-disk48                                                                                 253:58    0    15G  0 lvm  
    ├─fakedevs-disk49                                                                                 253:59    0    15G  0 lvm  
    ├─fakedevs-disk50                                                                                 253:60    0    15G  0 lvm  
    ├─fakedevs-disk51                                                                                 253:61    0    15G  0 lvm  
    ├─fakedevs-disk52                                                                                 253:62    0    15G  0 lvm  
    ├─fakedevs-disk53                                                                                 253:63    0    15G  0 lvm  
    ├─fakedevs-disk54                                                                                 253:64    0    15G  0 lvm  
    ├─fakedevs-disk55                                                                                 253:65    0    15G  0 lvm  
    ├─fakedevs-disk56                                                                                 253:66    0    15G  0 lvm  
    ├─fakedevs-disk57                                                                                 253:67    0    15G  0 lvm  
    ├─fakedevs-disk58                                                                                 253:68    0    15G  0 lvm  
    ├─fakedevs-disk59                                                                                 253:69    0    15G  0 lvm  
    ├─fakedevs-disk60                                                                                 253:70    0    15G  0 lvm  
    ├─fakedevs-disk61                                                                                 253:71    0    15G  0 lvm  
    ├─fakedevs-disk62                                                                                 253:72    0    15G  0 lvm  
    ├─fakedevs-disk63                                                                                 253:73    0    15G  0 lvm  
    └─fakedevs-disk64                                                                                 253:74    0    15G  0 lvm  
sdc                                                                                                     8:32    0   1.8T  0 disk 
├─data-thinpool_tmeta                                                                                 253:3     0   112M  0 lvm  
│ └─data-thinpool-tpool                                                                               253:6     0   1.7T  0 lvm  
│   ├─data-thinpool                                                                                   253:7     0   1.7T  1 lvm  
│   ├─data-prometheus                                                                                 253:8     0     1T  0 lvm  
│   └─data-prometheus--pacific                                                                        253:139   0    60G  0 lvm  
└─data-thinpool_tdata                                                                                 253:4     0   1.7T  0 lvm  
  └─data-thinpool-tpool                                                                               253:6     0   1.7T  0 lvm  
    ├─data-thinpool                                                                                   253:7     0   1.7T  1 lvm  
    ├─data-prometheus                                                                                 253:8     0     1T  0 lvm  
    └─data-prometheus--pacific                                                                        253:139   0    60G  0 lvm  
nbd1                                                                                                   43:32    0    15G  0 disk 
└─ceph--ed9435ce--f5af--4fc8--b2e4--a314fcac3854-osd--block--e8c73e72--765c--41aa--895c--98146b6a107d 253:75    0    15G  0 lvm  
nbd2                                                                                                   43:64    0    15G  0 disk 
└─ceph--17919713--dc6c--4c9c--b22b--49a1417cc580-osd--block--c8765267--c933--4e55--b3b1--5769780f6c46 253:86    0    15G  0 lvm  
nbd3                                                                                                   43:96    0    15G  0 disk 
└─ceph--ce68c898--9a3c--4005--8cdc--4c6cd1d40023-osd--block--e2ec785f--a44e--4ca3--800d--22769eeb352d 253:97    0    15G  0 lvm  
nbd4                                                                                                   43:128   0    15G  0 disk 
└─ceph--3aadf1e1--9f74--4d2a--baad--f608bc01a3b4-osd--block--f9e4fef7--b910--489e--b1db--60abfc769380 253:108   0    15G  0 lvm  
nbd5                                                                                                   43:160   0    15G  0 disk 
└─ceph--0c08747d--852a--4d4d--9725--baccdf0a01ce-osd--block--88295aed--2446--4c28--bcfa--3e0239c30501 253:119   0    15G  0 lvm  
nbd6                                                                                                   43:192   0    15G  0 disk 
└─ceph--c30f856d--fd98--4ea2--936f--6640a4fbf50f-osd--block--e0a36fd4--86ff--44e7--8e17--a3097093f852 253:130   0    15G  0 lvm  
nbd7                                                                                                   43:224   0    15G  0 disk 
└─ceph--a5792f13--e16a--421d--8262--bc23c1067d16-osd--block--ee77ac95--24ae--4fc2--a145--752204d5f1f5 253:136   0    15G  0 lvm  
nbd8                                                                                                   43:256   0    15G  0 disk 
└─ceph--998d0a92--1651--42f5--87c9--6ec7945f7e3c-osd--block--dc2bf347--f929--4320--ad42--f7450df67581 253:137   0    15G  0 lvm  
nbd9                                                                                                   43:288   0    15G  0 disk 
└─ceph--999a60b9--9720--4d43--9768--a4f7b578b28b-osd--block--1e64f7ad--a8ea--4cf0--ba1d--165a07aeab95 253:138   0    15G  0 lvm  
nbd10                                                                                                  43:320   0    15G  0 disk 
└─ceph--060ba7d6--baca--4fc8--8d93--0b98d6a67504-osd--block--66c937af--f02e--48df--86c6--f6e6920b8843 253:76    0    15G  0 lvm  
nbd11                                                                                                  43:352   0    15G  0 disk 
└─ceph--e18cdca0--bedd--4b7e--9899--bcd06a2b9638-osd--block--0f28580d--82e1--4122--b854--ad78b58a9e35 253:77    0    15G  0 lvm  
nbd12                                                                                                  43:384   0    15G  0 disk 
└─ceph--918d0965--d8db--453e--b016--8f481d04702a-osd--block--9925c9b2--66ee--4a6b--accc--56f0781868a0 253:78    0    15G  0 lvm  
nbd13                                                                                                  43:416   0    15G  0 disk 
└─ceph--1ad7d592--3fc8--48ee--8d6e--2146c35e0a22-osd--block--4edc5663--eac6--4197--a029--9aca49b1416c 253:79    0    15G  0 lvm  
nbd14                                                                                                  43:448   0    15G  0 disk 
└─ceph--77db4696--b703--4ae2--90a5--031b4e55cd27-osd--block--cb4bcb9e--c9cf--430a--ba43--20d7796cd2be 253:80    0    15G  0 lvm  
nbd15                                                                                                  43:480   0    15G  0 disk 
└─ceph--94b83544--d48d--484e--b2a9--2061f9a66d15-osd--block--9fa1d9ae--0d64--4b56--b0d6--7ae29cd81125 253:81    0    15G  0 lvm  
nbd16                                                                                                  43:512   0    15G  0 disk 
└─ceph--dea0d1dc--dd65--4d4e--ac19--2a735f005baf-osd--block--2eef97f0--1eb7--4a79--b931--cd643e4c7884 253:82    0    15G  0 lvm  
nbd17                                                                                                  43:544   0    15G  0 disk 
└─ceph--17c14312--7faa--482b--9518--ebc1be9269f2-osd--block--5a5cf883--d4a4--4ca5--af48--a41f1c04c009 253:83    0    15G  0 lvm  
nbd18                                                                                                  43:576   0    15G  0 disk 
└─ceph--a4a330cd--a7a5--4e80--a5e3--bdb3ee236f93-osd--block--a47743fd--fe5a--4791--a186--9fbf6bffe7f2 253:84    0    15G  0 lvm  
nbd19                                                                                                  43:608   0    15G  0 disk 
└─ceph--677d302d--2f13--485d--8371--f0b4129bf7e1-osd--block--e31bcef6--1446--4f17--8417--26c632edb767 253:85    0    15G  0 lvm  
nbd20                                                                                                  43:640   0    15G  0 disk 
└─ceph--de479b9b--4172--4831--b1b3--d571bdb5ecaf-osd--block--5f1fdec0--04b1--4b68--a530--93788980ad3e 253:87    0    15G  0 lvm  
nbd21                                                                                                  43:672   0    15G  0 disk 
└─ceph--fb98939b--a944--41c1--a742--437d04dd1c82-osd--block--0abe419c--4dd0--417f--80cc--c21d4b7176ca 253:88    0    15G  0 lvm  
nbd22                                                                                                  43:704   0    15G  0 disk 
└─ceph--18de1a02--78a8--4436--aa0d--6e36def24d96-osd--block--662416bb--364e--4a24--bbb2--d9cf27b09f2e 253:89    0    15G  0 lvm  
nbd23                                                                                                  43:736   0    15G  0 disk 
└─ceph--2f3c7386--0688--4bec--85ca--74b5f0fea561-osd--block--15083a72--a422--4b6e--a75d--4cfac2d0521c 253:90    0    15G  0 lvm  
nbd24                                                                                                  43:768   0    15G  0 disk 
└─ceph--f4573d4c--6ca6--4b25--ab2e--4953a20f2bb0-osd--block--9675d1fd--c047--45ce--b05d--db0600db243d 253:91    0    15G  0 lvm  
nbd25                                                                                                  43:800   0    15G  0 disk 
└─ceph--362b901c--6396--4b0a--9494--ebad8421f725-osd--block--b70fe4d3--21a8--491a--a015--7cf81d0aafa9 253:92    0    15G  0 lvm  
nbd26                                                                                                  43:832   0    15G  0 disk 
└─ceph--01b6f2f9--176a--4d5e--8c50--d68f861ba53d-osd--block--abbf33d4--ef87--4339--92ea--4fa4a5309ebe 253:93    0    15G  0 lvm  
nbd27                                                                                                  43:864   0    15G  0 disk 
└─ceph--77ba91ce--330d--44d4--bcbb--c4055a244d4e-osd--block--e0e8f695--a288--4101--9577--3708215d46c9 253:94    0    15G  0 lvm  
nbd28                                                                                                  43:896   0    15G  0 disk 
└─ceph--2395986d--d1e5--499a--b3f8--9e564f950562-osd--block--b920377c--f347--4a7a--84ea--d08e79460594 253:95    0    15G  0 lvm  
nbd29                                                                                                  43:928   0    15G  0 disk 
└─ceph--27de290c--a59d--469c--b14f--08ab2a585546-osd--block--518970de--677b--43cf--ac29--ce36d113b9a6 253:96    0    15G  0 lvm  
nbd30                                                                                                  43:960   0    15G  0 disk 
└─ceph--15e4ca29--a38c--4781--89b3--e10881b03ed4-osd--block--955d1460--dff5--46ba--9c3f--7dd3769a750c 253:98    0    15G  0 lvm  
nbd31                                                                                                  43:992   0    15G  0 disk 
└─ceph--c013ca44--da65--4443--b4ff--0fa1a4fcdd9f-osd--block--bfcff753--bf06--42f4--8eec--40de0f9565cc 253:99    0    15G  0 lvm  
nbd32                                                                                                  43:1024  0    15G  0 disk 
└─ceph--58af7bed--f7d6--41b8--9ae1--9f7a96794379-osd--block--4ad1f99f--a138--4560--bb85--9a500f8982a2 253:100   0    15G  0 lvm  
nbd33                                                                                                  43:1056  0    15G  0 disk 
└─ceph--dc8e8e2b--b307--43d7--a0bf--7946584f42c2-osd--block--3adbbd98--dbb1--4694--9fc9--681ef908f8ba 253:101   0    15G  0 lvm  
nbd34                                                                                                  43:1088  0    15G  0 disk 
└─ceph--ef006baf--5bd8--4a82--bfa1--31ba3a1a86fd-osd--block--08bc5eb4--dedd--4702--950c--306de0d57503 253:102   0    15G  0 lvm  
nbd35                                                                                                  43:1120  0    15G  0 disk 
└─ceph--43701b40--a241--4bad--91b9--1b6d6c7a5ffa-osd--block--3646f554--6bb3--4bbd--92f4--adb5c9da71e2 253:103   0    15G  0 lvm  
nbd36                                                                                                  43:1152  0    15G  0 disk 
└─ceph--042574fd--0a92--453a--913a--2e58599bbff8-osd--block--637fd6c6--df74--4af9--ae5e--c2fa344e5179 253:104   0    15G  0 lvm  
nbd37                                                                                                  43:1184  0    15G  0 disk 
└─ceph--38a03704--73df--4490--9704--3ca7987b2b6b-osd--block--869b5f63--f818--4801--9d37--7a336b056e44 253:105   0    15G  0 lvm  
nbd38                                                                                                  43:1216  0    15G  0 disk 
└─ceph--64fb3718--a340--4841--af0e--3ed65a319859-osd--block--47959160--a067--4714--b0e3--853b5cce208b 253:106   0    15G  0 lvm  
nbd39                                                                                                  43:1248  0    15G  0 disk 
└─ceph--f652500c--fe39--474b--baa8--ca20d6a9c90d-osd--block--913df422--01ce--44a8--b68d--706d511859d5 253:107   0    15G  0 lvm  
nbd40                                                                                                  43:1280  0    15G  0 disk 
└─ceph--ec0fed29--21f0--4e17--911e--583647deb93e-osd--block--15db49ff--aad7--458a--a656--0a403e19f0a9 253:109   0    15G  0 lvm  
nbd41                                                                                                  43:1312  0    15G  0 disk 
└─ceph--58edd60a--6f8e--4457--af15--0e358c31eff0-osd--block--725a1860--4da1--4857--8d78--e7494c255b0d 253:110   0    15G  0 lvm  
nbd42                                                                                                  43:1344  0    15G  0 disk 
└─ceph--8f7d4875--9950--46ba--964c--c0133fd0a4dd-osd--block--0e711027--deda--496a--b4a3--662d14c20db6 253:111   0    15G  0 lvm  
nbd43                                                                                                  43:1376  0    15G  0 disk 
└─ceph--29af190e--1f75--4a7c--acfe--db499ddcf4c4-osd--block--45d885d4--d5c5--4fac--b949--7e24765654eb 253:112   0    15G  0 lvm  
nbd44                                                                                                  43:1408  0    15G  0 disk 
└─ceph--ea3bfacc--f1bf--44b3--bf40--1740d2814ab0-osd--block--8488e659--cb8e--4a1a--9f8e--aea0a509d09d 253:113   0    15G  0 lvm  
nbd45                                                                                                  43:1440  0    15G  0 disk 
└─ceph--869c250f--6d79--419b--833e--df7d92ab32d9-osd--block--43a13e23--7ef3--4e2f--b24f--2151350a4b8f 253:114   0    15G  0 lvm  
nbd46                                                                                                  43:1472  0    15G  0 disk 
└─ceph--6bcf9c3f--f7cf--48a0--91fa--89b7c2389942-osd--block--ef0f707e--ad1d--44f3--bda0--8a24f6e4188d 253:115   0    15G  0 lvm  
nbd47                                                                                                  43:1504  0    15G  0 disk 
└─ceph--b06d1fab--a99d--4758--869d--ec0c805fe598-osd--block--d56bb7a3--6d59--4d82--a791--6797fbf77950 253:116   0    15G  0 lvm  
nbd48                                                                                                  43:1536  0    15G  0 disk 
└─ceph--14491d13--a715--4eb2--bf98--5d2e5c9bdd8a-osd--block--074fc8bf--088a--4a39--a3ed--283a560bd688 253:117   0    15G  0 lvm  
nbd49                                                                                                  43:1568  0    15G  0 disk 
└─ceph--e38be3e4--4c3a--4b58--8b28--09a295c0d137-osd--block--50af9301--eeca--492c--9d18--ed6861e68dbd 253:118   0    15G  0 lvm  
nbd50                                                                                                  43:1600  0    15G  0 disk 
└─ceph--7118af7c--13b2--4215--a540--347a53c49538-osd--block--dae5cff0--19fd--4578--93dd--da1a9c348b36 253:120   0    15G  0 lvm  
nbd51                                                                                                  43:1632  0    15G  0 disk 
└─ceph--45d99b8d--02a1--41a8--b272--473270369675-osd--block--6239ccd8--12e4--483e--9efd--2796a64bd337 253:121   0    15G  0 lvm  
nbd52                                                                                                  43:1664  0    15G  0 disk 
└─ceph--a2ee4182--9b64--412f--bedc--5fb8336bed4b-osd--block--082e7459--229d--4c67--8f24--b2b6a24e256f 253:122   0    15G  0 lvm  
nbd53                                                                                                  43:1696  0    15G  0 disk 
└─ceph--da606365--e257--4a7c--b8b1--92ae021d18f0-osd--block--5f651116--06a6--4113--b75d--b102a3fe8087 253:123   0    15G  0 lvm  
nbd54                                                                                                  43:1728  0    15G  0 disk 
└─ceph--a0b7a000--34d5--4c3a--8464--fa9f16cdfa2d-osd--block--e8ee1d99--a87f--4ad6--90ec--2d4c7d6bc668 253:124   0    15G  0 lvm  
nbd55                                                                                                  43:1760  0    15G  0 disk 
└─ceph--70809034--67f9--4530--96b5--287c17723955-osd--block--43f08dc4--18b4--49bd--ba3e--13b69266e38c 253:125   0    15G  0 lvm  
nbd56                                                                                                  43:1792  0    15G  0 disk 
└─ceph--cf57c6e1--cbce--428e--9718--42a3c9fe3f92-osd--block--d29ee6ce--546d--4346--a792--ba2bfff5b1f0 253:126   0    15G  0 lvm  
nbd57                                                                                                  43:1824  0    15G  0 disk 
└─ceph--48cf2cd1--d570--4208--954a--9487acca6211-osd--block--7cb2976c--14ef--4aac--b760--f67b0b324bf4 253:127   0    15G  0 lvm  
nbd58                                                                                                  43:1856  0    15G  0 disk 
└─ceph--f0013eed--033b--43e6--b73e--98b086d2d756-osd--block--d0e94d2c--d54d--43b2--a41b--c73ed395a5e0 253:128   0    15G  0 lvm  
nbd59                                                                                                  43:1888  0    15G  0 disk 
└─ceph--dbc54a27--e88f--4cd8--b9c8--1369887511ae-osd--block--897d9c1b--4490--4906--a831--f4835b49fc37 253:129   0    15G  0 lvm  
nbd60                                                                                                  43:1920  0    15G  0 disk 
└─ceph--ae2d5c10--3127--45b0--a329--a8f87a215571-osd--block--2aae29b7--f1c8--4ab3--b9c4--e4c217130755 253:131   0    15G  0 lvm  
nbd61                                                                                                  43:1952  0    15G  0 disk 
└─ceph--5ee15ce6--3bbb--4d0d--8afc--11ca96672e74-osd--block--a53003f7--75a8--466a--8f40--5839f9ac930d 253:132   0    15G  0 lvm  
nbd62                                                                                                  43:1984  0    15G  0 disk 
└─ceph--c82d068d--63b0--4f44--8712--6f30e3b6157c-osd--block--ce7cb138--0e82--422a--842a--637f19642b30 253:133   0    15G  0 lvm  
nbd63                                                                                                  43:2016  0    15G  0 disk 
└─ceph--e11387f2--b358--48df--a881--3b3228d395ae-osd--block--bfadf223--c694--4d7e--8a73--99217f09ea90 253:134   0    15G  0 lvm  
nbd64                                                                                                  43:2048  0    15G  0 disk 
└─ceph--de6c61b5--b0ef--424e--bc06--8d01d1cc1797-osd--block--a466d236--53e1--4469--8548--a88745857d40 253:135   0    15G  0 lvm  

#3 Updated by Vikhyat Umrao 10 months ago

  • Related to Bug #54263: cephadm upgrade pacific to quincy autoscaler is scaling pgs from 32 -> 32768 for cephfs meta pool added

Also available in: Atom PDF