Support #19747
closedOnly the first 3 OSD could be activated
0%
Description
I have 4 storage servers, each has 24 SSDs as ceph storage.
I use SLES12sp2 and SES4.0.
prepare stage completed successfully:
deploy osd prepare ses01:/dev/sdX
as for activate, the first 3 disks could be done. What ever select the disks by sequence or randomly.
The failed ones have message like this:
cephadm@mgt01:~/ceph> ceph-deploy osd activate ses03:sdb1
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadm/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.34): /usr/bin/ceph-deploy osd activate ses03:sdb1
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] subcommand : activate
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f1faf1cb3b0>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] func : <function osd at 0x7f1faf415b18>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.cli][INFO ] disk : [('ses03', '/dev/sdb1', None)]
[ceph_deploy.osd][DEBUG ] Activating cluster ceph disks ses03:/dev/sdb1:
[ses03][DEBUG ] connection detected need for sudo
[ses03][DEBUG ] connected to host: ses03
[ses03][DEBUG ] detect platform information from remote host
[ses03][DEBUG ] detect machine type
[ses03][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO ] Distro info: SUSE Linux Enterprise Server 12 x86_64
[ceph_deploy.osd][DEBUG ] activating host ses03 disk /dev/sdb1
[ceph_deploy.osd][DEBUG ] will use init type: systemd
[ses03][DEBUG ] find the location of an executable
[ses03][INFO ] Running command: sudo /usr/sbin/ceph-disk v activate --mark-init systemd --mount /dev/sdb1 /dev/sdb1
[ses03][WARNIN] main_activate: path = /dev/sdb1
[ses03][WARNIN] get_dm_uuid: get_dm_uuid /dev/sdb1 uuid path is /sys/dev/block/8:17/dm/uuid
[ses03][WARNIN] command: Running command: /usr/sbin/blkid -o udev -p /dev/sdb1
[ses03][WARNIN] command: Running command: /sbin/blkid -p -s TYPE -o value -
[ses03][WARNIN] command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs
[ses03][WARNIN] command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs
[ses03][WARNIN] mount: Mounting /dev/sdb1 on /var/lib/ceph/tmp/mnt.Qib3jB with options noatime,inode64
[ses03][WARNIN] command_check_call: Running command: /usr/bin/mount t xfs -o noatime,inode64 - /dev/sdb1 /var/lib/ceph/tmp/mnt.Qib3jB
[ses03][WARNIN] activate: Cluster uuid is 8b693493-8921-4640-9474-bb51a61c3ed4
[ses03][WARNIN] command: Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
[ses03][WARNIN] activate: Cluster name is ceph
[ses03][WARNIN] activate: OSD uuid is e9ddf344-c088-4754-aec8-c90f4de8c082
[ses03][WARNIN] activate: OSD id is 75
[ses03][WARNIN] activate: Initializing OSD...
[ses03][WARNIN] command_check_call: Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap o /var/lib/ceph/tmp/mnt.Qib3jB/activate.monmap /var/lib/ceph/tmp/mnt.Qib3jB
[ses03][WARNIN] got monmap epoch 1
[ses03][WARNIN] command: Running command: /usr/bin/timeout 300 ceph-osd --cluster ceph --mkfs --mkkey -i 75 --monmap /var/lib/ceph/tmp/mnt.Qib3jB/activate.monmap --osd-data /var/lib/ceph/tmp/mnt.Qib3jB --osd-journal /var/lib/ceph/tmp/mnt.Qib3jB/journal --osd-uuid e9ddf344-c088-4754-aec8-c90f4de8c082 --keyring /var/lib/ceph/tmp/mnt.Qib3jB/keyring --setuser ceph --setgroup ceph
[ses03][WARNIN] mount_activate: Failed to activate
[ses03][WARNIN] unmount: Unmounting /var/lib/ceph/tmp/mnt.Qib3jB
[ses03][WARNIN] command_check_call: Running command: /bin/umount -
[ses03][WARNIN] Traceback (most recent call last):
[ses03][WARNIN] File "/usr/sbin/ceph-disk", line 9, in <module>
[ses03][WARNIN] load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')()
[ses03][WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 5009, in run
[ses03][WARNIN] main(sys.argv[1:])
[ses03][WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4960, in main
[ses03][WARNIN] args.func(args)
[ses03][WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 3321, in main_activate
[ses03][WARNIN] reactivate=args.reactivate,
[ses03][WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 3078, in mount_activate
[ses03][WARNIN] (osd_id, cluster) = activate(path, activate_key_template, init)
[ses03][WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 3254, in activate
[ses03][WARNIN] keyring=keyring,
[ses03][WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2747, in mkfs
[ses03][WARNIN] '--setgroup', get_ceph_group(),
[ses03][WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2694, in ceph_osd_mkfs
[ses03][WARNIN] raise Error('%s failed : s' % (str(arguments), error))
[ses03][WARNIN] ceph_disk.main.Error: Error: ['ceph-osd', '--cluster', 'ceph', '--mkfs', '--mkkey', '-i', u'75', '--monmap', '/var/lib/ceph/tmp/mnt.Qib3jB/activate.monmap', '--osd-data', '/var/lib/ceph/tmp/mnt.Qib3jB', '--osd-journal', '/var/lib/ceph/tmp/mnt.Qib3jB/journal', '--osd-uuid', u'e9ddf344-c088-4754-aec8-c90f4de8c082', '--keyring', '/var/lib/ceph/tmp/mnt.Qib3jB/keyring', '--setuser', 'ceph', '--setgroup', 'ceph'] failed : tcmalloc: large alloc 17481375744 bytes == (nil) @ 0x7f08355428ba 0x7f0835563a74 0x560eb8d5bfa5 0x560eb8da5d18 0x560eb8da609d 0x560eb8d88234 0x560eb8a32b09 0x560eb89c86c0 0x7f08320d86e5 0x560eb8a12fa9 (nil)
[ses03][WARNIN] terminate called after throwing an instance of 'std::bad_alloc'
[ses03][WARNIN] what(): std::bad_alloc
[ses03][WARNIN] * Caught signal (Aborted) *
[ses03][WARNIN] in thread 7f0835993800 thread_name:ceph-osd
[ses03][WARNIN] ceph version 10.2.3-560-g4429782 (4429782e6c7b1bf08516e2ee2f6b2c47f3bf62d7)
[ses03][WARNIN] 1: (()+0x91e6a2) [0x560eb8fae6a2]
[ses03][WARNIN] 2: (()+0x10b00) [0x7f0834658b00]
[ses03][WARNIN] 3: (gsignal()+0x37) [0x7f08320ec8d7]
[ses03][WARNIN] 4: (abort()+0x13a) [0x7f08320edcaa]
[ses03][WARNIN] 5: (_gnu_cxx::_verbose_terminate_handler()+0x15d) [0x7f0832a0772d]
[ses03][WARNIN] 6: (()+0x96706) [0x7f0832a05706]
[ses03][WARNIN] 7: (()+0x96751) [0x7f0832a05751]
[ses03][WARNIN] 8: (_cxa_rethrow()+0x46) [0x7f0832a059b6]
[ses03][WARNIN] 9: (std::_Hashtable<ghobject_t, std::pair<ghobject_t const, std::_List_iterator<std::pair<ghobject_t, DBObjectMap::_Header> > >, std::allocator<std::pair<ghobject_t const, std::_List_iterator<std::pair<ghobject_t, DBObjectMap::_Header> > > >, std::_detail::_Select1st, std::equal_to<ghobject_t>, std::hash<ghobject_t>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::rehash(unsigned long)+0x15e) [0x560eb8da5dce]
[ses03][WARNIN] 10: (DBObjectMap::DBObjectMap(KeyValueDB)+0x2bd) [0x560eb8da609d]
[ses03][WARNIN] 11: (FileStore::mount()+0x2514) [0x560eb8d88234]
[ses03][WARNIN] 12: (OSD::mkfs(CephContext*, ObjectStore*, std::string constx%x, uuid_d, int)+0x399) [0x560eb8a32b09]
[ses03][WARNIN] 13: (main()+0x1030) [0x560eb89c86c0]
[ses03][WARNIN] 14: (_libc_start_main()+0xf5) [0x7f08320d86e5]
[ses03][WARNIN] 15: (_start()+0x29) [0x560eb8a12fa9]
[ses03][WARNIN] 2017-04-22 09:34:47.377352 7f0835993800 -1 Caught signal (Aborted)
[ses03][WARNIN] in thread 7f0835993800 thread_name:ceph-osd
[ses03][WARNIN]
[ses03][WARNIN] ceph version 10.2.3-560-g4429782 (4429782e6c7b1bf08516e2ee2f6b2c47f3bf62d7)
[ses03][WARNIN] 1: (()+0x91e6a2) [0x560eb8fae6a2]
[ses03][WARNIN] 2: (()+0x10b00) [0x7f0834658b00]
[ses03][WARNIN] 3: (gsignal()+0x37) [0x7f08320ec8d7]
[ses03][WARNIN] 4: (abort()+0x13a) [0x7f08320edcaa]
[ses03][WARNIN] 5: (_gnu_cxx::__verbose_terminate_handler()+0x15d) [0x7f0832a0772d]
[ses03][WARNIN] 6: (()+0x96706) [0x7f0832a05706]
[ses03][WARNIN] 7: (()+0x96751) [0x7f0832a05751]
[ses03][WARNIN] 8: (_cxa_rethrow()+0x46) [0x7f0832a059b6]
[ses03][WARNIN] 9: (std::_Hashtable<ghobject_t, std::pair<ghobject_t const, std::_List_iterator<std::pair<ghobject_t, DBObjectMap::_Header> > >, std::allocator<std::pair<ghobject_t const, std::_List_iterator<std::pair<ghobject_t, DBObjectMap::_Header> > > >, std::_detail::_Select1st, std::equal_to<ghobject_t>, std::hash<ghobject_t>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::rehash(unsigned long)+0x15e) [0x560eb8da5dce]
[ses03][WARNIN] 10: (DBObjectMap::DBObjectMap(KeyValueDB)+0x2bd) [0x560eb8da609d]
[ses03][WARNIN] 11: (FileStore::mount()+0x2514) [0x560eb8d88234]
[ses03][WARNIN] 12: (OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d, int)+0x399) [0x560eb8a32b09]
[ses03][WARNIN] 13: (main()+0x1030) [0x560eb89c86c0]
[ses03][WARNIN] 14: (_libc_start_main()+0xf5) [0x7f08320d86e5]
[ses03][WARNIN] 15: (_start()+0x29) [0x560eb8a12fa9]
[ses03][WARNIN] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
[ses03][WARNIN]
[ses03][WARNIN] 0> 2017-04-22 09:34:47.377352 7f0835993800 -1 Caught signal (Aborted) *
[ses03][WARNIN] in thread 7f0835993800 thread_name:ceph-osd
[ses03][WARNIN]
[ses03][WARNIN] ceph version 10.2.3-560-g4429782 (4429782e6c7b1bf08516e2ee2f6b2c47f3bf62d7)
[ses03][WARNIN] 1: (()+0x91e6a2) [0x560eb8fae6a2]
[ses03][WARNIN] 2: (()+0x10b00) [0x7f0834658b00]
[ses03][WARNIN] 3: (gsignal()+0x37) [0x7f08320ec8d7]
[ses03][WARNIN] 4: (abort()+0x13a) [0x7f08320edcaa]
[ses03][WARNIN] 5: (_gnu_cxx::__verbose_terminate_handler()+0x15d) [0x7f0832a0772d]
[ses03][WARNIN] 6: (()+0x96706) [0x7f0832a05706]
[ses03][WARNIN] 7: (()+0x96751) [0x7f0832a05751]
[ses03][WARNIN] 8: (_cxa_rethrow()+0x46) [0x7f0832a059b6]
[ses03][WARNIN] 9: (std::_Hashtable<ghobject_t, std::pair<ghobject_t const, std::_List_iterator<std::pair<ghobject_t, DBObjectMap::_Header> > >, std::allocator<std::pair<ghobject_t const, std::_List_iterator<std::pair<ghobject_t, DBObjectMap::_Header> > > >, std::_detail::_Select1st, std::equal_to<ghobject_t>, std::hash<ghobject_t>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::rehash(unsigned long)+0x15e) [0x560eb8da5dce]
[ses03][WARNIN] 10: (DBObjectMap::DBObjectMap(KeyValueDB*)+0x2bd) [0x560eb8da609d]
[ses03][WARNIN] 11: (FileStore::mount()+0x2514) [0x560eb8d88234]
[ses03][WARNIN] 12: (OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d, int)+0x399) [0x560eb8a32b09]
[ses03][WARNIN] 13: (main()+0x1030) [0x560eb89c86c0]
[ses03][WARNIN] 14: (__libc_start_main()+0xf5) [0x7f08320d86e5]
[ses03][WARNIN] 15: (_start()+0x29) [0x560eb8a12fa9]
[ses03][WARNIN] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
[ses03][WARNIN]
[ses03][WARNIN]
[ses03][ERROR ] RuntimeError: command returned non-zero exit status: 1
[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: /usr/sbin/ceph-disk -v activate --mark-init systemd --mount /dev/sdb1
Updated by Ming Gao about 7 years ago
If I mount /dev/sda1 manully to a directory, I can find there is not ready and keyring file, which is different from the good ones.
Updated by huang jun about 7 years ago
you can monitor the memory usage while adding the third osd.
The 'dmesg' output or /var/log/messages on suse will useful.
Updated by Nathan Cutler about 7 years ago
How much memory do you have in your OSD nodes? See [1] for background.
Updated by Ming Gao about 7 years ago
If I use the default ceph.conf file, everything would be OK. So I guess the problem was caused by the modified ceph.conf file.
Anyone knows which parametters might triger this issue?
@Nathan Weinberg Cutler, every storage node has 256G memory.
The default ceph.conf:
[global]
fsid = f407570e-1bce-475b-9afd-aadb8bc2b7e7
mon_initial_members = ses01, ses02, ses03
mon_host = 192.168.10.210,192.168.10.211,192.168.10.212,192.168.10.213
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
The modified ceph.conf:
[global]
fsid = f407570e-1bce-475b-9afd-aadb8bc2b7e7
mon_initial_members = ses01, ses02, ses03, ses04
mon_host = 192.168.10.210,192.168.10.211,192.168.10.212,192.168.10.213
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
cluster_network = 192.168.30.0/24
public_network = 192.168.10.0/24
osd_pool_default_size = 2
osd_pool_default_min_size = 1
debug_lockdep = 0/0
debug_lockdep = 0/0
debug_context = 0/0
debug_crush = 0/0
debug_buffer = 0/0
debug_timer = 0/0
debug_journaler = 0/0
debug_osd = 0/0
debug_optracker = 0/0
debug_objclass = 0/0
debug_filestore = 0/0
debug_journal = 0/0
debug_ms = 0/0
debug_monc = 0/0
debug_tp = 0/0
debug_auth = 0/0
debug_finisher = 0/0
debug_heartbeatmap = 0/0
debug_perfcounter = 0/0
debug_asok = 0/0
debug_throttle = 0/0
max_open_files = 262144
journal_queue_max_ops = 16384
journal_queue_max_bytes = 2048576000
objecter_inflight_ops = 16384
filestore_omap_header_cache_size = 2048576000
filestore_fd_cache_size = 2048576000
filestore_queue_max_ops = 1048576
filestore_queue_max_bytes = 17179869184
filestore_queue_committing_max_ops = 1048576
filestore_queue_committing_max_bytes = 17179869184
filestore_wbthrottle_enable = false
filestore_fiemap = true
filestore_max_sync_interval = 3600
filestore_op_threads = 8
[osd]
osd_crush_update_on_start = false
osd_op_threads = 8
Updated by Ming Gao about 7 years ago
filestore_omap_header_cache_size = 2048576000
This is the parameter which caused the problem. How much should this be? It's for performance tunning right?
Updated by Alex Lau about 7 years ago
Ming Gao wrote:
filestore_omap_header_cache_size = 2048576000
This is the parameter which caused the problem. How much should this be? It's for performance tunning right?
ceph_disk.main.Error: Error: ['ceph-osd', '--cluster', 'ceph', '--mkfs', '--mkkey', '-i', u'75', '--monmap', '/var/lib/ceph/tmp/mnt.Qib3jB/activate.monmap', '--osd-data', '/var/lib/ceph/tmp/mnt.Qib3jB', '--osd-journal', '/var/lib/ceph/tmp/mnt.Qib3jB/journal', '--osd-uuid', u'e9ddf344-c088-4754-aec8-c90f4de8c082', '--keyring', '/var/lib/ceph/tmp/mnt.Qib3jB/keyring', '--setuser', 'ceph', '--setgroup', 'ceph'] failed : tcmalloc: large alloc 17481375744 bytes == (nil) @ 0x7f08355428ba 0x7f0835563a74 0x560eb8d5bfa5 0x560eb8da5d18 0x560eb8da609d 0x560eb8d88234 0x560eb8a32b09 0x560eb89c86c0 0x7f08320d86e5 0x560eb8a12fa9 (nil)
[ses03][WARNIN] terminate called after throwing an instance of 'std::bad_alloc'
You log show about 16GB of allocation fail.
For filestore_omap_header_cache_size default is 1024
and assume you have 4M object with 1TB storage, you only need about 250K to map everything.
It is really depending on what you are doing.
If you have bigger object size then your cache size actually need less. e.g. 16M only need 62.5k to map everything.
Updated by Greg Farnum almost 7 years ago
- Tracker changed from Bug to Support
- Status changed from New to Closed