Project

General

Profile

Bug #58047

ceph-osd: osd numa affinity setting doesn't work

Added by xu wang 2 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
OSD
Target version:
-
% Done:

100%

Source:
Tags:
Backport:
Regression:
No
Severity:
1 - critical
Reviewed:
ceph-qa-suite:
knfs
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

After setting osd_numa_node parameter, the osd numa is not as expected.

  • cpu numa node
    # lscpu | grep NUMA
    NUMA node(s):        4
    NUMA node0 CPU(s):   0-2,6-8,24-26,30-32
    NUMA node1 CPU(s):   3-5,9-11,27-29,33-35
    NUMA node2 CPU(s):   12-14,18-20,36-38,42-44
    NUMA node3 CPU(s):   15-17,21-23,39-41,45-47
    
  • set osd.0 to numa 1
    # ceph daemon osd.0 config show | grep osd_numa_node
        "osd_numa_node": "1",
    
  • check osd numa node
    ceph       1121      1  0 Sep25 ?        10:39:11 /usr/bin/ceph-osd --cluster ceph -f -i 0 --setuser ceph --setgroup ceph
    
    # ps -L -o pid,lwp,pcpu,cpuid,time,comm 1121
       PID    LWP %CPU CPUID     TIME COMMAND
      1121   1121  0.0    19 00:01:30 ceph-osd
      1121   1122  0.0     0 00:00:08 log
      1121   1124  0.0    12 00:18:17 msgr-worker-0
      1121   1125  0.0    28 00:18:26 msgr-worker-1
      1121   1126  0.0    38 00:17:23 msgr-worker-2
      1121   1127  0.0    28 00:12:26 msgr-worker-3
      1121   1128  0.0    34 00:15:27 msgr-worker-4
      1121   1129  0.0    14 00:13:55 msgr-worker-5
      1121   1130  0.0    27 00:13:26 msgr-worker-6
      1121   1131  0.0    41 00:13:33 msgr-worker-7
      1121   1132  0.0    11 00:12:09 msgr-worker-8
      1121   1133  0.0    34 00:23:33 msgr-worker-9
      1121   1134  0.0     1 00:23:36 msgr-worker-10
      1121   1135  0.0    35 00:22:15 msgr-worker-11
      1121   1136  0.0    39 00:21:30 msgr-worker-12
      1121   1137  0.0    30 00:20:10 msgr-worker-13
      1121   1138  0.0    17 00:23:41 msgr-worker-14
      1121   1139  0.0    11 00:18:31 msgr-worker-15
      1121   1140  0.0    38 00:17:47 msgr-worker-16
      1121   1141  0.0    29 00:18:01 msgr-worker-17
      1121   1142  0.0    23 00:18:40 msgr-worker-18
      1121   1143  0.0    35 00:19:35 msgr-worker-19
      1121   1144  0.0     4 00:17:39 msgr-worker-20
      1121   1145  0.0    43 00:20:52 msgr-worker-21
      1121   1146  0.0    12 00:18:14 msgr-worker-22
      1121   1147  0.0    31 00:18:33 msgr-worker-23
      1121   1151  0.0    10 00:00:24 service
      1121   1152  0.0    42 00:12:03 admin_socket
      1121   1161  0.0    37 00:00:00 signal_handler
      1121   1162  0.0    14 00:00:00 io_context_pool
      1121   1163  0.0     7 00:00:00 io_context_pool
      1121   1166  0.0    38 00:05:02 OpHistorySvc
      1121   1167  0.0     7 00:00:14 ceph_timer
      1121   1186  0.0    21 00:02:03 safe_timer
      1121   1187  0.0    45 00:05:56 safe_timer
      1121   1188  0.0    42 00:00:00 safe_timer
      1121   1189  0.0    15 00:00:00 safe_timer
      1121   1190  0.0    40 00:00:00 fn_anonymous
      1121   1191  0.0    46 00:05:55 bstore_aio
      1121   1192  0.0    45 00:00:00 bstore_discard
      1121   1199  0.0    16 00:00:08 rocksdb:low0
      1121   1200  0.0    23 00:00:04 rocksdb:low1
      1121   1201  0.0    16 00:00:04 rocksdb:low2
      1121   1202  0.0    47 00:00:00 rocksdb:low3
      1121   1203  0.0    21 00:00:05 rocksdb:high0
      1121   1204  0.0    23 00:00:08 rocksdb:high1
      1121   1386  0.0    42 00:02:24 bstore_aio
      1121   1387  0.0    23 00:00:00 bstore_discard
      1121   1388  0.0    12 00:02:23 bstore_aio
      1121   1389  0.0    40 00:00:00 bstore_discard
      1121   1390  0.0    45 00:04:02 bstore_aio
      1121   1391  0.0    38 00:00:00 bstore_discard
      1121   1392  0.0    44 00:00:00 ceph-osd
      1121   1573  0.0    46 00:00:07 rocksdb:dump_st
      1121   1574  0.0    23 00:00:00 rocksdb:pst_st
      1121   1575  0.0    39 00:00:00 cfin
      1121   1576  0.0    16 00:11:41 bstore_kv_sync
      1121   1577  0.0    17 00:03:41 bstore_kv_final
      1121   1578  0.0    39 00:31:46 bstore_mempool
      1121   1579  0.0    46 00:00:59 ms_dispatch
      1121   1580  0.0     1 00:00:00 ms_local
      1121   1581  0.0    23 00:01:31 safe_timer
      1121   1582  0.0    41 00:07:18 safe_timer
      1121   1583  0.0     0 00:00:16 ms_dispatch
      1121   1584  0.0    32 00:00:00 ms_local
      1121   1585  0.0    34 00:00:00 ms_dispatch
      1121   1586  0.0    13 00:00:00 ms_local
      1121   1587  0.0    40 00:00:00 ms_dispatch
      1121   1588  0.0    44 00:00:00 ms_local
      1121   1589  0.0    45 00:00:00 ms_dispatch
      1121   1590  0.0    44 00:00:00 ms_local
      1121   1591  0.0    19 00:00:00 ms_dispatch
      1121   1592  0.0    20 00:00:00 ms_local
      1121   1593  0.0    12 00:00:00 ms_dispatch
      1121   1594  0.0    20 00:00:00 ms_local
      1121   1595  0.0    20 00:00:00 fn_anonymous
      1121   1596  0.0    44 00:00:00 finisher
      1121   1597  0.0    44 00:00:00 safe_timer
      1121   1598  0.0    44 00:00:00 safe_timer
      1121   1599  0.0    44 00:00:00 ceph-osd
      1121   1600  0.0    40 00:00:00 osd_srv_agent
      1121   1601  0.0    47 00:15:48 tp_osd_tp
      1121   1602  0.0    45 00:14:21 tp_osd_tp
      1121   1603  0.0    17 00:10:50 tp_osd_tp
      1121   1604  0.0    41 00:20:39 tp_osd_tp
      1121   1605  0.0    16 00:13:16 tp_osd_tp
      1121   1606  0.0    16 00:26:19 osd_srv_heartbt
    
    

Also available in: Atom PDF