Bug #59562: [pacific] segfault on FIPS enabled server at "radosgw-admin period update" command - rgw - Ceph

Actions

Copy link

Bug #59562

closed

[pacific] segfault on FIPS enabled server at "radosgw-admin period update" command

Added by Peter Razumovsky 12 months ago. Updated 11 months ago.

Status:

Resolved

Priority:

Normal

Assignee:

Mark Kogan

Target version:

% Done:

Source:

Community (dev)

Tags:

Backport:

Regression:

Severity:

2 - major

Reviewed:

Affected Versions:

Ceph - v16.2.11

ceph-qa-suite:

rgw

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

After https://tracker.ceph.com/issues/52900 PR merged we tried to enable RGW on FIPS enabled host and got the following error:

    -25> 2023-04-26T13:44:06.080+0000 7fe277b18600  5 monclient: authenticate success, global_id 10611936
   -24> 2023-04-26T13:44:06.080+0000 7fe277b18600 10 monclient: _renew_subs
   -23> 2023-04-26T13:44:06.080+0000 7fe277b18600 10 monclient: _send_mon_message to mon.a at v2:<IP>:3300/0
   -22> 2023-04-26T13:44:06.080+0000 7fe24afe5640  4 set_mon_vals no callback set
   -21> 2023-04-26T13:44:06.080+0000 7fe277b18600 10 monclient: _renew_subs
   -20> 2023-04-26T13:44:06.080+0000 7fe277b18600 10 monclient: _send_mon_message to mon.a at v2:<IP>:3300/0
   -19> 2023-04-26T13:44:06.080+0000 7fe277b18600  1 librados: init done
   -18> 2023-04-26T13:44:06.080+0000 7fe277b18600  5 asok(0x5610d50b4440) register_command cr dump hook 0x5610d5212f18
   -17> 2023-04-26T13:44:06.080+0000 7fe27570e640 10 monclient: get_auth_request con 0x5610d521d6a0 auth_method 0
   -16> 2023-04-26T13:44:06.084+0000 7fe2497e2640  4 mgrc handle_mgr_map Got map version 190
   -15> 2023-04-26T13:44:06.084+0000 7fe2497e2640  4 mgrc handle_mgr_map Active mgr is now [v2:<IP>:6800/44795856,v1:<IP>:6801/44795856]
   -14> 2023-04-26T13:44:06.084+0000 7fe2497e2640  4 mgrc reconnect Starting new session with [v2:<IP>:6800/44795856,v1:<IP>:6801/44795856]
   -13> 2023-04-26T13:44:06.084+0000 7fe275f0f640 10 monclient: get_auth_request con 0x5610d5220f70 auth_method 0
   -12> 2023-04-26T13:44:06.084+0000 7fe27570e640 10 monclient: get_auth_request con 0x7fe1c80528d0 auth_method 0
   -11> 2023-04-26T13:44:06.088+0000 7fe274f0d640 10 monclient: get_auth_request con 0x5610d52246e0 auth_method 0
   -10> 2023-04-26T13:44:06.092+0000 7fe275f0f640 10 monclient: get_auth_request con 0x5610d5227f70 auth_method 0
    -9> 2023-04-26T13:44:06.100+0000 7fe27570e640 10 monclient: get_auth_request con 0x7fe26802a070 auth_method 0
    -8> 2023-04-26T13:44:06.100+0000 7fe274f0d640 10 monclient: get_auth_request con 0x7fe2680389a0 auth_method 0
    -7> 2023-04-26T13:44:06.104+0000 7fe275f0f640 10 monclient: get_auth_request con 0x7fe26c076c10 auth_method 0
    -6> 2023-04-26T13:44:06.108+0000 7fe27570e640 10 monclient: get_auth_request con 0x7fe26801b3f0 auth_method 0
    -5> 2023-04-26T13:44:06.112+0000 7fe274f0d640 10 monclient: get_auth_request con 0x7fe26804b3f0 auth_method 0
    -4> 2023-04-26T13:44:06.116+0000 7fe275f0f640 10 monclient: get_auth_request con 0x7fe26c0aa040 auth_method 0
    -3> 2023-04-26T13:44:06.120+0000 7fe27570e640 10 monclient: get_auth_request con 0x7fe2600082f0 auth_method 0
    -2> 2023-04-26T13:44:06.124+0000 7fe274f0d640 10 monclient: get_auth_request con 0x7fe26c0ad0b0 auth_method 0
    -1> 2023-04-26T13:44:06.128+0000 7fe275f0f640 10 monclient: get_auth_request con 0x7fe27003e040 auth_method 0
     0> 2023-04-26T13:44:06.140+0000 7fe277b18600 -1 *** Caught signal (Segmentation fault) **
 in thread 7fe277b18600 thread_name:radosgw-admin

 ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)
 1: /lib64/libc.so.6(+0x54df0) [0x7fe279cb4df0]

--- logging levels ---
   ...
  -2/-2 (syslog threshold)
  99/99 (stderr threshold)
--- pthread ID / name mapping for recent threads ---
  140609872340544 / ms_dispatch
  140609897518656 / io_context_pool
  140610350593600 / ms_dispatch
  140610375771712 / io_context_pool
  140610384164416 / fn_anonymous
  140610601276992 / msgr-worker-2
  140610609669696 / msgr-worker-1
  140610618062400 / msgr-worker-0
  140610647459328 / radosgw-admin
  max_recent       500
  max_new          500

We made gdb run, threads info and backtraces could be found in attachments. This issue blocks using Ceph RGW on FIPS enabled cluster.

Files

Download all files

info_threads.rtf (7.71 KB) info_threads.rtf		Peter Razumovsky, 04/26/2023 02:35 PM
thread_apply_all_bt.rtf (34.1 KB) thread_apply_all_bt.rtf		Peter Razumovsky, 04/26/2023 02:35 PM

Related issues 2 (1 open — 1 closed)

Actions

Copy link

Updated by Casey Bodley 12 months ago

Related to Bug #52900: segfault on FIPS enabled server as result of EVP_md5 disabled in openssl added

Actions

Copy link

Updated by Casey Bodley 12 months ago

Assignee set to Mark Kogan

Actions

Copy link

Updated by Mark Kogan 12 months ago

@Peter Razumovsky

May ask which OS please for repro? (there are differences between openssl v1 and v3)

Actions

Copy link

Updated by Mark Kogan 12 months ago

reproduced, working on backport of https://tracker.ceph.com/issues/58332 to pacific at https://tracker.ceph.com/issues/59579

Actions

Copy link

Updated by Mark Kogan 12 months ago

Is duplicate of Bug #58332: On version 17.2.5-8.el9cp, "Segmentation fault" while uploading object(regular/multipart) on FIPS enabled cluster. added

Actions

Copy link