Bug #51244: radosgw crash - rgw - Ceph

Actions

Copy link

Bug #51244

closed

radosgw crash

Added by Matthew Darwin almost 3 years ago. Updated almost 3 years ago.

Status:

Duplicate

Priority:

Normal

Assignee:

Target version:

% Done:

Source:

Community (user)

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

Ceph - v16.2.4

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

In trying to rename files, ceph is crashing. debian 10.

Using this command:

```
mc cp --attr x-amz-acl=public-read ceph11/bos-snapshots/snapshot-2019-01-20-15-bos@0000540625.bin.zst ceph11/bos-snapshots/snapshot-2019-01-20-15-bos-0000540625.bin.zst
```

(might be related to my previous issue? https://tracker.ceph.com/issues/51097)

```
Jun 16 14:20:24 ceph-gw11 radosgw⁴⁰⁰³: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable)
Jun 16 14:20:24 ceph-gw11 radosgw⁴⁰⁰³: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730) [0x7fd133136730]
Jun 16 14:20:24 ceph-gw11 radosgw⁴⁰⁰³: 2: (rgw_bucket::rgw_bucket(rgw_bucket const&)+0x1f) [0x7fd13cd79eaf]
Jun 16 14:20:24 ceph-gw11 radosgw⁴⁰⁰³: 3: (rgw::sal::RGWObject::get_obj() const+0x1c) [0x7fd13cda6d7c]
Jun 16 14:20:24 ceph-gw11 radosgw⁴⁰⁰³: 4: (RGWInitMultipart::verify_permission(optional_yield)+0x68) [0x7fd13d078bf8]
Jun 16 14:20:24 ceph-gw11 radosgw⁴⁰⁰³: 5: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, optional_yield, bool)+0x80c) [0x7fd13cd5ca5c]
Jun 16 14:20:24 ceph-gw11 radosgw⁴⁰⁰³: 6: (process_request(rgw::sal::RGWRadosStore*, RGWREST*, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >, int*)+0x2895) [0x7fd13cd60975]
Jun 16 14:20:24 ceph-gw11 radosgw⁴⁰⁰³: 7: /usr/lib/libradosgw.so.2(+0x4b882b) [0x7fd13ccb182b]
Jun 16 14:20:24 ceph-gw11 radosgw⁴⁰⁰³: 8: /usr/lib/libradosgw.so.2(+0x4b9dc0) [0x7fd13ccb2dc0]
Jun 16 14:20:24 ceph-gw11 radosgw⁴⁰⁰³: 9: /usr/lib/libradosgw.so.2(+0x4b9f08) [0x7fd13ccb2f08]
Jun 16 14:20:24 ceph-gw11 radosgw⁴⁰⁰³: 10: make_fcontext()
Jun 16 14:20:24 ceph-gw11 radosgw⁴⁰⁰³: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
```

```
$ objdump -rdS /usr/bin/radosgw

/usr/bin/radosgw: file format elf64-x86-64

Disassembly of section .init:

0000000000001000 <.init>:
1000: 48 83 ec 08 sub $0x8,%rsp
1004: 48 8b 05 e5 2f 00 00 mov 0x2fe5(%rip),%rax # 3ff0 <__gmon_start__>
100b: 48 85 c0 test %rax,%rax
100e: 74 02 je 1012 <_Z12radosgw_MainiPPKc@plt-0x1e>
1010: ff d0 callq *%rax
1012: 48 83 c4 08 add $0x8,%rsp
1016: c3 retq

Disassembly of section .plt:

0000000000001020 <_Z12radosgw_MainiPPKc@plt-0x10>:
1020: ff 35 e2 2f 00 00 pushq 0x2fe2(%rip) # 4008 <_IO_stdin_used@Base+0x2008> 1026: ff 25 e4 2f 00 00 jmpq *0x2fe4(%rip) # 4010 <_IO_stdin_used@Base+0x2010>
102c: 0f 1f 40 00 nopl 0x0(%rax)

0000000000001030 <_Z12radosgw_MainiPPKc@plt>:
1030: ff 25 e2 2f 00 00 jmpq *0x2fe2(%rip) # 4018 <_Z12radosgw_MainiPPKc>
1036: 68 00 00 00 00 pushq $0x0
103b: e9 e0 ff ff ff jmpq 1020 <_Z12radosgw_MainiPPKc@plt-0x10>

Disassembly of section .plt.got:

0000000000001040 <__cxa_finalize@plt>:
1040: ff 25 a2 2f 00 00 jmpq *0x2fa2(%rip) # 3fe8 <__cxa_finalize@GLIBC_2.2.5>
1046: 66 90 xchg %ax,%ax

Disassembly of section .text:

0000000000001050 <main@@Base>:
1050: e9 db ff ff ff jmpq 1030 <_Z12radosgw_MainiPPKc@plt>
1055: 66 2e 0f 1f 84 00 00 nopw %cs:0x0(%rax,%rax,1)
105c: 00 00 00
105f: 90 nop

0000000000001060 <_start@Base>: 1060: 31 ed xor %ebp,%ebp 1062: 49 89 d1 mov %rdx,%r9 1065: 5e pop %rsi 1066: 48 89 e2 mov %rsp,%rdx 1069: 48 83 e4 f0 and $0xfffffffffffffff0,%rsp 106d: 50 push %rax 106e: 54 push %rsp 106f: 4c 8d 05 3a 01 00 00 lea 0x13a(%rip),%r8 # 11b0 <__libc_csu_fini@Base>
1076: 48 8d 0d d3 00 00 00 lea 0xd3(%rip),%rcx # 1150 <__libc_csu_init@Base> 107d: 48 8d 3d cc ff ff ff lea -0x34(%rip),%rdi # 1050 <main@Base>
1084: ff 15 6e 2f 00 00 callq *0x2f6e(%rip) # 3ff8 <__libc_start_main@GLIBC_2.2.5>
108a: f4 hlt
108b: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
1090: 48 8d 3d 99 2f 00 00 lea 0x2f99(%rip),%rdi # 4030 <_edata@Base> 1097: 48 8d 05 92 2f 00 00 lea 0x2f92(%rip),%rax # 4030 <_edata@Base>
109e: 48 39 f8 cmp %rdi,%rax
10a1: 74 15 je 10b8 <_start@Base+0x58> 10a3: 48 8b 05 36 2f 00 00 mov 0x2f36(%rip),%rax # 3fe0 <_ITM_deregisterTMCloneTable> 10aa: 48 85 c0 test %rax,%rax 10ad: 74 09 je 10b8 <_start@Base+0x58>
10af: ff e0 jmpq *%rax
10b1: 0f 1f 80 00 00 00 00 nopl 0x0(%rax)
10b8: c3 retq
10b9: 0f 1f 80 00 00 00 00 nopl 0x0(%rax)
10c0: 48 8d 3d 69 2f 00 00 lea 0x2f69(%rip),%rdi # 4030 <_edata@Base> 10c7: 48 8d 35 62 2f 00 00 lea 0x2f62(%rip),%rsi # 4030 <_edata@Base>
10ce: 48 29 fe sub %rdi,%rsi
10d1: 48 c1 fe 03 sar $0x3,%rsi
10d5: 48 89 f0 mov %rsi,%rax
10d8: 48 c1 e8 3f shr $0x3f,%rax
10dc: 48 01 c6 add %rax,%rsi
10df: 48 d1 fe sar %rsi
10e2: 74 14 je 10f8 <_start@Base+0x98> 10e4: 48 8b 05 ed 2e 00 00 mov 0x2eed(%rip),%rax # 3fd8 <_ITM_registerTMCloneTable> 10eb: 48 85 c0 test %rax,%rax 10ee: 74 08 je 10f8 <_start@Base+0x98>
10f0: ff e0 jmpq *%rax
10f2: 66 0f 1f 44 00 00 nopw 0x0(%rax,%rax,1)
10f8: c3 retq
10f9: 0f 1f 80 00 00 00 00 nopl 0x0(%rax)
1100: 80 3d 29 2f 00 00 00 cmpb $0x0,0x2f29(%rip) # 4030 <_edata@Base> 1107: 75 2f jne 1138 <_start@Base+0xd8>
1109: 55 push %rbp
110a: 48 83 3d d6 2e 00 00 cmpq $0x0,0x2ed6(%rip) # 3fe8 <__cxa_finalize@GLIBC_2.2.5>
1111: 00
1112: 48 89 e5 mov %rsp,%rbp
1115: 74 0c je 1123 <_start@Base+0xc3> 1117: 48 8b 3d 0a 2f 00 00 mov 0x2f0a(%rip),%rdi # 4028 <__data_start@Base+0x8>
111e: e8 1d ff ff ff callq 1040 <__cxa_finalize@plt>
1123: e8 68 ff ff ff callq 1090 <_start@Base+0x30> 1128: c6 05 01 2f 00 00 01 movb $0x1,0x2f01(%rip) # 4030 <_edata@Base>
112f: 5d pop %rbp
1130: c3 retq
1131: 0f 1f 80 00 00 00 00 nopl 0x0(%rax)
1138: c3 retq
1139: 0f 1f 80 00 00 00 00 nopl 0x0(%rax)
1140: e9 7b ff ff ff jmpq 10c0 <_start@@Base+0x60>
1145: 66 2e 0f 1f 84 00 00 nopw %cs:0x0(%rax,%rax,1)
114c: 00 00 00
114f: 90 nop

0000000000001150 <__libc_csu_init@Base>: 1150: 41 57 push %r15 1152: 49 89 d7 mov %rdx,%r15 1155: 41 56 push %r14 1157: 49 89 f6 mov %rsi,%r14 115a: 41 55 push %r13 115c: 41 89 fd mov %edi,%r13d 115f: 41 54 push %r12 1161: 4c 8d 25 40 2b 00 00 lea 0x2b40(%rip),%r12 # 3ca8 <_IO_stdin_used@Base+0x1ca8>
1168: 55 push %rbp
1169: 48 8d 2d 40 2b 00 00 lea 0x2b40(%rip),%rbp # 3cb0 <_IO_stdin_used@Base+0x1cb0> 1170: 53 push %rbx 1171: 4c 29 e5 sub %r12,%rbp 1174: 48 83 ec 08 sub $0x8,%rsp 1178: e8 83 fe ff ff callq 1000 <_Z12radosgw_MainiPPKc@plt-0x30> 117d: 48 c1 fd 03 sar $0x3,%rbp 1181: 74 1b je 119e <__libc_csu_init@Base+0x4e>
1183: 31 db xor %ebx,%ebx
1185: 0f 1f 00 nopl (%rax)
1188: 4c 89 fa mov %r15,%rdx
118b: 4c 89 f6 mov %r14,%rsi
118e: 44 89 ef mov %r13d,%edi
1191: 41 ff 14 dc callq *(%r12,%rbx,8)
1195: 48 83 c3 01 add $0x1,%rbx
1199: 48 39 dd cmp %rbx,%rbp
119c: 75 ea jne 1188 <__libc_csu_init@@Base+0x38>
119e: 48 83 c4 08 add $0x8,%rsp
11a2: 5b pop %rbx
11a3: 5d pop %rbp
11a4: 41 5c pop %r12
11a6: 41 5d pop %r13
11a8: 41 5e pop %r14
11aa: 41 5f pop %r15
11ac: c3 retq
11ad: 0f 1f 00 nopl (%rax)