Project

General

Profile

Actions
Copy link

Bug #54361

open

common/compressor: crash when using qat compressor concurrently

Added by Hualong Feng about 2 years ago. Updated over 1 year ago.

Status:
Pending Backport
Priority:
Normal
Assignee:
Hualong Feng
Target version:
-
% Done:

100%

Source:
Tags:
backport_processed
Backport:
quincy
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
45050
Crash signature (v1):
Crash signature (v2):

Description

export QAT_SECTION_NAME=SSL
MON=1 OSD=1 MDS=0 MGR=1 RGW=1 ../src/vstart.sh -n -d --bluestore -X \
-o "debug compressor = 20 " \
-o "qat compressor enabled=true"

radosgw-admin zonegroup placement add --rgw-zonegroup default --placement-id default-placement --storage-class COLD
radosgw-admin zone placement add --rgw-zone default --placement-id default-placement --storage-class COLD --compression zlib --data-pool default.rgw.cold.data

using `warp` to test

warp put --host=<ip:port> --access-key=<access-key> --secret-key=<secret-key> --storage-class COLD --obj.size 4MB --concurrent 100

Caught signal (Segmentation fault) 
   in thread 7f9f87636700 thread_name:radosgw
   ceph version 17.0.0-10799-gd98319f8fde (d98319f8fdec6f32febe6a715c3cb1dea67ffbeb) quincy (dev)
   1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7fa1844933c0]
   2: /usr/local/lib64/libqatzip.so.1(+0x4a39) [0x7fa182628a39]
   3: qzCompressCrc()
   4: qzCompress()
   5: (QatAccel::compress(ceph::buffer::v15_2_0::list const&, ceph::buffer::v15_2_0::list&, boost::optional&lt;int&gt;&)+0x51) [0x7fa1838183f9]
   6: (ZlibCompressor::compress(ceph::buffer::v15_2_0::list const&, ceph::buffer::v15_2_0::list&, boost::optional&lt;int&gt;&)+0x17) [0x7fa1685bb6ab]
   7: (RGWPutObj_Compress::process(ceph::buffer::v15_2_0::list&&, unsigned long)+0x92) [0x7fa184c8d936]
   8: (RGWPutObj::execute(optional_yield)+0x27c4) [0x7fa184eed106]
   9: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, optional_yield, rgw::sal::Store*, bool)+0x1926) [0x7fa184a31d44]
   10: (process_request(rgw::sal::Store*, RGWREST*, RGWRequest*, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSink*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; >, std::chrono::duration&lt;unsigned long, std::ratio&lt;1l, 1000000000l&gt; >, std::shared_ptr&lt;RateLimiter&gt;, int*)+0x29ed) [0x7fa184a354f4]
   11: /usr/local/lib/libradosgw.so.2(+0x4d3955) [0x7fa184974955]
   12: /usr/local/lib/libradosgw.so.2(+0x4d469a) [0x7fa18497569a]
   13: /usr/local/lib/libradosgw.so.2(+0x4d498d) [0x7fa18497598d]
   14: make_fcontext()
  2022-02-22T10:23:47.720+0800 7f9f87636700 -1  Caught signal (Segmentation fault) *
   in thread 7f9f87636700 thread_name:radosgw
ceph version 17.0.0-10799-gd98319f8fde (d98319f8fdec6f32febe6a715c3cb1dea67ffbeb) quincy (dev)
   1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7fa1844933c0]
   2: /usr/local/lib64/libqatzip.so.1(+0x4a39) [0x7fa182628a39]
   3: qzCompressCrc()
   4: qzCompress()
   5: (QatAccel::compress(ceph::buffer::v15_2_0::list const&, ceph::buffer::v15_2_0::list&, boost::optional&lt;int&gt;&)+0x51) [0x7fa1838183f9]
   6: (ZlibCompressor::compress(ceph::buffer::v15_2_0::list const&, ceph::buffer::v15_2_0::list&, boost::optional&lt;int&gt;&)+0x17) [0x7fa1685bb6ab]
   7: (RGWPutObj_Compress::process(ceph::buffer::v15_2_0::list&&, unsigned long)+0x92) [0x7fa184c8d936]
   8: (RGWPutObj::execute(optional_yield)+0x27c4) [0x7fa184eed106]
   9: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, optional_yield, rgw::sal::Store*, bool)+0x1926) [0x7fa184a31d44]
   10: (process_request(rgw::sal::Store*, RGWREST*, RGWRequest*, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSink*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; >, std::chrono::duration&lt;unsigned long, std::ratio&lt;1l, 1000000000l&gt; >, std::shared_ptr&lt;RateLimiter&gt;, int*)+0x29ed) [0x7fa184a354f4]
   11: /usr/local/lib/libradosgw.so.2(+0x4d3955) [0x7fa184974955]
   12: /usr/local/lib/libradosgw.so.2(+0x4d469a) [0x7fa18497569a]
   13: /usr/local/lib/libradosgw.so.2(+0x4d498d) [0x7fa18497598d]
   14: make_fcontext()
   NOTE: a copy of the executable, or `objdump -rdS &lt;executable&gt;` is needed to interpret this.
ERROR in Remove Instance 2 session
   -3416> 2022-02-22T10:22:37.002+0800 7fa17fe81300 -1 WARNING: all dangerous and experimental features are enabled.
   -3359> 2022-02-22T10:22:37.010+0800 7fa17fe81300 -1 WARNING: all dangerous and experimental features are enabled.
   -3354> 2022-02-22T10:22:37.010+0800 7fa17fe81300 -1 WARNING: all dangerous and experimental features are enabled.
  ERROR in Remove Instance 3 session
       0> 2022-02-22T10:23:47.720+0800 7f9f87636700 -1 ** Caught signal (Segmentation fault) *
   in thread 7f9f87636700 thread_name:radosgw
ceph version 17.0.0-10799-gd98319f8fde (d98319f8fdec6f32febe6a715c3cb1dea67ffbeb) quincy (dev)
   1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7fa1844933c0]
   2: /usr/local/lib64/libqatzip.so.1(+0x4a39) [0x7fa182628a39]
   3: qzCompressCrc()
   4: qzCompress()
   5: (QatAccel::compress(ceph::buffer::v15_2_0::list const&, ceph::buffer::v15_2_0::list&, boost::optional&lt;int&gt;&)+0x51) [0x7fa1838183f9]
   6: (ZlibCompressor::compress(ceph::buffer::v15_2_0::list const&, ceph::buffer::v15_2_0::list&, boost::optional&lt;int&gt;&)+0x17) [0x7fa1685bb6ab]
   7: (RGWPutObj_Compress::process(ceph::buffer::v15_2_0::list&&, unsigned long)+0x92) [0x7fa184c8d936]
   8: (RGWPutObj::execute(optional_yield)+0x27c4) [0x7fa184eed106]
   9: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, optional_yield, rgw::sal::Store*, bool)+0x1926) [0x7fa184a31d44]
   10: (process_request(rgw::sal::Store*, RGWREST*, RGWRequest*, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSink*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; >, std::chrono::duration&lt;unsigned long, std::ratio&lt;1l, 1000000000l&gt; >, std::shared_ptr&lt;RateLimiter&gt;, int*)+0x29ed) [0x7fa184a354f4]
   11: /usr/local/lib/libradosgw.so.2(+0x4d3955) [0x7fa184974955]
   12: /usr/local/lib/libradosgw.so.2(+0x4d469a) [0x7fa18497569a]
   13: /usr/local/lib/libradosgw.so.2(+0x4d498d) [0x7fa18497598d]
   14: make_fcontext()
   NOTE: a copy of the executable, or `objdump -rdS &lt;executable&gt;` is needed to interpret this.
-3458> 2022-02-22T10:22:37.002+0800 7fa17fe81300 -1 WARNING: all dangerous and experimental features are enabled.
   -3457> 2022-02-22T10:22:37.010+0800 7fa17fe81300 -1 WARNING: all dangerous and experimental features are enabled.
   -3456> 2022-02-22T10:22:37.010+0800 7fa17fe81300 -1 WARNING: all dangerous and experimental features are enabled.
   -3455> 2022-02-22T10:23:47.720+0800 7f9f87636700 -1 ** Caught signal (Segmentation fault) *
   in thread 7f9f87636700 thread_name:radosgw
ceph version 17.0.0-10799-gd98319f8fde (d98319f8fdec6f32febe6a715c3cb1dea67ffbeb) quincy (dev)
   1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7fa1844933c0]
   2: /usr/local/lib64/libqatzip.so.1(+0x4a39) [0x7fa182628a39]
   3: qzCompressCrc()
   4: qzCompress()
   5: (QatAccel::compress(ceph::buffer::v15_2_0::list const&, ceph::buffer::v15_2_0::list&, boost::optional&lt;int&gt;&)+0x51) [0x7fa1838183f9]
   6: (ZlibCompressor::compress(ceph::buffer::v15_2_0::list const&, ceph::buffer::v15_2_0::list&, boost::optional&lt;int&gt;&)+0x17) [0x7fa1685bb6ab]
   7: (RGWPutObj_Compress::process(ceph::buffer::v15_2_0::list&&, unsigned long)+0x92) [0x7fa184c8d936]
   8: (RGWPutObj::execute(optional_yield)+0x27c4) [0x7fa184eed106]
   9: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, optional_yield, rgw::sal::Store*, bool)+0x1926) [0x7fa184a31d44]
   10: (process_request(rgw::sal::Store*, RGWREST*, RGWRequest*, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSink*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; >, std::chrono::duration&lt;unsigned long, std::ratio&lt;1l, 1000000000l&gt; >, std::shared_ptr&lt;RateLimiter&gt;, int*)+0x29ed) [0x7fa184a354f4]
   11: /usr/local/lib/libradosgw.so.2(+0x4d3955) [0x7fa184974955]
   12: /usr/local/lib/libradosgw.so.2(+0x4d469a) [0x7fa18497569a]
   13: /usr/local/lib/libradosgw.so.2(+0x4d498d) [0x7fa18497598d]
   14: make_fcontext()
   NOTE: a copy of the executable, or `objdump -rdS &lt;executable&gt;` is needed to interpret this.
Segmentation fault (core dumped)

Related issues 3 (1 open — 2 closed)

Copied to rgw - Backport #54980: pacific: common/compressor: crash when using qat compressor concurrentlyRejectedHualong FengActions
Copied to rgw - Backport #54981: octopus: common/compressor: crash when using qat compressor concurrentlyRejectedActions
Copied to rgw - Backport #54982: quincy: common/compressor: crash when using qat compressor concurrentlyNewHualong FengActions
Actions
Copy link
 #1

Updated by Casey Bodley about 2 years ago

  • Status changed from In Progress to Fix Under Review
Actions
Copy link
 #2

Updated by Casey Bodley about 2 years ago

  • Status changed from Fix Under Review to Pending Backport
  • Backport changed from quincy to octopus pacific quincy
Actions
Copy link
 #3

Updated by Backport Bot about 2 years ago

  • Copied to Backport #54980: pacific: common/compressor: crash when using qat compressor concurrently added
Actions
Copy link
 #4

Updated by Backport Bot about 2 years ago

  • Copied to Backport #54981: octopus: common/compressor: crash when using qat compressor concurrently added
Actions
Copy link
 #5

Updated by Backport Bot about 2 years ago

  • Copied to Backport #54982: quincy: common/compressor: crash when using qat compressor concurrently added
Actions
Copy link
 #6

Updated by Hualong Feng about 2 years ago

  • % Done changed from 0 to 100
  • Backport changed from octopus pacific quincy to quincy
Actions
Copy link
 #7

Updated by Backport Bot over 1 year ago

  • Tags set to backport_processed
Actions
Copy link

Also available in: Atom PDF